AN-736 Add support for `container` attribute in WDL 1.1 #7809

jgainerdewar · 2025-09-12T20:56:24Z

Description

Add support for container as a preferred way to specify the image to run a task in in WDL 1.1. Also:

Allow docker:// prefix for images (Cromwell only supports Docker images currently)
Allow docker or container to be a list of images. For now, always choose the first one.

This ended up being very tricky because our runtime attribute validation system really, really wants each validation to be associated with a single key name and non-interacting with validation of other keys; and this validation system is pretty tightly coupled to call caching, metadata creation, and other things. I have many larger refactors that I got 75% of the way through implementing before giving up because they spiraled out of control. The current state of this code is technically not in-spec because it allows both container and docker to be defined for 1.1 WDLs (container is preferred if both are present).

Release Notes Confirmation

`CHANGELOG.md`

I updated CHANGELOG.md in this PR
I assert that this change shouldn't be included in CHANGELOG.md because it doesn't impact community users

Terra Release Notes

I added a suggested release notes entry in this Jira ticket
I assert that this change doesn't need Jira release notes because it doesn't impact Terra users

…sent

jgainerdewar · 2025-09-25T19:46:45Z

backend/src/main/scala/cromwell/backend/validation/Containers.scala

+ * Trait to handle validation of both 'docker' and 'container' runtime attributes, which are mutually exclusive
+ * ways of specifying the container image to use for a task.
+ */
+trait ContainersValidation extends OptionalRuntimeAttributesValidation[Containers] {


This needs to be optional because docker and container are each optional.

jgainerdewar · 2025-09-25T20:02:18Z

...ch/src/test/scala/cromwell/backend/google/batch/actors/GcpBatchInitializationActorSpec.scala

      val eventPattern =
        "Key/s [test] is/are not supported by backend. Unsupported attributes will not be part of job executions."
-      EventFilter.warning(pattern = escapePattern(eventPattern), occurrences = 1) intercept {
+      EventFilter.info(pattern = escapePattern(eventPattern), occurrences = 1) intercept {


Drive-by fix of something that's been annoying me for years.

jgainerdewar · 2025-09-25T20:11:10Z

...ch/src/test/scala/cromwell/backend/google/batch/actors/GcpBatchInitializationActorSpec.scala

-      }
-    }
-  }
+  // TODO


To discuss: This test fails now because the lack of a container or docker attr isn't detected at this stage of the validation process. It's detected when we actually try to get the container from the attrs via Containers.extractContainer. I did some work (not checked in yet) to optionally include a "must include one of these" check in the Containers.validate function, but that's not actually used in backend initialization - only validateValue is used. See BackendWorkflowInitializationActor.validateRuntimeAttributes (link)

jgainerdewar · 2025-09-25T20:12:17Z

supportedBackends/tes/src/test/scala/cromwell/backend/impl/tes/TesInitializationActorSpec.scala

-        }
-      }
-    }
+    // TODO


See comment on GcpBatchInitializationActorSpec

jgainerdewar · 2025-09-25T20:14:50Z

wdl/transforms/biscayne/src/main/scala/wdl/transforms/biscayne/wdlom2wom/package.scala

@@ -1,80 +1,36 @@
 package wdl.transforms.biscayne

-import cats.implicits.catsSyntaxValidatedId


Changes to this file and the below one are largely a revert of changes in #7389

jgainerdewar · 2025-09-25T20:23:35Z

.../src/main/scala/wdl/transforms/base/wdlom2wom/TaskDefinitionElementToWomTaskDefinition.scala

  final case class TaskDefinitionElementToWomInputs(taskDefinitionElement: TaskDefinitionElement,
                                                    typeAliases: Map[String, WomType]
  )



Continued revert of #7389

jgainerdewar · 2025-09-30T15:21:03Z

backend/src/main/scala/cromwell/backend/validation/DockerValidation.scala

-  override protected def validateValue: PartialFunction[WomValue, ErrorOr[String]] = { case WomString(value) =>
-    value.validNel
-  }
+class ContainerValidation extends ContainersValidation {


Make these not the same name

aednichols

Looking good! Holding off on a full thumb as it looks like there are still a few things to patch up

aednichols · 2025-10-03T17:59:55Z

CHANGELOG.md

 * ECR [pull-through caches](https://docs.aws.amazon.com/AmazonECR/latest/userguide/pull-through-cache.html) can now be used to access Docker images. See [ReadTheDocs](https://cromwell.readthedocs.io/en/develop/backends/AWSBatch/) for details.

+### Progress toward WDL 1.1 Support
+ * WDL 1.1 support is in progress. Users that would like to try out the current partial support can do so by using WDL version `development-1.1`. As of Cromwell 91, `development-1.1` includes:


Maybe clarify that this is not the first 1.1 feature?

Suggested change

* WDL 1.1 support is in progress. Users that would like to try out the current partial support can do so by using WDL version `development-1.1`. As of Cromwell 91, `development-1.1` includes:

* WDL 1.1 support is in progress. Users that would like to try out the current partial support can do so by using WDL version `development-1.1`. As of Cromwell 91, `development-1.1` adds support for:

aednichols · 2025-10-03T18:00:43Z

CHANGELOG.md

+
 ### Other changes
 * Removed unused code related to Azure cloud services.
+* Changed log level from WARN to INFO for messages about unsupported runtime attributes.


aednichols · 2025-10-03T18:16:16Z

backend/src/main/scala/cromwell/backend/validation/Containers.scala

+
+  val runtimeAttrKeys = List(RuntimeAttributesKeys.ContainerKey, RuntimeAttributesKeys.DockerKey)
+
+  def apply(value: String): Containers = Containers(List(value))


aednichols · 2025-10-03T18:17:46Z

backend/src/main/scala/cromwell/backend/validation/Containers.scala

+      case _ => None
+    }
+
+    // TODO enhance to select the best container from the list if multiple are provided.


It would be nice to link the relevant story here, I found its number: AN-734

aednichols · 2025-10-03T20:38:37Z

centaur/src/main/resources/standardTestCases/container_attr_wdl10.test

@@ -0,0 +1,11 @@
+name: container_attr_wdl10


As a reminder, the search for *.test cases is now recursive, so you can safely locate this file at standardTestCases/container_attr/container_attr_wdl10.test

jgainerdewar added 30 commits September 9, 2025 14:06

Add ContainerValidation

9646039

Prefer container attr as image source in JobPreparationActor

c6bb830

Support docker:// prefix on container names

1cd1d94

Improve handling of version-specific runtime attr munging

c7adf3d

Ignore container attr in pre-1.1 WDLs

4819d35

Centaur tests

3abf512

Fix build

7b4e86c

Merge branch 'develop' into jd_AN-736_container

5501acc

Fix test WDL version name

359f062

Fix test

46904f3

Changelog

f5e8268

Merge branch 'develop' into jd_AN-736_container

f30ede3

Scalafmt

392fb84

Comments

6a1df7c

Docker and container validation both ensure the other value isn't pre…

79cf0db

…sent

New Containers validation to unite Docker and Container

1f65b17

Canonical ordering

7de7b8f

Update JPA to play nicer with Containers

df24bd4

Be optional better

c86938d

Tests

51312c2

Adjust Centaur tests

ab99f09

Merge branch 'develop' into jd_AN-736_container

0bebe0b

Fix test

c36116b

Include containers key in call cache hash logic

000cff2

Attempted call caching fix

2ae7bfc

Downgrade log level for unsupported attributes

e233231

Scalafmt

a6f6a59

Changelog update

335ee10

Fix tests

2871e45

Stop call caching in Centaur

5db3c3c

jgainerdewar added 6 commits September 24, 2025 15:18

Temporarily comment out test

5ebc93a

Handle container attr correctly in config backend

ca0315d

Temporarily comment out test

6953da4

Fix imports

b2c2ac4

More tests

e9c62c0

Fix slurm Centaur test

b5d2db8

jgainerdewar marked this pull request as ready for review September 25, 2025 19:10

jgainerdewar requested a review from a team as a code owner September 25, 2025 19:10

jgainerdewar commented Sep 25, 2025

View reviewed changes

jgainerdewar added 6 commits September 25, 2025 16:24

Cleanup

eea51d2

More Centaur tests

5777357

Persist validate overrides

603d3d1

Revert some previous changes

9a5d294

Fine, you can set both

e4d0b62

Dedupe

c3beb19

jgainerdewar commented Sep 29, 2025

View reviewed changes

jgainerdewar commented Sep 30, 2025

View reviewed changes

aednichols reviewed Oct 3, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

AN-736 Add support for `container` attribute in WDL 1.1 #7809

AN-736 Add support for `container` attribute in WDL 1.1 #7809

Uh oh!

jgainerdewar commented Sep 12, 2025 •

edited

Loading

Uh oh!

jgainerdewar Sep 25, 2025

Uh oh!

jgainerdewar Sep 25, 2025

Uh oh!

jgainerdewar Sep 25, 2025

Uh oh!

jgainerdewar Sep 25, 2025

Uh oh!

jgainerdewar Sep 25, 2025

Uh oh!

jgainerdewar Sep 25, 2025

Uh oh!

jgainerdewar Sep 30, 2025

Uh oh!

aednichols left a comment

Uh oh!

aednichols Oct 3, 2025 •

edited

Loading

Uh oh!

aednichols Oct 3, 2025

Uh oh!

aednichols Oct 3, 2025

Uh oh!

aednichols Oct 3, 2025

Uh oh!

aednichols Oct 3, 2025

Uh oh!

Uh oh!

		@@ -1,80 +1,36 @@
		package wdl.transforms.biscayne

		import cats.implicits.catsSyntaxValidatedId

	* WDL 1.1 support is in progress. Users that would like to try out the current partial support can do so by using WDL version `development-1.1`. As of Cromwell 91, `development-1.1` includes:
	* WDL 1.1 support is in progress. Users that would like to try out the current partial support can do so by using WDL version `development-1.1`. As of Cromwell 91, `development-1.1` adds support for:


		val runtimeAttrKeys = List(RuntimeAttributesKeys.ContainerKey, RuntimeAttributesKeys.DockerKey)

		def apply(value: String): Containers = Containers(List(value))

AN-736 Add support for container attribute in WDL 1.1 #7809

Are you sure you want to change the base?

AN-736 Add support for container attribute in WDL 1.1 #7809

Uh oh!

Conversation

jgainerdewar commented Sep 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Release Notes Confirmation

CHANGELOG.md

Terra Release Notes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aednichols left a comment

Choose a reason for hiding this comment

Uh oh!

aednichols Oct 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

AN-736 Add support for `container` attribute in WDL 1.1 #7809

AN-736 Add support for `container` attribute in WDL 1.1 #7809

jgainerdewar commented Sep 12, 2025 •

edited

Loading

`CHANGELOG.md`

aednichols Oct 3, 2025 •

edited

Loading