feat: Container build process improvements to reduce build times and improve Dockerfile organization #8

nv-tusharma · 2025-06-05T19:58:31Z

This DEP introduces a container build process optimization strategy for Dynamo to enhance developer experience and build efficiency by reorganizing Dockerfiles. We are introducing a new Dynamo base container which contains all the build logic to build Dynamo + NIXL wheels along with an executable base container (with Dynamo + NIXL pre-installed) that can be used for CI. We also define a structure for the backend-specific Dockerfiles and introduce other build improvements such as external caching.

deps/NNNN-container-strategy.md

nv-anants · 2025-06-09T17:31:15Z

deps/NNNN-container-strategy.md

+### Non Goals
+
+- Slim backend-specific runtime containers to use for performance testing. 
+- Unified build environment


do we wanna add something about api-store and other k8 containers here?

not able to find this, is this updated?

…nd specific build

…with discussions around multi-arch support

…process into seperate DEPs

biswapanda

Lgtm! 🚀

nv-anants · 2025-07-22T16:09:08Z

deps/NNNN-container-strategy.md

+
+## Container Build Flow
+
+```mermaid


We should also consider a path for doing editable build in framework runtime containers. After build, there should be a path for mount local repo checkout in container, do pip editable install to replace wheel. Technically nothing to change in the build as such, but would be nice to include it in the DEP on how to do it.

nvda-mesharma · 2025-07-22T16:11:13Z

whenever we introduce a new container, customer has to do security scans, QA, etc. Do we want to reconsider the strategy to have backend specific containers?
LLM NIM has been doing this for a year now, one container with trt-llm, vllm, and sglang. Shall we also create a container with all backends included?

nv-anants · 2025-07-22T16:14:55Z

deps/NNNN-container-strategy.md

+
+**Not Supported:**
+
+


can we also add something about future caching we can leverage to speed things up, like sccache for rust builds. Not urgent, but would be list out some future optimizations

nv-tusharma · 2025-07-22T21:19:36Z

whenever we introduce a new container, customer has to do security scans, QA, etc. Do we want to reconsider the strategy to have backend specific containers?
LLM NIM has been doing this for a year now, one container with trt-llm, vllm, and sglang. Shall we also create a container with all backends included?

@nvda-mesharma It makes sense to have a single container to reduce complexity but there are a few things to consider with this approach:

Is it feasible for a user to want deploy multiple inference engines at once with dynamo?
Container size, if a user is only interested in one particular backend, we should remove the dependencies associated with other backends. We would need to provide tools for users to create backend-specific containers for their deployment.
It is expected that Backends can differ on performance, which could be a result of dependencies.
Build times, if we have to build each backend from source in one container, our build time would increase (since layers can't be parallelized without docker buildkit)

I think providing a Dynamo base image with minimal dependencies is sufficient for now but I'll add this suggestion in the Alternate Solutions section.

whoisj · 2025-07-23T18:23:50Z

@nv-tusharma please add an appropriate PR description.

whoisj · 2025-07-23T18:24:48Z

deps/NNNN-container-strategy.md

@@ -0,0 +1,338 @@
+# Dynamo Container Build Process Improvements


do we require a copyright header for DEP written by NVIDIA employees?

whoisj · 2025-07-24T15:36:46Z

deps/NNNN-container-strategy.md

+
+## Terminology & Definitions
+
+| Term | Definition |


please try to keep the markdown readable as plain-text.

| Term | Definition | | :----------------- | :--------------------------------------------------------------------------------------------- | | **Base Container** | Pre-built container with Dynamo and NIXL that serves as foundation for backend-specific builds | | **Backend Build** | Container stage that builds backend-specific code and dependencies | | **sccache** | Compiler cache tool that speeds up recompilation by caching previous compilation results | | **CI Stage** | Container stage with testing tools and validation requirements | | **Manylinux** | PEP 513 standard for Linux wheel distribution compatibility | | **NIXL** | High-throughput, low-latency point-to-point communication library for accelerating inference | | **Runtime Stage** | Minimal container stage with only production deployment requirements |

alec-flowers · 2025-08-05T23:57:05Z

deps/NNNN-container-strategy.md

+
+This document proposes solutions to the build process challenges, aiming to improve overall container build efficiency and developer experience.
+
+## Goals


What is the plan between github and gitlab?

How do we make sure we share a cache and that it is clear where builds should occur. I think we should do all building in github as it relates to CI/CD.

Gitlab in my opinion should only be for benchmarking and should just reference containers that we build from github.

@nvda-mesharma , @nv-tusharma - is it feasible to build all containers on github runners and then pull them into gitlab - that might be better discussed in test strategy than container strategy though -

lets scope this to how things are build and run -

nv-tusharma · 2025-08-20T16:44:49Z

Need to add some information about potentially publishing a base container which has NIXL + UCX pinned that we can build dynamo on top off. Since nixl is pretty much locked and does not change often and similar for other packages -- we could have a separate workflow that we can use to publish a public image that we can build on top of.
Not sure if we can do that on github or ngc, but something like -
Base image -- cuda+nixl+uv+cargo+..... We use this to build dynamo and other things
Dynamo image -- base image + dynamo rust/python builds
Framework image -- dynamo image + framework builds

whoisj

Overall this looks great. Added a comment about the readability of the Mermaid graph when using specified colors.

whoisj · 2025-08-21T22:02:09Z

deps/NNNN-container-strategy.md

+    E[Dynamo Base Container]:::gray
+    F[Build Backend from source]
+    G[Backend Build Image]:::gray
+    J[Cuda Runtime<br/>nvcr.io/nvidia/cuda.XX.YY-runtime]:::gray


selecting a color is always risky due to user's local experience.

example:

Changing this or not is your call. I just wanted to call this out and provide the information.

nv-anants · 2025-09-03T19:42:48Z

deps/NNNN-container-strategy.md

+
+The base container will be a pre-built container that will be used by the backends to build the final container image. This build base container will contain a Dynamo build for all backends to use for their framework-specific build. The base image will leverage a manylinux base image to enable support for multiple distributions (U22, U24, etc). The container will also include a NIXL build since this is common across all backends. This will be a new Dockerfile in the /containers directory. Multi-arch support is also a P0 Also, the base container can be used as a drop-in replacement for the current devel container used in public CI. This would significantly reduce public CI build times and enable faster validation of changes. 
+
+## Use-case of build stages along with relationship between stages (base, runtime, devel, ci_minimum)


to match with latest changes - can we add the dev stage here?

whoisj · 2025-09-03T21:45:28Z

LGTM, I'm happy with what we have here.

mc-nv · 2025-09-03T21:51:08Z

deps/NNNN-container-strategy.md

+
+### Implementation Considerations
+- **Cache Invalidation**: Implement smart cache invalidation based on dependency changes and version updates
+- **Monitoring**: Add build time metrics to measure cache effectiveness and identify optimization opportunities


Suggested change

- **Monitoring**: Add build time metrics to measure cache effectiveness and identify optimization opportunities

- **Monitoring**: Collect build time metrics to measure cache effectiveness and identify optimization opportunities

mc-nv · 2025-09-03T21:52:32Z

deps/NNNN-container-strategy.md

+### Implementation Considerations
+- **Cache Invalidation**: Implement smart cache invalidation based on dependency changes and version updates
+- **Monitoring**: Add build time metrics to measure cache effectiveness and identify optimization opportunities
+- **CI Integration**: Configure CI/CD pipelines to properly utilize remote caching storage.


Context of the document include Docker and compiler cache, can we be more specific on this task?

mc-nv · 2025-09-03T21:54:19Z

deps/NNNN-container-strategy.md

+
+## Deferred to Implementation
+
+TBD


Confused with this section... may be redundant

mc-nv · 2025-09-03T21:56:26Z

deps/NNNN-container-strategy.md

+
+Pinning/Fixing dependencies will ensure a unified build environment reducing "it works on my machine" problems or "this worked yesterday"
+
+* Minimize effort for providing multi-arch support across various backends for Dynamo by leveraging manylinux to build for multiple distributions


Is this effort elaborated somewhere ?

mc-nv · 2025-09-03T21:57:19Z

deps/NNNN-container-strategy.md

+
+* Minimize effort for providing multi-arch support across various backends for Dynamo by leveraging manylinux to build for multiple distributions
+
+* Implement remote compiler caching to dramatically reduce build times across development and CI/CD environments


This one could be plural, especially if we use different build tools it may use different endpoints.

mc-nv · 2025-09-03T21:58:50Z

deps/NNNN-container-strategy.md

+
+### REQ \<\#1\> \<Backend Integration with Base Container\>
+The build-base container must be designed such that backend-specific Dockerfiles can integrate with it with minimal changes to their existing build process. This includes:
+- Multi-arch support is a P0. The Base container should be able to support both x84_64 and arm64 builds. 


Struggling to understand this point, may I know what kind of support we are looking for?

mc-nv · 2025-09-03T22:01:34Z

deps/NNNN-container-strategy.md

+
+### REQ \<\#2\> \<Layered Container Structure\>
+Dockerfiles must follow a layered, super-set structure to optimize build efficiency:
+- Each stage should build upon the previous stage or use artifacts from the previous stage


isn't Docker build doesn't care about order if the cache available and concurrency possible isn't docker will try to utilize it?

mc-nv · 2025-09-03T22:02:34Z

deps/NNNN-container-strategy.md

+Dockerfiles must follow a layered, super-set structure to optimize build efficiency:
+- Each stage should build upon the previous stage or use artifacts from the previous stage
+- Artifacts should be built only once and reused across stages
+- Clear separation between build-time and runtime dependencies


That's a packaging issue IMO, do we have a conversation with developers about it?

nv-tusharma added 6 commits June 3, 2025 11:33

init commit to start outlining container strategy proposal

1c8c535

update container strategy proposal

64f95a2

updates to the container strategy

b0e99e0

resize container build flow image

5c38ad3

Resize image to better fit on screen

66cf8e8

Remove height parameter

0fe9448

nv-tusharma requested review from nv-anants and a team June 5, 2025 20:11

nv-anants requested changes Jun 9, 2025

View reviewed changes

deps/NNNN-container-strategy.md Outdated Show resolved Hide resolved

deps/NNNN-container-strategy.md Outdated Show resolved Hide resolved

deps/NNNN-container-strategy.md Show resolved Hide resolved

deps/NNNN-container-strategy.md Show resolved Hide resolved

nv-anants reviewed Jun 9, 2025

View reviewed changes

nv-tusharma added 20 commits June 10, 2025 23:14

Update strategy to build Dynamo in base container instead of in backe…

c7f34d0

…nd specific build

Fix mermaid diagram in markdown

8ed1fcd

nit: fix mermaid diiagram in proposal

75cbe69

Add descriptions to crosslinks in mermaid diagram instead of boxes

a29d831

Update container strategy to discuss container release process along …

d337ee3

…with discussions around multi-arch support

nit: fix minor formatting

cb3368b

nit: move build spreed improvement to non-goal

a9b9762

nit: fix minor formatting

97b4a8f

update container strategy to add release info

827a5f5

nit: fix mermaid diagram

dbff0ea

nit: fix mermaid diagram

e89d166

nit: fix mermaid diagram

e7a3e59

nit: fix mermaid diagram

342f361

nit: reorganize mermaid diagram

0b13afc

Add subgraphs for mermaid diagram

b3b2f11

nit: fix diagram formatting

3c7af80

nit: update mermaid subgraph

b70361f

Minor fixes + syntax fixing

30e5d85

nit: minor syntax fix

6d2bcee

Split out container build process optimization and container release …

1fbb42a

…process into seperate DEPs

nv-tusharma changed the title ~~feat: container strategy proposal~~ feat: container build process improvements Jul 15, 2025

nv-tusharma changed the title ~~feat: container build process improvements~~ feat: Container build process improvements to reduce build times and improve Dockerfile organization Jul 15, 2025

nv-tusharma added 2 commits July 15, 2025 13:38

Merge branch 'main' into tusharma/container-strategy

a935124

Revert changes to NNNN-complete-template.md

c8081b3

biswapanda approved these changes Jul 21, 2025

View reviewed changes

nv-tusharma requested review from nnshah1 and removed request for nnshah1 July 21, 2025 23:27

nv-anants reviewed Jul 22, 2025

View reviewed changes

nv-tusharma added 2 commits July 22, 2025 15:52

Address current comments and concerns called out

fd97542

nit: minor improvements

a98a776

nv-tusharma requested review from keivenchang, whoisj and mc-nv July 23, 2025 16:30

whoisj reviewed Jul 24, 2025

View reviewed changes

nv-tusharma requested review from dmitry-tokarev-nv, pvijayakrish and saturley-hall July 24, 2025 17:45

nv-tusharma mentioned this pull request Jul 31, 2025

build: introduce dynamo base container ai-dynamo/dynamo#2222

Merged

alec-flowers reviewed Aug 5, 2025

View reviewed changes

whoisj reviewed Aug 21, 2025

View reviewed changes

nit: minor updates to strategy

e180e78

nv-anants reviewed Sep 3, 2025

View reviewed changes

whoisj approved these changes Sep 3, 2025

View reviewed changes

mc-nv reviewed Sep 3, 2025

View reviewed changes

		@@ -0,0 +1,338 @@
		# Dynamo Container Build Process Improvements


		This document proposes solutions to the build process challenges, aiming to improve overall container build efficiency and developer experience.

		## Goals


		The base container will be a pre-built container that will be used by the backends to build the final container image. This build base container will contain a Dynamo build for all backends to use for their framework-specific build. The base image will leverage a manylinux base image to enable support for multiple distributions (U22, U24, etc). The container will also include a NIXL build since this is common across all backends. This will be a new Dockerfile in the /containers directory. Multi-arch support is also a P0 Also, the base container can be used as a drop-in replacement for the current devel container used in public CI. This would significantly reduce public CI build times and enable faster validation of changes.

		## Use-case of build stages along with relationship between stages (base, runtime, devel, ci_minimum)

	- Monitoring: Add build time metrics to measure cache effectiveness and identify optimization opportunities
	- Monitoring: Collect build time metrics to measure cache effectiveness and identify optimization opportunities


		Pinning/Fixing dependencies will ensure a unified build environment reducing "it works on my machine" problems or "this worked yesterday"

		* Minimize effort for providing multi-arch support across various backends for Dynamo by leveraging manylinux to build for multiple distributions


		* Minimize effort for providing multi-arch support across various backends for Dynamo by leveraging manylinux to build for multiple distributions

		* Implement remote compiler caching to dramatically reduce build times across development and CI/CD environments


		Not Supported:

feat: Container build process improvements to reduce build times and improve Dockerfile organization #8

Are you sure you want to change the base?

feat: Container build process improvements to reduce build times and improve Dockerfile organization #8

Uh oh!

Conversation

nv-tusharma commented Jun 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

biswapanda left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nvda-mesharma commented Jul 22, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nv-tusharma commented Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

whoisj commented Jul 23, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nv-tusharma commented Aug 20, 2025

Uh oh!

whoisj left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

whoisj commented Sep 3, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nv-tusharma commented Jun 5, 2025 •

edited

Loading

nv-tusharma commented Jul 22, 2025 •

edited

Loading