From 1c8c535598a71a14c7a5dd163ec57d68a894f465 Mon Sep 17 00:00:00 2001 From: Tushar Sharma Date: Tue, 3 Jun 2025 11:33:31 -0700 Subject: [PATCH 01/42] init commit to start outlining container strategy proposal --- deps/NNNN-container-strategy.md | 184 ++++++++++++++++++++++++++++++++ 1 file changed, 184 insertions(+) create mode 100644 deps/NNNN-container-strategy.md diff --git a/deps/NNNN-container-strategy.md b/deps/NNNN-container-strategy.md new file mode 100644 index 0000000..57ffba7 --- /dev/null +++ b/deps/NNNN-container-strategy.md @@ -0,0 +1,184 @@ +# Container Build Strategy + +**Status**: Draft + +**Authors**: [nv-tusharma/Dynamo Ops team] + +**Category**: Architecture + +**Replaces**: N/A + +**Replaced By**: N/A + +**Sponsor**: saturley-hall, nv-anants + +**Required Reviewers**: nnshah1, saturley-hall, nv-anants, nvda-mesharma, mc-nv, dmitry-tokarev-nv, pvijayakrish + +**Review Date**: 2025-06-03 + +**Pull Request**: TBD + +**Implementation PR / Tracking Issue**: TBD + +# Summary + +This document outlines a container strategy for Dynamo to further improve the build process by focusing on reducing build times and improving developer experience with pre-built Dynamo containers. With these goals in mind, This DEP covers various optimizations to the build process, including: +- Restructuring the build process to provide a build-base container with Dynamo pre-built, enabiling specific backends to use the build base container to build the final container image. +- Improve CI registry service to provide pre-built Dynamo containers for developers to use from their CI pipelines. This registry should be available to external contributors as well. +- Discuss possible further improvements including external caching to reduce build times. + +# Motivation + +Dynamo is primarily built from a collection of Dockerfiles hosted in the /containers directory of the[Dynamo repository](https://github.com/NVIDIA/Dynamo). These Dockerfiles build Dynamo along with the specific backend (vLLM, TRT-LLM, etc) and NIXL, the high-throughput, low-latency point to point communication library used by the Dynamo to accelerate inference. This approach has several drawbacks, including: +1. Build times are long because all Dynamo, NIXL, and the selected backend are built each time. This is especially problematic as we continue to add more backends to Dynamo as we have to rebuilt the entire stack each time. +2. Developers need to build Dynamo per code change, which is time consuming and results in a poor developer experience. +3. Currently, we do not have a way to provide pre-built Dynamo containers to external contributors. The Github registry is not performant enough for public CI use-cases. + + +## Goals + +* Reduce build times for Dynamo by providing a build-base container with Dynamo pre-built. This build-base container can be used by specific backends to build the final container image. + +* Provide a CI registry service to provide pre-built Dynamo containers for developers to use from the CI pipelines. This registry should be available to external contributors as well. The registry should be fast and reliable. + +* Outline possible further improvements including external caching/multi-context docker builds to reduce build times. + +### Non Goals + +- Slim backend-specific runtime containers to use for performance testing. +- Unified build environment + + +## Requirements + +**\[Optional \- if not applicable omit\]** + +List out any additional requirements in numbered subheadings. + +**\** + +### REQ \<\#\> \ + +Describe the requirement in as much detail as necessary for others to understand it and how it applies to the DEP. Keep in mind that requirements should be measurable and will be used to determine if a DEP has been successfully implemented or not. + +Requirement names should be prefixed using a monotonically increasing number such as “REQ 1 \” followed by “REQ 2 \” and so on. Use title casing when naming requirements. Requirement names should be as descriptive as possible while remaining as terse as possible. + +Use all-caps, bolded terms like **MUST** and **SHOULD** when describing each requirement. See [RFC-2119](https://datatracker.ietf.org/doc/html/rfc2119) for additional information. + + +# Proposal + +**\[Required\]** + +Describe the high level design / proposal. Use sub sections as needed, but start with an overview and then dig into the details. Try to provide images and diagrams to facilitate understanding. + +# Implementation Details + +**\[Optional \- if not applicable omit\]** + +Add additional detailed items here including interface signatures, etc. Add anything that is relevant but seems more of a detail than central to the proposal. Use sub sections / bullet points as needed. Try to provide images and diagrams to facilitate understanding. If applicable link to PR. + +## Deferred to Implementation + +**\[Optional \- if not applicable omit\]** + +List out items that are under discussion but that will be resolved only during implementation / code review. + +# Implementation Phases + +**\[Optional \- if not applicable omit\]** + +List out phases of implementation (can be single phase). Give each phase a monotonically increasing number; example “Phase 0” followed by “Phase 1” and so on. Give phases titles if it makes sense. + +## Phase \<\#\> \ + +**Release Target**: Date + +**Effort Estimate**: \ + +**Work Item(s):** \ + +**Supported API / Behavior:** + +* \ + +**Not Supported:** + +* \ + +# Related Proposals + +**\[Optional \- if not applicable omit\]** + +* File + +* File + +* File + +* File + +* File + +# Alternate Solutions + +**\[Required, if not applicable write N/A\]** + +List out solutions that were considered but ultimately rejected. Consider free form \- but a possible format shown below. + +## Alt \<\#\> \ + +**Pros:** + +\ + +**Cons:** + +\ + +**Reason Rejected:** + +\ + +**Notes:** + +\ + +# Background + +**\[Optional \- if not applicable omit\]** + +Add additional context and references as needed to help reviewers and authors understand the context of the problem and solution being proposed. + +## References + +**\[Optional \- if not applicable omit\]** + +Add additional references as needed to help reviewers and authors understand the context of the problem and solution being proposed. + +* \ + +## Terminology & Definitions + +**\[Optional \- if not applicable omit\]** + +List out additional terms / definitions (lexicon). Try to keep definitions as concise as possible and use links to external resources when additional information would be useful to the reader. + +Keep the list of terms sorted alphabetically to ease looking up definitions by readers. + +| \ | \ | +| :---- | :---- | +| **\** | \ | + +## Acronyms & Abbreviations + +**\[Optional \- if not applicable omit\]** + +Provide a list of frequently used acronyms and abbreviations which are uncommon or unlikely to be known by the reader. Do not include acronyms or abbreviations which the reader is likely to be familiar with. + +Keep the list of acronyms and abbreviations sorted alphabetically to ease looking up definitions by readers. + +Do not include the full definition in the expanded meaning of an abbreviation or acronym. If the reader needs the definition, please include it in the [Terminology & Definitions](#terminology--definitions) section. + +**\:** \ + From 64f95a24a4217de29f85872790eb36f5bcd8aedc Mon Sep 17 00:00:00 2001 From: Tushar Sharma Date: Thu, 5 Jun 2025 12:45:59 -0700 Subject: [PATCH 02/42] update container strategy proposal --- deps/NNNN-container-strategy.md | 112 ++++++++++++++++++++------- deps/container_strategy_proposal.png | Bin 0 -> 47514 bytes 2 files changed, 85 insertions(+), 27 deletions(-) create mode 100644 deps/container_strategy_proposal.png diff --git a/deps/NNNN-container-strategy.md b/deps/NNNN-container-strategy.md index 57ffba7..55f1585 100644 --- a/deps/NNNN-container-strategy.md +++ b/deps/NNNN-container-strategy.md @@ -1,8 +1,8 @@ -# Container Build Strategy +# Container Strategy **Status**: Draft -**Authors**: [nv-tusharma/Dynamo Ops team] +**Authors**: [nv-tusharma] **Category**: Architecture @@ -22,24 +22,36 @@ # Summary -This document outlines a container strategy for Dynamo to further improve the build process by focusing on reducing build times and improving developer experience with pre-built Dynamo containers. With these goals in mind, This DEP covers various optimizations to the build process, including: -- Restructuring the build process to provide a build-base container with Dynamo pre-built, enabiling specific backends to use the build base container to build the final container image. -- Improve CI registry service to provide pre-built Dynamo containers for developers to use from their CI pipelines. This registry should be available to external contributors as well. -- Discuss possible further improvements including external caching to reduce build times. +This document outlines a container strategy for Dynamo to enhance the developer experience by +organizing Dockerfiles to maximize coverage and reuse. The primary goal for this document is to define a clear and maintainable structure for our Dockerfiles—specifically, to determine how many Dockerfiles we need and clarify the relationships between base, runtime, development, and CI images. The aim is to ensure each environment's Dockerfile builds upon the previous (as supersets), maximizing environment consistency and coverage during daily development and testing. +To achieve this goal, this document proposes certain optimizations to improve the current build process: +- Restructuring the build process to provide a build-base container which contains all build dependencies, enabling specific backends to use the build base container to build the final binary. +- Defining a structure/template for all Dockerfiles to follow to ensure consistent and reproducible builds across backends along with specific roles/use cases targeted for each stage. # Motivation -Dynamo is primarily built from a collection of Dockerfiles hosted in the /containers directory of the[Dynamo repository](https://github.com/NVIDIA/Dynamo). These Dockerfiles build Dynamo along with the specific backend (vLLM, TRT-LLM, etc) and NIXL, the high-throughput, low-latency point to point communication library used by the Dynamo to accelerate inference. This approach has several drawbacks, including: -1. Build times are long because all Dynamo, NIXL, and the selected backend are built each time. This is especially problematic as we continue to add more backends to Dynamo as we have to rebuilt the entire stack each time. -2. Developers need to build Dynamo per code change, which is time consuming and results in a poor developer experience. -3. Currently, we do not have a way to provide pre-built Dynamo containers to external contributors. The Github registry is not performant enough for public CI use-cases. +Dynamo is primarily built from a collection of Dockerfiles hosted in the /containers directory of the [Dynamo repository](https://github.com/NVIDIA/Dynamo). Dockerfiles are split by backends (vLLM, sglang, TRT-LLM) and each Dockerfile contains multiple stages +(base, devel, ci, runtime) to account for different purposes. Each stage essentially provides a Dynamo build along with the specific backend (vLLM, TRT-LLM, etc) and NIXL, the high-throughput, low-latency point-to-point communication library used by Dynamo to accelerate inference. +This approach has several drawbacks, including: +1. Inefficient Build Times: Components such as Dynamo, NIXL, and the selected backend are rebuilt multiple times across stages, instead of leveraging a layered, superset structure. For instance, Dynamo is installed three separate times in the Dockerfile.vllm—once each in the base, ci_minimum, and runtime stages. +2. Poor Developer Experience: The lack of clear organization among Dockerfiles makes it difficult for developers to identify which build suits their needs. As a result, the devel build is often used by default, regardless of the use case. +3. Flaky Builds: Due to the large number of layers along with multiple repeated steps across stages, builds can fail intermittently resulting in flaky builds. +4. Lack of standardization across Dockerfiles: Currently, there is not a single, stand-alone Dockerfile to build Dynamo, NIXL, and dynamo dependencies resulting in duplicated/missing code across multiple Dockerfiles. Optimizations applied to one backend's Dockerfile are not immediately available to other backend-specific Dockerfiles. + +As Dynamo continues to scale to support multiple LLM backends along with efforts to provide pre-built Docker containers for external usage, we need to define a structure to our Dockerfiles to improve container usability. ## Goals -* Reduce build times for Dynamo by providing a build-base container with Dynamo pre-built. This build-base container can be used by specific backends to build the final container image. +* Remove duplicate code in current dockerfile implementations and define a single build base image containing all the necessary dependencies to build Dynamo/NIXL specific dependencies. + +This build-base image should operate as a single base container which can then be used as base containers for backend-specific images. By leveraging a build base container, We can reduce the redundant code across Dockerfiles and establish a single-source of truth for all Dynamo-builds. + +* Define the relationships between base, runtime, development, and CI images for each Dockerfile and provide a structure/template to follow for Dockerfiles. + +* Reduce build flakiness by pinning/fixing dependencies in the base image from package managers and squashing/reducing layers as necessary -* Provide a CI registry service to provide pre-built Dynamo containers for developers to use from the CI pipelines. This registry should be available to external contributors as well. The registry should be fast and reliable. +Pinning/Fixing dependencies will ensure a unified build environment reducing "it works on my machine" problems or "this worked yesterday" * Outline possible further improvements including external caching/multi-context docker builds to reduce build times. @@ -51,46 +63,92 @@ Dynamo is primarily built from a collection of Dockerfiles hosted in the /contai ## Requirements -**\[Optional \- if not applicable omit\]** +### REQ \<\#1\> \ +The build-base container must be designed such that backend-specific Dockerfiles can integrate with it with minimal changes to their existing build process. This includes: +- Clear documentation on how to use the base container +- Standardized environment variables and paths +- Well-defined extension points for backend-specific customizations -List out any additional requirements in numbered subheadings. +### REQ \<\#2\> \ +Dockerfiles must follow a layered, super-set structure to optimize build efficiency: +- Each stage should build upon the previous stage +- Artifacts should be built only once and reused across stages +- Clear separation between build-time and runtime dependencies +- Minimal layer count to reduce build complexity -**\** +### REQ \<\#3\> \ +Each build stage must have a clearly defined purpose and scope: +- Base: Common build dependencies and tools +- Development: Additional debugging and development tools +- Runtime: Minimal production deployment requirements +- CI: Testing tools and validation requirements -### REQ \<\#\> \ -Describe the requirement in as much detail as necessary for others to understand it and how it applies to the DEP. Keep in mind that requirements should be measurable and will be used to determine if a DEP has been successfully implemented or not. -Requirement names should be prefixed using a monotonically increasing number such as “REQ 1 \” followed by “REQ 2 \” and so on. Use title casing when naming requirements. Requirement names should be as descriptive as possible while remaining as terse as possible. +# Proposal -Use all-caps, bolded terms like **MUST** and **SHOULD** when describing each requirement. See [RFC-2119](https://datatracker.ietf.org/doc/html/rfc2119) for additional information. +**\[Required\]** +In order to address the requirements, we propose the following changes to the Dynamo build process: -# Proposal +## Build-Base Container -**\[Required\]** +The build-base container will be a pre-built container that will be used by the backends to build the final container image. This build base container will contain all the necessary dependencies to build Dynamo. The dependencies should either be pinned or fixed to a particular commit SHA to promote reproducibility. The container will also include a NIXL build + NATS + ETCD installation since this is common across all backends. We will create a new Dockerfile in the /containers directory for this container and provide the image through our CI registry for developers to use for local development. + +## Use-case of build stages along with relationship between stages (base, runtime, devel, ci_minimum) + +Each backend-specific Dockerfile should follow a specific format. The backend-specific Dockerfiles should be divided up into multiple stages, with each stage inheriting artifacts/leveraging the previous stage as the base container. The following stages should be defined in the backend-specific Dockerfile: + +| Stage | Targeted User | Base Image | Functionality | +|----------|---------------------|----------------------|----------------------------------------------------------------------------------------------------------------------| +| Devel | Developers | Dynamo Build base image | Builds targeted backend and Dynamo; includes development tools for debugging and continuous development. | +| Runtime | Customers/Production| Cuda base runtime image| Minimal image with only the dependencies required to deploy and run Dynamo; intended for production deployments. | +| CI | Internal CI Pipelines/Local CI Debugging | Runtime image | Adds CI-specific tools, QA test scripts, internal models, and other dependencies needed for automated testing. | Describe the high level design / proposal. Use sub sections as needed, but start with an overview and then dig into the details. Try to provide images and diagrams to facilitate understanding. # Implementation Details -**\[Optional \- if not applicable omit\]** +## Container Build Flow + +![Container Strategy Diagram](container_strategy_proposal.png) + +The diagram above illustrates the proposed container strategy showing the relationships between: +- Build Base Container with common dependencies +- Backend-specific development containers +- Runtime containers +- CI containers + +This layered approach ensures consistent builds, reduces duplication, and improves maintainability across all backend implementations. -Add additional detailed items here including interface signatures, etc. Add anything that is relevant but seems more of a detail than central to the proposal. Use sub sections / bullet points as needed. Try to provide images and diagrams to facilitate understanding. If applicable link to PR. ## Deferred to Implementation **\[Optional \- if not applicable omit\]** -List out items that are under discussion but that will be resolved only during implementation / code review. +TBD # Implementation Phases -**\[Optional \- if not applicable omit\]** +## Phase \<\#1\> \ + +**Release Target**: TBD + +**Release Target**: Date + +**Effort Estimate**: \ + +**Work Item(s):** \ -List out phases of implementation (can be single phase). Give each phase a monotonically increasing number; example “Phase 0” followed by “Phase 1” and so on. Give phases titles if it makes sense. +**Supported API / Behavior:** + +* \ + +**Not Supported:** + +* \ -## Phase \<\#\> \ +## Phase \<\#2\> \ **Release Target**: Date diff --git a/deps/container_strategy_proposal.png b/deps/container_strategy_proposal.png new file mode 100644 index 0000000000000000000000000000000000000000..2fb0c1285ce2c67443c814d088a763398833fe98 GIT binary patch literal 47514 zcmeFZc{r5s|2M2pq6k^CBvgtiMJPMdq!L+1iew8R>)4mUOeo5dFhaH|TYZ#$XG{^Y zP7Sh-$-d8EFvcwIOW)t~JMQ~_?&tpJx$p1s9QX5a94<50bzawbdY{YndcWSU_xTcP ztjEK3f{TrfjYt3Xjk|1Y>`*o~?iYs-0%x|u8Vi6wryOtJHDY6f%doLMea6PN3!Hko zz{VDEnT>7f5gVIEIvbmacUFU`78~1vGN|Fbn|phEV`F1)-@b*z;W!*_dwZKiBHg-m zOIcZ&PNzSzcYL4!0p{)N>guYid*jEz2yuGh!-o&8t*t|2zhmQH#l^*iBO=Cs&wehg ztgWr>?D<*S(EO$K`kKbMf@Iuz9R`{Z4U7#eFNskDtmibMwauL~94PWK>%AyMmbbBt0YZjI4KV zus|UBTZC*tUS+P*-=)?w`Yb17ly?dn2C3K(1e}sKvx5 zrle)&cJ%gJ+t~eGQWXG?$jr%GC6R-Ig5DSW{js<#IyOGP8<`Nx**nHZ?PQO%ECt5mQ;)VDI8pS<`^Qwq)eI*EKMUijH@M z1whShZW`Wy_A)UzJQCkOc-QigmagFhadK#6)XLt~Hz>Rb+meOO@B4wzFZv`k@bn$f zsRtbH=-ps@aY*7e@a2g2ZA(8kHqp%e-vjyqQ7GUfr@y|@P0mG*LxPe#>q(8uz(8Qr zzX7=yJi5w^a}-~Ei>4j`v28&F@;S~{-AjJ>G|O){%PT0b@6w2JWu_SAf-R&yGZWK! zHl@<%*NFAdO9@%yEHx2LDpH;Jnd|dsv9;?9!x#-SIERoIut8 zFMh84SUnef4a2&&tH;NmG8Az~5_Kf!`9PCPvCTH8r_tJDp9|JRrv4%84eH{nc+DM|Ql9##+&8rDD?w z#~}x5-dCfWI)egQki?mhV0!2Lmp9XN2S?k2>5!k1E_1V1st2a2Vdj1(rF5DuMQ&^} zDP3L7IE#si(54wehXZbLHHPnIoh>qq3SWIg!c$AJp-fU#V_WgORNUBo(kzC#F&4%< zKNz#|I~L8h{E)kgGOu|0G$_QD(N6!#2i-8*4W)dnBP_X~dz<)4S_#sTo?A3($3|i~>Te-R_~W zOU#i>liBw(kdW1st!x5?wmD}xz6Iw8Bg73hJFm`$(B2^O5YDArc%wKr=MBKkbJ+HbXmTLNobw7U-+G}ChtG*S@(r}46{Z1z|r=(boo~h@wnp$ zhV2Rx+-91Jjj~wh4|^aL$}fce;S9By8M4V;(kC@9pS{8Z?EoJ}nFd zk;k=`i};AC?TXF;o3uG})9$;vG>6MG*p0C2^oA<}Lg;>9XmhBu4rc3M1U{$J9eapX zHd-Ho(I7Je)cNdj9|TZkN5R+2riCbry%dY9ke7|8!FA^c=A|4PuG{KhdOH;R_zJ8h zj-f<~LqjqXUn34fLd4QG-X}PAuphcmEiTk}T8B^E-)B@`w5HBe2b;H1iS0ec~*_uGt%f~h;+WCVy37(l&)>{MPV^I#|wpDFRLZcS=llg|Drk^bP zp}&DgQbV;oevnsc7%E<`6k5y2yzk!Ds;TI@T{k+GhR~QbdeKWoL?u>!Bpao#GwoB0 zs?j^;hFQPErFgBtRGYQ)-}CNkEUvbrzlDa(r!g?%zh_cfN~heg;eGawzhy3Q&aM_Q z!V*nL0THEk%rzz6ljHZ=;}VJWQBAmsH?W10?Oj6A@ia0OonxRIWuzSF9pvg?hHTCMy~Er%Kf{^UZU^nLV-0#ftIS7xH_Gw zM%%!IEE@kEZi<*?gB-ZOQ9X0Lb7deKO&ycZYw zO|W~SVRg5_lG}h3{v;(-@qO;WK-sh=?@D=MxQOkWot4?q^FCXU^;$X7Y+f!poF<<& zKV@7QYX5LInqTO{Kp?bP&bFUoqqeZ&arGdb+1)iJ7G}{r8W&tW6nLAmK_^fBlv178 z*_8KKXI zue+`mwRe+1yTX2Gt%m-}bD12!xyxKGwfnr$xok0Oh4g$_Rk5=nt2*wJT`NxA&{h08 zK0#Lk+V#!q+QiCgrg!Y>o&M3dcy3?-@uM=Czir<$ScJ;0{*bY2P_zJ~zAaNtzn{hS z9DPkT(l^tB+>Bh;)vPRY2VNv+KuZ6^V^`G1dPt^y2)#MG*7sK&Fp)%2wF=^|?oDVF zYld!xu!1f%BQmKt0%Sy0;cc_(ml#ddJU45#bKgUm^m2|Hu-gI zz8KqMU+P6>>e7Az&1^$!!Gbq_74K3NLVd8qs$ zCxWl88;xxR-G#hFF8sjG7tlB9msvZ_jESRUb8?Claj8vjK0HhW7Wvn6Lx#I*GaD`! zi=eO9?~P@ZQd}cyC!>a>a?pEJ=i_TQ{-HH_7as04E+ig=}?b zpNzuj50^~egslz@72Q}#2Ah+X0~C!WS@BOKg@URZ?0+ul0HbHS)y=lRm$=Q{cJzBO zzT^4jE9WP`^6_9DbL#kY&)9eRJ95%6FpYc_5@A3K6{LN}vkaIsd*g+~qb)W36KuPA z$mzTt`3;4M^eMv9G*J#|$~u@6cxQK^l=_pnkZ1ROc_)<|H>R~WLoCRzDn|~7E%q$q zrA~cjW#flfsVd-S_mRhm>8_rYrS*xv&J=O{s_or&4v&k9ZZKBUl% z^xkS6GZ_RYiaI|;e3zoTo4FkWS~a;S!8y+=MKsi&htJxIQe9KoETTnVIt^iyg)xBP*G z-}Vq(a>AqGgwWobWY~u4lgiLju#Mty3rxV4P8Y;Sj&bJJ%zaw(YT5+j7%{V#(8foT zQo9stpA&A^u*n%(T_(P1v#3Pb-Ew>Ti|G{HXt0j)Il?8>Mbx^=%J!q$H3qO=@2xkb zKQ)<57-G-|v33bJHa>4=Bc*C;anlMrdvb-Onlg3=K2g{ zb=D3Onkw>M0OjKIL5lTv9YJkcWoNQ6OkL5$K7*x_ilt1BlIXv*5ub>Ti1Q`SJNdn@@aK^U@Pj!G^*Ul^fGR z?1$VwADV4-JACYxpw0nYxA+phE6|LVnY|8I2-n(p9(^P}Y=8o1swi9#rvsRkh+?%6 zm%9|0zR->7yRbOzSSm14;*kRCNaE64da!Ae^fpB+o+dDX2$Sqk3gnwN%9^YWpdcd# z_L8>eXTjHYYBa9W-03i3u0=`99iY{%D zlEOs^@RskGnrNk0(SmfQwfELMC??U(pC>Gue@_`UT|Q4&L{b>O&_Dlkmq1P$iJpr6 zl{rM5CAUb4*c|Di&W*(3?bypTU0}cFp|!K|V?|nROjYKRV!~#%K$BNnY4ZI=d@NXW^Y(mh zn1!uDv&eeqOzD$87-c1#6?BVo5KJl+KuXOk>R@79pHYbz_?)bqPI?-7qM@|$5Z0%7 z#>9(T8dXcgAiRD>-u078MSnZjv?Fh<(?Mq~bjX)7q|^dl(ta!K+OM`x)}?)HdLxTu z_GPLTUp)n2>kWzuc#wTIz^8{&4USMbDUtwrnR(Y|+rr0dkxvo8DzvXNm1x=`3y}mf z7pGJ3c9PXrb=d7p@6Ty0SZ*escs^XB(88{H-48VRMqy~b1+y|*6r}6GjxUD$&W_Mt ze0d1%d=_dNhYnD#Ox0~E?i9pMSHUG7sl_ti4Vy`Wf*-ahGXm6%D}_;Hb7a!oXHCBC z45Z-euxA_gSFe|g<1431U+Kgtzgvb_W-GllL24mgb1Us@nx2BI#Gc2GGx6==kFa=C^H-hmvnhem@G5_ zKRX^gyQj9umpZMQYaZ@XDiBt#(L@PXyV!pCkcnAXG`4Rh zbW92e+SqOnLWVN8mxth?bhorU{Ni$^*Y@9wjbOq)40nKTR1_r4>A-B(o&m@OmH5@} z3ZlE}D%mRB?_o6nL8ffTPRE2k+<-p*=lFlOKp4#V5A*UdL>A+zt>3;`uurhS=lZP{ z6PAzcKIs7{9=o?-T!H?wN9DH7i zC|3DFD}46F{Mu4kXDtc+y&n;mPV@vN5v~+iygR}n`^E8b)ENvHTNPTC?V&D|ttuML z_Jjv=;0FMk&{x?HT?4S=(_hDg*gl^I{U1_b^at*u<2ZH%3U^{#sd;HBcDwZGQ-X^1 z*5iv9dA2~p`@>!L;P=^98jgf&v5hJ{PUO7iz7d}Dax<1~I2`1PQHOZbG}frn1`u^}^*Z?uB{l2~PW-}=)+WhK(|UKhr@*CctiF++4Vw z@5Wr)v2#)%$E!Y;6dJ)vg|xvbUlpm=CFfN1#i=|wPN&xm&z~QkINJ3iJqfABU3s`R ztF2M$jk9}KrN+8rQ5PbEG zhHrLv&N(lUClE^(BZNdIM|{y~1(r@x6ffjJ69?qwp=o^e?Z( z+dr1qO3DQK`KM$IqKC(ai!U-j&cEZ1MJa7q)SU*|G=48HpDyx+-a@F|<%vzOQ#$hQ z;}HWK_{H<`pUd>n&Ac3+2j_yGyEaMH!kKI3claO&G!h~OicVjEtmh^8gm6`Tf~kJ7 zMSufCbH?l_g;zEKmhEw#F_M4oav_4=X9&~DH4#R9Rb_XY6xK}7Fn}_=A8UhZlxVMA zl|kWYJev6xwJS6mDeS3Q16^buWG8s#&BAuhO23ZPy|+1`tCkWPV}sUGWUnr}aZNOV#Rg11d( zZlO_uFpZB#`kUg;>bKj2Kj?rghfgn_b%&Hs$1cL58`^Ig(8CJbs|efxVe1ESN_d7k zYwf&pT?Vmb)>vk_xIY+9aJ21BlyW$Ez+JO6Fs@b^MfO#X%%>AZ(LK|DSy0ZcyS4hL z1?B62Fk6~@QVqH$gJ?sKoT+NJ$SIC<+^R1s?u%s%S6;MMx6C%3Hdj8clOq5$TUdx5 ziUrGYwvmp<@A|mRwR^VV=C)T$OnzWbgP4P|>&rCLoRTr>i%ilJZP27~=rTm?l3}S8 zZ>&{Zg!{E36_B5@{Nb4D*E>P9j8th2BMG2v(Oj-s^C7k6PHk=Od?LAv0GWE3C% z&1Q<4lj1hE>&|?)FhuOx^Bwe#vQO(^vX<#-5DVRXx#!7dF=Vm*%Eez%Q*Z7`zKa*O zx090ybc>pbozv4iSowmkF^lfm5PYubb-8kPzf;>G<(W6se?aKljbnDJl0sC6@u1bm$V?i}WkzviMly5w0JuCxwvdb%zbKZvK+cSxGB@`5C zi(Kir$9`zx3SaElQwrKYPP(0SKQiOI)NBBU&jsamaW_T-wsRO(B@M=?8(oKvM|%jL zmY5uT+9tp(>3_5WYZev?`l~pU_S+dczB>yM%U@q*Qy06t5SAbv5`{K>Hx zt`h*p-b{~DGHhnm9B7OM0C>kqp}~1P)GHzC678+U|NW{(K z@a}Bac;Nr1V)Xy1E8@AM%DPi;r(eqT)P}l@tCuRUIP|52-V39k8D}vU00@r@F<}`k zUOiAopVGm>kkXhlY=Oj5;4=;{_tzsTQvkkU_r{iw?F2)5CahzD8Nr6#||d>5EETPER!6eZA^v;dbUk zsx9{;?gmDM(B+5e&o?#LtbQmv0s|*@uKn`GXVGGa;Svi*r&=lTk^E+*D${rH|Z?)q-uglnoAI_Q;*W!GHv%+J&xF zz3ncU)va)A0i@5{$Enp~#=@iviI1-Vo*Vv6RPw7SJ)h~E(MyE;j_D7$E4}x`qMOf7 zPc8L*KVAsBYdz#MwWS$O&8?~nAF>^ZCtM%j5(uit<_mpDqKBO%H#fYooSy_<5mmG3 zbbSWMtIDxo88bLUF;Z6)%uN1SSLI1fSRCW-$|USI$h6hTR_i80Tn6|KaHS;{x{C75 z2C0beB=zh@8$zG{@=yg;*6oDlNu;-Z$uBPLFba{3k=e$!4d~%|#;>;5esbqWy|kQ> zes^%0gJ!VpUv&o};e(p-vzdHxPHiOP}CzEokF)uTU- zoPOUTQLkM%A)Ep%xqz*dxT2nENpLsbHzE9x-h(rqm}Za;!)5g44x+@Pl;Ra1V72B> z+l@N}EH`H!FpjrL342x{j2bwrT}dGGKq9UK;4SmHgz#c|BY%{HT)ep*-+2C#S~2fo z4zHQZ7wjB5j%J8}ajlA_fki?n|E+~N*5$hl|H=8lxq<*I?$|iosgbE$CO}haqVVfh zPKJ8vuO%{F7ni?&n5SwPEKezXLioKf7wP-s1dBx+n=->67Q;m{&3PxY}3Ux>`NQ3E?WvHW7)uI+~YpTqHb zaf4hFp}bC2ir>oOto;p&+Rmx6pBpzdfIbZ!K|nnO;FEclP)*swr$&a!f zZ{R3~pG9O!78X`b5*D>krn6e8MK_z4@K@trLt1Yf${xm<)_stQn&P{$m7#lpG=Ct{ zp*D!Rk@;TQYkKt!mUMk8S~Snn=SF@1maAyIk*kp3az;gpccBN?3|Z)*h0uPx(=F_* z;w#m-^?e0#zk}x`ob|<6%flm0S*q0Bh5L^+CC`9JPU;z%&e`7-dRnYI-O@Tp!x!Px zT`#4$Q??n!47@{a-sVF|g^wo#RXW?DQUPAACf7_KW7`5Oe9ypC;I6K%r zaRG#QVjpzx?^2bLrh6em&JO*-OIxo=DhXE}>Bp z{oM?9k}#GMKUW;e1#?gWVO%>O>8zHNzC_ZKP1mDkXQ{eJ>`I#!JD1?C_{Mxx{2UW+ zbM%f>N3DYx@AzGfNhl88*p_un2-K!oGdMi;*yGhc__-T}X6rqMI>CPENi=#O9kv$= z%%6Ywp?{kNeoBLWd{1xQyr8AO|BCKxf4KkN1kR=%_LFSBwInYMg`(L%@lr< zZKYWXDIU3z@`vCB$K)_9;o59Z)=Y*0Cie}IaeN>BZ7Bc2fXjebuNwf0=8U=8ea`nK zVV))naI)9pkef8sKd6ux&a#9bXM3{V4NkQ^$fhc0SSdlVORJ()+xG#s`ejs=0W;OjzQ@+Gv%yxqZs3GfXfBEg#@Du7_KkEn}&bf zh%f0S08(7NBNiGbnrR3>{&c&%L)y{V8K}D|5&ix9@Zd9m_~iiN^8XzFJr=Oy>3TvM z40BQwA3@GV4W26~er}b}yfqOMWb@1}gL5CqnMWz9+Kw)lAf4>t*~ShP8?h*zqpIB` ze_2%n((VsNm)2##tFMtjUF;SRkNG-k!9f3>LvL2-C!bK3M08__>Y#Y&(>ETPAQCw5 zo8eM(m>-m(5zr4CTOgejIs$~wftQemP0A>;vM14OcSMVo!s(=w$?G187ZVkvrtVg* zeLb}t6rEzLI3e~JbNNMv3+1c-n^`+i!xdX|NfZPoP%?ny*f!`7Q&W1|w_ALbtF|lV zMwDxEc>mhMzI+J&8g1Ac9AKd%U5wNSuf6lQ;w{=Ajwlv)a@EOGQhkXH#M#*o`$;KS zyDKSM)|oJFKl!a76@%_^VWpH{=MhU}IqUPiuw%a*n;$;xAh!1au9hsok)MuKvQBrM zFGTm)$b~7X47qv-Bs6J(`RiqWw&Ld!A~s>>5h)kUi2$*pJ1gk(NZdxFwM(x zHa$Pu^T9Uh{=Y>Y!(Q)9;=kK`)LeHwg_ZPj4R5>>)K;%g_p>TtZ8rmPtP7r+ApPXw z&+6grI@#&9x#iJpT1mJ*VJ^SL*n}A6oPO65rRTs zOAU(n!jF4V$`0i`(PQ>KaWE*A=$iWJ7FzJN zuCD39Xir<4jZaN)+71wi-Mp+9!r;*GEh@_JH|6R29w=?p8%$}7kl_hEXBM>aa@{4q zr}xa36y1nG@a6B*KhN1Fkza06Vdu-|xgr8<&vSz^QdfqA+h(T+@z%9Vwh;qo_Nlrv zplM5lTc}&|I|MWdAhk zw!`|+Vy(i@0X=Xx@eC+P#iCM7;J*#FS6M;CdTX_+*B=4Xdh|1T;h*T>zsCZKe{kQ& zecX4R81&>D%z)p|kpln^Ri&_=8*qu7oUg+d^M?)$M`tKvNVnmaa!_BdTrE_<2!BzD zZ*MmUOdhd3uN`qW&*LH{MEmNZ2Z(9LS>rN*3&hXkDA{KL?g=_gaap{|hUU=zKQVwW z%wWjPSJLP876T5HEwgt0NC4v6M|np4sN8>XfMydZe^4Glgc^dN%Pk%UC_o@~Z4xiK~q3jhMJAw=P7@U_KH;Mc5o2d$FzVRj~_8h?Zr zKb4(fdu)79_I?OISLC|#R@~2@afomETl$S%Sw0Vl^LBo1NzoIcXfz1=#KX8iwM2HY z$MBEhBVIjiq9!L_S~=-onK#$qWAJ48e?|4%&q4Ns`~P6Pe~SeQ+-TSSeYOW z6--76YmB>z+hk0(l-2b=Yh0ZN)e}+5S zjLaTrhBQLT`;1m=u-42eyEoghuUav=sG8{|nbjzV4gqHcTj)cA)8gNQ+MlKUX0d@y zT&A;JvNUrWcBkn^@BWsw)i5lu&bw)Y0?o?D5Y4Bh-kODeo!!b&ESh{J52T(Hb+BU7 zM7F1y3Tc6RGnc~xT){dT#G_sMJLIiYarjQDl-+1zX-l{ynVf;aYsFy#J-#%nYre9m zYk0-2Rg?Dt!<<&&v@iWoF?=vhl#YwA>24ivG&);efmLy)qPy82&QM-8fj)XZ<=aTvo6wcx<5{n* zc+!s5uDK(}lq-k&BqabivdTHzontejam<2blNlS!|Q50798DVb7y7<)n@e|Z= zrR&z?Ws|{{*5^D1y2HNsEC#*~6EqLKSe&P2e~F2u`{V^3e}L#ro0eCX|Akgc@Dqy$ zcj!E7*IOJeHPeZ?YwPi#rg6``cD(MM_$A~R^9>xJaQe+7eI;E?(-7{~o@=b`4o^k;!rPeFFCGi;eNUC-H9QbZ?p( z$IEgJF30>ZsNXS>O!CGzHd@clbE)TJHGzgSd>E@}Z!3msvbTzqVa7+fSsjkG%O}Ph ziAt^;iAHDHJ?Gc+gl?psKXf{5E0_rD?r!ylzqFd5y=Tn8i!U8_hD3F~ddN zeh&@1P*J`W4_6@TCQamShsl^W&#@z2$Z0H%2oK&{-Rr|r$Ohn9v?U$#QaZ`3F~p8` zss5x;v55m_{3KjO)w|cbtd#yeApr2cxF_mv)mAUJW)BFAXqzwm6cSM0#Ny^Sv zGDhN)f)~ioQXC~+83Yga_SS`NPqpj^>wDyJCVLDD(qdPgvke%erW1+91?qHw7WS@^ zu?|rN>3JQV)KhIUm>nwkOD@yk%#tFHB=Os@K2HEmDJEN;K0W?}Ncwi{L+5R_CiL0QBe2dJ@IFQ>v=EZ*dVwb2vop=J!Zv1ydk3-_{1%K`8!}oc zdEQ|~Cxxuy^4vw_8iz5v>O#l|V;J-!@&kRS#-468X!z|fyOY7GQtLKvcdozf%Kwy` zqOJLfuPd#9sJoc+DWEi!gOvsbKzY(FmaVH1?=Qqeo9 zw(U2we{s&Lg$+uH}MV$5m zSFKNiX}(R!harPT>BqDan3c&-e0s@1Vix*QsLb5T4frxVayx;a{-?mHtD$!mdzU-wFFVU@OUr_W-MaYX74$YV7N5ylPf!_xTDp}Xo=8d3bMj4yiTS5i8^-{p# z6yA!rG@K8<)}L`Hc3kv1Fv{oJ&q^<*2Kz>DH_8XjvmY8u$#@Rrxj<*tLw-5}uE+U-3B7L)pKzZ<;=mRdeYQWC$=lYb=n{~ikr zxIsDaiE|`(4X(WmXWfiJnLn~ynU~69USaQYw>7fumCoB$+lBok{yl^V_^Ge7GTwWc z_J+qd{gsWWlA|GxrRiA_4|{kZ@~`X@?B^7}IxydaHwFaG&0adEXvwk{hNVu_Y)yoF z&b5cN=J7(r3S_FKQPB`u;qgj_1gPlncv{wvHOqO7gf36*lcDpD>p%J<`R9 zwldd|Ox3X@e$W$7bh+FRlGMgE9-#0U5SColm^S+WzH&Y9~Hm?&Vg1-kkGO4LppoJj(Fu{vMv?oY!|b zK)k^!Q%y&|$K!{Xk)ejSh(a{Uv9_AJ@;gu2_v?ky6rxTeA~o5YXkX#H@vleno1I5chQgS zmjb+05;vZ$f(+65q`qlTA-V5xxw~MfQlj#ekMvm>!&y?sv*k(a=Nlo}&dwfKl@qkm zx+F7~1xd@>Z^W_wgboBwcA3vDGPzFr-nLg<-el`ec7`!$iHeRWcTn$T=W^;3jkXQ) zp3Pan@TSmq{p5TZFH``KSLIS9fO?Ke9kV$s$eZV-;uoY7I~LM}SkZTJYxk2qGFFD| z9kvY@1?RM{s=w*wJ&R|ZDzR1Dtty+%7)`p_j6xhXDtbULfZpvd)#=*!1QVt{gKu+0 z)iomyRp2+sF@U*Y>ALGtF`1G01Phy7U9nfpCyp9@MK|PP+!;;;@2&AqYUt#hVg-#! zUOC8Ooz1aYyNU)ARl1d}fv~j2C$DoL@t99tR*Z|J!vU*Kl;MJRRuTH;VSv84c@c#B z=DA!hNZ3Npbi7b~wXtc#zw%41uOaH}QFU_)-}aQH==G zDQNK9_~yvbF11fTmpszK1Uj-~l@j7I>LJF~5ihMGHvYprhhs}yzQ79es!pH~!zW~~ zLe{PBsx^LICTl5HUQW|E;_g`aNOcU7K9HX1LNS1j`=pwx6Em2fg0)>4QUM?7dv>X0gOeN%MZ3r_cCdtIF*nlI$XO4q+Me{kdlfg$Kry zUv1SHd3)$FQ~WQ10xMN9toLw90|(e_;GG6US2dYet5LI-uH`ys3>I&zUfi7qsT|fv zI`=r-mrGNLR$D`f8=sO{n1Rs=&04+);7d)3{w$M1z@$ zMqV=#HS#DRq>eMED>UP6aBxceA%)4`+SFU;n-eOwvbJU^+ z4K^#*e6JxnDP@1t3`Vb#EI3cJPAnOnm@B?%ZjP>CG)jY@`b5o|bDHZf07vD&PV>i$ zpgOaOx9;xyD7%pr;{V;{|GMZupZ^zHK!*L$2{9o5>$$9}2mD`fi(5I#_Ic5hQ=+~A z6w65W2VrhJMuD z0HmIA{OUO~Vr2HyRSxhkb^*l#Par?weEdxL$NBgl_+F2Cao7tC5JvuA-T>)O0rzW_?;&emi z=j{}YSv{4_pC?k~le3F}_w^z_`#FYcU`r`(Dc86Nd)c+LQ4H00>MFBu;37j-SG7mc z`ATKubu)0IlF5pfDW0eMq(Q2vhA}w(e~wAx_O-8Yk%#x^$KX3 z7N#SS+4jND3@T8$5X7NvCW2~i^mc}mrUwFA()~%ojb$S7%vt3jF_E{XE{_<0$+D6i z%BQX#zXo`=iN8UKWc_d`hy68s3RDnPFQ+$@pUtqFmUiI0uMaXo1%J+jq>D%@~f$lkUnN`C=P?BpNv*RAqJEoYl#<2aD6? zVyqt9b9RK|so9JEgn(QD*A4~*J7dyPLbW1FK!zZ}0c0wO`X=J=s1(m9o!|>yS#wdzP6V_hq8T%B zB4K~5GSAOmPuFkh@@Yr4Q;|%0|=Y>EZo%dOvJF5U_iHvqPau1(bK?sF$#BN5T=V zdjdRNXSypU&Sdf8Rh?BNGOJDqg)Y7)Dg(KJR=w>c!>h;$!sjC~ujCrFxcis4J5+n# z)g3fMlyjF`hyoIW&BMInjg>bTA2)wpvD7rHA@_lmEqmh*#(ll%qoJ=3BCJ)kibvwt zKkLi-7IgtP07E`x4%j)1c(kzJzKGlxgNsFKkOrfeKTj>s*PhrB54fW+(rJRl!6rq1 zify@!jN8sA@CB)og>r{YLf^xl0$U#8y|B(h;@mk9q9}lI8YNLb!yNDuVWsDQ855`t zG84r05fuS1Xf+I)+-a=C_OH|XX;S~>Lj4;o@ZaV_1q=bhzVfo*eufK2yp6gZA|pMQ#I@>jtWAKm z;*+N3ZP=sUFjm$|{>OzXU=KpKa27zOH3Bim>IMCOsM-GkGhH?}Z+9GlvQ7IJ;-RWj z#TW0701CMW)Q&SCKtbG9avJod0X(Y>2wUAN`~U*ROTbLi6l3y=_VII@L8j_{<>()7X2MIebT&o7A=L0Wt49_zHB|I}Hq zZp^lW66JvNlH=yNNI%C{udu)Evqe*0c(x>708XC;il=30FX~Z+R>h@NZiL)}Owwc> zLUGt~^;Cqn2|gpkz_ayF(Z?REG?$+GO~=sWS3Nh}#1VgrI=bV)j=cF&V=4ZpRbOs= z2byKl@+$|)xaMRnAe4iQtl4Hvb6$u-uQ5_?KvAdn>*N=5I?ROw>$Be!+b=EvF^&G` z^1sn68)`Yo?n?@LJJy?%gcoR#E>!X>h5UP=Co6lRsZ%PWyCE(i$RD02YLnq!-x0(e z`Ti&*3CXgFR!XnT?DBSp_N*2o9=#ez8sm(=NfK8Rv(X=}L%-!B=_=>aI=Y#1e!;C* zfz42D=4V+G3AKI>(3)nG!DIW~d>g#?3fJ>|@Ap~usV}hI4o;sNt&ffxT zPxyJX`gpI1OtGd}>b$v9E17cCf~k!Z_aNE^{CIl7%<%L5=H>%cIhXEOTGu1Pw5GmH z{rH$`Djn@UGQ;#1@hkQ2q(KwB7TX{DB3K!jQNEMyqVMFk!3g7T zugnZ;wJ=Ru1Z5quLG;dx9h-M__gWs4d*=>xlBw(RsC+77Y*lF#sX!tuoa&YuXICrt)lh7*~}Oqxk*fdZ;YvZ~{*+9O_{(-hck6 zzr1Rs18hF$u`2uQfOWm#L`k1omtSo)j}-~ALvG8_1o+Rj+1ZQ{P3=x>c?qRtjdMFj z#J0c*ODiIMM-IOxbi@GcWBCxJgOXTE?U|UZ*x?bJ$0sUKdj?tSX>ZZPvaf7%F|*~G zKYzEltc+w!V==OY&XftiDhgs$p=}SVE)>Vp@GH`9I==DyG zfIIa^MRG5>1Jh+(aPB*?7ZD8XN*by`6DUNEbM4N5dxxbz_*;S&fslZ_96$E$>N!Tc znl~;_4jeOul&b$|fhA8Fqpl7CA(bN=xLcgAl|yE?CDTCVnIBh2Guu5}$HDY10A?5EB8Xrginp}3Y|;;ak^Jvyds z6U+TwPT9mv!8c=@BY3Y-*G(0yN~_wgG?qfvu4&>v(|N8 z>%7iWG!8$>j#$rGH8HhF#_&b}&?l}PCHmy1lkPwi|JtO&Y`CyA`>}Kdede#*jr_cr zkBa8>tr*0i;&Vvw4uN9UU#o zi4~}mI|0^8qMl)F&_1lc1=j?KaDNM(MYS0|U4DTQq2{M{V~=Oc1qLh_2*wxP48n+| z$M_dRfRqBy+Ngh2XD&nwMpd3#sr~T8VQ*T0YMH@1BZpuK>u38pv#`)`GW{mFY}riE zh8x?JLD=BFn6N{p3A3){?mCw)YnoTV!Xx8RZK~EgMoM|7fl8 zMh>N4&HfV$5~QHh>{K#Xqgs!M}W&cHbhCwc< zLli;ZUkZbEa|Ww_vD%P^29;w^n*Y)wBUg}oC6rrcTmfrs9@YLe6yoPrW&4OoL8-u2 zk9bP+ph}+l%vCTf`%?X3CD*5ZjQPISi4+)OXE;k!zd?Lq^f>Q|wFMXPvS*% z2y%C~bL4S-j+^hR=MS9OLJoi$99x(pd-R17Zw%hOz8JT11vW;wHv~|ca}tyo=#Z1ia(*a_5n)p&o2$#T_3;{tbhtso3Ym-MZab3 zexxmxYj!%4LR)};GeeisY>g|)rCxW(&GaH!h%A=qf#7sfCN=rBx1_k zU7q_5ghLHN4w*4K*g#K$XC!@Ueh7!yyIu+ zdRzsRlXzl-Z`~`EZ5F*(_ZfHBbul$aa8#0!ku=PwNPGThdUu|!6dBc(CGLQl+s`-oZGik>bEHZlYE^bVrZG@56@!-qT>=Jz{O+)nvzRzVbj%m&yS zE$vMnPJ<4HMsq~i*N4XO6WEsYXo6F{8e`3?{>mebZ;$=5c_s~qd?Gxi^fFqM+kMmG z<0-;F?RF2<$BM-T+>Qis_g9H3zKKD@NN0mo64WKzuT!q-RLW5lzdjT#%02@c^(enZ zu?=3OkRR1dEUF|+U?Dbp-HA8f;ZrM05j@ik0jz_N&~?x4b7J}BiaJ5a+XU>gN^K>; zADpDMS(=T!!KX;32VG}Za;x8&hbxkoQk4F#H%uW-p)j#7rTOsjb=%GWgv}?mH!zEE zQOn39BOftKZi165@tc+X$(wrOusVuPVkRNpOGwdwVtv373brH$_&hqH1Wrb<_b^Xy zeGKH>kwuEbO4*Lbh=$*n9TX#h8K#XbLKEv(UNZIsW;J$n#`~Gc zP|&~?8QWRZOjhS;1zhV_g_;bQU*n>Tw4ckP6Y=Kt^rhS|yKQb@gZnUN%j(NJXE zhfBPrh!;Mtx3CFhik{jp8Vjl9982*#ZMRt!q}06K#>g5@qH!6Y2a)PRBU74{b8;|` zJ|4yvAvC4kx7Zi$te7!k=R(AKrf2t`vu^R5%<8S@F0vlA=7nMovTUJCIK@ zGV$A{5Sb_!Ury{2K60tKb>H(1Sl+BSksheCg?a2k+n_Fbvq+jG>QZ-PJk|!2NsUIUO-REFIf8N zG;Iq|(hgEt-M|cm$_5*ozu#G_BMJE0ET_-MbD<&~Twai_5)mR{Rd)5{jE-8!H)u8{ zaUH^#6e!Y453PupT3c#6(;m~?P2*OGT-Q0C1?-;wb$@q*`k*~?r5*AuIPx_82f-Ze zV!_q=ZFvO_(Q4R!8u=eZabjb&6x97-+pWK>xM&I~tq}}SK51m!O>7HS_Z^G!UMG=^ z-A24>31fBv5Q{j}Ch$3a^_8zcSrO*>Kvgy=hTg_amia$g@BjroNPP4L!@ z3fb0f(_x+c0p1O~$&+J)yIoo!)* zmbb6i$7xJePB5|t{@jomt)^Wo$Jmm|*^0>~iytbr8b9fCsJ@~c_=?GGU$kKe6myyP zm(wgGPLkvql<87B{nz}<7X1Mz<3@fr`km9gD)BXSgeXxIu$>v1V;SQjFGcnScYO~h zP#y9ea9h6m|IxT7dC74$>$k#~0%#o4_}vjB8v8Sb(sh2}AW2XOMeQoumbtU&K`k}3 zL2NAErX{jNZTNzxFNPZ zIsv~49JmUAnJKw{4zB?@d%wp$Oxn`2UuKaN*3Pk@t@Kga2)?{`DMNETky2~@i6&)d z_lZLq`Bt}Hm)r5_s5Qw^YxEdUxFNra+8EPtH_vLn(VVx2eD6?wb1yE>QBf`{UUE)DcMPM8nY!({UjbFa z^8|YbP-H24J?6yTseK>Wt+XHG$LN+G&9a5ZD>(f-`tvV|dexuWj|%$yBX(DAVh+Ua zT=&C#PJmY)TmKC1{TEtTo%Uuw9M`r}fZz!XIYVD4Kt;K<)lM1k0dc%DwSc|*+TBlv z{;MF5-NoH`;#c8OAc%KlB9alDbk0hEO=mGcBfmgMRqV;%0mC6Z<{m3|x%6n70Y7ks z6SY*@vKjvX_lKPR-$>%Rp9cKt_$Wry9Hpe2fXH-+$=_Ott}4~vqz%TR)Gk{50PaR0 zdf+g`lf55of7s|OOOb*uGw3uZ=~#O0umAoBeKkigxd%#7Ec6|57wb2JU zrVdiuUk9`nU*$>lS1C|{z&+{Pn7ZuKjvC6t%iQPpg*IDzt%5Gd4#D=f!bquzoxXXI&wBEaU z@#gXAqejHKEK(ai*#SrHa8g~G?Usoq!doX2*(htYq*Vm3!Lge>YqefX9ZxM#a}Yq7 z7;H5C)suw29mEt3y1KLR8oZFQshv#o@6yYDi&=^s6GAh0A7=2x>#S2VA2sIml}zg@ zr;HbdGp2I&-l5PhTgLqSeY<~MN+(I!^r&E*>fa_@N7Vk(k!)|f2F#%mCY#s zSgRvlCUJ~TCydjaANcU07x5^LD(JbKb9Kw9Ce!e*r>c=AtikMgygZT+>k1nnt%Q0Y z!n4Qmli5=%e(}9i;toM{e;GO77u_v)I#XyfDT&Xc4p8tM8<6p{JJ|(@-4Gxu->$x< zaYQ3uIdM9aoQw0RV{EE*ryXbIf!JjsheoZ~7?HnX>IzHkFUg;)B+*qM#s z{P|H?{l$_}y@c&UQeL^u6F92~$M%!k5fs7KH@UEF>DtpZJtu$)Z! zrnO!%Q0MqUpw1BjSz8HjA3$*8ScB7aH0IkT^~Vua@p48m|1#O}-~|i{?J5;G+yz z4d{H5+KbvU8caJ6fljkwyrya=zpM2*XZs6EtsD!Uj&0ICq^K1I)|XZC*F^SrI_(8n zgj+jhuu?L`Jfb>Fr*1F4OsPWa@VPq|J1y3+xWNN0SYi3r{F{F@s{$zFI9A@*->E?5 z2@-g0Z?&e-h2bMvyNAozBv)Rxl{HI&6i>&*dZkf4YPAb=-pY^WE6fGXDW2>!pUXIu zxzk-W&}u|0&Y!UjR~qtc_%Z+1d$-Wjxr5}#V}fZ2R|-a?FA?~}4fiT5MFiN29n zbuz^gr+yJCkQ_g|>ej5CS_RPeDs+lw8hWK{Xuky?Q{92-c6V|DNl;>!8Lu2@t`;_T z2@X|<9p%-T*-2M4w_U~5H#dR;O9I9;Cj)qkJnG}<+>xM7jY9v_901^{$>%gem3yi8p6|*+zem{NgqBMiMADeng9YN8l@okt$cWZ|} z*@H38HjJ&0zigXMqz`=R`Vc9G-Re7=Zru*T+e8BmcT%`d0yAYY?!OT=rf)o-ZrRVR5Urx1{zd*N+fwv^%?vVR z^CcD_DOPoB)i;6{d*e$vz08`p9RWDy{7(KqCOU;@y&+*!VyMjp?=Z^lu; zShMmRlNt|{@-^C7BhEWh!e%|I5&wSb%6R(f|8pz)^vpG zG>Ta^WOST~l}9M0KBr#PEmF6;B@fd#>}Z19jhr&>sD6u~ajnIlz)Wr?z9{3Ob8S>) z5F*^v@+A{r&x4UOtwmVsOWv&aq}R)0ki5gdp(@=CDbxcg4S-l?vJ2+gYsj$Ox4^QbXf z?@;F(0Z%)`kf#ju3Hb<_h@xEyzy0ln=uzy4NaD2;)vP=~{zeIGh|mFOI9<#ccfzlH zD*%3VX(6cle({3?>`NYte2|FW7SACla=xyoOZK7#0HO2Q4FmzHJIfDuj-_8cY$Ocd z(xZIbx{tcTQgl3eAP%W{?>!d;$hxT^`T?7qe{GtJsEx$^wx`x%B5Ia@ZyIo&l)JQl z&DGu5m=5gk>lwrQAufLzT9TM)RHfqY%cdd`&m9idB%stb#-df$G8iF(wMzNh;`E8_ zgj1j-z>IJuG{^pI9uN)foQyZHI2mRKSrn>%jHqVgcG2!;9+Q*>o&=b!0?G3S_8DG< zMreN+9ZPo?yKSLD_hT>xO*g@yq!ZBtUki_7(X{!#Uw3b8BZ6N3@q16g4j8{_Etx{+ z{%-tNaXpx*)9AV*X`tNw8akB!6y<#eunBpOrEBls@So8);`bl-g2mGQQyo2QB)V@z z{sDRW`*}}+f`6aJQuMp39-xNy4a$F?{NJX+9+OZUb?B;?P=WEFZ$g~4p$4$?C(k!Y zLI5)UljHR$iAnj!ZYi89y{yd3y_f`caWR3ZnfV!yN>DO~uaVG|2scSzgglEI5FK`% zk9K}l6EX3z#S%?N1tkf2I8J2pq(u#C%=(U=1+4=8<}mAQ335cK-qvCzI4&(#hyYh$ zBFok`dV0oI@8O!eH86JYdy`w0m-^sA!oZf}J;NkLEklm*9R@x5Mbl?RkLT@tAFK{{*K`YwUy&~^9G;B9?V|a7)z^8d~ zgZ7m$T$$9AlbHwQY=h{23%sCo&oe3JgywX2@(N9fk%eJ8(w9TKqM}toYys2VB~5XK zf5si+!n&CTd=B7wQ zu4Lv*gnajH=%^U_tT2GTUU@Czri;=w`W3_7`$VuwtAL$D96JowVcvxtkTAGARjq09zqpw3l4Z_J=ef;HZX`u7OHRpsSz{P#;${p-xCp9^QZ zGLl!U@9L}TF`<6ngd}~b-OiG8?MB-@uhvRDH`$Jt`}!@FYi~QQmXI90ZQXa+pClsb zZN+|SU2CDF!399o^>f*y2#^ab=1W^ik5n7sbF4El&VdAhO}i*6m2ExtHL$pUEIrQq zI)l#qrSvC*5!mHOH(*XMcg0Ek>5Gx1jtx%_JrFy2y~l4M1BLDlJa%y!qa6tSl^g0= ze|B0aNr$h=e{;yKg&DWjC~4Mmz~kHU2#edX4(|L#HNpewDXs>6kf&x3E^wlp6AOBO zvrhGC8ibv^t*O2)5PLmN?xj*3a-Ym8R*8dl#VPLuY0&sp;2%Uef?a4AXV$g&GouzmoH0QvuGQ+P~IX;zTNcmfd3? zMyefptyVmK*a){^yirO;jkz_n@2*Hf4~8`Q*jNARCI?Hs`~l5Wb|avdhDq0V~ zD)3H7R5X5EGxV2qALe59IOtJ5)hmAO+tGt{o80Jg|C~l7gS&V14nT9|y zD2d_mv8sY2Y1|B{LFE?50; zsf4>Hc3ekVzRT<+&b3EnO4%Ao%`KO(^8l}e+b9}ELo!zfJV9hFZ1t~+&qMvGaW?%Q zQM|%*cZ(L+#d|7>Ft_a8Bcsr4;b69>2JlnowlPx%Smvie;ABo+st$am3X7iPd&{ zu{3wm1D~-*qX_&JO=ssik^?L?uC;O*QW<5vRbmJboRci+v0dC?r7F**o5VTwRQQR^ zocffAj3@=nP+0Q`DLc;*Wb923EM(m(VE)r=7PY9CHsJju!5~QIua-utn|*KUf8sMG z!3EnXHB^^&Yh>Wcg`Ix1#gUD^RQpxSsAG~tIxwpi>fO3Npb@I)^>7o9z)i6sFA3$NDvv44QctaUwpu9%zp98dKZ2EtuA+){>(t^YhnhK~Cc0B5eve4_eO~qb zUPC_4E5uU73K@?|kp#h_M%#;%J4Ge)_>|8|5cNk9RA(Ao-P^z1paEj^Y9}NV@bBzPxce^4_W--BwS?0vU(cy>;yn9*PZwN49g*B zhDgFj^;S4a57_>no++?RL{$Qst^tg=n2sOEc)wD3*pfa1g>C)ij$ZgCu66G{2jn5j z5pY_8&)0;&Z`K0Qe;WSfgC0tSGp>Ylev^%U)$%Fpq%JCGMEn}_o4923D~ME6l-`v^ zi@i>U;BnqV;T`hDfx4*09)Mr&cc1%bR){996abAAm1A?QqUEOA`MA?qR$$Egtx95k z<9;!q6SoVDQXVE&;d}+J+e`os>h;s0#{TU3eTqUJsq5;;e~)3VEN(o&W&W#<{_hd+ zzjwV*b3JUV!bP!%6F8!86hbYEn<{N223hMwq#Tqc`rLnJu7VfFxv!F9p3v`(tut{C zn|5Y)yn3GI?8Rz3M)0;aPOjyz!4LkP(9}4v{@&5;bc9vEsc$4wVYPf^&X*Q|*W3MQ zoMcEZf&Czz*!(zexI{W7Bi6n;T0q!_uSJrD|0>&Ne<4Jxh@9La3+mXKKwQ2#&3bIz zSB3iJ8v{1tY_Sq8kblF%ge*7es{sPjYv4zI7fPGm1goJl2rp7eu1>U$V6ZPwZeI!f zX&ToZ#N~4FrerF#^ZO+*Z}CTzE1a@(f12`Xg8JiF(KZS=6SIe55M;BC?a$+VOgfUl zRQKf(rAn`#Az&8FQrqRbOXQ^r8QxO{;)eQ?7LgX-&M^hz+Ci&Uw6#!;iEreaqA`n_ zPh7ID!&H(tc}dIpB=7IluzLdFCzlE#newUc)y#}8&9oeAfq!FPULXZOOY)$Ewp6SK z@#+ih#qKtg)h@vE7{8Wh=Uv@N#7c4NS*t*pD#$X(`p&MaLwL=>#%JSSD^@j+lup73 z&*6S~v)5OZ-09y%qn+vV%nn3l!=rtiMlgdik04Lh=-(k7)`I$ODZL-rN^jnBqlYQ1KZD9roa5RMH4 zos;qOMqf{#)oao+de3^mOXGOpE=v*X%*TJ%usO3IF%PD7_mdlwgh^J43w)bLVz4VM zIBxfb^^;Jw4@ZqgQ$*d@joC3W^+~$8&;c2RQgL5_+RoQ^a!{H-hsahiRZ+dVGA^Q` z1=I`CYY4Ay2fa5?e!mhZA9>NvXTaZ_{$XI+4E_{9p9*9PaIT1i zNcjflo8tG{W-SF)%2kDD2Ibf>Y#Q8j(4nIFx@{YfDL>gq%IocB>&d!Ts+d=X&94eQ z#ne^T+TTbrK4j}$u05QlI}O)35(edN)4#<;B7LbOCVjpNvF*p`WM0@eL4|3{7$!5S zIR#U(;uO?Ud>j^HvO5~3zJrWSs>Y|7%SEGyi)?12c7UAYkv`c-B^hx zz2etolc|$=+k<+7e?r#3l&XaMP zbc-r98tqR$kOoy^Dax)-BA}Zw36*!r|@Q^6x7NeZ~;zpk!3zmww#4qN$yOJW{c^yJx#CJL>@sgyvV2PuJH4 zO+~==~M5+@42SijoHMB)Oo=HpuT3U^W}&S&8^S1=LTZ8F;mr$9b3 zy)S#_HzE4x>R%}RLT;PtOKTrt&S=O)4{*00t2@UTKWmn5d$Y%M=#Zp1M-bp5>1a{z zwXRN^XfLxX_HJhTKd49NZoGdZXGrKjp~MRPI^<~DvBz}JmAjK;)jNN+A%96o6OcH; zf@Ef9)yaA3t5g#|!KAg|KfiwyeI@K$&^mF0jW}~tlN=hbd?vmtzmC{!OK|nk#OD_^ zVV_(|3)TNfeIuR)RuZ$COJ3PtHg}K$`7GW}{e}sCL{t>v-V_z|;7e~`f(2!^bKI_t zsN++T!ZWN4K${zb?z}rHEs;y|1fh5GV|vl@8|*o4=CWsbVbsc|cRCVkNy{3Euyn^> z0#Ji^VFM8SF%gC!iuRU5y7~)*@IE(oNbyOzEJc`gohY%nIQ5bDMNMV9blIIVel`oA zwwmY(^5$?EP*Lms>o+*+cwk}HaOaM~P(QqIH^kxxF`ncrC#sOJ*q%FQL95No;AWq7 zTIkzwYR=Ua?VMIyo@la<6gdRcEb>y<=&8fgDYQ4(rSXP4{`~A3f^CVosFb+%j`q#A zGU`})I6*vm$JYJiV78;edx+bWM6GnSBY~hn`4x%ejr+TtK&nb4=}kmD3b)o)%i5@G z6P)mV(;ou$STPaU=Ij}rv5i(S>J3#c&wHA~iy;^RLVk=KtvhWVK$?Ie;-|~=?He~m zzLkm18mc0;TC60zCrb%oQ{*iPg<^hM9KVNQ=>)w;{IZZdkia88V*+H%5)G^1jbzvf z_=@l)r0YVz#}Rg$Ai9u3^~amH=`~*^%29G7%1Dl*9@w3TsiEc^Fk2_`DG*cHc=R=v zDje@F$yi5kdUwQpM4{Ji3n%0b{Arp82GbGY{#mo*GaYV3 zG_JuVb;6OZt_G@!o`P;tj1LI33nI)Cd=V;Vr=N&*cRkLEoki`+^-do>o?h17f9QM{ zisrd(GJb`&SPXqFUZ$RJ!(dq}?@%6I)z6#G#EGQ802UIR?h@;$p8B_J9R1f1(3<83 ztJYc4meTK^toSLAJ+oYYGyy!W{{CJ?!>{3-D)^O-QdPG!fZs-&u2Ue>_*M$Up7tXm zpgQRs$Z)~lvLTt-$p|sK^mKJ?HC6}Wu4b|_vQT`QO0uwKfJUN;ijHWWO}p*ZY#%29 zRgLLb%vti~H#qoImTzZ~;YoPo*xL6{JCTQS;AhQ_KfmccxME%RQutCaN_eI-Cf_7|ll3tz$aXS_Crn+uly;L;1fi`s> zTI)?f-s!*P@LYJ3+buKLc-riyh-IXS368RFeF|Y7BJ_?(FLaUN-`ipmiMVbB1zE&~?sKL!J(p)Y| zG;W8k))E9L(ZaalF$Im0ooC5s>$HC4*FP7px#hRUKh9kdA=n6z0eNe3c->-gTXOJQ z9G!6J=F?3qk!*A{i-Pgged#P1)A=Ch#HQl(zyqPjc>?($Rye<2RZb&@h-kbGr=Xuo ztUDM%)-f~0YjLQ@cbKBW9=?8Q{bl;$c-D&Y$%YrGUB5XAlWZVmVKl(Kba@v+8!9Bc z6L2~k?AWdG<(PHdhi#O^lvov-x}Z)kpu!3;x~COVO#MXgE<-<(GhE46&edee@LM z_-#%j0;g?n+1koTVBt@NPtU*bn~1sW0wT0^EEt~|$hzY~8C>NYX#Mf+F&SZ4tI6nn zo;(kmJ2kI))0IJsyISeW>WjOTy`QdK`b4h@qJ|o@2VBK_pHN>TT9RroLEB3`&AII| zdqYpSWp~<0YNm5$OXPR~`17gL)D)dAcZ-!ii=_^&nD>HUX(y2^t`R_Xqu2d7@AYBS zQ>)D~4Z-*QZ;5a~=OKY9${z?~M3b&s&^qBd+!Yv7eR-#;e|CCtCNOu9^IPmwd>k4m zUgxch9WhboLG<QOu1E#IZhSQ0cOKnNI$UIqd7t_RGcB_7mCmPYIFJi?BuB8Na zL);f5pkVb7g4Z2~Z`#ot%Ce9n9+64Djm-!cU-*C>{<;`h9I<FG(Jhrr)hSPG;=H5uPil+abPMy9$3AFTH} zRslA#*xN#*hl>1S&K>PcW+NFxSv!ykM3PhXp0K&mSwetC0~DH=gnGn+4oe@-0aYsS zkbMP8QIBrAwz?`nzNFpY;YFlgC90!!K@98Liw^+dV^&oJmz5GryN^s7ZRqC<)D~^( zBrG&iOh2D@o>RcMf5gx62=)x3>$0iEiN@07{l3eT!nP@NKgc9=*HQ>E)M+3uOlAct zCqaYHf!oM#p(c_T!h!uN%AyqQ49<=9vksM$kB%-WlmbZwSHE>fA2kY<%ek_soxgOr zopfPqo(Ge}h1coaT@5HuxIYiLZs>TmzVI38t@IcoSN<(u$0G@%m@WKq;(ZI8Tt3n3 zi#Jyz-A2=@v*}wY*uHJLRTQj6BCyM?*1yfR0gM^h^sTgpO^cOhE<%xK@Sa!&5 zhG3?e@5|~Y*oV&BvNYgJTudLA1>uRdOOBG(pG2BY9s_70?gL6WCJ|GxUpV@mU~{^! zl3ah1hYceIZ`Vq7g-JKT*NgF_S6_nEMfc9WFGU=UnRLx2RB6)T$Fc+&)ge?`GEq7( z0u~d--xr=-x6%6G*9&o2`A^q^3TKZ=c4?)K2thYS9A3UliIhbe%a<4vZ|{Dm6fCkW zZI5Vj{sCfw%4*A$nxooOg-F@% z3-Y9LVeb}1PH#*n%GNp2IWzBEwm_u@S-2A-3#xSp!CR)r=+D;bM8`2T%XndA5ma#?<<6#U$z&H3@riZ zVqxpENU(IHC$@&~*Fxkz{l$T;Q=kL?uL`$nUMc zu|x+}J~?(4&_4c_X!}Pr_P3-5#?6fXa~<_Bn9v^=9*Rfp)BEm=dpwEh0AzbLMATTn zWrCsK1TT0ol7U@6HUK2h!F%0G`#-z@BCmoMY~h^{7eMZpA^@nTeDIw#<`lEgUgmy{ zSooajIxsPMGo7mlvGUn+>nXse{`ue!aLpB`r++5Rb%=-}zO-telX}kRI%xY0pp__F zmieI(cls#r$=sWZNy~=Xsy7$K^}gx5^cB^WNiTjid5ze*Kz*w9Y+v1jUFtgOIOI9h z&|?4NU!y|ve|agXwI~sBHvf7{e4l7u|}V;4qPylZSLN-Q;=6ud5WFC8A&Al?D$I|fPF}TwO#RE zJ;II4ul`0WYK%Lj!o0u7$0gl3aFG!>Zx1XZ)KuV6W3L6PIRx_3)mPnUp$eecFKz2p z0hDkHn!5GqyXmw@_NMoC&0F0%ih2!hv&*F@k=+a(n{s?mARnd%4e*JeZcIQZRr8(4 zrG2> zThGt!4Eu`NtdRX03$a+qTy>tRHK&3#DjRB;_z6|`fhqOgu{5tn{Ew4pThZ)NRc7VH zEAEzADN2O!$&^wQQnJo4Void2-Z0jE}1E%+)O7 z$WL{-Vqxd2xVem+_kxoI0+&W;yBj@-Dh~kj+SK<$OA#Q^y0L5!)*|@X{DnOJo|_Qj z?b@0Wea=_Iux|Re^K^4KF`8H&KvLBHHAUH&eJ_$;8WR(fsQlH29SetR0LleBdg!B$ z+cCnJp{C(ulC14z?NBu1ieRRED_(pivGSaX?`))D^28n-Ab%TjwkS^q)Nsu>zHIDm zd{v;KbdP(%Brlq;j6{U~Tp7=tv3eq0&fKk;`>@W)NG-Q~{u7n|LeU|oiGv}W6A`YXql?>}%#kUxxm6kO8cW`p5FTXuy~=^Kk@ zNw1EjM_!nY69tk8ZMWfCpopxjtcWSW{yP>}AG9PW%YE7|b1BIZQvV4%N;}2zs%0rZ zb0iCOK;s@Al!B~AtIz$~_`D`=*V(J${!`YZtHCg-`zPu|dK3mpysGRK2%2{?%{?^x zL!+y{{QVl>f;5QOi^-?7jA8q;?BXTt`Ie0>9thWn@z(VRh$M@OciM|#rhj_eUXg0h z-Mp04EtqWa^tJ^6Im_NQ96!rMrhUjj5|lt}MdM==F(+W2*g#Op=^&5QNaW$8YF>IE zgSj){#?=9($o~GDk2)UNq$<}hl^BxAW?4YiC6b~LUJpQ(&G2!Sq9ZJ`OLM5PqP-*8 zrL8g8Jzv&#@g-@Zao>&3eC6ww*M)+kEpP zMyZTl|FHq3>5=S?J)*I@v^ohj_Og4CVhVxslp2ig z6niWK7?v34#-J$p$&pA;nwHFzE^owiYalF6EI#R>@JIOAuDnPh)U36of}TYy7OCC3 zMDAL5Lhr5LvA{j4@|6L=+V+j3JdL^bek|=3pVv)+tVXwf<7Hc6-bMp=nn_CIOemqe z+e!20j;2X>!N;IIJz!ueJ2b1^6BZ}XF(xnKkcX)M#Bne>Z7@;OMcCo9fV+CX5_c=d zyo#L&=Dt91Xw)Y=QzXB*Lp-;cBH6qpc$%}k6`zN0l(UB)%M`1q`J~&V30ACFs|w;{ z9)0j(HxSl@zQ(+WqzzHix?2-R94xK@c4)1J_5I;y?r)1rGZ{O;xN+ISH{^bKUrH9$ zqqxk9CG+oHk-_mAA@?8ev#|vvPA?BOm3EhXx-9|R^ZsJvzyJR$YM|o~ASTw4XACH4 z3}h@CG8Y~KmTBx(?KZDM(xTD+SJR&HCtD-iEC5{=J&mpQ?z0z7MO$;*Gj@WiX@J=? z@$GzvoxRA)>u!8dxO-2i1U-hW_LbkSpW$09_k*R`7dhWQ??v{5DzvbN~nHJZ2vFp_cZS>AI;j;C0|G{-00*4;8vz*Hr+7XhzJjhh+;w)!BXP z4~kW7c{@Pg?)jRir0x^y-NP&cLW6+f96#WJ@R}w$J^nNHpMU@dYWe|LVPMt&NzOGu zkXj!Q9tIp0)4$~At^x|S|29wmHS*vC$%ovIF;si@V*2i#YEmD$H?$J6&*HBKN^=cG z9+EO-m^*3PX4@c!Ts@bUo%jBM1-BiNSVXjnO2TkBOw|bufE0Xj8Di+$*fAI9VZtz1 zGcr-8FAX%mn~wq5J@cY`Y|Tt|ObMBd_9>_XZUn*fUQWNWAa)mix)Z_O?bpYmsC+_q zPEMsT=`TFE=O+RnTQ>kep6z-CIIdE9b-WzyxUoa8@#s985=GFgX;oENCb*pA;7Hav z!D0ET!$xkIE}?@e@x$DSY!=M}Z??KRqPUhTU_3yug(R5X<7#t%pn!0Tl}Beukc`g~ zq0YXf-ph+U6e0^T+~kaS*5Fm%dH23w{X!DWCr;HRb$9l8QzJAFCgheFIvR3b8_M?1 zdb_@9IP}h)8I*_Lk0glD*3+Ee*6FV+0sJ=d51TGVYIZzbIm18reQ5&{c_75@V>Smd zqk{x*$(U=xwsO|Jc$93i5Axm@yoGUL$6kjd-g#s^AaTBUB4_NRMArw^8lZsWJyn0=M$dWW) z?yGe8v~i17+J})Tc15L2S#6pX)0xd59COD~41A$)sW%pl>=8wkLKAZJS|VBebwh@X zYu&$b+`#M@G>;Obmg3`D8TMragONPnv7WdMFSvkfb$vf zo?(^B>3=55-qTs^k>+IL_p>-8?O3|A5PaCf-KlQ9{2ayiD!QTRZR64q;v@2fa^8z^ zB?aoL_^Rd66<}D#%Tvyk-SO%YUfGg>oyKd`Kl;I$KsCu zt|y}m-e4{2c)joaVRNY}+Rx z;iE{>_SFk%Nd47uZET)vdS^1Yh zEYt{X4r`C}Y_^>`9u@p969^RQu(kV0g{Yd+ian~F;G3ak5U}v*AvZu~A@tNZw{nkn zff5$LV*BQ1dy~)#=a0NxSC3X9Xo{pc;@R322Nk3OpBoZiJY5xNaG%No2U;Z){;V6i zF0#0rI_@+1Kf&0%ga^$+okebn4m<9S;0YjNR@7*YZAh7it1c+w)UJSWILDH@uO3Dk zurU5yW7|XR{d2`O+3BL~=TFY@R4}wX4=2?fPw!m~@ZN|q%IQv$ne*Rl)#nj@{vNIS zmPK-HP?<$?R8H(V`o`kv+ZEZoDGOcz0{hR57cA`zvyelJ!5gbRjEhLO&Fl6SS4$n| z2j9X%+d$QqA((?Z;*h9khMC@o4#u#hzDFfFp+$Ebv?ix1^#}H@9sgak|}-QR?U4i$|huU=j9 zwSYO>_CM-kLkRo_1pClm5bVeb%aYiw7Fn)oMRQEe8PCWzdSv+P4TWnhx1M0WNUU_d zk1f2pBG8*il{c+TH^IW}09YabqwV7N8UF^_!_mOAl74WBgE(m-ojfGBxzImE+#icLo<*rT5U(7j06Y7{I@?UXn0 zfHSIu(wlR8bEJx521!fwE1KNx>Q2v1=~}9Aq7<~^fd6tnfV-ATuCj#W%*#2oWVn-@ z6s`g^hH~|RlN;Ui0e85kx=*!lU8f>8W| z^3{zXy27CUQZ^FDQG(^_WUx;?s3d%LVFOZs&D?{HtxcmF0$uQ7?-}px5jP2vjlvlK z=9+SO>C;V4BRf0r3f8nG!)TruFW1`|Rj!D9pUE%&-HH`Zgwh6PktkT3b3v?UjPj~x zUSIJ#di}Hc$K<_gz^g6RH)is(;@<-+er2vhYo-lQ)0LL+)?qw9*$JGpP48{F-{$5D zz1}+cdQb3bj=ihh^!K0v_BvA&^`)JJw@FU!YoWL9m^Hg^jt4yhKX6h{>`oS#u{t^9 zW#8Z&`FO`n3g0&hjBHz>fO>VG(l4Kd2?sb%9cY++fY^Wh7c;jY?Ug5*MomOqtlfX% zI(gDd01knF0a-C1+Vg*TDgT+Sx#{QyD!=;~Ks4<8gJ+kD{t2#p=q@|!TN{r3#Ks)J z2XJ7!EBwDGR{ZZ=0Jk4ULxc1I;){a^!jJJD_-`48J{kQd3ZO&204%aR82Mjs@=FQ^ zO9wT{9tgp&>CXcq()(NeS1cH3_O~Yg#ta`=X(rIrS!VqJ#PI(|jJk_$7B*1&$AwtT z)Nd>Qub4Fe6W)(m2mB6A@Bc7DEBGC`)^YFzTm>-m3=riX>5X{%<)D!V{oxk3=vb;r zI5j+B;N@0|=vYF*xd(+Y+KVPqtFT?jg^y4NQ!r1$~h?ss`wzC?5I!@oyQK ze?bj|M?$o0Fn7anp*EepehqGb6QE~dkttx{zILXDIJ58uOx=WHVTPSq!IW9qsj}l}{+B58Ve>%Gr)!3$y|*r71}>Cf`BERYxfqeUTPNg->g(v)fE6FFei~ z1?LrE7T1>&i?}f&N*LU)uKfVd)-g!Ex<_}UYQr6F?8_snvh(h1Uk}ZyFan$AubwYI zMmoBK^lF?4+MS)js^4RF-@mtNJCW8JlXM*W@m#Zd$=oYI54hbi7XMN(y{TF? z;!7-kx1Xjk0~B4`?WC*S5JXYv;coT=|^PTpMCH4qnnN4Kw`)3H5UcYC|sn< z4;Cj2fi_ABygYLnG#!|BU3yopNWDP7HNQs>RWwk4|ZEOUnfX_F;V>w%eAo8 zH66t9>Gn}EK=t#!-VpVr$O@9wxw!@~!}WJ}7h&4CkCZC`c#(GS`o=Jrbh3Nk&&({U z^wg^l*)ba#MB*l5<4T7P4vYgo>E@{TGeMdl>gbr~0B$P{+rNoOzYklVxoA$)n(b8o z{T)jXEseR2nQqGWy~YpT!a&`a>w!qI(UYxQ7Ke`pg>OJ8f01L-zpb?a%Rv8{aKyh$ z0crpWESWw(H~)JkbPFEk=v^@e7WDVO?vsZBGb)7Iag90iP(g5?HLNL2 zPfMVq=QU#E_X_?$(}fu_e*%FzK*d5#@8KPM*TwLIUhdCVih-ma1Fi%AfBruE?lBKG z`tUPrzhhN_3rPlDu?})|3UbqS32*~G4yY)roL5j*S5Q8G>ztOhs;B^zUNO5|e(`Sf F{{WFMgDn66 literal 0 HcmV?d00001 From b0e99e0da6048cba627c1cdabda1ab3fcc8ea41a Mon Sep 17 00:00:00 2001 From: Tushar Sharma Date: Thu, 5 Jun 2025 12:52:20 -0700 Subject: [PATCH 03/42] updates to the container strategy --- deps/NNNN-container-strategy.md | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/deps/NNNN-container-strategy.md b/deps/NNNN-container-strategy.md index 55f1585..4a168f9 100644 --- a/deps/NNNN-container-strategy.md +++ b/deps/NNNN-container-strategy.md @@ -67,7 +67,6 @@ Pinning/Fixing dependencies will ensure a unified build environment reducing "it The build-base container must be designed such that backend-specific Dockerfiles can integrate with it with minimal changes to their existing build process. This includes: - Clear documentation on how to use the base container - Standardized environment variables and paths -- Well-defined extension points for backend-specific customizations ### REQ \<\#2\> \ Dockerfiles must follow a layered, super-set structure to optimize build efficiency: @@ -87,8 +86,6 @@ Each build stage must have a clearly defined purpose and scope: # Proposal -**\[Required\]** - In order to address the requirements, we propose the following changes to the Dynamo build process: ## Build-Base Container @@ -105,13 +102,12 @@ Each backend-specific Dockerfile should follow a specific format. The backend-sp | Runtime | Customers/Production| Cuda base runtime image| Minimal image with only the dependencies required to deploy and run Dynamo; intended for production deployments. | | CI | Internal CI Pipelines/Local CI Debugging | Runtime image | Adds CI-specific tools, QA test scripts, internal models, and other dependencies needed for automated testing. | -Describe the high level design / proposal. Use sub sections as needed, but start with an overview and then dig into the details. Try to provide images and diagrams to facilitate understanding. # Implementation Details ## Container Build Flow -![Container Strategy Diagram](container_strategy_proposal.png) +Container Strategy Diagram The diagram above illustrates the proposed container strategy showing the relationships between: - Build Base Container with common dependencies @@ -124,8 +120,6 @@ This layered approach ensures consistent builds, reduces duplication, and improv ## Deferred to Implementation -**\[Optional \- if not applicable omit\]** - TBD # Implementation Phases From 5c38ad31116b453dbe9ed91e3356b997ab8dcdf1 Mon Sep 17 00:00:00 2001 From: Tushar Sharma Date: Thu, 5 Jun 2025 12:54:36 -0700 Subject: [PATCH 04/42] resize container build flow image --- deps/NNNN-container-strategy.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/deps/NNNN-container-strategy.md b/deps/NNNN-container-strategy.md index 4a168f9..52d7b36 100644 --- a/deps/NNNN-container-strategy.md +++ b/deps/NNNN-container-strategy.md @@ -107,7 +107,7 @@ Each backend-specific Dockerfile should follow a specific format. The backend-sp ## Container Build Flow -Container Strategy Diagram +Container Strategy Diagram The diagram above illustrates the proposed container strategy showing the relationships between: - Build Base Container with common dependencies From 66cf8e873d5dd33c15ddba7cb0ce7246bbc2dbdc Mon Sep 17 00:00:00 2001 From: Tushar Sharma Date: Thu, 5 Jun 2025 12:55:46 -0700 Subject: [PATCH 05/42] Resize image to better fit on screen --- deps/NNNN-container-strategy.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/deps/NNNN-container-strategy.md b/deps/NNNN-container-strategy.md index 52d7b36..d9ada9d 100644 --- a/deps/NNNN-container-strategy.md +++ b/deps/NNNN-container-strategy.md @@ -107,7 +107,7 @@ Each backend-specific Dockerfile should follow a specific format. The backend-sp ## Container Build Flow -Container Strategy Diagram +Container Strategy Diagram The diagram above illustrates the proposed container strategy showing the relationships between: - Build Base Container with common dependencies From 0fe9448a815b41c2e28dba2a09c895827a3380f9 Mon Sep 17 00:00:00 2001 From: Tushar Sharma Date: Thu, 5 Jun 2025 12:56:32 -0700 Subject: [PATCH 06/42] Remove height parameter --- deps/NNNN-container-strategy.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/deps/NNNN-container-strategy.md b/deps/NNNN-container-strategy.md index d9ada9d..6b9296e 100644 --- a/deps/NNNN-container-strategy.md +++ b/deps/NNNN-container-strategy.md @@ -107,7 +107,7 @@ Each backend-specific Dockerfile should follow a specific format. The backend-sp ## Container Build Flow -Container Strategy Diagram +Container Strategy Diagram The diagram above illustrates the proposed container strategy showing the relationships between: - Build Base Container with common dependencies From c7f34d085ad61d391fcf139ac5080c356debf40b Mon Sep 17 00:00:00 2001 From: Tushar Sharma Date: Tue, 10 Jun 2025 23:14:12 -0700 Subject: [PATCH 07/42] Update strategy to build Dynamo in base container instead of in backend specific build --- deps/NNNN-container-strategy.md | 66 +++++++++++++++++++++++++++------ 1 file changed, 55 insertions(+), 11 deletions(-) diff --git a/deps/NNNN-container-strategy.md b/deps/NNNN-container-strategy.md index 6b9296e..1c8f3e4 100644 --- a/deps/NNNN-container-strategy.md +++ b/deps/NNNN-container-strategy.md @@ -25,7 +25,7 @@ This document outlines a container strategy for Dynamo to enhance the developer experience by organizing Dockerfiles to maximize coverage and reuse. The primary goal for this document is to define a clear and maintainable structure for our Dockerfiles—specifically, to determine how many Dockerfiles we need and clarify the relationships between base, runtime, development, and CI images. The aim is to ensure each environment's Dockerfile builds upon the previous (as supersets), maximizing environment consistency and coverage during daily development and testing. To achieve this goal, this document proposes certain optimizations to improve the current build process: -- Restructuring the build process to provide a build-base container which contains all build dependencies, enabling specific backends to use the build base container to build the final binary. +- Restructuring the build process to provide a base container with a pre-built version of Dynamo + NIXL available on all distributions, enabling splitting of specific backends from the dynamo build process. - Defining a structure/template for all Dockerfiles to follow to ensure consistent and reproducible builds across backends along with specific roles/use cases targeted for each stage. # Motivation @@ -43,9 +43,9 @@ As Dynamo continues to scale to support multiple LLM backends along with efforts ## Goals -* Remove duplicate code in current dockerfile implementations and define a single build base image containing all the necessary dependencies to build Dynamo/NIXL specific dependencies. +* Remove duplicate code in current dockerfile implementations and define a single build base image which provides a pre-built container with Dynamo + NIXL. -This build-base image should operate as a single base container which can then be used as base containers for backend-specific images. By leveraging a build base container, We can reduce the redundant code across Dockerfiles and establish a single-source of truth for all Dynamo-builds. +This base image should operate as a single base container which can then be used as base containers for backend-specific images. By leveraging a base container, We can reduce the redundant code across Dockerfiles and establish a single-source of truth for all Dynamo-builds. * Define the relationships between base, runtime, development, and CI images for each Dockerfile and provide a structure/template to follow for Dockerfiles. @@ -77,10 +77,10 @@ Dockerfiles must follow a layered, super-set structure to optimize build efficie ### REQ \<\#3\> \ Each build stage must have a clearly defined purpose and scope: -- Base: Common build dependencies and tools -- Development: Additional debugging and development tools +- Base: NIXL + Dynamo build from a manylinux container (Enables support on multiple platforms) +- Backend Build: Builds the specified backend along with any dependencies required for the backend - Runtime: Minimal production deployment requirements -- CI: Testing tools and validation requirements +- CI: Testing tools and validation requirements built on runtime @@ -90,7 +90,7 @@ In order to address the requirements, we propose the following changes to the Dy ## Build-Base Container -The build-base container will be a pre-built container that will be used by the backends to build the final container image. This build base container will contain all the necessary dependencies to build Dynamo. The dependencies should either be pinned or fixed to a particular commit SHA to promote reproducibility. The container will also include a NIXL build + NATS + ETCD installation since this is common across all backends. We will create a new Dockerfile in the /containers directory for this container and provide the image through our CI registry for developers to use for local development. +The base container will be a pre-built container that will be used by the backends to build the final container image. This build base container will contain a Dynamo build for all backends to use for their framework-specific build. The base image will leverage a manylinux base image to enable support for multiple distributions (U22, U24, etc). The container will also include a NIXL build since this is common across all backends. We will create a new Dockerfile in the /containers directory for this container and provide the image through our CI registry for developers to use for local development. ## Use-case of build stages along with relationship between stages (base, runtime, devel, ci_minimum) @@ -98,16 +98,60 @@ Each backend-specific Dockerfile should follow a specific format. The backend-sp | Stage | Targeted User | Base Image | Functionality | |----------|---------------------|----------------------|----------------------------------------------------------------------------------------------------------------------| -| Devel | Developers | Dynamo Build base image | Builds targeted backend and Dynamo; includes development tools for debugging and continuous development. | -| Runtime | Customers/Production| Cuda base runtime image| Minimal image with only the dependencies required to deploy and run Dynamo; intended for production deployments. | -| CI | Internal CI Pipelines/Local CI Debugging | Runtime image | Adds CI-specific tools, QA test scripts, internal models, and other dependencies needed for automated testing. | +| Backend Build | Developers | Cuda base devel image | Builds targeted backend along with backend-specific dependencies. +| Runtime | Customers/Production| Cuda base runtime image| Minimal image with only the dependencies required to deploy and run Dynamo w/backend from the backend build stage; intended for production deployments. Copies dynamo artifacts from base image and backend artifaces from backend build image. | +| CI | Developers/Internal CI Pipelines/Local Debugging | Runtime image | Adds CI-specific tools, QA test scripts, internal models, and other dependencies needed for automated testing. | # Implementation Details ## Container Build Flow -Container Strategy Diagram +```mermaid +flowchart TD + A[Manylinux build base image]:::gray + B[NIXL Setup/Build NIXL Wheel] + C[Build Dependencies] + D[Build Dynamo] + E[Dynamo Base Container]:::gray + F[Build Backend-specific code] + G[Backend Build Image] + H[Copy Backend Build] + I[Copy Dynamo + NIXL] + J[Cuda Runtime
nvcr.io/nvidia/cuda.XX.YY-runtime]:::gray + K[Install NATS + ETCD] + L[Runtime-specific dependencies] + M[pip install dynamo[Backend] && NIXL] + N[Backend Runtime Image]:::gray + O[Install CI/Test/Dev dependencies] + P[CI Minimum image]:::gray + + %% Main build flow (left) + A --> B + B --> C + C --> D + D --> E + E --> F + F --> G + G --> H + + %% Runtime flow (right) + J --> K + K --> L + L --> M + M --> N + N --> O + O --> P + + %% Cross-links + E -.-> I + D -.-> I + I -.-> M + H -.-> M + + %% Styling + classDef gray fill:#e5e7eb,stroke:#333,stroke-width:1px; + The diagram above illustrates the proposed container strategy showing the relationships between: - Build Base Container with common dependencies From 8ed1fcd27b7d3f887ac2ea272894af554d8f84ea Mon Sep 17 00:00:00 2001 From: Tushar Sharma Date: Tue, 10 Jun 2025 23:15:49 -0700 Subject: [PATCH 08/42] Fix mermaid diagram in markdown --- deps/NNNN-container-strategy.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/deps/NNNN-container-strategy.md b/deps/NNNN-container-strategy.md index 1c8f3e4..07abb27 100644 --- a/deps/NNNN-container-strategy.md +++ b/deps/NNNN-container-strategy.md @@ -121,7 +121,7 @@ flowchart TD J[Cuda Runtime
nvcr.io/nvidia/cuda.XX.YY-runtime]:::gray K[Install NATS + ETCD] L[Runtime-specific dependencies] - M[pip install dynamo[Backend] && NIXL] + M[pip install dynamo + Backend && NIXL] N[Backend Runtime Image]:::gray O[Install CI/Test/Dev dependencies] P[CI Minimum image]:::gray From 75cbe692120c39816279d28fd3c8eed1123404e0 Mon Sep 17 00:00:00 2001 From: Tushar Sharma Date: Tue, 10 Jun 2025 23:17:44 -0700 Subject: [PATCH 09/42] nit: fix mermaid diiagram in proposal --- deps/NNNN-container-strategy.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/deps/NNNN-container-strategy.md b/deps/NNNN-container-strategy.md index 07abb27..dc861ed 100644 --- a/deps/NNNN-container-strategy.md +++ b/deps/NNNN-container-strategy.md @@ -151,7 +151,7 @@ flowchart TD %% Styling classDef gray fill:#e5e7eb,stroke:#333,stroke-width:1px; - +``` The diagram above illustrates the proposed container strategy showing the relationships between: - Build Base Container with common dependencies From a29d831d96f4dcddec47fffbc1fda24bfecaf21d Mon Sep 17 00:00:00 2001 From: Tushar Sharma Date: Tue, 10 Jun 2025 23:22:47 -0700 Subject: [PATCH 10/42] Add descriptions to crosslinks in mermaid diagram instead of boxes --- deps/NNNN-container-strategy.md | 13 ++++--------- 1 file changed, 4 insertions(+), 9 deletions(-) diff --git a/deps/NNNN-container-strategy.md b/deps/NNNN-container-strategy.md index dc861ed..c845593 100644 --- a/deps/NNNN-container-strategy.md +++ b/deps/NNNN-container-strategy.md @@ -115,9 +115,7 @@ flowchart TD D[Build Dynamo] E[Dynamo Base Container]:::gray F[Build Backend-specific code] - G[Backend Build Image] - H[Copy Backend Build] - I[Copy Dynamo + NIXL] + G[Backend Build Image]:::gray J[Cuda Runtime
nvcr.io/nvidia/cuda.XX.YY-runtime]:::gray K[Install NATS + ETCD] L[Runtime-specific dependencies] @@ -133,7 +131,6 @@ flowchart TD D --> E E --> F F --> G - G --> H %% Runtime flow (right) J --> K @@ -143,11 +140,9 @@ flowchart TD N --> O O --> P - %% Cross-links - E -.-> I - D -.-> I - I -.-> M - H -.-> M + %% Cross-links with text + E -.->|Copy Dynamo & NIXL Build Wheels| M + G -.->|Copy Backend build| M %% Styling classDef gray fill:#e5e7eb,stroke:#333,stroke-width:1px; From d337ee39212de1db901feefe575c272e83352ea5 Mon Sep 17 00:00:00 2001 From: Tushar Sharma Date: Thu, 12 Jun 2025 23:37:19 -0700 Subject: [PATCH 11/42] Update container strategy to discuss container release process along with discussions around multi-arch support --- deps/NNNN-container-strategy.md | 65 ++++++++++++++++++++++----- deps/container_strategy_proposal.png | Bin 47514 -> 0 bytes 2 files changed, 53 insertions(+), 12 deletions(-) delete mode 100644 deps/container_strategy_proposal.png diff --git a/deps/NNNN-container-strategy.md b/deps/NNNN-container-strategy.md index c845593..45ea714 100644 --- a/deps/NNNN-container-strategy.md +++ b/deps/NNNN-container-strategy.md @@ -23,22 +23,35 @@ # Summary This document outlines a container strategy for Dynamo to enhance the developer experience by -organizing Dockerfiles to maximize coverage and reuse. The primary goal for this document is to define a clear and maintainable structure for our Dockerfiles—specifically, to determine how many Dockerfiles we need and clarify the relationships between base, runtime, development, and CI images. The aim is to ensure each environment's Dockerfile builds upon the previous (as supersets), maximizing environment consistency and coverage during daily development and testing. +organizing Dockerfiles to maximize coverage and reuse along with defining a strategy for releasing pre-built Dynamo containers publicly. + +One of the goals for this document is to define a clear and maintainable structure for our Dockerfiles—specifically, to determine how many Dockerfiles we need and clarify the relationships between base, runtime, development, and CI images. The aim is to ensure each environment's Dockerfile builds upon the previous (as supersets), maximizing environment consistency and coverage during daily development and testing. To achieve this goal, this document proposes certain optimizations to improve the current build process: - Restructuring the build process to provide a base container with a pre-built version of Dynamo + NIXL available on all distributions, enabling splitting of specific backends from the dynamo build process. - Defining a structure/template for all Dockerfiles to follow to ensure consistent and reproducible builds across backends along with specific roles/use cases targeted for each stage. +This DEP also outlines Dynamo's container release strategy. Since pre-built containers represent an important distribution method, we need to establish clear guidelines for: (1) which containers to release, (2) the requirements and processes for releasing these containers, and (3) the container registry to publish these containers to. + # Motivation -Dynamo is primarily built from a collection of Dockerfiles hosted in the /containers directory of the [Dynamo repository](https://github.com/NVIDIA/Dynamo). Dockerfiles are split by backends (vLLM, sglang, TRT-LLM) and each Dockerfile contains multiple stages -(base, devel, ci, runtime) to account for different purposes. Each stage essentially provides a Dynamo build along with the specific backend (vLLM, TRT-LLM, etc) and NIXL, the high-throughput, low-latency point-to-point communication library used by Dynamo to accelerate inference. -This approach has several drawbacks, including: -1. Inefficient Build Times: Components such as Dynamo, NIXL, and the selected backend are rebuilt multiple times across stages, instead of leveraging a layered, superset structure. For instance, Dynamo is installed three separate times in the Dockerfile.vllm—once each in the base, ci_minimum, and runtime stages. -2. Poor Developer Experience: The lack of clear organization among Dockerfiles makes it difficult for developers to identify which build suits their needs. As a result, the devel build is often used by default, regardless of the use case. -3. Flaky Builds: Due to the large number of layers along with multiple repeated steps across stages, builds can fail intermittently resulting in flaky builds. -4. Lack of standardization across Dockerfiles: Currently, there is not a single, stand-alone Dockerfile to build Dynamo, NIXL, and dynamo dependencies resulting in duplicated/missing code across multiple Dockerfiles. Optimizations applied to one backend's Dockerfile are not immediately available to other backend-specific Dockerfiles. +Dynamo's current container architecture consists of multiple Dockerfiles in the `/containers` directory of the [Dynamo repository](https://github.com/NVIDIA/Dynamo). These Dockerfiles are organized by backend (vLLM, sglang, TRT-LLM) and contain multiple stages (base, devel, ci, runtime) for different use cases. Each stage includes a Dynamo build, the specific backend, and NIXL - our high-throughput, low-latency point-to-point communication library for accelerating inference. +The current approach faces several challenges: + +1. **Inefficient Build Process**: Components like Dynamo, NIXL, and backends are rebuilt across stages instead of using a layered approach. For example, in Dockerfile.vllm, Dynamo is installed three times - in base, ci_minimum, and runtime stages. + +2. **Developer Experience Issues**: The unclear organization of Dockerfiles makes it difficult for developers to choose the right build for their needs, often defaulting to the devel build regardless of use case. + +3. **Build Reliability**: The complex layering and repeated steps across stages lead to intermittent build failures. + +4. **Inconsistent Standards**: Without a unified Dockerfile for building Dynamo, NIXL, and dependencies, code is duplicated or missing across backend-specific Dockerfiles, and optimizations aren't shared effectively. -As Dynamo continues to scale to support multiple LLM backends along with efforts to provide pre-built Docker containers for external usage, we need to define a structure to our Dockerfiles to improve container usability. +Additionally, as Dynamo expands its support for multiple LLM backends, we need a clear container release strategy to: +- Provide pre-built backend containers along with Dynamo deploy containers (api-operator, api-store) to a public container registry +- Ensure consistent quality and security across released containers +- Simplify the deployment process for end users +- Maintain version control and compatibility across Dynamo releases + +This document proposes solutions to both the build process challenges and establishes a framework for container releases, aiming to improve overall container usability and deployment experience. ## Goals @@ -55,6 +68,10 @@ Pinning/Fixing dependencies will ensure a unified build environment reducing "it * Outline possible further improvements including external caching/multi-context docker builds to reduce build times. +# Minimize effort for providing multi-arch support across various backends for Dynamo by leveraging manylinux to build for multiple distributions + +* Outline a container release strategy which enables Dynamo team to release containers as part of Dynamo releases + ### Non Goals - Slim backend-specific runtime containers to use for performance testing. @@ -65,12 +82,13 @@ Pinning/Fixing dependencies will ensure a unified build environment reducing "it ### REQ \<\#1\> \ The build-base container must be designed such that backend-specific Dockerfiles can integrate with it with minimal changes to their existing build process. This includes: +- Multi-arch support is a P0. The Base container should be able to support both x84_64 and arm64 builds. - Clear documentation on how to use the base container - Standardized environment variables and paths ### REQ \<\#2\> \ Dockerfiles must follow a layered, super-set structure to optimize build efficiency: -- Each stage should build upon the previous stage +- Each stage should build upon the previous stage or use artifacts from the previous stage - Artifacts should be built only once and reused across stages - Clear separation between build-time and runtime dependencies - Minimal layer count to reduce build complexity @@ -82,7 +100,11 @@ Each build stage must have a clearly defined purpose and scope: - Runtime: Minimal production deployment requirements - CI: Testing tools and validation requirements built on runtime - +### REQ \<\#4\> \ +The container release strategy must be defined to allow the Dynamo team to release containers as part of Dynamo releases. This should include: +- Which containers to release (backend, api-operator, api-store, etc) +- The minimum requirements and processes required for releasing these containers +- The container registry to publish these containers to (NGC, Github, etc). # Proposal @@ -90,7 +112,7 @@ In order to address the requirements, we propose the following changes to the Dy ## Build-Base Container -The base container will be a pre-built container that will be used by the backends to build the final container image. This build base container will contain a Dynamo build for all backends to use for their framework-specific build. The base image will leverage a manylinux base image to enable support for multiple distributions (U22, U24, etc). The container will also include a NIXL build since this is common across all backends. We will create a new Dockerfile in the /containers directory for this container and provide the image through our CI registry for developers to use for local development. +The base container will be a pre-built container that will be used by the backends to build the final container image. This build base container will contain a Dynamo build for all backends to use for their framework-specific build. The base image will leverage a manylinux base image to enable support for multiple distributions (U22, U24, etc). The container will also include a NIXL build since this is common across all backends. We will create a new Dockerfile in the /containers directory for this container and provide the image through our CI registry for developers to use for local development. The base container must provide multi-arch support. ## Use-case of build stages along with relationship between stages (base, runtime, devel, ci_minimum) @@ -102,6 +124,7 @@ Each backend-specific Dockerfile should follow a specific format. The backend-sp | Runtime | Customers/Production| Cuda base runtime image| Minimal image with only the dependencies required to deploy and run Dynamo w/backend from the backend build stage; intended for production deployments. Copies dynamo artifacts from base image and backend artifaces from backend build image. | | CI | Developers/Internal CI Pipelines/Local Debugging | Runtime image | Adds CI-specific tools, QA test scripts, internal models, and other dependencies needed for automated testing. | +Although the Dynamo base container is required to have multi-arch support, the backend builds aren't required to have immediate multi-arch support since backends often require different build logic across multiple architectures. # Implementation Details @@ -156,6 +179,24 @@ The diagram above illustrates the proposed container strategy showing the relati This layered approach ensures consistent builds, reduces duplication, and improves maintainability across all backend implementations. +## Container Release Strategy + +Which containers to release: +- Slim backend runtime containers (vLLM, sglang, TRT-LLM, etc) on all supported platforms (x86_64, arm64). As long as the backend is supported on the platform, the container should be released for that platform. +- Dynamo cloud API-Operator container +- Dynamo cloud API-Store container +- Dynamo cloud helm charts + +The minimum requirements and processes required for releasing these containers: +- Containers image must be built in CI and built in the release pipeline. These containers can then be staged in the release pipeline before being pushed to NGC. +- Container images must go through CVE & secrets scanning in CI to ensure no vulnerabilties exist or secrets are exposed. +- Container images must be pushed to NGC, a public container registry for customers to download NVIDIA containers. To publish to NGC, the container must meet the following requirements: +1. Container must not container any high/critical vulnerabilities +2. Container must not container any secrets +3. Container must be approved to be released as an open-source container + Given the requirements to push to NGC include several checks (such as CVE & secrets scanning), we could temporarily use Github container registry to push the release containers. +- To ensure container freshness, The base container should be updated use the latest CUDA runtime images when available. This will reduce the vulnerability surface area for the containers. Can be a separate effort but is required for getting the containers published to NGC. + ## Deferred to Implementation diff --git a/deps/container_strategy_proposal.png b/deps/container_strategy_proposal.png deleted file mode 100644 index 2fb0c1285ce2c67443c814d088a763398833fe98..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 47514 zcmeFZc{r5s|2M2pq6k^CBvgtiMJPMdq!L+1iew8R>)4mUOeo5dFhaH|TYZ#$XG{^Y zP7Sh-$-d8EFvcwIOW)t~JMQ~_?&tpJx$p1s9QX5a94<50bzawbdY{YndcWSU_xTcP ztjEK3f{TrfjYt3Xjk|1Y>`*o~?iYs-0%x|u8Vi6wryOtJHDY6f%doLMea6PN3!Hko zz{VDEnT>7f5gVIEIvbmacUFU`78~1vGN|Fbn|phEV`F1)-@b*z;W!*_dwZKiBHg-m zOIcZ&PNzSzcYL4!0p{)N>guYid*jEz2yuGh!-o&8t*t|2zhmQH#l^*iBO=Cs&wehg ztgWr>?D<*S(EO$K`kKbMf@Iuz9R`{Z4U7#eFNskDtmibMwauL~94PWK>%AyMmbbBt0YZjI4KV zus|UBTZC*tUS+P*-=)?w`Yb17ly?dn2C3K(1e}sKvx5 zrle)&cJ%gJ+t~eGQWXG?$jr%GC6R-Ig5DSW{js<#IyOGP8<`Nx**nHZ?PQO%ECt5mQ;)VDI8pS<`^Qwq)eI*EKMUijH@M z1whShZW`Wy_A)UzJQCkOc-QigmagFhadK#6)XLt~Hz>Rb+meOO@B4wzFZv`k@bn$f zsRtbH=-ps@aY*7e@a2g2ZA(8kHqp%e-vjyqQ7GUfr@y|@P0mG*LxPe#>q(8uz(8Qr zzX7=yJi5w^a}-~Ei>4j`v28&F@;S~{-AjJ>G|O){%PT0b@6w2JWu_SAf-R&yGZWK! zHl@<%*NFAdO9@%yEHx2LDpH;Jnd|dsv9;?9!x#-SIERoIut8 zFMh84SUnef4a2&&tH;NmG8Az~5_Kf!`9PCPvCTH8r_tJDp9|JRrv4%84eH{nc+DM|Ql9##+&8rDD?w z#~}x5-dCfWI)egQki?mhV0!2Lmp9XN2S?k2>5!k1E_1V1st2a2Vdj1(rF5DuMQ&^} zDP3L7IE#si(54wehXZbLHHPnIoh>qq3SWIg!c$AJp-fU#V_WgORNUBo(kzC#F&4%< zKNz#|I~L8h{E)kgGOu|0G$_QD(N6!#2i-8*4W)dnBP_X~dz<)4S_#sTo?A3($3|i~>Te-R_~W zOU#i>liBw(kdW1st!x5?wmD}xz6Iw8Bg73hJFm`$(B2^O5YDArc%wKr=MBKkbJ+HbXmTLNobw7U-+G}ChtG*S@(r}46{Z1z|r=(boo~h@wnp$ zhV2Rx+-91Jjj~wh4|^aL$}fce;S9By8M4V;(kC@9pS{8Z?EoJ}nFd zk;k=`i};AC?TXF;o3uG})9$;vG>6MG*p0C2^oA<}Lg;>9XmhBu4rc3M1U{$J9eapX zHd-Ho(I7Je)cNdj9|TZkN5R+2riCbry%dY9ke7|8!FA^c=A|4PuG{KhdOH;R_zJ8h zj-f<~LqjqXUn34fLd4QG-X}PAuphcmEiTk}T8B^E-)B@`w5HBe2b;H1iS0ec~*_uGt%f~h;+WCVy37(l&)>{MPV^I#|wpDFRLZcS=llg|Drk^bP zp}&DgQbV;oevnsc7%E<`6k5y2yzk!Ds;TI@T{k+GhR~QbdeKWoL?u>!Bpao#GwoB0 zs?j^;hFQPErFgBtRGYQ)-}CNkEUvbrzlDa(r!g?%zh_cfN~heg;eGawzhy3Q&aM_Q z!V*nL0THEk%rzz6ljHZ=;}VJWQBAmsH?W10?Oj6A@ia0OonxRIWuzSF9pvg?hHTCMy~Er%Kf{^UZU^nLV-0#ftIS7xH_Gw zM%%!IEE@kEZi<*?gB-ZOQ9X0Lb7deKO&ycZYw zO|W~SVRg5_lG}h3{v;(-@qO;WK-sh=?@D=MxQOkWot4?q^FCXU^;$X7Y+f!poF<<& zKV@7QYX5LInqTO{Kp?bP&bFUoqqeZ&arGdb+1)iJ7G}{r8W&tW6nLAmK_^fBlv178 z*_8KKXI zue+`mwRe+1yTX2Gt%m-}bD12!xyxKGwfnr$xok0Oh4g$_Rk5=nt2*wJT`NxA&{h08 zK0#Lk+V#!q+QiCgrg!Y>o&M3dcy3?-@uM=Czir<$ScJ;0{*bY2P_zJ~zAaNtzn{hS z9DPkT(l^tB+>Bh;)vPRY2VNv+KuZ6^V^`G1dPt^y2)#MG*7sK&Fp)%2wF=^|?oDVF zYld!xu!1f%BQmKt0%Sy0;cc_(ml#ddJU45#bKgUm^m2|Hu-gI zz8KqMU+P6>>e7Az&1^$!!Gbq_74K3NLVd8qs$ zCxWl88;xxR-G#hFF8sjG7tlB9msvZ_jESRUb8?Claj8vjK0HhW7Wvn6Lx#I*GaD`! zi=eO9?~P@ZQd}cyC!>a>a?pEJ=i_TQ{-HH_7as04E+ig=}?b zpNzuj50^~egslz@72Q}#2Ah+X0~C!WS@BOKg@URZ?0+ul0HbHS)y=lRm$=Q{cJzBO zzT^4jE9WP`^6_9DbL#kY&)9eRJ95%6FpYc_5@A3K6{LN}vkaIsd*g+~qb)W36KuPA z$mzTt`3;4M^eMv9G*J#|$~u@6cxQK^l=_pnkZ1ROc_)<|H>R~WLoCRzDn|~7E%q$q zrA~cjW#flfsVd-S_mRhm>8_rYrS*xv&J=O{s_or&4v&k9ZZKBUl% z^xkS6GZ_RYiaI|;e3zoTo4FkWS~a;S!8y+=MKsi&htJxIQe9KoETTnVIt^iyg)xBP*G z-}Vq(a>AqGgwWobWY~u4lgiLju#Mty3rxV4P8Y;Sj&bJJ%zaw(YT5+j7%{V#(8foT zQo9stpA&A^u*n%(T_(P1v#3Pb-Ew>Ti|G{HXt0j)Il?8>Mbx^=%J!q$H3qO=@2xkb zKQ)<57-G-|v33bJHa>4=Bc*C;anlMrdvb-Onlg3=K2g{ zb=D3Onkw>M0OjKIL5lTv9YJkcWoNQ6OkL5$K7*x_ilt1BlIXv*5ub>Ti1Q`SJNdn@@aK^U@Pj!G^*Ul^fGR z?1$VwADV4-JACYxpw0nYxA+phE6|LVnY|8I2-n(p9(^P}Y=8o1swi9#rvsRkh+?%6 zm%9|0zR->7yRbOzSSm14;*kRCNaE64da!Ae^fpB+o+dDX2$Sqk3gnwN%9^YWpdcd# z_L8>eXTjHYYBa9W-03i3u0=`99iY{%D zlEOs^@RskGnrNk0(SmfQwfELMC??U(pC>Gue@_`UT|Q4&L{b>O&_Dlkmq1P$iJpr6 zl{rM5CAUb4*c|Di&W*(3?bypTU0}cFp|!K|V?|nROjYKRV!~#%K$BNnY4ZI=d@NXW^Y(mh zn1!uDv&eeqOzD$87-c1#6?BVo5KJl+KuXOk>R@79pHYbz_?)bqPI?-7qM@|$5Z0%7 z#>9(T8dXcgAiRD>-u078MSnZjv?Fh<(?Mq~bjX)7q|^dl(ta!K+OM`x)}?)HdLxTu z_GPLTUp)n2>kWzuc#wTIz^8{&4USMbDUtwrnR(Y|+rr0dkxvo8DzvXNm1x=`3y}mf z7pGJ3c9PXrb=d7p@6Ty0SZ*escs^XB(88{H-48VRMqy~b1+y|*6r}6GjxUD$&W_Mt ze0d1%d=_dNhYnD#Ox0~E?i9pMSHUG7sl_ti4Vy`Wf*-ahGXm6%D}_;Hb7a!oXHCBC z45Z-euxA_gSFe|g<1431U+Kgtzgvb_W-GllL24mgb1Us@nx2BI#Gc2GGx6==kFa=C^H-hmvnhem@G5_ zKRX^gyQj9umpZMQYaZ@XDiBt#(L@PXyV!pCkcnAXG`4Rh zbW92e+SqOnLWVN8mxth?bhorU{Ni$^*Y@9wjbOq)40nKTR1_r4>A-B(o&m@OmH5@} z3ZlE}D%mRB?_o6nL8ffTPRE2k+<-p*=lFlOKp4#V5A*UdL>A+zt>3;`uurhS=lZP{ z6PAzcKIs7{9=o?-T!H?wN9DH7i zC|3DFD}46F{Mu4kXDtc+y&n;mPV@vN5v~+iygR}n`^E8b)ENvHTNPTC?V&D|ttuML z_Jjv=;0FMk&{x?HT?4S=(_hDg*gl^I{U1_b^at*u<2ZH%3U^{#sd;HBcDwZGQ-X^1 z*5iv9dA2~p`@>!L;P=^98jgf&v5hJ{PUO7iz7d}Dax<1~I2`1PQHOZbG}frn1`u^}^*Z?uB{l2~PW-}=)+WhK(|UKhr@*CctiF++4Vw z@5Wr)v2#)%$E!Y;6dJ)vg|xvbUlpm=CFfN1#i=|wPN&xm&z~QkINJ3iJqfABU3s`R ztF2M$jk9}KrN+8rQ5PbEG zhHrLv&N(lUClE^(BZNdIM|{y~1(r@x6ffjJ69?qwp=o^e?Z( z+dr1qO3DQK`KM$IqKC(ai!U-j&cEZ1MJa7q)SU*|G=48HpDyx+-a@F|<%vzOQ#$hQ z;}HWK_{H<`pUd>n&Ac3+2j_yGyEaMH!kKI3claO&G!h~OicVjEtmh^8gm6`Tf~kJ7 zMSufCbH?l_g;zEKmhEw#F_M4oav_4=X9&~DH4#R9Rb_XY6xK}7Fn}_=A8UhZlxVMA zl|kWYJev6xwJS6mDeS3Q16^buWG8s#&BAuhO23ZPy|+1`tCkWPV}sUGWUnr}aZNOV#Rg11d( zZlO_uFpZB#`kUg;>bKj2Kj?rghfgn_b%&Hs$1cL58`^Ig(8CJbs|efxVe1ESN_d7k zYwf&pT?Vmb)>vk_xIY+9aJ21BlyW$Ez+JO6Fs@b^MfO#X%%>AZ(LK|DSy0ZcyS4hL z1?B62Fk6~@QVqH$gJ?sKoT+NJ$SIC<+^R1s?u%s%S6;MMx6C%3Hdj8clOq5$TUdx5 ziUrGYwvmp<@A|mRwR^VV=C)T$OnzWbgP4P|>&rCLoRTr>i%ilJZP27~=rTm?l3}S8 zZ>&{Zg!{E36_B5@{Nb4D*E>P9j8th2BMG2v(Oj-s^C7k6PHk=Od?LAv0GWE3C% z&1Q<4lj1hE>&|?)FhuOx^Bwe#vQO(^vX<#-5DVRXx#!7dF=Vm*%Eez%Q*Z7`zKa*O zx090ybc>pbozv4iSowmkF^lfm5PYubb-8kPzf;>G<(W6se?aKljbnDJl0sC6@u1bm$V?i}WkzviMly5w0JuCxwvdb%zbKZvK+cSxGB@`5C zi(Kir$9`zx3SaElQwrKYPP(0SKQiOI)NBBU&jsamaW_T-wsRO(B@M=?8(oKvM|%jL zmY5uT+9tp(>3_5WYZev?`l~pU_S+dczB>yM%U@q*Qy06t5SAbv5`{K>Hx zt`h*p-b{~DGHhnm9B7OM0C>kqp}~1P)GHzC678+U|NW{(K z@a}Bac;Nr1V)Xy1E8@AM%DPi;r(eqT)P}l@tCuRUIP|52-V39k8D}vU00@r@F<}`k zUOiAopVGm>kkXhlY=Oj5;4=;{_tzsTQvkkU_r{iw?F2)5CahzD8Nr6#||d>5EETPER!6eZA^v;dbUk zsx9{;?gmDM(B+5e&o?#LtbQmv0s|*@uKn`GXVGGa;Svi*r&=lTk^E+*D${rH|Z?)q-uglnoAI_Q;*W!GHv%+J&xF zz3ncU)va)A0i@5{$Enp~#=@iviI1-Vo*Vv6RPw7SJ)h~E(MyE;j_D7$E4}x`qMOf7 zPc8L*KVAsBYdz#MwWS$O&8?~nAF>^ZCtM%j5(uit<_mpDqKBO%H#fYooSy_<5mmG3 zbbSWMtIDxo88bLUF;Z6)%uN1SSLI1fSRCW-$|USI$h6hTR_i80Tn6|KaHS;{x{C75 z2C0beB=zh@8$zG{@=yg;*6oDlNu;-Z$uBPLFba{3k=e$!4d~%|#;>;5esbqWy|kQ> zes^%0gJ!VpUv&o};e(p-vzdHxPHiOP}CzEokF)uTU- zoPOUTQLkM%A)Ep%xqz*dxT2nENpLsbHzE9x-h(rqm}Za;!)5g44x+@Pl;Ra1V72B> z+l@N}EH`H!FpjrL342x{j2bwrT}dGGKq9UK;4SmHgz#c|BY%{HT)ep*-+2C#S~2fo z4zHQZ7wjB5j%J8}ajlA_fki?n|E+~N*5$hl|H=8lxq<*I?$|iosgbE$CO}haqVVfh zPKJ8vuO%{F7ni?&n5SwPEKezXLioKf7wP-s1dBx+n=->67Q;m{&3PxY}3Ux>`NQ3E?WvHW7)uI+~YpTqHb zaf4hFp}bC2ir>oOto;p&+Rmx6pBpzdfIbZ!K|nnO;FEclP)*swr$&a!f zZ{R3~pG9O!78X`b5*D>krn6e8MK_z4@K@trLt1Yf${xm<)_stQn&P{$m7#lpG=Ct{ zp*D!Rk@;TQYkKt!mUMk8S~Snn=SF@1maAyIk*kp3az;gpccBN?3|Z)*h0uPx(=F_* z;w#m-^?e0#zk}x`ob|<6%flm0S*q0Bh5L^+CC`9JPU;z%&e`7-dRnYI-O@Tp!x!Px zT`#4$Q??n!47@{a-sVF|g^wo#RXW?DQUPAACf7_KW7`5Oe9ypC;I6K%r zaRG#QVjpzx?^2bLrh6em&JO*-OIxo=DhXE}>Bp z{oM?9k}#GMKUW;e1#?gWVO%>O>8zHNzC_ZKP1mDkXQ{eJ>`I#!JD1?C_{Mxx{2UW+ zbM%f>N3DYx@AzGfNhl88*p_un2-K!oGdMi;*yGhc__-T}X6rqMI>CPENi=#O9kv$= z%%6Ywp?{kNeoBLWd{1xQyr8AO|BCKxf4KkN1kR=%_LFSBwInYMg`(L%@lr< zZKYWXDIU3z@`vCB$K)_9;o59Z)=Y*0Cie}IaeN>BZ7Bc2fXjebuNwf0=8U=8ea`nK zVV))naI)9pkef8sKd6ux&a#9bXM3{V4NkQ^$fhc0SSdlVORJ()+xG#s`ejs=0W;OjzQ@+Gv%yxqZs3GfXfBEg#@Du7_KkEn}&bf zh%f0S08(7NBNiGbnrR3>{&c&%L)y{V8K}D|5&ix9@Zd9m_~iiN^8XzFJr=Oy>3TvM z40BQwA3@GV4W26~er}b}yfqOMWb@1}gL5CqnMWz9+Kw)lAf4>t*~ShP8?h*zqpIB` ze_2%n((VsNm)2##tFMtjUF;SRkNG-k!9f3>LvL2-C!bK3M08__>Y#Y&(>ETPAQCw5 zo8eM(m>-m(5zr4CTOgejIs$~wftQemP0A>;vM14OcSMVo!s(=w$?G187ZVkvrtVg* zeLb}t6rEzLI3e~JbNNMv3+1c-n^`+i!xdX|NfZPoP%?ny*f!`7Q&W1|w_ALbtF|lV zMwDxEc>mhMzI+J&8g1Ac9AKd%U5wNSuf6lQ;w{=Ajwlv)a@EOGQhkXH#M#*o`$;KS zyDKSM)|oJFKl!a76@%_^VWpH{=MhU}IqUPiuw%a*n;$;xAh!1au9hsok)MuKvQBrM zFGTm)$b~7X47qv-Bs6J(`RiqWw&Ld!A~s>>5h)kUi2$*pJ1gk(NZdxFwM(x zHa$Pu^T9Uh{=Y>Y!(Q)9;=kK`)LeHwg_ZPj4R5>>)K;%g_p>TtZ8rmPtP7r+ApPXw z&+6grI@#&9x#iJpT1mJ*VJ^SL*n}A6oPO65rRTs zOAU(n!jF4V$`0i`(PQ>KaWE*A=$iWJ7FzJN zuCD39Xir<4jZaN)+71wi-Mp+9!r;*GEh@_JH|6R29w=?p8%$}7kl_hEXBM>aa@{4q zr}xa36y1nG@a6B*KhN1Fkza06Vdu-|xgr8<&vSz^QdfqA+h(T+@z%9Vwh;qo_Nlrv zplM5lTc}&|I|MWdAhk zw!`|+Vy(i@0X=Xx@eC+P#iCM7;J*#FS6M;CdTX_+*B=4Xdh|1T;h*T>zsCZKe{kQ& zecX4R81&>D%z)p|kpln^Ri&_=8*qu7oUg+d^M?)$M`tKvNVnmaa!_BdTrE_<2!BzD zZ*MmUOdhd3uN`qW&*LH{MEmNZ2Z(9LS>rN*3&hXkDA{KL?g=_gaap{|hUU=zKQVwW z%wWjPSJLP876T5HEwgt0NC4v6M|np4sN8>XfMydZe^4Glgc^dN%Pk%UC_o@~Z4xiK~q3jhMJAw=P7@U_KH;Mc5o2d$FzVRj~_8h?Zr zKb4(fdu)79_I?OISLC|#R@~2@afomETl$S%Sw0Vl^LBo1NzoIcXfz1=#KX8iwM2HY z$MBEhBVIjiq9!L_S~=-onK#$qWAJ48e?|4%&q4Ns`~P6Pe~SeQ+-TSSeYOW z6--76YmB>z+hk0(l-2b=Yh0ZN)e}+5S zjLaTrhBQLT`;1m=u-42eyEoghuUav=sG8{|nbjzV4gqHcTj)cA)8gNQ+MlKUX0d@y zT&A;JvNUrWcBkn^@BWsw)i5lu&bw)Y0?o?D5Y4Bh-kODeo!!b&ESh{J52T(Hb+BU7 zM7F1y3Tc6RGnc~xT){dT#G_sMJLIiYarjQDl-+1zX-l{ynVf;aYsFy#J-#%nYre9m zYk0-2Rg?Dt!<<&&v@iWoF?=vhl#YwA>24ivG&);efmLy)qPy82&QM-8fj)XZ<=aTvo6wcx<5{n* zc+!s5uDK(}lq-k&BqabivdTHzontejam<2blNlS!|Q50798DVb7y7<)n@e|Z= zrR&z?Ws|{{*5^D1y2HNsEC#*~6EqLKSe&P2e~F2u`{V^3e}L#ro0eCX|Akgc@Dqy$ zcj!E7*IOJeHPeZ?YwPi#rg6``cD(MM_$A~R^9>xJaQe+7eI;E?(-7{~o@=b`4o^k;!rPeFFCGi;eNUC-H9QbZ?p( z$IEgJF30>ZsNXS>O!CGzHd@clbE)TJHGzgSd>E@}Z!3msvbTzqVa7+fSsjkG%O}Ph ziAt^;iAHDHJ?Gc+gl?psKXf{5E0_rD?r!ylzqFd5y=Tn8i!U8_hD3F~ddN zeh&@1P*J`W4_6@TCQamShsl^W&#@z2$Z0H%2oK&{-Rr|r$Ohn9v?U$#QaZ`3F~p8` zss5x;v55m_{3KjO)w|cbtd#yeApr2cxF_mv)mAUJW)BFAXqzwm6cSM0#Ny^Sv zGDhN)f)~ioQXC~+83Yga_SS`NPqpj^>wDyJCVLDD(qdPgvke%erW1+91?qHw7WS@^ zu?|rN>3JQV)KhIUm>nwkOD@yk%#tFHB=Os@K2HEmDJEN;K0W?}Ncwi{L+5R_CiL0QBe2dJ@IFQ>v=EZ*dVwb2vop=J!Zv1ydk3-_{1%K`8!}oc zdEQ|~Cxxuy^4vw_8iz5v>O#l|V;J-!@&kRS#-468X!z|fyOY7GQtLKvcdozf%Kwy` zqOJLfuPd#9sJoc+DWEi!gOvsbKzY(FmaVH1?=Qqeo9 zw(U2we{s&Lg$+uH}MV$5m zSFKNiX}(R!harPT>BqDan3c&-e0s@1Vix*QsLb5T4frxVayx;a{-?mHtD$!mdzU-wFFVU@OUr_W-MaYX74$YV7N5ylPf!_xTDp}Xo=8d3bMj4yiTS5i8^-{p# z6yA!rG@K8<)}L`Hc3kv1Fv{oJ&q^<*2Kz>DH_8XjvmY8u$#@Rrxj<*tLw-5}uE+U-3B7L)pKzZ<;=mRdeYQWC$=lYb=n{~ikr zxIsDaiE|`(4X(WmXWfiJnLn~ynU~69USaQYw>7fumCoB$+lBok{yl^V_^Ge7GTwWc z_J+qd{gsWWlA|GxrRiA_4|{kZ@~`X@?B^7}IxydaHwFaG&0adEXvwk{hNVu_Y)yoF z&b5cN=J7(r3S_FKQPB`u;qgj_1gPlncv{wvHOqO7gf36*lcDpD>p%J<`R9 zwldd|Ox3X@e$W$7bh+FRlGMgE9-#0U5SColm^S+WzH&Y9~Hm?&Vg1-kkGO4LppoJj(Fu{vMv?oY!|b zK)k^!Q%y&|$K!{Xk)ejSh(a{Uv9_AJ@;gu2_v?ky6rxTeA~o5YXkX#H@vleno1I5chQgS zmjb+05;vZ$f(+65q`qlTA-V5xxw~MfQlj#ekMvm>!&y?sv*k(a=Nlo}&dwfKl@qkm zx+F7~1xd@>Z^W_wgboBwcA3vDGPzFr-nLg<-el`ec7`!$iHeRWcTn$T=W^;3jkXQ) zp3Pan@TSmq{p5TZFH``KSLIS9fO?Ke9kV$s$eZV-;uoY7I~LM}SkZTJYxk2qGFFD| z9kvY@1?RM{s=w*wJ&R|ZDzR1Dtty+%7)`p_j6xhXDtbULfZpvd)#=*!1QVt{gKu+0 z)iomyRp2+sF@U*Y>ALGtF`1G01Phy7U9nfpCyp9@MK|PP+!;;;@2&AqYUt#hVg-#! zUOC8Ooz1aYyNU)ARl1d}fv~j2C$DoL@t99tR*Z|J!vU*Kl;MJRRuTH;VSv84c@c#B z=DA!hNZ3Npbi7b~wXtc#zw%41uOaH}QFU_)-}aQH==G zDQNK9_~yvbF11fTmpszK1Uj-~l@j7I>LJF~5ihMGHvYprhhs}yzQ79es!pH~!zW~~ zLe{PBsx^LICTl5HUQW|E;_g`aNOcU7K9HX1LNS1j`=pwx6Em2fg0)>4QUM?7dv>X0gOeN%MZ3r_cCdtIF*nlI$XO4q+Me{kdlfg$Kry zUv1SHd3)$FQ~WQ10xMN9toLw90|(e_;GG6US2dYet5LI-uH`ys3>I&zUfi7qsT|fv zI`=r-mrGNLR$D`f8=sO{n1Rs=&04+);7d)3{w$M1z@$ zMqV=#HS#DRq>eMED>UP6aBxceA%)4`+SFU;n-eOwvbJU^+ z4K^#*e6JxnDP@1t3`Vb#EI3cJPAnOnm@B?%ZjP>CG)jY@`b5o|bDHZf07vD&PV>i$ zpgOaOx9;xyD7%pr;{V;{|GMZupZ^zHK!*L$2{9o5>$$9}2mD`fi(5I#_Ic5hQ=+~A z6w65W2VrhJMuD z0HmIA{OUO~Vr2HyRSxhkb^*l#Par?weEdxL$NBgl_+F2Cao7tC5JvuA-T>)O0rzW_?;&emi z=j{}YSv{4_pC?k~le3F}_w^z_`#FYcU`r`(Dc86Nd)c+LQ4H00>MFBu;37j-SG7mc z`ATKubu)0IlF5pfDW0eMq(Q2vhA}w(e~wAx_O-8Yk%#x^$KX3 z7N#SS+4jND3@T8$5X7NvCW2~i^mc}mrUwFA()~%ojb$S7%vt3jF_E{XE{_<0$+D6i z%BQX#zXo`=iN8UKWc_d`hy68s3RDnPFQ+$@pUtqFmUiI0uMaXo1%J+jq>D%@~f$lkUnN`C=P?BpNv*RAqJEoYl#<2aD6? zVyqt9b9RK|so9JEgn(QD*A4~*J7dyPLbW1FK!zZ}0c0wO`X=J=s1(m9o!|>yS#wdzP6V_hq8T%B zB4K~5GSAOmPuFkh@@Yr4Q;|%0|=Y>EZo%dOvJF5U_iHvqPau1(bK?sF$#BN5T=V zdjdRNXSypU&Sdf8Rh?BNGOJDqg)Y7)Dg(KJR=w>c!>h;$!sjC~ujCrFxcis4J5+n# z)g3fMlyjF`hyoIW&BMInjg>bTA2)wpvD7rHA@_lmEqmh*#(ll%qoJ=3BCJ)kibvwt zKkLi-7IgtP07E`x4%j)1c(kzJzKGlxgNsFKkOrfeKTj>s*PhrB54fW+(rJRl!6rq1 zify@!jN8sA@CB)og>r{YLf^xl0$U#8y|B(h;@mk9q9}lI8YNLb!yNDuVWsDQ855`t zG84r05fuS1Xf+I)+-a=C_OH|XX;S~>Lj4;o@ZaV_1q=bhzVfo*eufK2yp6gZA|pMQ#I@>jtWAKm z;*+N3ZP=sUFjm$|{>OzXU=KpKa27zOH3Bim>IMCOsM-GkGhH?}Z+9GlvQ7IJ;-RWj z#TW0701CMW)Q&SCKtbG9avJod0X(Y>2wUAN`~U*ROTbLi6l3y=_VII@L8j_{<>()7X2MIebT&o7A=L0Wt49_zHB|I}Hq zZp^lW66JvNlH=yNNI%C{udu)Evqe*0c(x>708XC;il=30FX~Z+R>h@NZiL)}Owwc> zLUGt~^;Cqn2|gpkz_ayF(Z?REG?$+GO~=sWS3Nh}#1VgrI=bV)j=cF&V=4ZpRbOs= z2byKl@+$|)xaMRnAe4iQtl4Hvb6$u-uQ5_?KvAdn>*N=5I?ROw>$Be!+b=EvF^&G` z^1sn68)`Yo?n?@LJJy?%gcoR#E>!X>h5UP=Co6lRsZ%PWyCE(i$RD02YLnq!-x0(e z`Ti&*3CXgFR!XnT?DBSp_N*2o9=#ez8sm(=NfK8Rv(X=}L%-!B=_=>aI=Y#1e!;C* zfz42D=4V+G3AKI>(3)nG!DIW~d>g#?3fJ>|@Ap~usV}hI4o;sNt&ffxT zPxyJX`gpI1OtGd}>b$v9E17cCf~k!Z_aNE^{CIl7%<%L5=H>%cIhXEOTGu1Pw5GmH z{rH$`Djn@UGQ;#1@hkQ2q(KwB7TX{DB3K!jQNEMyqVMFk!3g7T zugnZ;wJ=Ru1Z5quLG;dx9h-M__gWs4d*=>xlBw(RsC+77Y*lF#sX!tuoa&YuXICrt)lh7*~}Oqxk*fdZ;YvZ~{*+9O_{(-hck6 zzr1Rs18hF$u`2uQfOWm#L`k1omtSo)j}-~ALvG8_1o+Rj+1ZQ{P3=x>c?qRtjdMFj z#J0c*ODiIMM-IOxbi@GcWBCxJgOXTE?U|UZ*x?bJ$0sUKdj?tSX>ZZPvaf7%F|*~G zKYzEltc+w!V==OY&XftiDhgs$p=}SVE)>Vp@GH`9I==DyG zfIIa^MRG5>1Jh+(aPB*?7ZD8XN*by`6DUNEbM4N5dxxbz_*;S&fslZ_96$E$>N!Tc znl~;_4jeOul&b$|fhA8Fqpl7CA(bN=xLcgAl|yE?CDTCVnIBh2Guu5}$HDY10A?5EB8Xrginp}3Y|;;ak^Jvyds z6U+TwPT9mv!8c=@BY3Y-*G(0yN~_wgG?qfvu4&>v(|N8 z>%7iWG!8$>j#$rGH8HhF#_&b}&?l}PCHmy1lkPwi|JtO&Y`CyA`>}Kdede#*jr_cr zkBa8>tr*0i;&Vvw4uN9UU#o zi4~}mI|0^8qMl)F&_1lc1=j?KaDNM(MYS0|U4DTQq2{M{V~=Oc1qLh_2*wxP48n+| z$M_dRfRqBy+Ngh2XD&nwMpd3#sr~T8VQ*T0YMH@1BZpuK>u38pv#`)`GW{mFY}riE zh8x?JLD=BFn6N{p3A3){?mCw)YnoTV!Xx8RZK~EgMoM|7fl8 zMh>N4&HfV$5~QHh>{K#Xqgs!M}W&cHbhCwc< zLli;ZUkZbEa|Ww_vD%P^29;w^n*Y)wBUg}oC6rrcTmfrs9@YLe6yoPrW&4OoL8-u2 zk9bP+ph}+l%vCTf`%?X3CD*5ZjQPISi4+)OXE;k!zd?Lq^f>Q|wFMXPvS*% z2y%C~bL4S-j+^hR=MS9OLJoi$99x(pd-R17Zw%hOz8JT11vW;wHv~|ca}tyo=#Z1ia(*a_5n)p&o2$#T_3;{tbhtso3Ym-MZab3 zexxmxYj!%4LR)};GeeisY>g|)rCxW(&GaH!h%A=qf#7sfCN=rBx1_k zU7q_5ghLHN4w*4K*g#K$XC!@Ueh7!yyIu+ zdRzsRlXzl-Z`~`EZ5F*(_ZfHBbul$aa8#0!ku=PwNPGThdUu|!6dBc(CGLQl+s`-oZGik>bEHZlYE^bVrZG@56@!-qT>=Jz{O+)nvzRzVbj%m&yS zE$vMnPJ<4HMsq~i*N4XO6WEsYXo6F{8e`3?{>mebZ;$=5c_s~qd?Gxi^fFqM+kMmG z<0-;F?RF2<$BM-T+>Qis_g9H3zKKD@NN0mo64WKzuT!q-RLW5lzdjT#%02@c^(enZ zu?=3OkRR1dEUF|+U?Dbp-HA8f;ZrM05j@ik0jz_N&~?x4b7J}BiaJ5a+XU>gN^K>; zADpDMS(=T!!KX;32VG}Za;x8&hbxkoQk4F#H%uW-p)j#7rTOsjb=%GWgv}?mH!zEE zQOn39BOftKZi165@tc+X$(wrOusVuPVkRNpOGwdwVtv373brH$_&hqH1Wrb<_b^Xy zeGKH>kwuEbO4*Lbh=$*n9TX#h8K#XbLKEv(UNZIsW;J$n#`~Gc zP|&~?8QWRZOjhS;1zhV_g_;bQU*n>Tw4ckP6Y=Kt^rhS|yKQb@gZnUN%j(NJXE zhfBPrh!;Mtx3CFhik{jp8Vjl9982*#ZMRt!q}06K#>g5@qH!6Y2a)PRBU74{b8;|` zJ|4yvAvC4kx7Zi$te7!k=R(AKrf2t`vu^R5%<8S@F0vlA=7nMovTUJCIK@ zGV$A{5Sb_!Ury{2K60tKb>H(1Sl+BSksheCg?a2k+n_Fbvq+jG>QZ-PJk|!2NsUIUO-REFIf8N zG;Iq|(hgEt-M|cm$_5*ozu#G_BMJE0ET_-MbD<&~Twai_5)mR{Rd)5{jE-8!H)u8{ zaUH^#6e!Y453PupT3c#6(;m~?P2*OGT-Q0C1?-;wb$@q*`k*~?r5*AuIPx_82f-Ze zV!_q=ZFvO_(Q4R!8u=eZabjb&6x97-+pWK>xM&I~tq}}SK51m!O>7HS_Z^G!UMG=^ z-A24>31fBv5Q{j}Ch$3a^_8zcSrO*>Kvgy=hTg_amia$g@BjroNPP4L!@ z3fb0f(_x+c0p1O~$&+J)yIoo!)* zmbb6i$7xJePB5|t{@jomt)^Wo$Jmm|*^0>~iytbr8b9fCsJ@~c_=?GGU$kKe6myyP zm(wgGPLkvql<87B{nz}<7X1Mz<3@fr`km9gD)BXSgeXxIu$>v1V;SQjFGcnScYO~h zP#y9ea9h6m|IxT7dC74$>$k#~0%#o4_}vjB8v8Sb(sh2}AW2XOMeQoumbtU&K`k}3 zL2NAErX{jNZTNzxFNPZ zIsv~49JmUAnJKw{4zB?@d%wp$Oxn`2UuKaN*3Pk@t@Kga2)?{`DMNETky2~@i6&)d z_lZLq`Bt}Hm)r5_s5Qw^YxEdUxFNra+8EPtH_vLn(VVx2eD6?wb1yE>QBf`{UUE)DcMPM8nY!({UjbFa z^8|YbP-H24J?6yTseK>Wt+XHG$LN+G&9a5ZD>(f-`tvV|dexuWj|%$yBX(DAVh+Ua zT=&C#PJmY)TmKC1{TEtTo%Uuw9M`r}fZz!XIYVD4Kt;K<)lM1k0dc%DwSc|*+TBlv z{;MF5-NoH`;#c8OAc%KlB9alDbk0hEO=mGcBfmgMRqV;%0mC6Z<{m3|x%6n70Y7ks z6SY*@vKjvX_lKPR-$>%Rp9cKt_$Wry9Hpe2fXH-+$=_Ott}4~vqz%TR)Gk{50PaR0 zdf+g`lf55of7s|OOOb*uGw3uZ=~#O0umAoBeKkigxd%#7Ec6|57wb2JU zrVdiuUk9`nU*$>lS1C|{z&+{Pn7ZuKjvC6t%iQPpg*IDzt%5Gd4#D=f!bquzoxXXI&wBEaU z@#gXAqejHKEK(ai*#SrHa8g~G?Usoq!doX2*(htYq*Vm3!Lge>YqefX9ZxM#a}Yq7 z7;H5C)suw29mEt3y1KLR8oZFQshv#o@6yYDi&=^s6GAh0A7=2x>#S2VA2sIml}zg@ zr;HbdGp2I&-l5PhTgLqSeY<~MN+(I!^r&E*>fa_@N7Vk(k!)|f2F#%mCY#s zSgRvlCUJ~TCydjaANcU07x5^LD(JbKb9Kw9Ce!e*r>c=AtikMgygZT+>k1nnt%Q0Y z!n4Qmli5=%e(}9i;toM{e;GO77u_v)I#XyfDT&Xc4p8tM8<6p{JJ|(@-4Gxu->$x< zaYQ3uIdM9aoQw0RV{EE*ryXbIf!JjsheoZ~7?HnX>IzHkFUg;)B+*qM#s z{P|H?{l$_}y@c&UQeL^u6F92~$M%!k5fs7KH@UEF>DtpZJtu$)Z! zrnO!%Q0MqUpw1BjSz8HjA3$*8ScB7aH0IkT^~Vua@p48m|1#O}-~|i{?J5;G+yz z4d{H5+KbvU8caJ6fljkwyrya=zpM2*XZs6EtsD!Uj&0ICq^K1I)|XZC*F^SrI_(8n zgj+jhuu?L`Jfb>Fr*1F4OsPWa@VPq|J1y3+xWNN0SYi3r{F{F@s{$zFI9A@*->E?5 z2@-g0Z?&e-h2bMvyNAozBv)Rxl{HI&6i>&*dZkf4YPAb=-pY^WE6fGXDW2>!pUXIu zxzk-W&}u|0&Y!UjR~qtc_%Z+1d$-Wjxr5}#V}fZ2R|-a?FA?~}4fiT5MFiN29n zbuz^gr+yJCkQ_g|>ej5CS_RPeDs+lw8hWK{Xuky?Q{92-c6V|DNl;>!8Lu2@t`;_T z2@X|<9p%-T*-2M4w_U~5H#dR;O9I9;Cj)qkJnG}<+>xM7jY9v_901^{$>%gem3yi8p6|*+zem{NgqBMiMADeng9YN8l@okt$cWZ|} z*@H38HjJ&0zigXMqz`=R`Vc9G-Re7=Zru*T+e8BmcT%`d0yAYY?!OT=rf)o-ZrRVR5Urx1{zd*N+fwv^%?vVR z^CcD_DOPoB)i;6{d*e$vz08`p9RWDy{7(KqCOU;@y&+*!VyMjp?=Z^lu; zShMmRlNt|{@-^C7BhEWh!e%|I5&wSb%6R(f|8pz)^vpG zG>Ta^WOST~l}9M0KBr#PEmF6;B@fd#>}Z19jhr&>sD6u~ajnIlz)Wr?z9{3Ob8S>) z5F*^v@+A{r&x4UOtwmVsOWv&aq}R)0ki5gdp(@=CDbxcg4S-l?vJ2+gYsj$Ox4^QbXf z?@;F(0Z%)`kf#ju3Hb<_h@xEyzy0ln=uzy4NaD2;)vP=~{zeIGh|mFOI9<#ccfzlH zD*%3VX(6cle({3?>`NYte2|FW7SACla=xyoOZK7#0HO2Q4FmzHJIfDuj-_8cY$Ocd z(xZIbx{tcTQgl3eAP%W{?>!d;$hxT^`T?7qe{GtJsEx$^wx`x%B5Ia@ZyIo&l)JQl z&DGu5m=5gk>lwrQAufLzT9TM)RHfqY%cdd`&m9idB%stb#-df$G8iF(wMzNh;`E8_ zgj1j-z>IJuG{^pI9uN)foQyZHI2mRKSrn>%jHqVgcG2!;9+Q*>o&=b!0?G3S_8DG< zMreN+9ZPo?yKSLD_hT>xO*g@yq!ZBtUki_7(X{!#Uw3b8BZ6N3@q16g4j8{_Etx{+ z{%-tNaXpx*)9AV*X`tNw8akB!6y<#eunBpOrEBls@So8);`bl-g2mGQQyo2QB)V@z z{sDRW`*}}+f`6aJQuMp39-xNy4a$F?{NJX+9+OZUb?B;?P=WEFZ$g~4p$4$?C(k!Y zLI5)UljHR$iAnj!ZYi89y{yd3y_f`caWR3ZnfV!yN>DO~uaVG|2scSzgglEI5FK`% zk9K}l6EX3z#S%?N1tkf2I8J2pq(u#C%=(U=1+4=8<}mAQ335cK-qvCzI4&(#hyYh$ zBFok`dV0oI@8O!eH86JYdy`w0m-^sA!oZf}J;NkLEklm*9R@x5Mbl?RkLT@tAFK{{*K`YwUy&~^9G;B9?V|a7)z^8d~ zgZ7m$T$$9AlbHwQY=h{23%sCo&oe3JgywX2@(N9fk%eJ8(w9TKqM}toYys2VB~5XK zf5si+!n&CTd=B7wQ zu4Lv*gnajH=%^U_tT2GTUU@Czri;=w`W3_7`$VuwtAL$D96JowVcvxtkTAGARjq09zqpw3l4Z_J=ef;HZX`u7OHRpsSz{P#;${p-xCp9^QZ zGLl!U@9L}TF`<6ngd}~b-OiG8?MB-@uhvRDH`$Jt`}!@FYi~QQmXI90ZQXa+pClsb zZN+|SU2CDF!399o^>f*y2#^ab=1W^ik5n7sbF4El&VdAhO}i*6m2ExtHL$pUEIrQq zI)l#qrSvC*5!mHOH(*XMcg0Ek>5Gx1jtx%_JrFy2y~l4M1BLDlJa%y!qa6tSl^g0= ze|B0aNr$h=e{;yKg&DWjC~4Mmz~kHU2#edX4(|L#HNpewDXs>6kf&x3E^wlp6AOBO zvrhGC8ibv^t*O2)5PLmN?xj*3a-Ym8R*8dl#VPLuY0&sp;2%Uef?a4AXV$g&GouzmoH0QvuGQ+P~IX;zTNcmfd3? zMyefptyVmK*a){^yirO;jkz_n@2*Hf4~8`Q*jNARCI?Hs`~l5Wb|avdhDq0V~ zD)3H7R5X5EGxV2qALe59IOtJ5)hmAO+tGt{o80Jg|C~l7gS&V14nT9|y zD2d_mv8sY2Y1|B{LFE?50; zsf4>Hc3ekVzRT<+&b3EnO4%Ao%`KO(^8l}e+b9}ELo!zfJV9hFZ1t~+&qMvGaW?%Q zQM|%*cZ(L+#d|7>Ft_a8Bcsr4;b69>2JlnowlPx%Smvie;ABo+st$am3X7iPd&{ zu{3wm1D~-*qX_&JO=ssik^?L?uC;O*QW<5vRbmJboRci+v0dC?r7F**o5VTwRQQR^ zocffAj3@=nP+0Q`DLc;*Wb923EM(m(VE)r=7PY9CHsJju!5~QIua-utn|*KUf8sMG z!3EnXHB^^&Yh>Wcg`Ix1#gUD^RQpxSsAG~tIxwpi>fO3Npb@I)^>7o9z)i6sFA3$NDvv44QctaUwpu9%zp98dKZ2EtuA+){>(t^YhnhK~Cc0B5eve4_eO~qb zUPC_4E5uU73K@?|kp#h_M%#;%J4Ge)_>|8|5cNk9RA(Ao-P^z1paEj^Y9}NV@bBzPxce^4_W--BwS?0vU(cy>;yn9*PZwN49g*B zhDgFj^;S4a57_>no++?RL{$Qst^tg=n2sOEc)wD3*pfa1g>C)ij$ZgCu66G{2jn5j z5pY_8&)0;&Z`K0Qe;WSfgC0tSGp>Ylev^%U)$%Fpq%JCGMEn}_o4923D~ME6l-`v^ zi@i>U;BnqV;T`hDfx4*09)Mr&cc1%bR){996abAAm1A?QqUEOA`MA?qR$$Egtx95k z<9;!q6SoVDQXVE&;d}+J+e`os>h;s0#{TU3eTqUJsq5;;e~)3VEN(o&W&W#<{_hd+ zzjwV*b3JUV!bP!%6F8!86hbYEn<{N223hMwq#Tqc`rLnJu7VfFxv!F9p3v`(tut{C zn|5Y)yn3GI?8Rz3M)0;aPOjyz!4LkP(9}4v{@&5;bc9vEsc$4wVYPf^&X*Q|*W3MQ zoMcEZf&Czz*!(zexI{W7Bi6n;T0q!_uSJrD|0>&Ne<4Jxh@9La3+mXKKwQ2#&3bIz zSB3iJ8v{1tY_Sq8kblF%ge*7es{sPjYv4zI7fPGm1goJl2rp7eu1>U$V6ZPwZeI!f zX&ToZ#N~4FrerF#^ZO+*Z}CTzE1a@(f12`Xg8JiF(KZS=6SIe55M;BC?a$+VOgfUl zRQKf(rAn`#Az&8FQrqRbOXQ^r8QxO{;)eQ?7LgX-&M^hz+Ci&Uw6#!;iEreaqA`n_ zPh7ID!&H(tc}dIpB=7IluzLdFCzlE#newUc)y#}8&9oeAfq!FPULXZOOY)$Ewp6SK z@#+ih#qKtg)h@vE7{8Wh=Uv@N#7c4NS*t*pD#$X(`p&MaLwL=>#%JSSD^@j+lup73 z&*6S~v)5OZ-09y%qn+vV%nn3l!=rtiMlgdik04Lh=-(k7)`I$ODZL-rN^jnBqlYQ1KZD9roa5RMH4 zos;qOMqf{#)oao+de3^mOXGOpE=v*X%*TJ%usO3IF%PD7_mdlwgh^J43w)bLVz4VM zIBxfb^^;Jw4@ZqgQ$*d@joC3W^+~$8&;c2RQgL5_+RoQ^a!{H-hsahiRZ+dVGA^Q` z1=I`CYY4Ay2fa5?e!mhZA9>NvXTaZ_{$XI+4E_{9p9*9PaIT1i zNcjflo8tG{W-SF)%2kDD2Ibf>Y#Q8j(4nIFx@{YfDL>gq%IocB>&d!Ts+d=X&94eQ z#ne^T+TTbrK4j}$u05QlI}O)35(edN)4#<;B7LbOCVjpNvF*p`WM0@eL4|3{7$!5S zIR#U(;uO?Ud>j^HvO5~3zJrWSs>Y|7%SEGyi)?12c7UAYkv`c-B^hx zz2etolc|$=+k<+7e?r#3l&XaMP zbc-r98tqR$kOoy^Dax)-BA}Zw36*!r|@Q^6x7NeZ~;zpk!3zmww#4qN$yOJW{c^yJx#CJL>@sgyvV2PuJH4 zO+~==~M5+@42SijoHMB)Oo=HpuT3U^W}&S&8^S1=LTZ8F;mr$9b3 zy)S#_HzE4x>R%}RLT;PtOKTrt&S=O)4{*00t2@UTKWmn5d$Y%M=#Zp1M-bp5>1a{z zwXRN^XfLxX_HJhTKd49NZoGdZXGrKjp~MRPI^<~DvBz}JmAjK;)jNN+A%96o6OcH; zf@Ef9)yaA3t5g#|!KAg|KfiwyeI@K$&^mF0jW}~tlN=hbd?vmtzmC{!OK|nk#OD_^ zVV_(|3)TNfeIuR)RuZ$COJ3PtHg}K$`7GW}{e}sCL{t>v-V_z|;7e~`f(2!^bKI_t zsN++T!ZWN4K${zb?z}rHEs;y|1fh5GV|vl@8|*o4=CWsbVbsc|cRCVkNy{3Euyn^> z0#Ji^VFM8SF%gC!iuRU5y7~)*@IE(oNbyOzEJc`gohY%nIQ5bDMNMV9blIIVel`oA zwwmY(^5$?EP*Lms>o+*+cwk}HaOaM~P(QqIH^kxxF`ncrC#sOJ*q%FQL95No;AWq7 zTIkzwYR=Ua?VMIyo@la<6gdRcEb>y<=&8fgDYQ4(rSXP4{`~A3f^CVosFb+%j`q#A zGU`})I6*vm$JYJiV78;edx+bWM6GnSBY~hn`4x%ejr+TtK&nb4=}kmD3b)o)%i5@G z6P)mV(;ou$STPaU=Ij}rv5i(S>J3#c&wHA~iy;^RLVk=KtvhWVK$?Ie;-|~=?He~m zzLkm18mc0;TC60zCrb%oQ{*iPg<^hM9KVNQ=>)w;{IZZdkia88V*+H%5)G^1jbzvf z_=@l)r0YVz#}Rg$Ai9u3^~amH=`~*^%29G7%1Dl*9@w3TsiEc^Fk2_`DG*cHc=R=v zDje@F$yi5kdUwQpM4{Ji3n%0b{Arp82GbGY{#mo*GaYV3 zG_JuVb;6OZt_G@!o`P;tj1LI33nI)Cd=V;Vr=N&*cRkLEoki`+^-do>o?h17f9QM{ zisrd(GJb`&SPXqFUZ$RJ!(dq}?@%6I)z6#G#EGQ802UIR?h@;$p8B_J9R1f1(3<83 ztJYc4meTK^toSLAJ+oYYGyy!W{{CJ?!>{3-D)^O-QdPG!fZs-&u2Ue>_*M$Up7tXm zpgQRs$Z)~lvLTt-$p|sK^mKJ?HC6}Wu4b|_vQT`QO0uwKfJUN;ijHWWO}p*ZY#%29 zRgLLb%vti~H#qoImTzZ~;YoPo*xL6{JCTQS;AhQ_KfmccxME%RQutCaN_eI-Cf_7|ll3tz$aXS_Crn+uly;L;1fi`s> zTI)?f-s!*P@LYJ3+buKLc-riyh-IXS368RFeF|Y7BJ_?(FLaUN-`ipmiMVbB1zE&~?sKL!J(p)Y| zG;W8k))E9L(ZaalF$Im0ooC5s>$HC4*FP7px#hRUKh9kdA=n6z0eNe3c->-gTXOJQ z9G!6J=F?3qk!*A{i-Pgged#P1)A=Ch#HQl(zyqPjc>?($Rye<2RZb&@h-kbGr=Xuo ztUDM%)-f~0YjLQ@cbKBW9=?8Q{bl;$c-D&Y$%YrGUB5XAlWZVmVKl(Kba@v+8!9Bc z6L2~k?AWdG<(PHdhi#O^lvov-x}Z)kpu!3;x~COVO#MXgE<-<(GhE46&edee@LM z_-#%j0;g?n+1koTVBt@NPtU*bn~1sW0wT0^EEt~|$hzY~8C>NYX#Mf+F&SZ4tI6nn zo;(kmJ2kI))0IJsyISeW>WjOTy`QdK`b4h@qJ|o@2VBK_pHN>TT9RroLEB3`&AII| zdqYpSWp~<0YNm5$OXPR~`17gL)D)dAcZ-!ii=_^&nD>HUX(y2^t`R_Xqu2d7@AYBS zQ>)D~4Z-*QZ;5a~=OKY9${z?~M3b&s&^qBd+!Yv7eR-#;e|CCtCNOu9^IPmwd>k4m zUgxch9WhboLG<QOu1E#IZhSQ0cOKnNI$UIqd7t_RGcB_7mCmPYIFJi?BuB8Na zL);f5pkVb7g4Z2~Z`#ot%Ce9n9+64Djm-!cU-*C>{<;`h9I<FG(Jhrr)hSPG;=H5uPil+abPMy9$3AFTH} zRslA#*xN#*hl>1S&K>PcW+NFxSv!ykM3PhXp0K&mSwetC0~DH=gnGn+4oe@-0aYsS zkbMP8QIBrAwz?`nzNFpY;YFlgC90!!K@98Liw^+dV^&oJmz5GryN^s7ZRqC<)D~^( zBrG&iOh2D@o>RcMf5gx62=)x3>$0iEiN@07{l3eT!nP@NKgc9=*HQ>E)M+3uOlAct zCqaYHf!oM#p(c_T!h!uN%AyqQ49<=9vksM$kB%-WlmbZwSHE>fA2kY<%ek_soxgOr zopfPqo(Ge}h1coaT@5HuxIYiLZs>TmzVI38t@IcoSN<(u$0G@%m@WKq;(ZI8Tt3n3 zi#Jyz-A2=@v*}wY*uHJLRTQj6BCyM?*1yfR0gM^h^sTgpO^cOhE<%xK@Sa!&5 zhG3?e@5|~Y*oV&BvNYgJTudLA1>uRdOOBG(pG2BY9s_70?gL6WCJ|GxUpV@mU~{^! zl3ah1hYceIZ`Vq7g-JKT*NgF_S6_nEMfc9WFGU=UnRLx2RB6)T$Fc+&)ge?`GEq7( z0u~d--xr=-x6%6G*9&o2`A^q^3TKZ=c4?)K2thYS9A3UliIhbe%a<4vZ|{Dm6fCkW zZI5Vj{sCfw%4*A$nxooOg-F@% z3-Y9LVeb}1PH#*n%GNp2IWzBEwm_u@S-2A-3#xSp!CR)r=+D;bM8`2T%XndA5ma#?<<6#U$z&H3@riZ zVqxpENU(IHC$@&~*Fxkz{l$T;Q=kL?uL`$nUMc zu|x+}J~?(4&_4c_X!}Pr_P3-5#?6fXa~<_Bn9v^=9*Rfp)BEm=dpwEh0AzbLMATTn zWrCsK1TT0ol7U@6HUK2h!F%0G`#-z@BCmoMY~h^{7eMZpA^@nTeDIw#<`lEgUgmy{ zSooajIxsPMGo7mlvGUn+>nXse{`ue!aLpB`r++5Rb%=-}zO-telX}kRI%xY0pp__F zmieI(cls#r$=sWZNy~=Xsy7$K^}gx5^cB^WNiTjid5ze*Kz*w9Y+v1jUFtgOIOI9h z&|?4NU!y|ve|agXwI~sBHvf7{e4l7u|}V;4qPylZSLN-Q;=6ud5WFC8A&Al?D$I|fPF}TwO#RE zJ;II4ul`0WYK%Lj!o0u7$0gl3aFG!>Zx1XZ)KuV6W3L6PIRx_3)mPnUp$eecFKz2p z0hDkHn!5GqyXmw@_NMoC&0F0%ih2!hv&*F@k=+a(n{s?mARnd%4e*JeZcIQZRr8(4 zrG2> zThGt!4Eu`NtdRX03$a+qTy>tRHK&3#DjRB;_z6|`fhqOgu{5tn{Ew4pThZ)NRc7VH zEAEzADN2O!$&^wQQnJo4Void2-Z0jE}1E%+)O7 z$WL{-Vqxd2xVem+_kxoI0+&W;yBj@-Dh~kj+SK<$OA#Q^y0L5!)*|@X{DnOJo|_Qj z?b@0Wea=_Iux|Re^K^4KF`8H&KvLBHHAUH&eJ_$;8WR(fsQlH29SetR0LleBdg!B$ z+cCnJp{C(ulC14z?NBu1ieRRED_(pivGSaX?`))D^28n-Ab%TjwkS^q)Nsu>zHIDm zd{v;KbdP(%Brlq;j6{U~Tp7=tv3eq0&fKk;`>@W)NG-Q~{u7n|LeU|oiGv}W6A`YXql?>}%#kUxxm6kO8cW`p5FTXuy~=^Kk@ zNw1EjM_!nY69tk8ZMWfCpopxjtcWSW{yP>}AG9PW%YE7|b1BIZQvV4%N;}2zs%0rZ zb0iCOK;s@Al!B~AtIz$~_`D`=*V(J${!`YZtHCg-`zPu|dK3mpysGRK2%2{?%{?^x zL!+y{{QVl>f;5QOi^-?7jA8q;?BXTt`Ie0>9thWn@z(VRh$M@OciM|#rhj_eUXg0h z-Mp04EtqWa^tJ^6Im_NQ96!rMrhUjj5|lt}MdM==F(+W2*g#Op=^&5QNaW$8YF>IE zgSj){#?=9($o~GDk2)UNq$<}hl^BxAW?4YiC6b~LUJpQ(&G2!Sq9ZJ`OLM5PqP-*8 zrL8g8Jzv&#@g-@Zao>&3eC6ww*M)+kEpP zMyZTl|FHq3>5=S?J)*I@v^ohj_Og4CVhVxslp2ig z6niWK7?v34#-J$p$&pA;nwHFzE^owiYalF6EI#R>@JIOAuDnPh)U36of}TYy7OCC3 zMDAL5Lhr5LvA{j4@|6L=+V+j3JdL^bek|=3pVv)+tVXwf<7Hc6-bMp=nn_CIOemqe z+e!20j;2X>!N;IIJz!ueJ2b1^6BZ}XF(xnKkcX)M#Bne>Z7@;OMcCo9fV+CX5_c=d zyo#L&=Dt91Xw)Y=QzXB*Lp-;cBH6qpc$%}k6`zN0l(UB)%M`1q`J~&V30ACFs|w;{ z9)0j(HxSl@zQ(+WqzzHix?2-R94xK@c4)1J_5I;y?r)1rGZ{O;xN+ISH{^bKUrH9$ zqqxk9CG+oHk-_mAA@?8ev#|vvPA?BOm3EhXx-9|R^ZsJvzyJR$YM|o~ASTw4XACH4 z3}h@CG8Y~KmTBx(?KZDM(xTD+SJR&HCtD-iEC5{=J&mpQ?z0z7MO$;*Gj@WiX@J=? z@$GzvoxRA)>u!8dxO-2i1U-hW_LbkSpW$09_k*R`7dhWQ??v{5DzvbN~nHJZ2vFp_cZS>AI;j;C0|G{-00*4;8vz*Hr+7XhzJjhh+;w)!BXP z4~kW7c{@Pg?)jRir0x^y-NP&cLW6+f96#WJ@R}w$J^nNHpMU@dYWe|LVPMt&NzOGu zkXj!Q9tIp0)4$~At^x|S|29wmHS*vC$%ovIF;si@V*2i#YEmD$H?$J6&*HBKN^=cG z9+EO-m^*3PX4@c!Ts@bUo%jBM1-BiNSVXjnO2TkBOw|bufE0Xj8Di+$*fAI9VZtz1 zGcr-8FAX%mn~wq5J@cY`Y|Tt|ObMBd_9>_XZUn*fUQWNWAa)mix)Z_O?bpYmsC+_q zPEMsT=`TFE=O+RnTQ>kep6z-CIIdE9b-WzyxUoa8@#s985=GFgX;oENCb*pA;7Hav z!D0ET!$xkIE}?@e@x$DSY!=M}Z??KRqPUhTU_3yug(R5X<7#t%pn!0Tl}Beukc`g~ zq0YXf-ph+U6e0^T+~kaS*5Fm%dH23w{X!DWCr;HRb$9l8QzJAFCgheFIvR3b8_M?1 zdb_@9IP}h)8I*_Lk0glD*3+Ee*6FV+0sJ=d51TGVYIZzbIm18reQ5&{c_75@V>Smd zqk{x*$(U=xwsO|Jc$93i5Axm@yoGUL$6kjd-g#s^AaTBUB4_NRMArw^8lZsWJyn0=M$dWW) z?yGe8v~i17+J})Tc15L2S#6pX)0xd59COD~41A$)sW%pl>=8wkLKAZJS|VBebwh@X zYu&$b+`#M@G>;Obmg3`D8TMragONPnv7WdMFSvkfb$vf zo?(^B>3=55-qTs^k>+IL_p>-8?O3|A5PaCf-KlQ9{2ayiD!QTRZR64q;v@2fa^8z^ zB?aoL_^Rd66<}D#%Tvyk-SO%YUfGg>oyKd`Kl;I$KsCu zt|y}m-e4{2c)joaVRNY}+Rx z;iE{>_SFk%Nd47uZET)vdS^1Yh zEYt{X4r`C}Y_^>`9u@p969^RQu(kV0g{Yd+ian~F;G3ak5U}v*AvZu~A@tNZw{nkn zff5$LV*BQ1dy~)#=a0NxSC3X9Xo{pc;@R322Nk3OpBoZiJY5xNaG%No2U;Z){;V6i zF0#0rI_@+1Kf&0%ga^$+okebn4m<9S;0YjNR@7*YZAh7it1c+w)UJSWILDH@uO3Dk zurU5yW7|XR{d2`O+3BL~=TFY@R4}wX4=2?fPw!m~@ZN|q%IQv$ne*Rl)#nj@{vNIS zmPK-HP?<$?R8H(V`o`kv+ZEZoDGOcz0{hR57cA`zvyelJ!5gbRjEhLO&Fl6SS4$n| z2j9X%+d$QqA((?Z;*h9khMC@o4#u#hzDFfFp+$Ebv?ix1^#}H@9sgak|}-QR?U4i$|huU=j9 zwSYO>_CM-kLkRo_1pClm5bVeb%aYiw7Fn)oMRQEe8PCWzdSv+P4TWnhx1M0WNUU_d zk1f2pBG8*il{c+TH^IW}09YabqwV7N8UF^_!_mOAl74WBgE(m-ojfGBxzImE+#icLo<*rT5U(7j06Y7{I@?UXn0 zfHSIu(wlR8bEJx521!fwE1KNx>Q2v1=~}9Aq7<~^fd6tnfV-ATuCj#W%*#2oWVn-@ z6s`g^hH~|RlN;Ui0e85kx=*!lU8f>8W| z^3{zXy27CUQZ^FDQG(^_WUx;?s3d%LVFOZs&D?{HtxcmF0$uQ7?-}px5jP2vjlvlK z=9+SO>C;V4BRf0r3f8nG!)TruFW1`|Rj!D9pUE%&-HH`Zgwh6PktkT3b3v?UjPj~x zUSIJ#di}Hc$K<_gz^g6RH)is(;@<-+er2vhYo-lQ)0LL+)?qw9*$JGpP48{F-{$5D zz1}+cdQb3bj=ihh^!K0v_BvA&^`)JJw@FU!YoWL9m^Hg^jt4yhKX6h{>`oS#u{t^9 zW#8Z&`FO`n3g0&hjBHz>fO>VG(l4Kd2?sb%9cY++fY^Wh7c;jY?Ug5*MomOqtlfX% zI(gDd01knF0a-C1+Vg*TDgT+Sx#{QyD!=;~Ks4<8gJ+kD{t2#p=q@|!TN{r3#Ks)J z2XJ7!EBwDGR{ZZ=0Jk4ULxc1I;){a^!jJJD_-`48J{kQd3ZO&204%aR82Mjs@=FQ^ zO9wT{9tgp&>CXcq()(NeS1cH3_O~Yg#ta`=X(rIrS!VqJ#PI(|jJk_$7B*1&$AwtT z)Nd>Qub4Fe6W)(m2mB6A@Bc7DEBGC`)^YFzTm>-m3=riX>5X{%<)D!V{oxk3=vb;r zI5j+B;N@0|=vYF*xd(+Y+KVPqtFT?jg^y4NQ!r1$~h?ss`wzC?5I!@oyQK ze?bj|M?$o0Fn7anp*EepehqGb6QE~dkttx{zILXDIJ58uOx=WHVTPSq!IW9qsj}l}{+B58Ve>%Gr)!3$y|*r71}>Cf`BERYxfqeUTPNg->g(v)fE6FFei~ z1?LrE7T1>&i?}f&N*LU)uKfVd)-g!Ex<_}UYQr6F?8_snvh(h1Uk}ZyFan$AubwYI zMmoBK^lF?4+MS)js^4RF-@mtNJCW8JlXM*W@m#Zd$=oYI54hbi7XMN(y{TF? z;!7-kx1Xjk0~B4`?WC*S5JXYv;coT=|^PTpMCH4qnnN4Kw`)3H5UcYC|sn< z4;Cj2fi_ABygYLnG#!|BU3yopNWDP7HNQs>RWwk4|ZEOUnfX_F;V>w%eAo8 zH66t9>Gn}EK=t#!-VpVr$O@9wxw!@~!}WJ}7h&4CkCZC`c#(GS`o=Jrbh3Nk&&({U z^wg^l*)ba#MB*l5<4T7P4vYgo>E@{TGeMdl>gbr~0B$P{+rNoOzYklVxoA$)n(b8o z{T)jXEseR2nQqGWy~YpT!a&`a>w!qI(UYxQ7Ke`pg>OJ8f01L-zpb?a%Rv8{aKyh$ z0crpWESWw(H~)JkbPFEk=v^@e7WDVO?vsZBGb)7Iag90iP(g5?HLNL2 zPfMVq=QU#E_X_?$(}fu_e*%FzK*d5#@8KPM*TwLIUhdCVih-ma1Fi%AfBruE?lBKG z`tUPrzhhN_3rPlDu?})|3UbqS32*~G4yY)roL5j*S5Q8G>ztOhs;B^zUNO5|e(`Sf F{{WFMgDn66 From cb3368bbdca32acda442cd35c04b8a1de80afe6a Mon Sep 17 00:00:00 2001 From: Tushar Sharma Date: Thu, 12 Jun 2025 23:39:29 -0700 Subject: [PATCH 12/42] nit: fix minor formatting --- deps/NNNN-container-strategy.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/deps/NNNN-container-strategy.md b/deps/NNNN-container-strategy.md index 45ea714..7a9ebdc 100644 --- a/deps/NNNN-container-strategy.md +++ b/deps/NNNN-container-strategy.md @@ -68,7 +68,7 @@ Pinning/Fixing dependencies will ensure a unified build environment reducing "it * Outline possible further improvements including external caching/multi-context docker builds to reduce build times. -# Minimize effort for providing multi-arch support across various backends for Dynamo by leveraging manylinux to build for multiple distributions +* Minimize effort for providing multi-arch support across various backends for Dynamo by leveraging manylinux to build for multiple distributions * Outline a container release strategy which enables Dynamo team to release containers as part of Dynamo releases From a9b9762ffee34ff74a5c4640d1a7c7af54d03bdd Mon Sep 17 00:00:00 2001 From: Tushar Sharma Date: Thu, 12 Jun 2025 23:41:43 -0700 Subject: [PATCH 13/42] nit: move build spreed improvement to non-goal --- deps/NNNN-container-strategy.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/deps/NNNN-container-strategy.md b/deps/NNNN-container-strategy.md index 7a9ebdc..5f5af04 100644 --- a/deps/NNNN-container-strategy.md +++ b/deps/NNNN-container-strategy.md @@ -66,8 +66,6 @@ This base image should operate as a single base container which can then be used Pinning/Fixing dependencies will ensure a unified build environment reducing "it works on my machine" problems or "this worked yesterday" -* Outline possible further improvements including external caching/multi-context docker builds to reduce build times. - * Minimize effort for providing multi-arch support across various backends for Dynamo by leveraging manylinux to build for multiple distributions * Outline a container release strategy which enables Dynamo team to release containers as part of Dynamo releases @@ -76,6 +74,7 @@ Pinning/Fixing dependencies will ensure a unified build environment reducing "it - Slim backend-specific runtime containers to use for performance testing. - Unified build environment +- Outline possible further improvements including external caching/multi-context docker builds to reduce build times. ## Requirements From 97b4a8f9afd02e0b611915634eb6b6ef687ae904 Mon Sep 17 00:00:00 2001 From: Tushar Sharma Date: Thu, 12 Jun 2025 23:58:33 -0700 Subject: [PATCH 14/42] nit: fix minor formatting --- deps/NNNN-container-strategy.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/deps/NNNN-container-strategy.md b/deps/NNNN-container-strategy.md index 5f5af04..6743869 100644 --- a/deps/NNNN-container-strategy.md +++ b/deps/NNNN-container-strategy.md @@ -190,10 +190,10 @@ The minimum requirements and processes required for releasing these containers: - Containers image must be built in CI and built in the release pipeline. These containers can then be staged in the release pipeline before being pushed to NGC. - Container images must go through CVE & secrets scanning in CI to ensure no vulnerabilties exist or secrets are exposed. - Container images must be pushed to NGC, a public container registry for customers to download NVIDIA containers. To publish to NGC, the container must meet the following requirements: -1. Container must not container any high/critical vulnerabilities -2. Container must not container any secrets -3. Container must be approved to be released as an open-source container - Given the requirements to push to NGC include several checks (such as CVE & secrets scanning), we could temporarily use Github container registry to push the release containers. + * No high/critical vulnerabilities present in the container + * No secrets or sensitive information exposed in the container + * Container must be approved for open-source release + Given the requirements to push to NGC include several checks (such as CVE & secrets scanning), we could temporarily use Github container registry to push the release containers. - To ensure container freshness, The base container should be updated use the latest CUDA runtime images when available. This will reduce the vulnerability surface area for the containers. Can be a separate effort but is required for getting the containers published to NGC. From 827a5f5c14cdcb9fceabc8e8c06ada9879e170d1 Mon Sep 17 00:00:00 2001 From: Tushar Sharma Date: Fri, 11 Jul 2025 14:49:23 -0700 Subject: [PATCH 15/42] update container strategy to add release info --- NNNN-complete-template.md | 187 -------------------------------- deps/NNNN-container-strategy.md | 95 +++++++++++++--- 2 files changed, 82 insertions(+), 200 deletions(-) delete mode 100644 NNNN-complete-template.md diff --git a/NNNN-complete-template.md b/NNNN-complete-template.md deleted file mode 100644 index 95468d1..0000000 --- a/NNNN-complete-template.md +++ /dev/null @@ -1,187 +0,0 @@ -# - -**Status**: Draft | Under Review | Approved | Replaced | Deferred | Rejected - -**Authors**: [Name/Team] - -**Category**: Architecture | Process | Guidelines - -**Replaces**: [Link of previous proposal if applicable] - -**Replaced By**: [Link of previous proposal if applicable] - -**Sponsor**: [Name of code owner or maintainer to shepard process] - -**Required Reviewers**: [Names of technical leads that are required for acceptance] - -**Review Date**: [Date for review] - -**Pull Request**: [Link to Pull Request of the Proposal itself] - -**Implementation PR / Tracking Issue**: [Link to Pull Request or Tracking Issue for Implementation] - -# Summary - -**\[Required\]** - -# Motivation - -**\[Required\]** - -Describe the problem that needs to be addressed with enough detail for -someone familiar with the project to understand. Generally one to two -short paragraphs. Additional details can be placed in the background -section as needed. Cover **what** the issue is and **why** it needs to -be addressed. Link to github issues if relevant. - -## Goals - -**\[Optional \- if not applicable omit\]** - -List out any additional goals in bullet points. Goals may be aspirational / difficult to measure but guide the proposal. - -* Goal - -* Goal - -* Goal - -### Non Goals - -**\[Optional \- if not applicable omit\]** - -List out any items which are out of scope / specifically not required in bullet points. Indicates the scope of the proposal and issue being resolved. - -## Requirements - -**\[Optional \- if not applicable omit\]** - -List out any additional requirements in numbered subheadings. - -**\<numbered subheadings\>** - -### REQ \<\#\> \<Title\> - -Describe the requirement in as much detail as necessary for others to understand it and how it applies to the DEP. Keep in mind that requirements should be measurable and will be used to determine if a DEP has been successfully implemented or not. - -Requirement names should be prefixed using a monotonically increasing number such as “REQ 1 \<Title\>” followed by “REQ 2 \<Title\>” and so on. Use title casing when naming requirements. Requirement names should be as descriptive as possible while remaining as terse as possible. - -Use all-caps, bolded terms like **MUST** and **SHOULD** when describing each requirement. See [RFC-2119](https://datatracker.ietf.org/doc/html/rfc2119) for additional information. - - -# Proposal - -**\[Required\]** - -Describe the high level design / proposal. Use sub sections as needed, but start with an overview and then dig into the details. Try to provide images and diagrams to facilitate understanding. - -# Implementation Details - -**\[Optional \- if not applicable omit\]** - -Add additional detailed items here including interface signatures, etc. Add anything that is relevant but seems more of a detail than central to the proposal. Use sub sections / bullet points as needed. Try to provide images and diagrams to facilitate understanding. If applicable link to PR. - -## Deferred to Implementation - -**\[Optional \- if not applicable omit\]** - -List out items that are under discussion but that will be resolved only during implementation / code review. - -# Implementation Phases - -**\[Optional \- if not applicable omit\]** - -List out phases of implementation (can be single phase). Give each phase a monotonically increasing number; example “Phase 0” followed by “Phase 1” and so on. Give phases titles if it makes sense. - -## Phase \<\#\> \<Optional Title\> - -**Release Target**: Date - -**Effort Estimate**: \<estimate of time and number of engineers to complete the phase\> - -**Work Item(s):** \<one or more links to github issues\> - -**Supported API / Behavior:** - -* \<name and concise description of the API / behavior\> - -**Not Supported:** - -* \<name and concise description of the API / behavior\> - -# Related Proposals - -**\[Optional \- if not applicable omit\]** - -* File - -* File - -* File - -* File - -* File - -# Alternate Solutions - -**\[Required, if not applicable write N/A\]** - -List out solutions that were considered but ultimately rejected. Consider free form \- but a possible format shown below. - -## Alt \<\#\> \<Title\> - -**Pros:** - -\<bulleted list or pros describing the positive aspects of this solution\> - -**Cons:** - -\<bulleted list or pros describing the negative aspects of this solution\> - -**Reason Rejected:** - -\<bulleted list or pros describing why this option was not used\> - -**Notes:** - -\<optional: additional comments about this solution\> - -# Background - -**\[Optional \- if not applicable omit\]** - -Add additional context and references as needed to help reviewers and authors understand the context of the problem and solution being proposed. - -## References - -**\[Optional \- if not applicable omit\]** - -Add additional references as needed to help reviewers and authors understand the context of the problem and solution being proposed. - -* \<hyper-linked title of an external reference resource\> - -## Terminology & Definitions - -**\[Optional \- if not applicable omit\]** - -List out additional terms / definitions (lexicon). Try to keep definitions as concise as possible and use links to external resources when additional information would be useful to the reader. - -Keep the list of terms sorted alphabetically to ease looking up definitions by readers. - -| \<Term\> | \<Definition\> | -| :---- | :---- | -| **\<Term\>** | \<Definition\> | - -## Acronyms & Abbreviations - -**\[Optional \- if not applicable omit\]** - -Provide a list of frequently used acronyms and abbreviations which are uncommon or unlikely to be known by the reader. Do not include acronyms or abbreviations which the reader is likely to be familiar with. - -Keep the list of acronyms and abbreviations sorted alphabetically to ease looking up definitions by readers. - -Do not include the full definition in the expanded meaning of an abbreviation or acronym. If the reader needs the definition, please include it in the [Terminology & Definitions](#terminology--definitions) section. - -**\<Acronym/Abbreviation\>:** \<Expanded Meaning\> - diff --git a/deps/NNNN-container-strategy.md b/deps/NNNN-container-strategy.md index 6743869..891dd95 100644 --- a/deps/NNNN-container-strategy.md +++ b/deps/NNNN-container-strategy.md @@ -23,9 +23,9 @@ # Summary This document outlines a container strategy for Dynamo to enhance the developer experience by -organizing Dockerfiles to maximize coverage and reuse along with defining a strategy for releasing pre-built Dynamo containers publicly. +re-organizing Dockerfile along with defining a strategy for releasing pre-built Dynamo containers publicly. -One of the goals for this document is to define a clear and maintainable structure for our Dockerfiles—specifically, to determine how many Dockerfiles we need and clarify the relationships between base, runtime, development, and CI images. The aim is to ensure each environment's Dockerfile builds upon the previous (as supersets), maximizing environment consistency and coverage during daily development and testing. +One of the goals for this document is to define a clear and maintainable structure for our Dockerfiles. Specifically, to determine how many Dockerfiles we need and clarify the relationships between base, runtime, development, and CI images. The aim is to ensure each environment's Dockerfile builds upon the previous (as supersets), maximizing environment consistency and coverage during daily development and testing. To achieve this goal, this document proposes certain optimizations to improve the current build process: - Restructuring the build process to provide a base container with a pre-built version of Dynamo + NIXL available on all distributions, enabling splitting of specific backends from the dynamo build process. - Defining a structure/template for all Dockerfiles to follow to ensure consistent and reproducible builds across backends along with specific roles/use cases targeted for each stage. @@ -100,10 +100,11 @@ Each build stage must have a clearly defined purpose and scope: - CI: Testing tools and validation requirements built on runtime ### REQ \<\#4\> \<Container Release Process\> -The container release strategy must be defined to allow the Dynamo team to release containers as part of Dynamo releases. This should include: -- Which containers to release (backend, api-operator, api-store, etc) +The container release strategy must be defined to define a process on how containers should be released as part of Dynamo releases. This should include: +- Justification and approval process for releasing containers - The minimum requirements and processes required for releasing these containers -- The container registry to publish these containers to (NGC, Github, etc). +- The container registry to publish these containers to along with location of staged container images. +- The process for releasing the containers to the container registry. # Proposal @@ -178,22 +179,90 @@ The diagram above illustrates the proposed container strategy showing the relati This layered approach ensures consistent builds, reduces duplication, and improves maintainability across all backend implementations. +## Container Release Process + +```mermaid +flowchart TD + A[Request for New Container]:::grey + B{Org3 Approval?}:::grey + C[Rejected request to release container]:::red + D[Container Build in CI]:::grey + E[CVE & Secrets Scanning]:::grey + F[Passes Sanity Tests]:::grey + G[Stage Container in Gitlab Registry]:::grey + H{Scan Passed?}:::grey + I[Fix Vulnerabilities]:::grey + J{Can Fix CVEs?}:::grey + K[Exception Filed with Org3]:::grey + L{Org3 Exception Approved?}:::grey + N[OSRB Approval]:::grey + O{OSRB Approved?}:::grey + P[Exception Filed]:::red + Q[Push to NGC Registry]:::green + R[Public Release]:::green + + %% Main flow + A --> B + B -->|No| C + B -->|Yes| N + D --> E + E --> F + F --> G + E --> H + H -->|No| J + J -->|No| K + J -->|Yes| I + H --> G + H -->|Yes| J + J -->|No| K + K --> L + L -->|No| C + L -->|Yes| N + J -->|Yes| N + N --> O + O -->|No| C + O -->|Yes| Q + Q --> R + + %% Styling + classDef grey fill:#f3f4f6,stroke:#6b7280,stroke-width:2px; + classDef green fill:#dcfce7,stroke:#166534,stroke-width:2px; + classDef red fill:#fecaca,stroke:#dc2626,stroke-width:2px; +``` + +The diagram above illustrates the container release process showing: +- Initial approval process through Org3 +- CI/CD pipeline integration +- Security scanning requirements +- OSRB approval process +- Final release to NGC registry + +This process ensures quality, security, and proper governance for all released containers. + ## Container Release Strategy Which containers to release: -- Slim backend runtime containers (vLLM, sglang, TRT-LLM, etc) on all supported platforms (x86_64, arm64). As long as the backend is supported on the platform, the container should be released for that platform. -- Dynamo cloud API-Operator container -- Dynamo cloud API-Store container -- Dynamo cloud helm charts +Containers need proper justification and Org3 approval from the team to be released. Approval is required for all containers to be released. The following information is required: +- Development/Operations PIC for the container +- Functionality of the container +- Justification for the container release +- Expected container size +- Name of the container +- The container registry to publish these containers to (NGC, Github, etc) along with location of staged container images. +- When will the container be released? +- Why are we releasing this container? +- OSRB request? + +This information can be provided in the Request for new asset form. If the container is not approved, the container will not be released. + The minimum requirements and processes required for releasing these containers: -- Containers image must be built in CI and built in the release pipeline. These containers can then be staged in the release pipeline before being pushed to NGC. -- Container images must go through CVE & secrets scanning in CI to ensure no vulnerabilties exist or secrets are exposed. +- Containers image must be built in CI and built in the release pipeline. These containers can then be staged in the Gitlab release pipeline before being pushed to NGC. Internal users can pull the containers from Gitlab container registry before these images are pushed to NGC. +- Container images must go through CVE & secrets scanning in CI to ensure no vulnerabilties exist or secrets are exposed. If the container contains CVE's that are either past the release cutoff date or are critical but cannot be fixed, an exception must be filed with Org 3 approval to release the container. - Container images must be pushed to NGC, a public container registry for customers to download NVIDIA containers. To publish to NGC, the container must meet the following requirements: * No high/critical vulnerabilities present in the container * No secrets or sensitive information exposed in the container - * Container must be approved for open-source release - Given the requirements to push to NGC include several checks (such as CVE & secrets scanning), we could temporarily use Github container registry to push the release containers. + * Container must be approved for open-source release (OSRB approval) - To ensure container freshness, The base container should be updated use the latest CUDA runtime images when available. This will reduce the vulnerability surface area for the containers. Can be a separate effort but is required for getting the containers published to NGC. From dbff0ea2fecc7525a3029e743611936d1e8222b5 Mon Sep 17 00:00:00 2001 From: Tushar Sharma <tusharma@nvidia.com> Date: Fri, 11 Jul 2025 15:22:26 -0700 Subject: [PATCH 16/42] nit: fix mermaid diagram --- deps/NNNN-container-strategy.md | 32 +++++++++++++------------------- 1 file changed, 13 insertions(+), 19 deletions(-) diff --git a/deps/NNNN-container-strategy.md b/deps/NNNN-container-strategy.md index 891dd95..e520f85 100644 --- a/deps/NNNN-container-strategy.md +++ b/deps/NNNN-container-strategy.md @@ -191,39 +191,33 @@ flowchart TD F[Passes Sanity Tests]:::grey G[Stage Container in Gitlab Registry]:::grey H{Scan Passed?}:::grey - I[Fix Vulnerabilities]:::grey - J{Can Fix CVEs?}:::grey + I{Can Fix CVEs?}:::grey + J[Fix Vulnerabilities]:::grey K[Exception Filed with Org3]:::grey L{Org3 Exception Approved?}:::grey N[OSRB Approval]:::grey O{OSRB Approved?}:::grey - P[Exception Filed]:::red - Q[Push to NGC Registry]:::green - R[Public Release]:::green + Q[Push to NGC Staging Registry]:::grey + R[Push to NGC Registry]:::green + S[Public Release]:::green %% Main flow A --> B B -->|No| C - B -->|Yes| N + B -->|Yes| D D --> E E --> F - F --> G E --> H - H -->|No| J - J -->|No| K - J -->|Yes| I - H --> G - H -->|Yes| J - J -->|No| K + H -->|No| I + I -->|No| K + I -->|Yes| J + J -->|No| P + J -->|Yes| Q K --> L - L -->|No| C - L -->|Yes| N - J -->|Yes| N - N --> O - O -->|No| C - O -->|Yes| Q + L --> Q Q --> R + %% Styling classDef grey fill:#f3f4f6,stroke:#6b7280,stroke-width:2px; classDef green fill:#dcfce7,stroke:#166534,stroke-width:2px; From e89d166a5151aafdabb45179362e9985744c667c Mon Sep 17 00:00:00 2001 From: Tushar Sharma <tusharma@nvidia.com> Date: Fri, 11 Jul 2025 15:27:37 -0700 Subject: [PATCH 17/42] nit: fix mermaid diagram --- deps/NNNN-container-strategy.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/deps/NNNN-container-strategy.md b/deps/NNNN-container-strategy.md index e520f85..83a0c2f 100644 --- a/deps/NNNN-container-strategy.md +++ b/deps/NNNN-container-strategy.md @@ -205,14 +205,20 @@ flowchart TD A --> B B -->|No| C B -->|Yes| D + B -->|Yes| N D --> E E --> F + F --> G E --> H H -->|No| I I -->|No| K I -->|Yes| J - J -->|No| P + J --> G J -->|Yes| Q + N --> O + O -->|No| C + O -->|Yes| Q + Q --> R K --> L L --> Q Q --> R From e7a3e59b1ea0618f7a71a4a461f64a05fbee25f7 Mon Sep 17 00:00:00 2001 From: Tushar Sharma <tusharma@nvidia.com> Date: Fri, 11 Jul 2025 15:37:07 -0700 Subject: [PATCH 18/42] nit: fix mermaid diagram --- deps/NNNN-container-strategy.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/deps/NNNN-container-strategy.md b/deps/NNNN-container-strategy.md index 83a0c2f..76b4a79 100644 --- a/deps/NNNN-container-strategy.md +++ b/deps/NNNN-container-strategy.md @@ -218,11 +218,11 @@ flowchart TD N --> O O -->|No| C O -->|Yes| Q - Q --> R K --> L L --> Q + G --> Q Q --> R - + R --> S %% Styling classDef grey fill:#f3f4f6,stroke:#6b7280,stroke-width:2px; From 342f36169cad62131c689e89adc3904772921e96 Mon Sep 17 00:00:00 2001 From: Tushar Sharma <tusharma@nvidia.com> Date: Fri, 11 Jul 2025 15:38:26 -0700 Subject: [PATCH 19/42] nit: fix mermaid diagram --- deps/NNNN-container-strategy.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/deps/NNNN-container-strategy.md b/deps/NNNN-container-strategy.md index 76b4a79..9b10b7a 100644 --- a/deps/NNNN-container-strategy.md +++ b/deps/NNNN-container-strategy.md @@ -219,7 +219,7 @@ flowchart TD O -->|No| C O -->|Yes| Q K --> L - L --> Q + L --> G G --> Q Q --> R R --> S From 0b13afc529f3d1265612719285d86ac0185b16f8 Mon Sep 17 00:00:00 2001 From: Tushar Sharma <tusharma@nvidia.com> Date: Fri, 11 Jul 2025 15:43:22 -0700 Subject: [PATCH 20/42] nit: reorganize mermaid diagram --- deps/NNNN-container-strategy.md | 72 ++++++++++++++++----------------- 1 file changed, 35 insertions(+), 37 deletions(-) diff --git a/deps/NNNN-container-strategy.md b/deps/NNNN-container-strategy.md index 9b10b7a..bd2df63 100644 --- a/deps/NNNN-container-strategy.md +++ b/deps/NNNN-container-strategy.md @@ -179,29 +179,63 @@ The diagram above illustrates the proposed container strategy showing the relati This layered approach ensures consistent builds, reduces duplication, and improves maintainability across all backend implementations. + +## Container Release Strategy + +Which containers to release: +Containers need proper justification and Org3 approval from the team to be released. Approval is required for all containers to be released. The following information is required: +- Development/Operations PIC for the container +- Functionality of the container +- Justification for the container release +- Expected container size +- Name of the container +- The container registry to publish these containers to (NGC, Github, etc) along with location of staged container images. +- When will the container be released? +- Why are we releasing this container? +- OSRB request? + +This information can be provided in the Request for new asset form. If the container is not approved, the container will not be released. + ## Container Release Process +The diagram below illustrates the container release process showing: +- Initial approval process through Org3 +- CI/CD pipeline integration +- Security scanning requirements +- OSRB approval process +- Final release to NGC registry + +This process ensures quality, security, and proper governance for all released containers. ```mermaid flowchart TD + %% Column 1: Initial Request & Approval A[Request for New Container]:::grey B{Org3 Approval?}:::grey C[Rejected request to release container]:::red + + %% Column 2: Build Process D[Container Build in CI]:::grey E[CVE & Secrets Scanning]:::grey F[Passes Sanity Tests]:::grey G[Stage Container in Gitlab Registry]:::grey + + %% Column 3: Security & Exception Handling H{Scan Passed?}:::grey I{Can Fix CVEs?}:::grey J[Fix Vulnerabilities]:::grey K[Exception Filed with Org3]:::grey L{Org3 Exception Approved?}:::grey + + %% Column 4: OSRB Approval N[OSRB Approval]:::grey O{OSRB Approved?}:::grey + + %% Column 5: Release Process Q[Push to NGC Staging Registry]:::grey R[Push to NGC Registry]:::green S[Public Release]:::green - %% Main flow + %% Main flow - keeping all connections exactly the same A --> B B -->|No| C B -->|Yes| D @@ -230,42 +264,6 @@ flowchart TD classDef red fill:#fecaca,stroke:#dc2626,stroke-width:2px; ``` -The diagram above illustrates the container release process showing: -- Initial approval process through Org3 -- CI/CD pipeline integration -- Security scanning requirements -- OSRB approval process -- Final release to NGC registry - -This process ensures quality, security, and proper governance for all released containers. - -## Container Release Strategy - -Which containers to release: -Containers need proper justification and Org3 approval from the team to be released. Approval is required for all containers to be released. The following information is required: -- Development/Operations PIC for the container -- Functionality of the container -- Justification for the container release -- Expected container size -- Name of the container -- The container registry to publish these containers to (NGC, Github, etc) along with location of staged container images. -- When will the container be released? -- Why are we releasing this container? -- OSRB request? - -This information can be provided in the Request for new asset form. If the container is not approved, the container will not be released. - - -The minimum requirements and processes required for releasing these containers: -- Containers image must be built in CI and built in the release pipeline. These containers can then be staged in the Gitlab release pipeline before being pushed to NGC. Internal users can pull the containers from Gitlab container registry before these images are pushed to NGC. -- Container images must go through CVE & secrets scanning in CI to ensure no vulnerabilties exist or secrets are exposed. If the container contains CVE's that are either past the release cutoff date or are critical but cannot be fixed, an exception must be filed with Org 3 approval to release the container. -- Container images must be pushed to NGC, a public container registry for customers to download NVIDIA containers. To publish to NGC, the container must meet the following requirements: - * No high/critical vulnerabilities present in the container - * No secrets or sensitive information exposed in the container - * Container must be approved for open-source release (OSRB approval) -- To ensure container freshness, The base container should be updated use the latest CUDA runtime images when available. This will reduce the vulnerability surface area for the containers. Can be a separate effort but is required for getting the containers published to NGC. - - ## Deferred to Implementation TBD From b3b2f11293bfaa0b004e49938a3f5ef95db3c306 Mon Sep 17 00:00:00 2001 From: Tushar Sharma <tusharma@nvidia.com> Date: Fri, 11 Jul 2025 15:45:21 -0700 Subject: [PATCH 21/42] Add subgraphs for mermaid diagram --- deps/NNNN-container-strategy.md | 57 ++++++++++++++++----------------- 1 file changed, 27 insertions(+), 30 deletions(-) diff --git a/deps/NNNN-container-strategy.md b/deps/NNNN-container-strategy.md index bd2df63..e0fc9b8 100644 --- a/deps/NNNN-container-strategy.md +++ b/deps/NNNN-container-strategy.md @@ -198,42 +198,39 @@ This information can be provided in the Request for new asset form. If the conta ## Container Release Process -The diagram below illustrates the container release process showing: -- Initial approval process through Org3 -- CI/CD pipeline integration -- Security scanning requirements -- OSRB approval process -- Final release to NGC registry - -This process ensures quality, security, and proper governance for all released containers. ```mermaid flowchart TD - %% Column 1: Initial Request & Approval - A[Request for New Container]:::grey - B{Org3 Approval?}:::grey - C[Rejected request to release container]:::red + subgraph "Initial Request & Approval" + A[Request for New Container]:::grey + B{Org3 Approval?}:::grey + C[Rejected request to release container]:::red + end - %% Column 2: Build Process - D[Container Build in CI]:::grey - E[CVE & Secrets Scanning]:::grey - F[Passes Sanity Tests]:::grey - G[Stage Container in Gitlab Registry]:::grey + subgraph "Build Process" + D[Container Build in CI]:::grey + E[CVE & Secrets Scanning]:::grey + F[Passes Sanity Tests]:::grey + G[Stage Container in Gitlab Registry]:::grey + end - %% Column 3: Security & Exception Handling - H{Scan Passed?}:::grey - I{Can Fix CVEs?}:::grey - J[Fix Vulnerabilities]:::grey - K[Exception Filed with Org3]:::grey - L{Org3 Exception Approved?}:::grey + subgraph "Security & Exception Handling" + H{Scan Passed?}:::grey + I{Can Fix CVEs?}:::grey + J[Fix Vulnerabilities]:::grey + K[Exception Filed with Org3]:::grey + L{Org3 Exception Approved?}:::grey + end - %% Column 4: OSRB Approval - N[OSRB Approval]:::grey - O{OSRB Approved?}:::grey + subgraph "OSRB Approval" + N[OSRB Approval]:::grey + O{OSRB Approved?}:::grey + end - %% Column 5: Release Process - Q[Push to NGC Staging Registry]:::grey - R[Push to NGC Registry]:::green - S[Public Release]:::green + subgraph "Release Process" + Q[Push to NGC Staging Registry]:::grey + R[Push to NGC Registry]:::green + S[Public Release]:::green + end %% Main flow - keeping all connections exactly the same A --> B From 3c7af80e013e7fbe8ae29432cb2748d9a22e4263 Mon Sep 17 00:00:00 2001 From: Tushar Sharma <tusharma@nvidia.com> Date: Fri, 11 Jul 2025 15:53:00 -0700 Subject: [PATCH 22/42] nit: fix diagram formatting --- deps/NNNN-container-strategy.md | 19 ++++++++----------- 1 file changed, 8 insertions(+), 11 deletions(-) diff --git a/deps/NNNN-container-strategy.md b/deps/NNNN-container-strategy.md index e0fc9b8..0f1cf7c 100644 --- a/deps/NNNN-container-strategy.md +++ b/deps/NNNN-container-strategy.md @@ -199,31 +199,28 @@ This information can be provided in the Request for new asset form. If the conta ## Container Release Process ```mermaid -flowchart TD +flowchart LR subgraph "Initial Request & Approval" A[Request for New Container]:::grey - B{Org3 Approval?}:::grey + B{Approval?}:::grey C[Rejected request to release container]:::red end - subgraph "Build Process" + subgraph "Build & Security Process" D[Container Build in CI]:::grey E[CVE & Secrets Scanning]:::grey F[Passes Sanity Tests]:::grey G[Stage Container in Gitlab Registry]:::grey - end - - subgraph "Security & Exception Handling" H{Scan Passed?}:::grey I{Can Fix CVEs?}:::grey J[Fix Vulnerabilities]:::grey - K[Exception Filed with Org3]:::grey - L{Org3 Exception Approved?}:::grey + K[Exception Filed]:::grey + L{Exception Approved?}:::grey end - subgraph "OSRB Approval" - N[OSRB Approval]:::grey - O{OSRB Approved?}:::grey + subgraph "Open Source Approval" + N[Open Source Approval]:::grey + O{Open Source Approved?}:::grey end subgraph "Release Process" From b70361fdb98c35b40b40836eab13d2c48b065442 Mon Sep 17 00:00:00 2001 From: Tushar Sharma <tusharma@nvidia.com> Date: Fri, 11 Jul 2025 15:56:15 -0700 Subject: [PATCH 23/42] nit: update mermaid subgraph --- deps/NNNN-container-strategy.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/deps/NNNN-container-strategy.md b/deps/NNNN-container-strategy.md index 0f1cf7c..7d7ec68 100644 --- a/deps/NNNN-container-strategy.md +++ b/deps/NNNN-container-strategy.md @@ -207,7 +207,7 @@ flowchart LR end subgraph "Build & Security Process" - D[Container Build in CI]:::grey + D[Container Build in Gitlab CI]:::grey E[CVE & Secrets Scanning]:::grey F[Passes Sanity Tests]:::grey G[Stage Container in Gitlab Registry]:::grey @@ -242,7 +242,6 @@ flowchart LR I -->|No| K I -->|Yes| J J --> G - J -->|Yes| Q N --> O O -->|No| C O -->|Yes| Q From 30e5d854bb9452bb758a7e98ad5270ab922c1659 Mon Sep 17 00:00:00 2001 From: Tushar Sharma <tusharma@nvidia.com> Date: Fri, 11 Jul 2025 16:00:20 -0700 Subject: [PATCH 24/42] Minor fixes + syntax fixing --- deps/NNNN-container-strategy.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/deps/NNNN-container-strategy.md b/deps/NNNN-container-strategy.md index 7d7ec68..d75e9d1 100644 --- a/deps/NNNN-container-strategy.md +++ b/deps/NNNN-container-strategy.md @@ -30,7 +30,7 @@ To achieve this goal, this document proposes certain optimizations to improve th - Restructuring the build process to provide a base container with a pre-built version of Dynamo + NIXL available on all distributions, enabling splitting of specific backends from the dynamo build process. - Defining a structure/template for all Dockerfiles to follow to ensure consistent and reproducible builds across backends along with specific roles/use cases targeted for each stage. -This DEP also outlines Dynamo's container release strategy. Since pre-built containers represent an important distribution method, we need to establish clear guidelines for: (1) which containers to release, (2) the requirements and processes for releasing these containers, and (3) the container registry to publish these containers to. +This DEP also outlines Dynamo's container release process. Since pre-built containers represent an important distribution method, we need to establish clear guidelines for: (1) which containers to release, (2) the requirements and processes for releasing these containers, and (3) the container registry to publish these containers to. # Motivation @@ -100,7 +100,7 @@ Each build stage must have a clearly defined purpose and scope: - CI: Testing tools and validation requirements built on runtime ### REQ \<\#4\> \<Container Release Process\> -The container release strategy must be defined to define a process on how containers should be released as part of Dynamo releases. This should include: +Define a process on how containers should be released as part of Dynamo releases. This should include: - Justification and approval process for releasing containers - The minimum requirements and processes required for releasing these containers - The container registry to publish these containers to along with location of staged container images. @@ -198,6 +198,8 @@ This information can be provided in the Request for new asset form. If the conta ## Container Release Process +The container release process follows a structured workflow that ensures quality, security, and proper governance. The process begins with a request for a new container that requires approval before proceeding. Once approved, the container undergoes a build and security process in Gitlab CI, including CVE scanning and vulnerability fixes. If vulnerabilities cannot be fixed, an exception must be filed with the required approval before proceeding with the release process. The container must also receive open source approval before being released to NGC registry for public distribution. + ```mermaid flowchart LR subgraph "Initial Request & Approval" @@ -223,7 +225,7 @@ flowchart LR O{Open Source Approved?}:::grey end - subgraph "Release Process" + subgraph "Publish Containers to NGC" Q[Push to NGC Staging Registry]:::grey R[Push to NGC Registry]:::green S[Public Release]:::green From 6d2bceedade9070d2fff412891603bb5b62c27f1 Mon Sep 17 00:00:00 2001 From: Tushar Sharma <tusharma@nvidia.com> Date: Fri, 11 Jul 2025 16:04:11 -0700 Subject: [PATCH 25/42] nit: minor syntax fix --- deps/NNNN-container-strategy.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/deps/NNNN-container-strategy.md b/deps/NNNN-container-strategy.md index d75e9d1..27cd3c2 100644 --- a/deps/NNNN-container-strategy.md +++ b/deps/NNNN-container-strategy.md @@ -183,7 +183,7 @@ This layered approach ensures consistent builds, reduces duplication, and improv ## Container Release Strategy Which containers to release: -Containers need proper justification and Org3 approval from the team to be released. Approval is required for all containers to be released. The following information is required: +Containers need proper justification and approval from the team to be released. Approval is required for all containers to be released. The following information is required: - Development/Operations PIC for the container - Functionality of the container - Justification for the container release From 1fbb42ad8c11060a84f6fe37ce6d1a7aa98eeed6 Mon Sep 17 00:00:00 2001 From: Tushar Sharma <tusharma@nvidia.com> Date: Tue, 15 Jul 2025 10:17:30 -0700 Subject: [PATCH 26/42] Split out container build process optimization and container release process into seperate DEPs --- deps/NNNN-container-strategy.md | 192 ++++++++++---------------------- 1 file changed, 59 insertions(+), 133 deletions(-) diff --git a/deps/NNNN-container-strategy.md b/deps/NNNN-container-strategy.md index 27cd3c2..a3deeb5 100644 --- a/deps/NNNN-container-strategy.md +++ b/deps/NNNN-container-strategy.md @@ -1,4 +1,4 @@ -# Container Strategy +# Container Build Process Optimization **Status**: Draft @@ -22,19 +22,18 @@ # Summary -This document outlines a container strategy for Dynamo to enhance the developer experience by -re-organizing Dockerfile along with defining a strategy for releasing pre-built Dynamo containers publicly. +This document outlines a container build process optimization strategy for Dynamo to enhance the developer experience by re-organizing Dockerfiles along with defining a clear and maintainable structure for our Dockerfiles. One of the goals for this document is to define a clear and maintainable structure for our Dockerfiles. Specifically, to determine how many Dockerfiles we need and clarify the relationships between base, runtime, development, and CI images. The aim is to ensure each environment's Dockerfile builds upon the previous (as supersets), maximizing environment consistency and coverage during daily development and testing. + To achieve this goal, this document proposes certain optimizations to improve the current build process: - Restructuring the build process to provide a base container with a pre-built version of Dynamo + NIXL available on all distributions, enabling splitting of specific backends from the dynamo build process. - Defining a structure/template for all Dockerfiles to follow to ensure consistent and reproducible builds across backends along with specific roles/use cases targeted for each stage. -This DEP also outlines Dynamo's container release process. Since pre-built containers represent an important distribution method, we need to establish clear guidelines for: (1) which containers to release, (2) the requirements and processes for releasing these containers, and (3) the container registry to publish these containers to. - # Motivation Dynamo's current container architecture consists of multiple Dockerfiles in the `/containers` directory of the [Dynamo repository](https://github.com/NVIDIA/Dynamo). These Dockerfiles are organized by backend (vLLM, sglang, TRT-LLM) and contain multiple stages (base, devel, ci, runtime) for different use cases. Each stage includes a Dynamo build, the specific backend, and NIXL - our high-throughput, low-latency point-to-point communication library for accelerating inference. + The current approach faces several challenges: 1. **Inefficient Build Process**: Components like Dynamo, NIXL, and backends are rebuilt across stages instead of using a layered approach. For example, in Dockerfile.vllm, Dynamo is installed three times - in base, ci_minimum, and runtime stages. @@ -45,14 +44,7 @@ The current approach faces several challenges: 4. **Inconsistent Standards**: Without a unified Dockerfile for building Dynamo, NIXL, and dependencies, code is duplicated or missing across backend-specific Dockerfiles, and optimizations aren't shared effectively. -Additionally, as Dynamo expands its support for multiple LLM backends, we need a clear container release strategy to: -- Provide pre-built backend containers along with Dynamo deploy containers (api-operator, api-store) to a public container registry -- Ensure consistent quality and security across released containers -- Simplify the deployment process for end users -- Maintain version control and compatibility across Dynamo releases - -This document proposes solutions to both the build process challenges and establishes a framework for container releases, aiming to improve overall container usability and deployment experience. - +This document proposes solutions to the build process challenges, aiming to improve overall container build efficiency and developer experience. ## Goals @@ -68,15 +60,12 @@ Pinning/Fixing dependencies will ensure a unified build environment reducing "it * Minimize effort for providing multi-arch support across various backends for Dynamo by leveraging manylinux to build for multiple distributions -* Outline a container release strategy which enables Dynamo team to release containers as part of Dynamo releases - ### Non Goals -- Slim backend-specific runtime containers to use for performance testing. +- Container release strategy and processes (covered in separate DEP) - Unified build environment - Outline possible further improvements including external caching/multi-context docker builds to reduce build times. - ## Requirements ### REQ \<\#1\> \<Backend Integration with Base Container\> @@ -99,13 +88,6 @@ Each build stage must have a clearly defined purpose and scope: - Runtime: Minimal production deployment requirements - CI: Testing tools and validation requirements built on runtime -### REQ \<\#4\> \<Container Release Process\> -Define a process on how containers should be released as part of Dynamo releases. This should include: -- Justification and approval process for releasing containers -- The minimum requirements and processes required for releasing these containers -- The container registry to publish these containers to along with location of staged container images. -- The process for releasing the containers to the container registry. - # Proposal In order to address the requirements, we propose the following changes to the Dynamo build process: @@ -124,7 +106,7 @@ Each backend-specific Dockerfile should follow a specific format. The backend-sp | Runtime | Customers/Production| Cuda base runtime image| Minimal image with only the dependencies required to deploy and run Dynamo w/backend from the backend build stage; intended for production deployments. Copies dynamo artifacts from base image and backend artifaces from backend build image. | | CI | Developers/Internal CI Pipelines/Local Debugging | Runtime image | Adds CI-specific tools, QA test scripts, internal models, and other dependencies needed for automated testing. | -Although the Dynamo base container is required to have multi-arch support, the backend builds aren't required to have immediate multi-arch support since backends often require different build logic across multiple architectures. +The CUDA base images will be used from the [NVIDIA CUDA Container Registry](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/cuda). Please refer to the Pros and Cons section for more details on why we chose to use the cuda runtime image instead of the Deep Learning CUDA image. # Implementation Details @@ -171,7 +153,7 @@ flowchart TD classDef gray fill:#e5e7eb,stroke:#333,stroke-width:1px; ``` -The diagram above illustrates the proposed container strategy showing the relationships between: +The diagram above illustrates the proposed container build strategy showing the relationships between: - Build Base Container with common dependencies - Backend-specific development containers - Runtime containers @@ -179,85 +161,32 @@ The diagram above illustrates the proposed container strategy showing the relati This layered approach ensures consistent builds, reduces duplication, and improves maintainability across all backend implementations. +## Dockerfile Structure Template -## Container Release Strategy +Each backend-specific Dockerfile will follow this standardized structure: -Which containers to release: -Containers need proper justification and approval from the team to be released. Approval is required for all containers to be released. The following information is required: -- Development/Operations PIC for the container -- Functionality of the container -- Justification for the container release -- Expected container size -- Name of the container -- The container registry to publish these containers to (NGC, Github, etc) along with location of staged container images. -- When will the container be released? -- Why are we releasing this container? -- OSRB request? +```dockerfile +# Backend Build Stage +FROM nvcr.io/nvidia/cuda:XX.YY-devel-ubuntuXX.XX as backend-build +# Install backend-specific dependencies and build backend -This information can be provided in the Request for new asset form. If the container is not approved, the container will not be released. +# Runtime Stage +FROM nvcr.io/nvidia/cuda:XX.YY-runtime-ubuntuXX.XX as runtime +# Copy dynamo and NIXL wheels from base container +# Copy backend artifacts from backend-build stage +# Install runtime dependencies only -## Container Release Process - -The container release process follows a structured workflow that ensures quality, security, and proper governance. The process begins with a request for a new container that requires approval before proceeding. Once approved, the container undergoes a build and security process in Gitlab CI, including CVE scanning and vulnerability fixes. If vulnerabilities cannot be fixed, an exception must be filed with the required approval before proceeding with the release process. The container must also receive open source approval before being released to NGC registry for public distribution. +# CI Stage +FROM runtime as ci +# Add CI-specific tools and test dependencies +``` -```mermaid -flowchart LR - subgraph "Initial Request & Approval" - A[Request for New Container]:::grey - B{Approval?}:::grey - C[Rejected request to release container]:::red - end - - subgraph "Build & Security Process" - D[Container Build in Gitlab CI]:::grey - E[CVE & Secrets Scanning]:::grey - F[Passes Sanity Tests]:::grey - G[Stage Container in Gitlab Registry]:::grey - H{Scan Passed?}:::grey - I{Can Fix CVEs?}:::grey - J[Fix Vulnerabilities]:::grey - K[Exception Filed]:::grey - L{Exception Approved?}:::grey - end - - subgraph "Open Source Approval" - N[Open Source Approval]:::grey - O{Open Source Approved?}:::grey - end - - subgraph "Publish Containers to NGC" - Q[Push to NGC Staging Registry]:::grey - R[Push to NGC Registry]:::green - S[Public Release]:::green - end - - %% Main flow - keeping all connections exactly the same - A --> B - B -->|No| C - B -->|Yes| D - B -->|Yes| N - D --> E - E --> F - F --> G - E --> H - H -->|No| I - I -->|No| K - I -->|Yes| J - J --> G - N --> O - O -->|No| C - O -->|Yes| Q - K --> L - L --> G - G --> Q - Q --> R - R --> S +## Dependency Management - %% Styling - classDef grey fill:#f3f4f6,stroke:#6b7280,stroke-width:2px; - classDef green fill:#dcfce7,stroke:#166534,stroke-width:2px; - classDef red fill:#fecaca,stroke:#dc2626,stroke-width:2px; -``` +- **Pinned Dependencies**: All dependencies will be pinned to specific versions in the base container +- **Multi-arch Support**: Base container will support both x86_64 and arm64 architectures +- **Minimal Runtime**: Runtime images will only include necessary dependencies for production deployment +- **Layered Caching**: Build layers will be optimized for Docker build cache efficiency ## Deferred to Implementation @@ -277,11 +206,15 @@ TBD **Supported API / Behavior:** -* \<name and concise description of the API / behavior\> +* Pre-built Dynamo base container with multi-arch support +* NIXL integration in base container +* Manylinux base for broad distribution compatibility +* Standardized environment variables and paths +* Pinned dependencies for base container **Not Supported:** -* \<name and concise description of the API / behavior\> +* Advanced caching mechanisms (e.g. multi-context docker builds) ## Phase \<\#2\> \<Restructure backend Dockerfiles to follow proposed structure\> @@ -293,49 +226,46 @@ TBD **Supported API / Behavior:** -* \<name and concise description of the API / behavior\> +* Updated vLLM, sglang, and TRT-LLM Dockerfiles following new structure +* Clear separation between backend build, runtime, and CI stages +* Integration with base container for Dynamo and NIXL dependencies +* Reduced build times through improved layering +* Backward compatibility with existing Dockerfiles +* Pinned dependencies for backend builds **Not Supported:** -* \<name and concise description of the API / behavior\> # Related Proposals **\[Optional \- if not applicable omit\]** -* File - -* File - -* File - -* File - -* File - # Alternate Solutions **\[Required, if not applicable write N/A\]** List out solutions that were considered but ultimately rejected. Consider free form \- but a possible format shown below. -## Alt \<\#\> \<Title\> +## Alt \<\# 1\> \<Using Deep Learning CUDA images as base image instead of cuda runtime\> **Pros:** -\<bulleted list or pros describing the positive aspects of this solution\> +- Dynamo would be in sync with the latest Deep Learning CUDA release +- Deep Learning CUDA image can provide additional libraries such as cuDNN, NCCL, etc which can accelerate deep learning workloads. +- TritonServer is built on top of the Deep Learning CUDA image, so would be a good reference for the use-case. **Cons:** -\<bulleted list or pros describing the negative aspects of this solution\> +- Container Size is too large for the use-case. The cuda runtime image we use is around 6 GB whereas the Deep Learning Cuda runtime is around 11 GB, uncompressed. +- Introduces additional dependencies that are not required for Dynamo, which can increase the container size and build time. +- Dynamo has NIXL as it's communication library, so a dependency on NCCL is not required. **Reason Rejected:** -\<bulleted list or pros describing why this option was not used\> +- Container Size is too large for the use-case. The cuda runtime image we use is around 6 GB whereas the Deep Learning cuda runtime is around 11 GB, uncompressed. +- Introduces additional dependencies that are not required for Dynamo, which can increase the container size and build time. -**Notes:** -\<optional: additional comments about this solution\> # Background @@ -353,25 +283,21 @@ Add additional references as needed to help reviewers and authors understand the ## Terminology & Definitions -**\[Optional \- if not applicable omit\]** - -List out additional terms / definitions (lexicon). Try to keep definitions as concise as possible and use links to external resources when additional information would be useful to the reader. - -Keep the list of terms sorted alphabetically to ease looking up definitions by readers. - -| \<Term\> | \<Definition\> | +| Term | Definition | | :---- | :---- | -| **\<Term\>** | \<Definition\> | +| **Base Container** | Pre-built container with Dynamo and NIXL that serves as foundation for backend-specific builds | +| **Backend Build** | Container stage that builds backend-specific code and dependencies | +| **CI Stage** | Container stage with testing tools and validation requirements | +| **Manylinux** | PEP 513 standard for Linux wheel distribution compatibility | +| **NIXL** | High-throughput, low-latency point-to-point communication library for accelerating inference | +| **Runtime Stage** | Minimal container stage with only production deployment requirements | ## Acronyms & Abbreviations -**\[Optional \- if not applicable omit\]** - -Provide a list of frequently used acronyms and abbreviations which are uncommon or unlikely to be known by the reader. Do not include acronyms or abbreviations which the reader is likely to be familiar with. +**CI:** Continuous Integration -Keep the list of acronyms and abbreviations sorted alphabetically to ease looking up definitions by readers. +**DEP:** Design Enhancement Proposal -Do not include the full definition in the expanded meaning of an abbreviation or acronym. If the reader needs the definition, please include it in the [Terminology & Definitions](#terminology--definitions) section. +**NIXL:** NVIDIA Inference Exchange Library -**\<Acronym/Abbreviation\>:** \<Expanded Meaning\> From 7bc9681e3731bd0553c9117798661f3cd344fafc Mon Sep 17 00:00:00 2001 From: Tushar Sharma <tusharma@nvidia.com> Date: Tue, 15 Jul 2025 10:19:37 -0700 Subject: [PATCH 27/42] Re-add the complete template into the MD file --- NNNN-complete-template.md | 186 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 186 insertions(+) create mode 100644 NNNN-complete-template.md diff --git a/NNNN-complete-template.md b/NNNN-complete-template.md new file mode 100644 index 0000000..28ab2d9 --- /dev/null +++ b/NNNN-complete-template.md @@ -0,0 +1,186 @@ +# <Title> + +**Status**: Draft | Under Review | Approved | Replaced | Deferred | Rejected + +**Authors**: [Name/Team] + +**Category**: Architecture | Process | Guidelines + +**Replaces**: [Link of previous proposal if applicable] + +**Replaced By**: [Link of previous proposal if applicable] + +**Sponsor**: [Name of code owner or maintainer to shepard process] + +**Required Reviewers**: [Names of technical leads that are required for acceptance] + +**Review Date**: [Date for review] + +**Pull Request**: [Link to Pull Request of the Proposal itself] + +**Implementation PR / Tracking Issue**: [Link to Pull Request or Tracking Issue for Implementation] + +# Summary + +**\[Required\]** + +# Motivation + +**\[Required\]** + +Describe the problem that needs to be addressed with enough detail for +someone familiar with the project to understand. Generally one to two +short paragraphs. Additional details can be placed in the background +section as needed. Cover **what** the issue is and **why** it needs to +be addressed. Link to github issues if relevant. + +## Goals + +**\[Optional \- if not applicable omit\]** + +List out any additional goals in bullet points. Goals may be aspirational / difficult to measure but guide the proposal. + +* Goal + +* Goal + +* Goal + +### Non Goals + +**\[Optional \- if not applicable omit\]** + +List out any items which are out of scope / specifically not required in bullet points. Indicates the scope of the proposal and issue being resolved. + +## Requirements + +**\[Optional \- if not applicable omit\]** + +List out any additional requirements in numbered subheadings. + +**\<numbered subheadings\>** + +### REQ \<\#\> \<Title\> + +Describe the requirement in as much detail as necessary for others to understand it and how it applies to the DEP. Keep in mind that requirements should be measurable and will be used to determine if a DEP has been successfully implemented or not. + +Requirement names should be prefixed using a monotonically increasing number such as “REQ 1 \<Title\>” followed by “REQ 2 \<Title\>” and so on. Use title casing when naming requirements. Requirement names should be as descriptive as possible while remaining as terse as possible. + +Use all-caps, bolded terms like **MUST** and **SHOULD** when describing each requirement. See [RFC-2119](https://datatracker.ietf.org/doc/html/rfc2119) for additional information. + + +# Proposal + +**\[Required\]** + +Describe the high level design / proposal. Use sub sections as needed, but start with an overview and then dig into the details. Try to provide images and diagrams to facilitate understanding. + +# Implementation Details + +**\[Optional \- if not applicable omit\]** + +Add additional detailed items here including interface signatures, etc. Add anything that is relevant but seems more of a detail than central to the proposal. Use sub sections / bullet points as needed. Try to provide images and diagrams to facilitate understanding. If applicable link to PR. + +## Deferred to Implementation + +**\[Optional \- if not applicable omit\]** + +List out items that are under discussion but that will be resolved only during implementation / code review. + +# Implementation Phases + +**\[Optional \- if not applicable omit\]** + +List out phases of implementation (can be single phase). Give each phase a monotonically increasing number; example “Phase 0” followed by “Phase 1” and so on. Give phases titles if it makes sense. + +## Phase \<\#\> \<Optional Title\> + +**Release Target**: Date + +**Effort Estimate**: \<estimate of time and number of engineers to complete the phase\> + +**Work Item(s):** \<one or more links to github issues\> + +**Supported API / Behavior:** + +* \<name and concise description of the API / behavior\> + +**Not Supported:** + +* \<name and concise description of the API / behavior\> + +# Related Proposals + +**\[Optional \- if not applicable omit\]** + +* File + +* File + +* File + +* File + +* File + +# Alternate Solutions + +**\[Required, if not applicable write N/A\]** + +List out solutions that were considered but ultimately rejected. Consider free form \- but a possible format shown below. + +## Alt \<\#\> \<Title\> + +**Pros:** + +\<bulleted list or pros describing the positive aspects of this solution\> + +**Cons:** + +\<bulleted list or pros describing the negative aspects of this solution\> + +**Reason Rejected:** + +\<bulleted list or pros describing why this option was not used\> + +**Notes:** + +\<optional: additional comments about this solution\> + +# Background + +**\[Optional \- if not applicable omit\]** + +Add additional context and references as needed to help reviewers and authors understand the context of the problem and solution being proposed. + +## References + +**\[Optional \- if not applicable omit\]** + +Add additional references as needed to help reviewers and authors understand the context of the problem and solution being proposed. + +* \<hyper-linked title of an external reference resource\> + +## Terminology & Definitions + +**\[Optional \- if not applicable omit\]** + +List out additional terms / definitions (lexicon). Try to keep definitions as concise as possible and use links to external resources when additional information would be useful to the reader. + +Keep the list of terms sorted alphabetically to ease looking up definitions by readers. + +| \<Term\> | \<Definition\> | +| :---- | :---- | +| **\<Term\>** | \<Definition\> | + +## Acronyms & Abbreviations + +**\[Optional \- if not applicable omit\]** + +Provide a list of frequently used acronyms and abbreviations which are uncommon or unlikely to be known by the reader. Do not include acronyms or abbreviations which the reader is likely to be familiar with. + +Keep the list of acronyms and abbreviations sorted alphabetically to ease looking up definitions by readers. + +Do not include the full definition in the expanded meaning of an abbreviation or acronym. If the reader needs the definition, please include it in the [Terminology & Definitions](#terminology--definitions) section. + +**\<Acronym/Abbreviation\>:** \<Expanded Meaning\> \ No newline at end of file From 831778ad142e94b0923661f8dcefdf727cc504b9 Mon Sep 17 00:00:00 2001 From: Tushar Sharma <tusharma@nvidia.com> Date: Tue, 15 Jul 2025 10:20:16 -0700 Subject: [PATCH 28/42] nit: add extra line --- NNNN-complete-template.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/NNNN-complete-template.md b/NNNN-complete-template.md index 28ab2d9..4b9c1a3 100644 --- a/NNNN-complete-template.md +++ b/NNNN-complete-template.md @@ -183,4 +183,4 @@ Keep the list of acronyms and abbreviations sorted alphabetically to ease lookin Do not include the full definition in the expanded meaning of an abbreviation or acronym. If the reader needs the definition, please include it in the [Terminology & Definitions](#terminology--definitions) section. -**\<Acronym/Abbreviation\>:** \<Expanded Meaning\> \ No newline at end of file +**\<Acronym/Abbreviation\>:** \<Expanded Meaning\> From 595b0d2d713193f626e22b3701292d8ca0987a08 Mon Sep 17 00:00:00 2001 From: Tushar Sharma <tusharma@nvidia.com> Date: Tue, 15 Jul 2025 10:22:04 -0700 Subject: [PATCH 29/42] Update NNNN-complete-template.md Signed-off-by: Tushar Sharma <tusharma@nvidia.com> From 747fcc4dfde31cf892f75dd085a0399ce24ca6f2 Mon Sep 17 00:00:00 2001 From: Tushar Sharma <tusharma@nvidia.com> Date: Tue, 15 Jul 2025 13:26:48 -0700 Subject: [PATCH 30/42] Minor updates to motivation section --- deps/NNNN-container-strategy.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/deps/NNNN-container-strategy.md b/deps/NNNN-container-strategy.md index a3deeb5..1b83102 100644 --- a/deps/NNNN-container-strategy.md +++ b/deps/NNNN-container-strategy.md @@ -36,11 +36,11 @@ Dynamo's current container architecture consists of multiple Dockerfiles in the The current approach faces several challenges: -1. **Inefficient Build Process**: Components like Dynamo, NIXL, and backends are rebuilt across stages instead of using a layered approach. For example, in Dockerfile.vllm, Dynamo is installed three times - in base, ci_minimum, and runtime stages. +1. **Inefficient Build Process**: Components like Dynamo, NIXL, and backends are installed multiple times across stages instead of using a layered approach. Stages do not build upon each other which leads to repeated steps and inefficient build times. 2. **Developer Experience Issues**: The unclear organization of Dockerfiles makes it difficult for developers to choose the right build for their needs, often defaulting to the devel build regardless of use case. -3. **Build Reliability**: The complex layering and repeated steps across stages lead to intermittent build failures. +3. **Build Reliability**: The complex layering and repeated steps across stages can lead to intermittent build failures. 4. **Inconsistent Standards**: Without a unified Dockerfile for building Dynamo, NIXL, and dependencies, code is duplicated or missing across backend-specific Dockerfiles, and optimizations aren't shared effectively. From e2e7caba1b97ea1f5feea6d1e1bed4c3889a1bda Mon Sep 17 00:00:00 2001 From: Tushar Sharma <tusharma@nvidia.com> Date: Tue, 15 Jul 2025 13:31:31 -0700 Subject: [PATCH 31/42] Set to RFR and update Review data --- deps/NNNN-container-strategy.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/deps/NNNN-container-strategy.md b/deps/NNNN-container-strategy.md index 1b83102..11eca39 100644 --- a/deps/NNNN-container-strategy.md +++ b/deps/NNNN-container-strategy.md @@ -1,6 +1,6 @@ # Container Build Process Optimization -**Status**: Draft +**Status**: Ready for Review **Authors**: [nv-tusharma] @@ -14,7 +14,7 @@ **Required Reviewers**: nnshah1, saturley-hall, nv-anants, nvda-mesharma, mc-nv, dmitry-tokarev-nv, pvijayakrish -**Review Date**: 2025-06-03 +**Review Date**: 2025-07-15 **Pull Request**: TBD From ada82ac1294536fa78f265b03bc70fdc0245f9d9 Mon Sep 17 00:00:00 2001 From: Tushar Sharma <tusharma@nvidia.com> Date: Tue, 15 Jul 2025 13:32:19 -0700 Subject: [PATCH 32/42] Rename title --- deps/NNNN-container-strategy.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/deps/NNNN-container-strategy.md b/deps/NNNN-container-strategy.md index 11eca39..f9c32a2 100644 --- a/deps/NNNN-container-strategy.md +++ b/deps/NNNN-container-strategy.md @@ -1,4 +1,4 @@ -# Container Build Process Optimization +# Dynamo Container Build Process Improvements **Status**: Ready for Review From 696e49aeacdd2e20983df5b1bd181c60e3f357dc Mon Sep 17 00:00:00 2001 From: Tushar Sharma <tusharma@nvidia.com> Date: Tue, 15 Jul 2025 13:34:12 -0700 Subject: [PATCH 33/42] Update Review data --- deps/NNNN-container-strategy.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/deps/NNNN-container-strategy.md b/deps/NNNN-container-strategy.md index f9c32a2..c790e90 100644 --- a/deps/NNNN-container-strategy.md +++ b/deps/NNNN-container-strategy.md @@ -14,7 +14,7 @@ **Required Reviewers**: nnshah1, saturley-hall, nv-anants, nvda-mesharma, mc-nv, dmitry-tokarev-nv, pvijayakrish -**Review Date**: 2025-07-15 +**Review Date**: 2025-07-18 **Pull Request**: TBD From c8081b34c6521cddab7742966f508ceeb71840a0 Mon Sep 17 00:00:00 2001 From: Tushar Sharma <tusharma@nvidia.com> Date: Tue, 15 Jul 2025 13:40:44 -0700 Subject: [PATCH 34/42] Revert changes to NNNN-complete-template.md --- NNNN-complete-template.md | 1 + 1 file changed, 1 insertion(+) diff --git a/NNNN-complete-template.md b/NNNN-complete-template.md index 4b9c1a3..95468d1 100644 --- a/NNNN-complete-template.md +++ b/NNNN-complete-template.md @@ -184,3 +184,4 @@ Keep the list of acronyms and abbreviations sorted alphabetically to ease lookin Do not include the full definition in the expanded meaning of an abbreviation or acronym. If the reader needs the definition, please include it in the [Terminology & Definitions](#terminology--definitions) section. **\<Acronym/Abbreviation\>:** \<Expanded Meaning\> + From fd9754222fb12361d68d8b748735b7c609d1bd66 Mon Sep 17 00:00:00 2001 From: Tushar Sharma <tusharma@nvidia.com> Date: Tue, 22 Jul 2025 15:52:41 -0700 Subject: [PATCH 35/42] Address current comments and concerns called out --- deps/NNNN-container-strategy.md | 69 +++++++++++++++++++++++++-------- 1 file changed, 52 insertions(+), 17 deletions(-) diff --git a/deps/NNNN-container-strategy.md b/deps/NNNN-container-strategy.md index c790e90..3558595 100644 --- a/deps/NNNN-container-strategy.md +++ b/deps/NNNN-container-strategy.md @@ -29,6 +29,7 @@ One of the goals for this document is to define a clear and maintainable structu To achieve this goal, this document proposes certain optimizations to improve the current build process: - Restructuring the build process to provide a base container with a pre-built version of Dynamo + NIXL available on all distributions, enabling splitting of specific backends from the dynamo build process. - Defining a structure/template for all Dockerfiles to follow to ensure consistent and reproducible builds across backends along with specific roles/use cases targeted for each stage. +- Implementing remote compiler caching strategies using tools like sccache to significantly reduce rust compilation times across builds and CI/CD pipelines. # Motivation @@ -50,7 +51,7 @@ This document proposes solutions to the build process challenges, aiming to impr * Remove duplicate code in current dockerfile implementations and define a single build base image which provides a pre-built container with Dynamo + NIXL. -This base image should operate as a single base container which can then be used as base containers for backend-specific images. By leveraging a base container, We can reduce the redundant code across Dockerfiles and establish a single-source of truth for all Dynamo-builds. +This base image should operate as a single base container which can then be used as base containers for backend-specific images. By leveraging a base container, We can reduce the redundant code across Dockerfiles and establish a single-source of truth for all Dynamo-builds. This would also enable us to replace the current devel container with the base container for local development/public CI for faster validation of changes. * Define the relationships between base, runtime, development, and CI images for each Dockerfile and provide a structure/template to follow for Dockerfiles. @@ -60,11 +61,14 @@ Pinning/Fixing dependencies will ensure a unified build environment reducing "it * Minimize effort for providing multi-arch support across various backends for Dynamo by leveraging manylinux to build for multiple distributions +* Implement remote compiler caching to dramatically reduce build times across development and CI/CD environments + +By integrating tools like sccache for remote compilation caching, we can avoid redundant compilation work across builds, significantly speeding up the container build process for both developers and CI pipelines. + ### Non Goals - Container release strategy and processes (covered in separate DEP) -- Unified build environment -- Outline possible further improvements including external caching/multi-context docker builds to reduce build times. +- Unified build environment ## Requirements @@ -73,6 +77,7 @@ The build-base container must be designed such that backend-specific Dockerfiles - Multi-arch support is a P0. The Base container should be able to support both x84_64 and arm64 builds. - Clear documentation on how to use the base container - Standardized environment variables and paths +- Replace the current devel container with the base container for local development/public CI for faster validation of changes. ### REQ \<\#2\> \<Layered Container Structure\> Dockerfiles must follow a layered, super-set structure to optimize build efficiency: @@ -94,7 +99,7 @@ In order to address the requirements, we propose the following changes to the Dy ## Build-Base Container -The base container will be a pre-built container that will be used by the backends to build the final container image. This build base container will contain a Dynamo build for all backends to use for their framework-specific build. The base image will leverage a manylinux base image to enable support for multiple distributions (U22, U24, etc). The container will also include a NIXL build since this is common across all backends. We will create a new Dockerfile in the /containers directory for this container and provide the image through our CI registry for developers to use for local development. The base container must provide multi-arch support. +The base container will be a pre-built container that will be used by the backends to build the final container image. This build base container will contain a Dynamo build for all backends to use for their framework-specific build. The base image will leverage a manylinux base image to enable support for multiple distributions (U22, U24, etc). The container will also include a NIXL build since this is common across all backends. This will be a new Dockerfile in the /containers directory. Multi-arch support is also a P0 Also, the base container can be used as a drop-in replacement for the current devel container used in public CI. This would significantly reduce public CI build times and enable faster validation of changes. ## Use-case of build stages along with relationship between stages (base, runtime, devel, ci_minimum) @@ -102,7 +107,7 @@ Each backend-specific Dockerfile should follow a specific format. The backend-sp | Stage | Targeted User | Base Image | Functionality | |----------|---------------------|----------------------|----------------------------------------------------------------------------------------------------------------------| -| Backend Build | Developers | Cuda base devel image | Builds targeted backend along with backend-specific dependencies. +| Backend Build | Developers | Cuda base devel image | Builds targeted backend along with backend-specific dependencies in editable mode. | Runtime | Customers/Production| Cuda base runtime image| Minimal image with only the dependencies required to deploy and run Dynamo w/backend from the backend build stage; intended for production deployments. Copies dynamo artifacts from base image and backend artifaces from backend build image. | | CI | Developers/Internal CI Pipelines/Local Debugging | Runtime image | Adds CI-specific tools, QA test scripts, internal models, and other dependencies needed for automated testing. | @@ -188,6 +193,21 @@ FROM runtime as ci - **Minimal Runtime**: Runtime images will only include necessary dependencies for production deployment - **Layered Caching**: Build layers will be optimized for Docker build cache efficiency +## Build Caching Strategy (Phase 3) + +To further optimize build times after the initial Dockerfile restructuring, We will explore remote compiler caching (further optimizations to be added in future): + +### Remote Compiler Caching +- **Compiler cache Integration**: Leverage compiler cache service like [sccache](https://github.com/mozilla/sccache) in the build-base container to cache compilation results for Dynamo, NIXL, and backend dependencies. +- **Remote Cache Storage**: Use a remote cache storage service to store the cached compilation results in CI/CD pipelines. +- **Cache Size Management**: Configure appropriate cache size limits and cleanup policies to balance storage usage with build performance. + + +### Implementation Considerations +- **Cache Invalidation**: Implement smart cache invalidation based on dependency changes and version updates +- **Monitoring**: Add build time metrics to measure cache effectiveness and identify optimization opportunities +- **CI Integration**: Configure CI/CD pipelines to properly utilize remote caching storage. + ## Deferred to Implementation TBD @@ -214,7 +234,6 @@ TBD **Not Supported:** -* Advanced caching mechanisms (e.g. multi-context docker builds) ## Phase \<\#2\> \<Restructure backend Dockerfiles to follow proposed structure\> @@ -235,6 +254,23 @@ TBD **Not Supported:** +## Phase \<\#3\> \<Build Caching Optimization\> + +**Release Target**: TBD + +**Effort Estimate**: \<estimate of time and number of engineers to complete the phase\> + +**Work Item(s):** \<one or more links to github issues\> + +**Supported API / Behavior:** + +* Integration of sccache for rust compilation caching across container builds +* Remote cache storage for CI/CD pipelines to reduce cold build times +* Cache invalidation strategies for dependency updates and smart cache layer management + +**Not Supported:** + +* Advanced distributed caching mechanisms # Related Proposals @@ -246,26 +282,24 @@ TBD List out solutions that were considered but ultimately rejected. Consider free form \- but a possible format shown below. -## Alt \<\# 1\> \<Using Deep Learning CUDA images as base image instead of cuda runtime\> +## Alt \<\# 2\> \<Provide a single container with multi-backend support instead of multiple backend-specific containers\> **Pros:** -- Dynamo would be in sync with the latest Deep Learning CUDA release -- Deep Learning CUDA image can provide additional libraries such as cuDNN, NCCL, etc which can accelerate deep learning workloads. -- TritonServer is built on top of the Deep Learning CUDA image, so would be a good reference for the use-case. +- Reduce overall complexity (less containers, less Dockerfiles) +- No need foradditonal security scans and QA validation. +- Simpler Open Source approval process. **Cons:** -- Container Size is too large for the use-case. The cuda runtime image we use is around 6 GB whereas the Deep Learning Cuda runtime is around 11 GB, uncompressed. -- Introduces additional dependencies that are not required for Dynamo, which can increase the container size and build time. -- Dynamo has NIXL as it's communication library, so a dependency on NCCL is not required. +- Container size, if a user is only interested in one particular backend, we should remove the dependencies associated with other backends. We would need to provide tools for users to create backend-specific containers for their deployment. +- It is expected that Backends can differ on performance, which could be a result of backend-specific dependencies. +- Build times are expected to be longer for a single container with multi-backend support. +- Is it feasible for a user to want deploy multiple inference engines at once with dynamo? **Reason Rejected:** -- Container Size is too large for the use-case. The cuda runtime image we use is around 6 GB whereas the Deep Learning cuda runtime is around 11 GB, uncompressed. -- Introduces additional dependencies that are not required for Dynamo, which can increase the container size and build time. - - +- There are more cons than pros for this approach. Along with the cons, the Dynamo base container is a good drop-in replacement for the multi-backend container. # Background @@ -287,6 +321,7 @@ Add additional references as needed to help reviewers and authors understand the | :---- | :---- | | **Base Container** | Pre-built container with Dynamo and NIXL that serves as foundation for backend-specific builds | | **Backend Build** | Container stage that builds backend-specific code and dependencies | +| **sccache** | Compiler cache tool that speeds up recompilation by caching previous compilation results | | **CI Stage** | Container stage with testing tools and validation requirements | | **Manylinux** | PEP 513 standard for Linux wheel distribution compatibility | | **NIXL** | High-throughput, low-latency point-to-point communication library for accelerating inference | From a98a776500f568b7094a2c7da06828e662d6e54c Mon Sep 17 00:00:00 2001 From: Tushar Sharma <tusharma@nvidia.com> Date: Tue, 22 Jul 2025 16:07:53 -0700 Subject: [PATCH 36/42] nit: minor improvements --- deps/NNNN-container-strategy.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/deps/NNNN-container-strategy.md b/deps/NNNN-container-strategy.md index 3558595..249cfd1 100644 --- a/deps/NNNN-container-strategy.md +++ b/deps/NNNN-container-strategy.md @@ -29,7 +29,7 @@ One of the goals for this document is to define a clear and maintainable structu To achieve this goal, this document proposes certain optimizations to improve the current build process: - Restructuring the build process to provide a base container with a pre-built version of Dynamo + NIXL available on all distributions, enabling splitting of specific backends from the dynamo build process. - Defining a structure/template for all Dockerfiles to follow to ensure consistent and reproducible builds across backends along with specific roles/use cases targeted for each stage. -- Implementing remote compiler caching strategies using tools like sccache to significantly reduce rust compilation times across builds and CI/CD pipelines. +- Implementing remote compiler caching strategies tools like sccache to significantly reduce rust compilation times across builds and CI/CD pipelines. # Motivation @@ -124,7 +124,7 @@ flowchart TD C[Build Dependencies] D[Build Dynamo] E[Dynamo Base Container]:::gray - F[Build Backend-specific code] + F[Build Backend from source] G[Backend Build Image]:::gray J[Cuda Runtime<br/>nvcr.io/nvidia/cuda.XX.YY-runtime]:::gray K[Install NATS + ETCD] From e180e78b0f5e9e0ba3ad82fd95233342ea16bc30 Mon Sep 17 00:00:00 2001 From: Tushar Sharma <tusharma@nvidia.com> Date: Mon, 1 Sep 2025 20:36:43 -0700 Subject: [PATCH 37/42] nit: minor updates to strategy --- deps/NNNN-container-strategy.md | 114 +++++++++++++++++++++++--------- 1 file changed, 83 insertions(+), 31 deletions(-) diff --git a/deps/NNNN-container-strategy.md b/deps/NNNN-container-strategy.md index 249cfd1..53b9d93 100644 --- a/deps/NNNN-container-strategy.md +++ b/deps/NNNN-container-strategy.md @@ -105,11 +105,20 @@ The base container will be a pre-built container that will be used by the backen Each backend-specific Dockerfile should follow a specific format. The backend-specific Dockerfiles should be divided up into multiple stages, with each stage inheriting artifacts/leveraging the previous stage as the base container. The following stages should be defined in the backend-specific Dockerfile: -| Stage | Targeted User | Base Image | Functionality | -|----------|---------------------|----------------------|----------------------------------------------------------------------------------------------------------------------| -| Backend Build | Developers | Cuda base devel image | Builds targeted backend along with backend-specific dependencies in editable mode. -| Runtime | Customers/Production| Cuda base runtime image| Minimal image with only the dependencies required to deploy and run Dynamo w/backend from the backend build stage; intended for production deployments. Copies dynamo artifacts from base image and backend artifaces from backend build image. | -| CI | Developers/Internal CI Pipelines/Local Debugging | Runtime image | Adds CI-specific tools, QA test scripts, internal models, and other dependencies needed for automated testing. | +**Backend Build Stage:** +- Targeted User: Developers +- Base Image: Cuda base devel image +- Functionality: Builds targeted backend along with backend-specific dependencies in editable mode. + +**Runtime Stage:** +- Targeted User: Customers/Production +- Base Image: Cuda base runtime image +- Functionality: Minimal image with only the dependencies required to deploy and run Dynamo w/backend from the backend build stage; intended for production deployments. Copies dynamo artifacts from base image and backend artifacts from backend build image. + +**CI Stage:** +- Targeted User: Developers/Internal CI Pipelines/Local Debugging +- Base Image: Runtime image +- Functionality: Adds CI-specific tools, QA test scripts, internal models, and other dependencies needed for automated testing. The CUDA base images will be used from the [NVIDIA CUDA Container Registry](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/cuda). Please refer to the Pros and Cons section for more details on why we chose to use the cuda runtime image instead of the Deep Learning CUDA image. @@ -119,20 +128,20 @@ The CUDA base images will be used from the [NVIDIA CUDA Container Registry](http ```mermaid flowchart TD - A[Manylinux build base image]:::gray + A[Manylinux build base image] B[NIXL Setup/Build NIXL Wheel] C[Build Dependencies] D[Build Dynamo] - E[Dynamo Base Container]:::gray + E[Build Base Container] F[Build Backend from source] - G[Backend Build Image]:::gray - J[Cuda Runtime<br/>nvcr.io/nvidia/cuda.XX.YY-runtime]:::gray + G[Backend Build Image] + J[Cuda Runtime<br/>nvcr.io/nvidia/cuda.XX.YY-runtime] K[Install NATS + ETCD] L[Runtime-specific dependencies] M[pip install dynamo + Backend && NIXL] - N[Backend Runtime Image]:::gray + N[Backend Runtime Image] O[Install CI/Test/Dev dependencies] - P[CI Minimum image]:::gray + P[CI Minimum image] %% Main build flow (left) A --> B @@ -216,17 +225,15 @@ TBD ## Phase \<\#1\> \<Build Base Container Development\> -**Release Target**: TBD +**Release Target**: v0.5.0 -**Release Target**: Date +**Effort Estimate**: 1 engineer, 1 week -**Effort Estimate**: \<estimate of time and number of engineers to complete the phase\> - -**Work Item(s):** \<one or more links to github issues\> +**Work Item(s):** N/A **Supported API / Behavior:** -* Pre-built Dynamo base container with multi-arch support +* Pre-built Build base container with multi-arch support containing Dynamo and NIXL * NIXL integration in base container * Manylinux base for broad distribution compatibility * Standardized environment variables and paths @@ -237,9 +244,9 @@ TBD ## Phase \<\#2\> \<Restructure backend Dockerfiles to follow proposed structure\> -**Release Target**: Date +**Release Target**: v0.5.1 -**Effort Estimate**: \<estimate of time and number of engineers to complete the phase\> +**Effort Estimate**: 1 engineer, 1 week **Work Item(s):** \<one or more links to github issues\> @@ -256,11 +263,11 @@ TBD ## Phase \<\#3\> \<Build Caching Optimization\> -**Release Target**: TBD +**Release Target**: v0.5.1 -**Effort Estimate**: \<estimate of time and number of engineers to complete the phase\> +**Effort Estimate**: 1 engineer, 1 week -**Work Item(s):** \<one or more links to github issues\> +**Work Item(s):** N/A **Supported API / Behavior:** @@ -301,6 +308,47 @@ List out solutions that were considered but ultimately rejected. Consider free f - There are more cons than pros for this approach. Along with the cons, the Dynamo base container is a good drop-in replacement for the multi-backend container. +## Alt #3 Published Base Container with Pinned Dependencies + +**Description:** + +Instead of building NIXL, UCX, and other stable dependencies from source in each build, publish a pre-built base container with these pinned components. This would create a three-tier container hierarchy: + +1. **Base Image (Published):** CUDA + NIXL + UCX + uv + cargo + other stable dependencies +2. **Dynamo Image:** Base Image + Dynamo Rust/Python builds +3. **Framework Image:** Dynamo Image + Framework builds (vLLM, sglang, TRT-LLM) + +The base image would be published to a public registry (GitHub Container Registry or NGC) and updated infrequently when NIXL or other core dependencies change. + +**Pros:** + +- **Dramatically reduced build times:** Skip compilation of NIXL, UCX, and other stable components that rarely change +- **Consistent environment:** All builds use the same pinned versions of core dependencies +- **Simplified maintenance:** Base image updates are centralized and infrequent +- **Better caching:** Base image can be cached across all builds and CI pipelines +- **Reduced CI resource usage:** Less compilation work in CI/CD pipelines +- **Public availability:** Base image could be made available to external users/partners + +**Cons:** + +- **Additional publishing workflow:** Need separate CI/CD pipeline to build and publish base images +- **Registry management:** Need to manage storage and access for published base images +- **Storage costs:** Published images consume registry storage space +- **Security scanning:** Published base images need regular security scanning and updates +- **Dependency on external registry:** Builds depend on availability of published base image + +**Implementation Considerations:** + +- **Publishing cadence:** Base image updates triggered by NIXL/UCX version changes or monthly schedule +- **Versioning strategy:** Semantic versioning for base images (e.g., `nvidia/dynamo-base:v1.2.0`) +- **Multi-arch support:** Publish both x86_64 and arm64 variants +- **Registry choice:** GitHub Container Registry (ghcr.io) for open source, NGC for enterprise +- **Fallback strategy:** Ability to build from source if published image unavailable + +**Reason Rejected:** + +N/A + # Background **\[Optional \- if not applicable omit\]** @@ -317,15 +365,19 @@ Add additional references as needed to help reviewers and authors understand the ## Terminology & Definitions -| Term | Definition | -| :---- | :---- | -| **Base Container** | Pre-built container with Dynamo and NIXL that serves as foundation for backend-specific builds | -| **Backend Build** | Container stage that builds backend-specific code and dependencies | -| **sccache** | Compiler cache tool that speeds up recompilation by caching previous compilation results | -| **CI Stage** | Container stage with testing tools and validation requirements | -| **Manylinux** | PEP 513 standard for Linux wheel distribution compatibility | -| **NIXL** | High-throughput, low-latency point-to-point communication library for accelerating inference | -| **Runtime Stage** | Minimal container stage with only production deployment requirements | +**Base Container:** Pre-built container with Dynamo and NIXL that serves as foundation for backend-specific builds + +**Backend Build:** Container stage that builds backend-specific code and dependencies + +**sccache:** Compiler cache tool that speeds up recompilation by caching previous compilation results + +**CI Stage:** Container stage with testing tools and validation requirements + +**Manylinux:** PEP 513 standard for Linux wheel distribution compatibility + +**NIXL:** High-throughput, low-latency point-to-point communication library for accelerating inference + +**Runtime Stage:** Minimal container stage with only production deployment requirements ## Acronyms & Abbreviations From d3d252a99552d1fdbd5b31b7c9009e986429f9c0 Mon Sep 17 00:00:00 2001 From: Tushar Sharma <tusharma@nvidia.com> Date: Fri, 19 Sep 2025 10:21:09 -0700 Subject: [PATCH 38/42] minor fixes to update recent changes in container strateegy --- deps/NNNN-container-strategy.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/deps/NNNN-container-strategy.md b/deps/NNNN-container-strategy.md index 53b9d93..3219737 100644 --- a/deps/NNNN-container-strategy.md +++ b/deps/NNNN-container-strategy.md @@ -24,7 +24,7 @@ This document outlines a container build process optimization strategy for Dynamo to enhance the developer experience by re-organizing Dockerfiles along with defining a clear and maintainable structure for our Dockerfiles. -One of the goals for this document is to define a clear and maintainable structure for our Dockerfiles. Specifically, to determine how many Dockerfiles we need and clarify the relationships between base, runtime, development, and CI images. The aim is to ensure each environment's Dockerfile builds upon the previous (as supersets), maximizing environment consistency and coverage during daily development and testing. +One of the goals for this document is to define a clear and maintainable structure for our Dockerfiles. Specifically, to determine how many Dockerfiles we need and clarify the relationships between base, runtime, and development images. The aim is to ensure each environment's Dockerfile builds upon the previous (as supersets), maximizing environment consistency and coverage during daily development and testing. To achieve this goal, this document proposes certain optimizations to improve the current build process: - Restructuring the build process to provide a base container with a pre-built version of Dynamo + NIXL available on all distributions, enabling splitting of specific backends from the dynamo build process. @@ -53,7 +53,7 @@ This document proposes solutions to the build process challenges, aiming to impr This base image should operate as a single base container which can then be used as base containers for backend-specific images. By leveraging a base container, We can reduce the redundant code across Dockerfiles and establish a single-source of truth for all Dynamo-builds. This would also enable us to replace the current devel container with the base container for local development/public CI for faster validation of changes. -* Define the relationships between base, runtime, development, and CI images for each Dockerfile and provide a structure/template to follow for Dockerfiles. +* Define the relationships between base, runtime, and development images for each Dockerfile and provide a structure/template to follow for Dockerfiles. * Reduce build flakiness by pinning/fixing dependencies in the base image from package managers and squashing/reducing layers as necessary @@ -115,10 +115,10 @@ Each backend-specific Dockerfile should follow a specific format. The backend-sp - Base Image: Cuda base runtime image - Functionality: Minimal image with only the dependencies required to deploy and run Dynamo w/backend from the backend build stage; intended for production deployments. Copies dynamo artifacts from base image and backend artifacts from backend build image. -**CI Stage:** +**Development Stage:** - Targeted User: Developers/Internal CI Pipelines/Local Debugging - Base Image: Runtime image -- Functionality: Adds CI-specific tools, QA test scripts, internal models, and other dependencies needed for automated testing. +- Functionality: Adds development-specific tools, QA test scripts, internal models, and other dependencies needed for developers. We also want to integrate dev container features into this stage. The CUDA base images will be used from the [NVIDIA CUDA Container Registry](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/cuda). Please refer to the Pros and Cons section for more details on why we chose to use the cuda runtime image instead of the Deep Learning CUDA image. @@ -141,7 +141,7 @@ flowchart TD M[pip install dynamo + Backend && NIXL] N[Backend Runtime Image] O[Install CI/Test/Dev dependencies] - P[CI Minimum image] + P[Development image] %% Main build flow (left) A --> B @@ -171,7 +171,7 @@ The diagram above illustrates the proposed container build strategy showing the - Build Base Container with common dependencies - Backend-specific development containers - Runtime containers -- CI containers +- Development containers This layered approach ensures consistent builds, reduces duplication, and improves maintainability across all backend implementations. @@ -190,9 +190,9 @@ FROM nvcr.io/nvidia/cuda:XX.YY-runtime-ubuntuXX.XX as runtime # Copy backend artifacts from backend-build stage # Install runtime dependencies only -# CI Stage -FROM runtime as ci -# Add CI-specific tools and test dependencies +# Development Stage +FROM runtime as dev +# Add development-specific tools and test dependencies ``` ## Dependency Management From fe9b03748df2ab4f9e65ea898313628db4b12f69 Mon Sep 17 00:00:00 2001 From: Tushar Sharma <tusharma@nvidia.com> Date: Fri, 19 Sep 2025 10:24:59 -0700 Subject: [PATCH 39/42] remove deferred to implementation section --- deps/NNNN-container-strategy.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/deps/NNNN-container-strategy.md b/deps/NNNN-container-strategy.md index 3219737..5924224 100644 --- a/deps/NNNN-container-strategy.md +++ b/deps/NNNN-container-strategy.md @@ -217,9 +217,6 @@ To further optimize build times after the initial Dockerfile restructuring, We w - **Monitoring**: Add build time metrics to measure cache effectiveness and identify optimization opportunities - **CI Integration**: Configure CI/CD pipelines to properly utilize remote caching storage. -## Deferred to Implementation - -TBD # Implementation Phases From f6a596a895e43c1499d71cb7441824e13ddbaf7e Mon Sep 17 00:00:00 2001 From: Tushar Sharma <tusharma@nvidia.com> Date: Fri, 19 Sep 2025 10:27:58 -0700 Subject: [PATCH 40/42] set to remote compiler caching strategies --- deps/NNNN-container-strategy.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/deps/NNNN-container-strategy.md b/deps/NNNN-container-strategy.md index 5924224..fdb90b1 100644 --- a/deps/NNNN-container-strategy.md +++ b/deps/NNNN-container-strategy.md @@ -61,7 +61,7 @@ Pinning/Fixing dependencies will ensure a unified build environment reducing "it * Minimize effort for providing multi-arch support across various backends for Dynamo by leveraging manylinux to build for multiple distributions -* Implement remote compiler caching to dramatically reduce build times across development and CI/CD environments +* Implement remote compiler caching strategies to dramatically reduce build times across development and CI/CD environments By integrating tools like sccache for remote compilation caching, we can avoid redundant compilation work across builds, significantly speeding up the container build process for both developers and CI pipelines. @@ -204,9 +204,9 @@ FROM runtime as dev ## Build Caching Strategy (Phase 3) -To further optimize build times after the initial Dockerfile restructuring, We will explore remote compiler caching (further optimizations to be added in future): +To further optimize build times after the initial Dockerfile restructuring, We will explore remote compiler caching strategies (further optimizations to be added in future): -### Remote Compiler Caching +### Remote Compiler Caching Strategies - **Compiler cache Integration**: Leverage compiler cache service like [sccache](https://github.com/mozilla/sccache) in the build-base container to cache compilation results for Dynamo, NIXL, and backend dependencies. - **Remote Cache Storage**: Use a remote cache storage service to store the cached compilation results in CI/CD pipelines. - **Cache Size Management**: Configure appropriate cache size limits and cleanup policies to balance storage usage with build performance. From cf2cf59cad0d21b6128a59204c1229581161b13f Mon Sep 17 00:00:00 2001 From: Tushar Sharma <tusharma@nvidia.com> Date: Fri, 19 Sep 2025 13:35:14 -0700 Subject: [PATCH 41/42] Remove ci_minimum stage from header --- deps/NNNN-container-strategy.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/deps/NNNN-container-strategy.md b/deps/NNNN-container-strategy.md index fdb90b1..09897f2 100644 --- a/deps/NNNN-container-strategy.md +++ b/deps/NNNN-container-strategy.md @@ -101,7 +101,7 @@ In order to address the requirements, we propose the following changes to the Dy The base container will be a pre-built container that will be used by the backends to build the final container image. This build base container will contain a Dynamo build for all backends to use for their framework-specific build. The base image will leverage a manylinux base image to enable support for multiple distributions (U22, U24, etc). The container will also include a NIXL build since this is common across all backends. This will be a new Dockerfile in the /containers directory. Multi-arch support is also a P0 Also, the base container can be used as a drop-in replacement for the current devel container used in public CI. This would significantly reduce public CI build times and enable faster validation of changes. -## Use-case of build stages along with relationship between stages (base, runtime, devel, ci_minimum) +## Use-case of build stages along with relationship between stages (base, runtime, dev) Each backend-specific Dockerfile should follow a specific format. The backend-specific Dockerfiles should be divided up into multiple stages, with each stage inheriting artifacts/leveraging the previous stage as the base container. The following stages should be defined in the backend-specific Dockerfile: From de448a242a0ff3112322c819e1ef9f9779acabe8 Mon Sep 17 00:00:00 2001 From: Tushar Sharma <tusharma@nvidia.com> Date: Thu, 25 Sep 2025 13:51:35 -0700 Subject: [PATCH 42/42] add dep id to container strategy DEP Signed-off-by: Tushar Sharma <tusharma@nvidia.com> --- deps/{NNNN-container-strategy.md => 0010-container-strategy.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename deps/{NNNN-container-strategy.md => 0010-container-strategy.md} (100%) diff --git a/deps/NNNN-container-strategy.md b/deps/0010-container-strategy.md similarity index 100% rename from deps/NNNN-container-strategy.md rename to deps/0010-container-strategy.md