From c107545a564d4307ce33832c08ffc184deb5c9e3 Mon Sep 17 00:00:00 2001 From: Dianna Ma Date: Sun, 2 Feb 2025 00:45:10 +0000 Subject: [PATCH 01/12] Replaceable Library Plugins --- proposals/0461-replaceable-library-plugins.md | 154 ++++++++++++++++++ 1 file changed, 154 insertions(+) create mode 100644 proposals/0461-replaceable-library-plugins.md diff --git a/proposals/0461-replaceable-library-plugins.md b/proposals/0461-replaceable-library-plugins.md new file mode 100644 index 0000000000..10b3603c35 --- /dev/null +++ b/proposals/0461-replaceable-library-plugins.md @@ -0,0 +1,154 @@ +# Replaceable Library Plugins + +* Proposal: [SE-0461](0641-replaceable-library-plugins.md) +* Authors: [tayloraswift](https://github.com/tayloraswift) +* Review Manager: TBD +* Implementation: [swiftlang/swift-package-manager#8249](https://github.com/swiftlang/swift-package-manager/pull/8249) +* Bugs: [SR-5714](https://github.com/swiftlang/swift-package-manager/issues/5714) + +## Introduction + +SwiftPM currently has no support for non-system binary library dependencies on Linux. This proposal adds support for **Replaceable Library Plugins**, which are a type of dynamic library that is shared across a fleet of machines and can be upgraded without recompiling and redeploying all applications running on those machines. We will distribute Replaceable Library Plugins through the existing `.artifactbundle` format. + +Swift-evolution thread: [Discussion thread topic for that +proposal](https://forums.swift.org/) + + +## Motivation + +Many of us in the Server World have a Big App with a small component that changes very rapidly, much more rapidly than the rest of the App. This component might be something like a filter, or an algorithm, or a plugin that is being constantly tuned. + +We could, for argument’s sake, try and turn this component into data that can be consumed by the Big App, which would probably involve designing a bytecode and an interpreter, and maybe even a whole interpreted domain-specific programming language. But that is Really Hard and we would rather just write this thing In Swift, and let Swift code call Swift code. + +While macOS has Dynamic Library support through XCFrameworks, on Linux we currently have to recompile the Big App from source and redeploy the Big App every time the filter changes, and we don’t want to do that. What we really want instead is to have the Big App link the filter as a Dynamic Library, and redeploy the Dynamic Library as needed. + + +## Proposed solution + +On Linux, there are a lot of obstacles to having fully general support for Dynamic Libraries. Swift is not ABI stable on Linux, and Linux itself is not a single platform but a wide range of similar platforms that provide few binary compatibility guarantees. This means it is pretty much impossible for a public Swift library to vend precompiled binaries that will Just Work for everyone, and we are not going to try to solve that problem in this proposal. + +Instead, we will focus on **Replaceable Library Plugins** (RLPs). We choose this term to emphasize the distinction between our use case and fully general Dynamic Libraries. + +### Organization-Defined Platforms (ODPs) + +Unlike fully general Dynamic Libraries, you would distribute Replaceable Library Plugins strictly for internal consumption within an organization, or to a small set of paying clients. + +The organization that distributes an RLP is responsible for defining what exactly constitutes a “platform” for their purposes. An Organization-Defined Platform (ODP) is not necessarily an operating system or architecture, or even a specific distribution of an operating system. A trivial example of two ODPs might be: + +1. Ubuntu 24.04 with the Swift 6.0.3 runtime installed at `/home/ubuntu/swift` +2. Ubuntu 24.04 with the Swift 6.0.3 runtime installed at `/home/ubuntu/swift-runtime` + +Concepts like Platform Triples are not sufficient to describe an ODP. Even though both ODPs above would probably share the Triple `aarch64-unknown-linux-gnu`, code compiled for one would never be able to run on the other. + +Organizations add and remove ODPs as needed, and trying to define a global registry of all possible ODPs is a non-goal. + +To keep things simple, we identify ODPs by the URL of the Artifact Bundle that contains the RLP. + +### Creating RLPs + +To compile an RLP, you just need to build an ordinary SwiftPM library product with the `-enable-library-evolution` flag. This requires no modifications to SwiftPM. + +You would package an RLP as an `.artifactbundle` just as you would an executable, with the following differences: + +- The `info.json` must have `schemaVersion` set to `1.2` or higher. +- The artifact type must be `library`, a new enum case introduced in this proposal. +- The artifact must have exactly one variant in the `variants` list, and the `supportedTriples` field is forbidden. +- The artifact payload must include the `.swiftinterface` file corresponding to the actual library object. + +Because SwiftPM is not (and cannot be) aware of a particular organization’s ODPs, this enforces the requirement that each ODP must have its own Artifact Bundle. + +The organization that distributes the RLP is responsible for upholding ABI stability guarantees, including the exact Swift compiler and runtime versions needed to safely consume the RLP. + + +### Consuming RLPs + +To consume an RLP, you would add a `binaryTarget` to your `Package.swift` manifest, just as you would for an executable. Because ODPs are identified by the URL of the Artifact Bundle, there are no new fields in the `PackageDescription` API. + +We expect that the logic for selecting the correct RLP for a given ODP would live within the `Package.swift` file, that it would be highly organization-specific, and that it would be manipulated using existing means such as environment variables. + + +### Deploying RLPs + +Deploying RLPs does not involve SwiftPM or Artifact Bundles at all. You would deploy an RLP by copying the latest binaries to the appropriate `@rpath` location on each machine in your fleet. The `@rpath` location is part of the ODP definition, and is not modeled by SwiftPM. + +Some organizations might choose to forgo the `@rpath` mechanism entirely and simply install the RLPs in a system-wide location. + + +## Detailed design + +### Schema extensions + +We will extend the `ArtifactsArchiveMetadata` schema to include a new `library` case in the `ArtifactType` enum. + +```diff +public enum ArtifactType: String, RawRepresentable, Decodable { + case executable ++ case library + case swiftSDK +} +``` + +This also bumps the latest `schemaVersion` to `1.2`. + + +### Artifact Bundle layout + +Below is an example of an `info.json` file for an Artifact Bundle containing a single library called `MyLibrary`. + +```json +{ + "schemaVersion": "1.2", + "artifacts": { + "MyLibrary": { + "type": "library", + "version": "1.0.0", + "variants": [{ "path": "MyLibrary" }] + } + } +} +``` + +The artifact must have exactly one variant in the `variants` list, and the `supportedTriples` field is forbidden. An RLP Artifact Bundle can contain multiple libraries at the top level. + +Below is an example of the layout of an Artifact Bundle containing a single library called `MyLibrary`. Only the `info.json` must appear at the root of the Artifact Bundle; all other files can appear at whatever paths are defined in the `info.json`, as long as they are within the Artifact Bundle. + +```text +📂 example.artifactbundle + 📂 MyLibrary + ⚙️ libMyLibrary.so + 📝 MyLibrary.swiftinterface + 📝 info.json +``` + +A macOS Artifact Bundle would contain a `.dylib` instead of a `.so`. RLPs will be supported on macOS, although we expect this will be an exceedingly rare use case. + + +## Security + +RLPs are not intended for public distribution, and are not subject to the same security concerns as public libraries. Organizations that distribute RLPs are responsible for ensuring that the RLPs are safe to consume. + + +## Impact on existing packages + +There will be no impact on existing packages. All Artifact Bundle schema changes are additive. + + +## Alternatives considered + +### Extending Platform Triples to model ODPs + +SwiftPM currently uses Platform Triples to select among artifact variants when consuming executables. This is workable because it is usually feasible to build executables that are portable across the range of platforms encompassed by a single Platform Triple. + +We could extend Platform Triples to model ODPs, but this would privilege a narrow set of predefined deployment architectures, and if you wanted to add a new ODP, you would have to modify SwiftPM to teach it to recognize the new ODP. + +### Supporting multiple variants of an RLP in the same Artifact Bundle + +We could allow an Artifact Bundle to contain multiple variants of an RLP, but we would still need to support a way to identify those variants, which in practice makes SwiftPM aware of ODPs. + +We also don’t see much value in this feature, as you would probably package and upload RLPs using one CI/CD workflow per ODP anyway. Combining artifacts would require some kind of synchronization mechanism to await all pipelines before fetching and merging bundles. + +One benefit of merging bundles would be that it reduces the number of checksums you need to keep track of, but we expect that most organizations will have a very small number of ODPs, with new ODPs continously phasing out old ODPs. + +### Using a different `ArtifactType` name besides `library` + +We intentionally preserved the structure of the `variants` list in the `info.json` file, despite imposing the current restriction of one variant per library, in order to allow this format to be extended in the future to support fully general Dynamic Libraries. From f3a13de34a3b2b8c957630e2f30a38ef4cad67b5 Mon Sep 17 00:00:00 2001 From: Dianna Ma Date: Sun, 2 Feb 2025 00:56:25 +0000 Subject: [PATCH 02/12] links to example projects --- proposals/0461-replaceable-library-plugins.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/proposals/0461-replaceable-library-plugins.md b/proposals/0461-replaceable-library-plugins.md index 10b3603c35..6b952b95ae 100644 --- a/proposals/0461-replaceable-library-plugins.md +++ b/proposals/0461-replaceable-library-plugins.md @@ -13,12 +13,15 @@ SwiftPM currently has no support for non-system binary library dependencies on L Swift-evolution thread: [Discussion thread topic for that proposal](https://forums.swift.org/) +Example Producer: [swift-rlp-example](https://github.com/tayloraswift/swift-rlp-example) + +Example Consumer: [swift-rlp-example-client](https://github.com/tayloraswift/swift-rlp-example-client) ## Motivation Many of us in the Server World have a Big App with a small component that changes very rapidly, much more rapidly than the rest of the App. This component might be something like a filter, or an algorithm, or a plugin that is being constantly tuned. -We could, for argument’s sake, try and turn this component into data that can be consumed by the Big App, which would probably involve designing a bytecode and an interpreter, and maybe even a whole interpreted domain-specific programming language. But that is Really Hard and we would rather just write this thing In Swift, and let Swift code call Swift code. +We could, for argument’s sake, try and turn this component into data that can be consumed by the Big App, which would probably involve designing a bytecode and an interpreter, and maybe even a whole interpreted domain-specific programming language. But that is very hard and we would rather just write this thing in Swift, and let Swift code call Swift code. While macOS has Dynamic Library support through XCFrameworks, on Linux we currently have to recompile the Big App from source and redeploy the Big App every time the filter changes, and we don’t want to do that. What we really want instead is to have the Big App link the filter as a Dynamic Library, and redeploy the Dynamic Library as needed. @@ -38,7 +41,7 @@ The organization that distributes an RLP is responsible for defining what exactl 1. Ubuntu 24.04 with the Swift 6.0.3 runtime installed at `/home/ubuntu/swift` 2. Ubuntu 24.04 with the Swift 6.0.3 runtime installed at `/home/ubuntu/swift-runtime` -Concepts like Platform Triples are not sufficient to describe an ODP. Even though both ODPs above would probably share the Triple `aarch64-unknown-linux-gnu`, code compiled for one would never be able to run on the other. +Concepts like Platform Triples are not sufficient to describe an ODP. Even though both ODPs above would probably share the Triple `aarch64-unknown-linux-gnu`, Swift code compiled (without `--static-swift-stdlib`) for one would never be able to run on the other. Organizations add and remove ODPs as needed, and trying to define a global registry of all possible ODPs is a non-goal. From 686a8c96f0121cd183444815390985045b1086ed Mon Sep 17 00:00:00 2001 From: Dianna Ma Date: Sun, 2 Feb 2025 01:11:11 +0000 Subject: [PATCH 03/12] link to Forums thread --- proposals/0461-replaceable-library-plugins.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/proposals/0461-replaceable-library-plugins.md b/proposals/0461-replaceable-library-plugins.md index 6b952b95ae..a89ccffc52 100644 --- a/proposals/0461-replaceable-library-plugins.md +++ b/proposals/0461-replaceable-library-plugins.md @@ -10,8 +10,7 @@ SwiftPM currently has no support for non-system binary library dependencies on Linux. This proposal adds support for **Replaceable Library Plugins**, which are a type of dynamic library that is shared across a fleet of machines and can be upgraded without recompiling and redeploying all applications running on those machines. We will distribute Replaceable Library Plugins through the existing `.artifactbundle` format. -Swift-evolution thread: [Discussion thread topic for that -proposal](https://forums.swift.org/) +Swift-evolution thread: [Discussion thread](https://forums.swift.org/t/pitch-replaceable-library-plugins/77605) Example Producer: [swift-rlp-example](https://github.com/tayloraswift/swift-rlp-example) From b02a4ac7241a110e46e0273a81631b575162f7c8 Mon Sep 17 00:00:00 2001 From: Dianna Date: Sat, 1 Feb 2025 21:02:44 -0600 Subject: [PATCH 04/12] unassigned number --- ...e-library-plugins.md => NNNN-replaceable-library-plugins.md} | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename proposals/{0461-replaceable-library-plugins.md => NNNN-replaceable-library-plugins.md} (99%) diff --git a/proposals/0461-replaceable-library-plugins.md b/proposals/NNNN-replaceable-library-plugins.md similarity index 99% rename from proposals/0461-replaceable-library-plugins.md rename to proposals/NNNN-replaceable-library-plugins.md index a89ccffc52..35f2d9e1c3 100644 --- a/proposals/0461-replaceable-library-plugins.md +++ b/proposals/NNNN-replaceable-library-plugins.md @@ -1,6 +1,6 @@ # Replaceable Library Plugins -* Proposal: [SE-0461](0641-replaceable-library-plugins.md) +* Proposal: [SE-NNNN](NNNN-replaceable-library-plugins.md) * Authors: [tayloraswift](https://github.com/tayloraswift) * Review Manager: TBD * Implementation: [swiftlang/swift-package-manager#8249](https://github.com/swiftlang/swift-package-manager/pull/8249) From f30047248257f74ac8a4aae61a6dd5c04ef3b519 Mon Sep 17 00:00:00 2001 From: Dianna Ma Date: Tue, 25 Feb 2025 01:46:13 +0000 Subject: [PATCH 05/12] replace term RLP with EDSL --- ...environment-dependent-shared-libraries.md} | 46 +++++++++---------- 1 file changed, 23 insertions(+), 23 deletions(-) rename proposals/{NNNN-replaceable-library-plugins.md => NNNN-environment-dependent-shared-libraries.md} (63%) diff --git a/proposals/NNNN-replaceable-library-plugins.md b/proposals/NNNN-environment-dependent-shared-libraries.md similarity index 63% rename from proposals/NNNN-replaceable-library-plugins.md rename to proposals/NNNN-environment-dependent-shared-libraries.md index 35f2d9e1c3..e7a0569ab5 100644 --- a/proposals/NNNN-replaceable-library-plugins.md +++ b/proposals/NNNN-environment-dependent-shared-libraries.md @@ -1,6 +1,6 @@ -# Replaceable Library Plugins +# Environment Dependent Shared Libraries -* Proposal: [SE-NNNN](NNNN-replaceable-library-plugins.md) +* Proposal: [SE-NNNN](NNNN-environment-dependent-shared-libraries.md) * Authors: [tayloraswift](https://github.com/tayloraswift) * Review Manager: TBD * Implementation: [swiftlang/swift-package-manager#8249](https://github.com/swiftlang/swift-package-manager/pull/8249) @@ -8,7 +8,7 @@ ## Introduction -SwiftPM currently has no support for non-system binary library dependencies on Linux. This proposal adds support for **Replaceable Library Plugins**, which are a type of dynamic library that is shared across a fleet of machines and can be upgraded without recompiling and redeploying all applications running on those machines. We will distribute Replaceable Library Plugins through the existing `.artifactbundle` format. +SwiftPM currently has no support for non-system binary library dependencies on Linux. This proposal adds support for **Environment Dependent Shared Libraries**, which are a type of dynamic library that is shared across a fleet of machines and can be upgraded without recompiling and redeploying all applications running on those machines. We will distribute Environment Dependent Shared Libraries through the existing `.artifactbundle` format. Swift-evolution thread: [Discussion thread](https://forums.swift.org/t/pitch-replaceable-library-plugins/77605) @@ -29,13 +29,13 @@ While macOS has Dynamic Library support through XCFrameworks, on Linux we curren On Linux, there are a lot of obstacles to having fully general support for Dynamic Libraries. Swift is not ABI stable on Linux, and Linux itself is not a single platform but a wide range of similar platforms that provide few binary compatibility guarantees. This means it is pretty much impossible for a public Swift library to vend precompiled binaries that will Just Work for everyone, and we are not going to try to solve that problem in this proposal. -Instead, we will focus on **Replaceable Library Plugins** (RLPs). We choose this term to emphasize the distinction between our use case and fully general Dynamic Libraries. +Instead, we will focus on **Environment Dependent Shared Libraries** (EDSLs). We choose this term to emphasize the distinction between our use case and fully general Dynamic Libraries. ### Organization-Defined Platforms (ODPs) -Unlike fully general Dynamic Libraries, you would distribute Replaceable Library Plugins strictly for internal consumption within an organization, or to a small set of paying clients. +Unlike fully general Dynamic Libraries, you would distribute Environment Dependent Shared Libraries strictly for internal consumption within an organization, or to a small set of paying clients. -The organization that distributes an RLP is responsible for defining what exactly constitutes a “platform” for their purposes. An Organization-Defined Platform (ODP) is not necessarily an operating system or architecture, or even a specific distribution of an operating system. A trivial example of two ODPs might be: +The organization that distributes an EDSL is responsible for defining what exactly constitutes a “platform” for their purposes. An Organization-Defined Platform (ODP) is not necessarily an operating system or architecture, or even a specific distribution of an operating system. A trivial example of two ODPs might be: 1. Ubuntu 24.04 with the Swift 6.0.3 runtime installed at `/home/ubuntu/swift` 2. Ubuntu 24.04 with the Swift 6.0.3 runtime installed at `/home/ubuntu/swift-runtime` @@ -44,13 +44,13 @@ Concepts like Platform Triples are not sufficient to describe an ODP. Even thoug Organizations add and remove ODPs as needed, and trying to define a global registry of all possible ODPs is a non-goal. -To keep things simple, we identify ODPs by the URL of the Artifact Bundle that contains the RLP. +To keep things simple, we identify ODPs by the URL of the Artifact Bundle that contains the EDSL. -### Creating RLPs +### Creating EDSLs -To compile an RLP, you just need to build an ordinary SwiftPM library product with the `-enable-library-evolution` flag. This requires no modifications to SwiftPM. +To compile an EDSL, you just need to build an ordinary SwiftPM library product with the `-enable-library-evolution` flag. This requires no modifications to SwiftPM. -You would package an RLP as an `.artifactbundle` just as you would an executable, with the following differences: +You would package an EDSL as an `.artifactbundle` just as you would an executable, with the following differences: - The `info.json` must have `schemaVersion` set to `1.2` or higher. - The artifact type must be `library`, a new enum case introduced in this proposal. @@ -59,21 +59,21 @@ You would package an RLP as an `.artifactbundle` just as you would an executable Because SwiftPM is not (and cannot be) aware of a particular organization’s ODPs, this enforces the requirement that each ODP must have its own Artifact Bundle. -The organization that distributes the RLP is responsible for upholding ABI stability guarantees, including the exact Swift compiler and runtime versions needed to safely consume the RLP. +The organization that distributes the EDSL is responsible for upholding ABI stability guarantees, including the exact Swift compiler and runtime versions needed to safely consume the EDSL. -### Consuming RLPs +### Consuming EDSLs -To consume an RLP, you would add a `binaryTarget` to your `Package.swift` manifest, just as you would for an executable. Because ODPs are identified by the URL of the Artifact Bundle, there are no new fields in the `PackageDescription` API. +To consume an EDSL, you would add a `binaryTarget` to your `Package.swift` manifest, just as you would for an executable. Because ODPs are identified by the URL of the Artifact Bundle, there are no new fields in the `PackageDescription` API. -We expect that the logic for selecting the correct RLP for a given ODP would live within the `Package.swift` file, that it would be highly organization-specific, and that it would be manipulated using existing means such as environment variables. +We expect that the logic for selecting the correct EDSL for a given ODP would live within the `Package.swift` file, that it would be highly organization-specific, and that it would be manipulated using existing means such as environment variables. -### Deploying RLPs +### Deploying EDSLs -Deploying RLPs does not involve SwiftPM or Artifact Bundles at all. You would deploy an RLP by copying the latest binaries to the appropriate `@rpath` location on each machine in your fleet. The `@rpath` location is part of the ODP definition, and is not modeled by SwiftPM. +Deploying EDSLs does not involve SwiftPM or Artifact Bundles at all. You would deploy an EDSL by copying the latest binaries to the appropriate `@rpath` location on each machine in your fleet. The `@rpath` location is part of the ODP definition, and is not modeled by SwiftPM. -Some organizations might choose to forgo the `@rpath` mechanism entirely and simply install the RLPs in a system-wide location. +Some organizations might choose to forgo the `@rpath` mechanism entirely and simply install the EDSLs in a system-wide location. ## Detailed design @@ -110,7 +110,7 @@ Below is an example of an `info.json` file for an Artifact Bundle containing a s } ``` -The artifact must have exactly one variant in the `variants` list, and the `supportedTriples` field is forbidden. An RLP Artifact Bundle can contain multiple libraries at the top level. +The artifact must have exactly one variant in the `variants` list, and the `supportedTriples` field is forbidden. An EDSL Artifact Bundle can contain multiple libraries at the top level. Below is an example of the layout of an Artifact Bundle containing a single library called `MyLibrary`. Only the `info.json` must appear at the root of the Artifact Bundle; all other files can appear at whatever paths are defined in the `info.json`, as long as they are within the Artifact Bundle. @@ -122,12 +122,12 @@ Below is an example of the layout of an Artifact Bundle containing a single libr 📝 info.json ``` -A macOS Artifact Bundle would contain a `.dylib` instead of a `.so`. RLPs will be supported on macOS, although we expect this will be an exceedingly rare use case. +A macOS Artifact Bundle would contain a `.dylib` instead of a `.so`. EDSLs will be supported on macOS, although we expect this will be an exceedingly rare use case. ## Security -RLPs are not intended for public distribution, and are not subject to the same security concerns as public libraries. Organizations that distribute RLPs are responsible for ensuring that the RLPs are safe to consume. +EDSLs are not intended for public distribution, and are not subject to the same security concerns as public libraries. Organizations that distribute EDSLs are responsible for ensuring that the EDSLs are safe to consume. ## Impact on existing packages @@ -143,11 +143,11 @@ SwiftPM currently uses Platform Triples to select among artifact variants when c We could extend Platform Triples to model ODPs, but this would privilege a narrow set of predefined deployment architectures, and if you wanted to add a new ODP, you would have to modify SwiftPM to teach it to recognize the new ODP. -### Supporting multiple variants of an RLP in the same Artifact Bundle +### Supporting multiple variants of an EDSL in the same Artifact Bundle -We could allow an Artifact Bundle to contain multiple variants of an RLP, but we would still need to support a way to identify those variants, which in practice makes SwiftPM aware of ODPs. +We could allow an Artifact Bundle to contain multiple variants of an EDSL, but we would still need to support a way to identify those variants, which in practice makes SwiftPM aware of ODPs. -We also don’t see much value in this feature, as you would probably package and upload RLPs using one CI/CD workflow per ODP anyway. Combining artifacts would require some kind of synchronization mechanism to await all pipelines before fetching and merging bundles. +We also don’t see much value in this feature, as you would probably package and upload EDSLs using one CI/CD workflow per ODP anyway. Combining artifacts would require some kind of synchronization mechanism to await all pipelines before fetching and merging bundles. One benefit of merging bundles would be that it reduces the number of checksums you need to keep track of, but we expect that most organizations will have a very small number of ODPs, with new ODPs continously phasing out old ODPs. From 45b3f89a19db3f91db192288473e1ff1cdda64bc Mon Sep 17 00:00:00 2001 From: Dianna Ma Date: Wed, 5 Mar 2025 00:43:17 +0000 Subject: [PATCH 06/12] add documentation link --- proposals/NNNN-environment-dependent-shared-libraries.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/proposals/NNNN-environment-dependent-shared-libraries.md b/proposals/NNNN-environment-dependent-shared-libraries.md index e7a0569ab5..6a6cdbe8e9 100644 --- a/proposals/NNNN-environment-dependent-shared-libraries.md +++ b/proposals/NNNN-environment-dependent-shared-libraries.md @@ -4,6 +4,7 @@ * Authors: [tayloraswift](https://github.com/tayloraswift) * Review Manager: TBD * Implementation: [swiftlang/swift-package-manager#8249](https://github.com/swiftlang/swift-package-manager/pull/8249) +* Documentation: [How to use Environment-Dependent Shared Libraries](https://github.com/tayloraswift/swift-edsl-example-client/blob/master/Sources/KrustyKrab/docs.docc/Getting%20Started.md) * Bugs: [SR-5714](https://github.com/swiftlang/swift-package-manager/issues/5714) ## Introduction @@ -12,9 +13,9 @@ SwiftPM currently has no support for non-system binary library dependencies on L Swift-evolution thread: [Discussion thread](https://forums.swift.org/t/pitch-replaceable-library-plugins/77605) -Example Producer: [swift-rlp-example](https://github.com/tayloraswift/swift-rlp-example) +Example Producer: [swift-edsl-example](https://github.com/tayloraswift/swift-edsl-example) -Example Consumer: [swift-rlp-example-client](https://github.com/tayloraswift/swift-rlp-example-client) +Example Consumer: [swift-edsl-example-client](https://github.com/tayloraswift/swift-edsl-example-client) ## Motivation From 47d96fc2add76b8ae808790a75706a1c22f4fe6d Mon Sep 17 00:00:00 2001 From: Dianna Ma Date: Fri, 7 Mar 2025 00:42:27 +0000 Subject: [PATCH 07/12] update link --- proposals/NNNN-environment-dependent-shared-libraries.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/NNNN-environment-dependent-shared-libraries.md b/proposals/NNNN-environment-dependent-shared-libraries.md index 6a6cdbe8e9..d1068526ea 100644 --- a/proposals/NNNN-environment-dependent-shared-libraries.md +++ b/proposals/NNNN-environment-dependent-shared-libraries.md @@ -4,7 +4,7 @@ * Authors: [tayloraswift](https://github.com/tayloraswift) * Review Manager: TBD * Implementation: [swiftlang/swift-package-manager#8249](https://github.com/swiftlang/swift-package-manager/pull/8249) -* Documentation: [How to use Environment-Dependent Shared Libraries](https://github.com/tayloraswift/swift-edsl-example-client/blob/master/Sources/KrustyKrab/docs.docc/Getting%20Started.md) +* Documentation: [How to use Environment-Dependent Shared Libraries](https://github.com/swiftlang/swift-package-manager/blob/b586467575580f2365e8f5a29c949379724db795/Documentation/EDSLs.md) * Bugs: [SR-5714](https://github.com/swiftlang/swift-package-manager/issues/5714) ## Introduction From e95300cb1c50c88f189fcbc85837014a7906b933 Mon Sep 17 00:00:00 2001 From: Dianna Ma Date: Thu, 10 Jul 2025 18:20:38 +0000 Subject: [PATCH 08/12] revise proposal --- ...-environment-dependent-shared-libraries.md | 73 ++++++++++--------- 1 file changed, 38 insertions(+), 35 deletions(-) diff --git a/proposals/NNNN-environment-dependent-shared-libraries.md b/proposals/NNNN-environment-dependent-shared-libraries.md index d1068526ea..5c7a228bf9 100644 --- a/proposals/NNNN-environment-dependent-shared-libraries.md +++ b/proposals/NNNN-environment-dependent-shared-libraries.md @@ -1,21 +1,21 @@ -# Environment Dependent Shared Libraries +# Environment Constrained Shared Libraries -* Proposal: [SE-NNNN](NNNN-environment-dependent-shared-libraries.md) +* Proposal: [SE-NNNN](NNNN-environment-constrained-shared-libraries.md) * Authors: [tayloraswift](https://github.com/tayloraswift) * Review Manager: TBD * Implementation: [swiftlang/swift-package-manager#8249](https://github.com/swiftlang/swift-package-manager/pull/8249) -* Documentation: [How to use Environment-Dependent Shared Libraries](https://github.com/swiftlang/swift-package-manager/blob/b586467575580f2365e8f5a29c949379724db795/Documentation/EDSLs.md) +* Documentation: [How to use Environment-Constrained Shared Libraries](https://github.com/swiftlang/swift-package-manager/blob/b586467575580f2365e8f5a29c949379724db795/Documentation/ECSLs.md) * Bugs: [SR-5714](https://github.com/swiftlang/swift-package-manager/issues/5714) ## Introduction -SwiftPM currently has no support for non-system binary library dependencies on Linux. This proposal adds support for **Environment Dependent Shared Libraries**, which are a type of dynamic library that is shared across a fleet of machines and can be upgraded without recompiling and redeploying all applications running on those machines. We will distribute Environment Dependent Shared Libraries through the existing `.artifactbundle` format. +SwiftPM currently has no support for non-system binary library dependencies on Linux. This proposal adds support for **Environment Constrained Shared Libraries**, which are a type of dynamic library that is shared across a fleet of machines and can be upgraded without recompiling and redeploying all applications running on those machines. We will distribute Environment Constrained Shared Libraries through the existing `.artifactbundle` format. Swift-evolution thread: [Discussion thread](https://forums.swift.org/t/pitch-replaceable-library-plugins/77605) -Example Producer: [swift-edsl-example](https://github.com/tayloraswift/swift-edsl-example) +Example Producer: [swift-ECSL-example](https://github.com/tayloraswift/swift-ECSL-example) -Example Consumer: [swift-edsl-example-client](https://github.com/tayloraswift/swift-edsl-example-client) +Example Consumer: [swift-ECSL-example-client](https://github.com/tayloraswift/swift-ECSL-example-client) ## Motivation @@ -30,51 +30,54 @@ While macOS has Dynamic Library support through XCFrameworks, on Linux we curren On Linux, there are a lot of obstacles to having fully general support for Dynamic Libraries. Swift is not ABI stable on Linux, and Linux itself is not a single platform but a wide range of similar platforms that provide few binary compatibility guarantees. This means it is pretty much impossible for a public Swift library to vend precompiled binaries that will Just Work for everyone, and we are not going to try to solve that problem in this proposal. -Instead, we will focus on **Environment Dependent Shared Libraries** (EDSLs). We choose this term to emphasize the distinction between our use case and fully general Dynamic Libraries. +Instead, we will focus on **Environment Constrained Shared Libraries** (ECSLs). We choose this term to emphasize the distinction between our use case and fully general Dynamic Libraries. -### Organization-Defined Platforms (ODPs) +### Target environment -Unlike fully general Dynamic Libraries, you would distribute Environment Dependent Shared Libraries strictly for internal consumption within an organization, or to a small set of paying clients. +Unlike fully general Dynamic Libraries, you would distribute Environment Constrained Shared Libraries strictly for controlled consumption within a known environment, such as a fleet of servers maintained by a single organization. -The organization that distributes an EDSL is responsible for defining what exactly constitutes a “platform” for their purposes. An Organization-Defined Platform (ODP) is not necessarily an operating system or architecture, or even a specific distribution of an operating system. A trivial example of two ODPs might be: +ECSLs are an advanced tool, and maintaining the prerequisite environment to deploy them safely is neither trivial nor recommended for most users. -1. Ubuntu 24.04 with the Swift 6.0.3 runtime installed at `/home/ubuntu/swift` -2. Ubuntu 24.04 with the Swift 6.0.3 runtime installed at `/home/ubuntu/swift-runtime` +The organization that distributes an ECSL is responsible for defining what exactly constitutes a “platform” for their purposes. An organization-defined platform is not necessarily an operating system or architecture, or even a specific distribution of an operating system. A trivial example of two such platforms might be: -Concepts like Platform Triples are not sufficient to describe an ODP. Even though both ODPs above would probably share the Triple `aarch64-unknown-linux-gnu`, Swift code compiled (without `--static-swift-stdlib`) for one would never be able to run on the other. +1. Ubuntu 24.04 with the Swift 6.1.2 runtime installed at `/home/ubuntu/swift` +2. Ubuntu 24.04 with the Swift 6.1.2 runtime installed at `/home/ubuntu/swift-runtime` -Organizations add and remove ODPs as needed, and trying to define a global registry of all possible ODPs is a non-goal. +Concepts like Platform Triples are not sufficient to describe an ECSL deployment target. Even though both “platforms” above would probably share the Triple `aarch64-unknown-linux-gnu`, Swift code compiled (without `--static-swift-stdlib`) for one would never be able to run on the other. -To keep things simple, we identify ODPs by the URL of the Artifact Bundle that contains the EDSL. +Organizations will add and remove environments as needed, and trying to define a global registry of all possible environments is a non-goal. -### Creating EDSLs +The proposed ECSL distribution format does not support shipping multiple variants of ECSLs targeting multiple environments in the same Artifact Bundle, nor does it specify a standardized means for identifying the environment in which a particular ECSL is intended to execute in. +Users are responsible for computing the correct URL of the Artifact Bundle for the environment they are building for, possibly within the package manifest. Swift tooling will not, on its own, diagnose or prevent the installation of an incompatible ECSL. -To compile an EDSL, you just need to build an ordinary SwiftPM library product with the `-enable-library-evolution` flag. This requires no modifications to SwiftPM. +### Creating ECSLs -You would package an EDSL as an `.artifactbundle` just as you would an executable, with the following differences: +To compile an ECSL, you just need to build an ordinary SwiftPM library product with the `-enable-library-evolution` flag. This requires no modifications to SwiftPM. + +You would package an ECSL as an `.artifactbundle` just as you would an executable, with the following differences: - The `info.json` must have `schemaVersion` set to `1.2` or higher. - The artifact type must be `library`, a new enum case introduced in this proposal. - The artifact must have exactly one variant in the `variants` list, and the `supportedTriples` field is forbidden. - The artifact payload must include the `.swiftinterface` file corresponding to the actual library object. -Because SwiftPM is not (and cannot be) aware of a particular organization’s ODPs, this enforces the requirement that each ODP must have its own Artifact Bundle. +Because SwiftPM is not (and cannot be) aware of a particular organization’s set of deployment environments, this enforces the requirement that each environment must have its own Artifact Bundle. -The organization that distributes the EDSL is responsible for upholding ABI stability guarantees, including the exact Swift compiler and runtime versions needed to safely consume the EDSL. +The organization that distributes the ECSL is responsible for upholding ABI stability guarantees, including the exact Swift compiler and runtime versions needed to safely consume the ECSL. -### Consuming EDSLs +### Consuming ECSLs -To consume an EDSL, you would add a `binaryTarget` to your `Package.swift` manifest, just as you would for an executable. Because ODPs are identified by the URL of the Artifact Bundle, there are no new fields in the `PackageDescription` API. +To consume an ECSL, you would add a `binaryTarget` to your `Package.swift` manifest, just as you would for an executable. Because organizations are responsible for defining their set of supported environments, they are also responsible for defining the URLs that the Artifact Bundles for each environment are hosted under, so there are no new fields in the `PackageDescription` API. -We expect that the logic for selecting the correct EDSL for a given ODP would live within the `Package.swift` file, that it would be highly organization-specific, and that it would be manipulated using existing means such as environment variables. +We expect that the logic for selecting the correct ECSL for a given environment would live within the `Package.swift` file, that it would be highly organization-specific, and that it would be manipulated using existing means such as environment variables. -### Deploying EDSLs +### Deploying ECSLs -Deploying EDSLs does not involve SwiftPM or Artifact Bundles at all. You would deploy an EDSL by copying the latest binaries to the appropriate `@rpath` location on each machine in your fleet. The `@rpath` location is part of the ODP definition, and is not modeled by SwiftPM. +Deploying ECSLs does not involve SwiftPM or Artifact Bundles at all. You would deploy an ECSL by copying the latest binaries to the appropriate `@rpath` location on each machine in your fleet. The `@rpath` location is part of the organization-specific environment definition, and is not modeled by SwiftPM. -Some organizations might choose to forgo the `@rpath` mechanism entirely and simply install the EDSLs in a system-wide location. +Some organizations might choose to forgo the `@rpath` mechanism entirely and simply install the ECSLs in a system-wide location. ## Detailed design @@ -111,7 +114,7 @@ Below is an example of an `info.json` file for an Artifact Bundle containing a s } ``` -The artifact must have exactly one variant in the `variants` list, and the `supportedTriples` field is forbidden. An EDSL Artifact Bundle can contain multiple libraries at the top level. +The artifact must have exactly one variant in the `variants` list, and the `supportedTriples` field is forbidden. An ECSL Artifact Bundle can contain multiple libraries at the top level. Below is an example of the layout of an Artifact Bundle containing a single library called `MyLibrary`. Only the `info.json` must appear at the root of the Artifact Bundle; all other files can appear at whatever paths are defined in the `info.json`, as long as they are within the Artifact Bundle. @@ -123,12 +126,12 @@ Below is an example of the layout of an Artifact Bundle containing a single libr 📝 info.json ``` -A macOS Artifact Bundle would contain a `.dylib` instead of a `.so`. EDSLs will be supported on macOS, although we expect this will be an exceedingly rare use case. +A macOS Artifact Bundle would contain a `.dylib` instead of a `.so`. ECSLs will be supported on macOS, although we expect this will be an exceedingly rare use case. ## Security -EDSLs are not intended for public distribution, and are not subject to the same security concerns as public libraries. Organizations that distribute EDSLs are responsible for ensuring that the EDSLs are safe to consume. +ECSLs are not intended for public distribution, and are not subject to the same security concerns as public libraries. Organizations that distribute ECSLs are responsible for ensuring that the ECSLs are safe to consume. ## Impact on existing packages @@ -138,19 +141,19 @@ There will be no impact on existing packages. All Artifact Bundle schema changes ## Alternatives considered -### Extending Platform Triples to model ODPs +### Extending Platform Triples to model deployment targets SwiftPM currently uses Platform Triples to select among artifact variants when consuming executables. This is workable because it is usually feasible to build executables that are portable across the range of platforms encompassed by a single Platform Triple. -We could extend Platform Triples to model ODPs, but this would privilege a narrow set of predefined deployment architectures, and if you wanted to add a new ODP, you would have to modify SwiftPM to teach it to recognize the new ODP. +We could extend Platform Triples to model ECSL deployment targets, but this would privilege a narrow set of predefined deployment architectures, and if you wanted to add a new environment, you would have to modify SwiftPM to teach it to recognize the new environment. -### Supporting multiple variants of an EDSL in the same Artifact Bundle +### Supporting multiple variants of an ECSL in the same Artifact Bundle -We could allow an Artifact Bundle to contain multiple variants of an EDSL, but we would still need to support a way to identify those variants, which in practice makes SwiftPM aware of ODPs. +We could allow an Artifact Bundle to contain multiple variants of an ECSL, but we would still need to support a way to identify those variants, which in practice forces SwiftPM to become aware of organization-defined environments. -We also don’t see much value in this feature, as you would probably package and upload EDSLs using one CI/CD workflow per ODP anyway. Combining artifacts would require some kind of synchronization mechanism to await all pipelines before fetching and merging bundles. +We also don’t see much value in this feature, as you would probably package and upload ECSLs using one CI/CD workflow per environment anyway. Combining artifacts would require some kind of synchronization mechanism to await all pipelines before fetching and merging bundles. -One benefit of merging bundles would be that it reduces the number of checksums you need to keep track of, but we expect that most organizations will have a very small number of ODPs, with new ODPs continously phasing out old ODPs. +One benefit of merging bundles would be that it reduces the number of checksums you need to keep track of, but we expect that most organizations will have a very small number of supported environments, with new environments continously phasing out old environments. ### Using a different `ArtifactType` name besides `library` From e731f51d247ac853175f214c3b59f5b80a4c2f8d Mon Sep 17 00:00:00 2001 From: Dianna Ma Date: Thu, 10 Jul 2025 18:24:53 +0000 Subject: [PATCH 09/12] rename proposal --- ...raries.md => NNNN-environment-constrained-shared-libraries.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename proposals/{NNNN-environment-dependent-shared-libraries.md => NNNN-environment-constrained-shared-libraries.md} (100%) diff --git a/proposals/NNNN-environment-dependent-shared-libraries.md b/proposals/NNNN-environment-constrained-shared-libraries.md similarity index 100% rename from proposals/NNNN-environment-dependent-shared-libraries.md rename to proposals/NNNN-environment-constrained-shared-libraries.md From 345640f7a3d924a816a455ea21b8ba4245162514 Mon Sep 17 00:00:00 2001 From: Dianna Ma Date: Thu, 10 Jul 2025 21:07:58 +0000 Subject: [PATCH 10/12] example URLs --- proposals/NNNN-environment-constrained-shared-libraries.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/proposals/NNNN-environment-constrained-shared-libraries.md b/proposals/NNNN-environment-constrained-shared-libraries.md index 5c7a228bf9..2d963c6c57 100644 --- a/proposals/NNNN-environment-constrained-shared-libraries.md +++ b/proposals/NNNN-environment-constrained-shared-libraries.md @@ -13,9 +13,9 @@ SwiftPM currently has no support for non-system binary library dependencies on L Swift-evolution thread: [Discussion thread](https://forums.swift.org/t/pitch-replaceable-library-plugins/77605) -Example Producer: [swift-ECSL-example](https://github.com/tayloraswift/swift-ECSL-example) +Example Producer: [swift-dynamic-library-example](https://github.com/tayloraswift/swift-dynamic-library-example) -Example Consumer: [swift-ECSL-example-client](https://github.com/tayloraswift/swift-ECSL-example-client) +Example Consumer: [swift-dynamic-library-example-client](https://github.com/tayloraswift/swift-dynamic-library-example-client) ## Motivation From e3b6bb27df3f1992da04a2ae859d0ec74f5eb91e Mon Sep 17 00:00:00 2001 From: Dianna Ma Date: Thu, 10 Jul 2025 21:11:13 +0000 Subject: [PATCH 11/12] =?UTF-8?q?library=20=E2=86=92=20dynamicLibrary?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../NNNN-environment-constrained-shared-libraries.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/proposals/NNNN-environment-constrained-shared-libraries.md b/proposals/NNNN-environment-constrained-shared-libraries.md index 2d963c6c57..f2394d4d2f 100644 --- a/proposals/NNNN-environment-constrained-shared-libraries.md +++ b/proposals/NNNN-environment-constrained-shared-libraries.md @@ -57,7 +57,7 @@ To compile an ECSL, you just need to build an ordinary SwiftPM library product w You would package an ECSL as an `.artifactbundle` just as you would an executable, with the following differences: - The `info.json` must have `schemaVersion` set to `1.2` or higher. -- The artifact type must be `library`, a new enum case introduced in this proposal. +- The artifact type must be `dynamicLibrary`, a new enum case introduced in this proposal. - The artifact must have exactly one variant in the `variants` list, and the `supportedTriples` field is forbidden. - The artifact payload must include the `.swiftinterface` file corresponding to the actual library object. @@ -84,12 +84,12 @@ Some organizations might choose to forgo the `@rpath` mechanism entirely and sim ### Schema extensions -We will extend the `ArtifactsArchiveMetadata` schema to include a new `library` case in the `ArtifactType` enum. +We will extend the `ArtifactsArchiveMetadata` schema to include a new `dynamicLibrary` case in the `ArtifactType` enum. ```diff public enum ArtifactType: String, RawRepresentable, Decodable { case executable -+ case library ++ case dynamicLibrary case swiftSDK } ``` @@ -106,7 +106,7 @@ Below is an example of an `info.json` file for an Artifact Bundle containing a s "schemaVersion": "1.2", "artifacts": { "MyLibrary": { - "type": "library", + "type": "dynamicLibrary", "version": "1.0.0", "variants": [{ "path": "MyLibrary" }] } @@ -155,6 +155,6 @@ We also don’t see much value in this feature, as you would probably package an One benefit of merging bundles would be that it reduces the number of checksums you need to keep track of, but we expect that most organizations will have a very small number of supported environments, with new environments continously phasing out old environments. -### Using a different `ArtifactType` name besides `library` +### Using a different `ArtifactType` name besides `dynamicLibrary` We intentionally preserved the structure of the `variants` list in the `info.json` file, despite imposing the current restriction of one variant per library, in order to allow this format to be extended in the future to support fully general Dynamic Libraries. From e320e31d49ce6833736ab268368034f0fe68f0c6 Mon Sep 17 00:00:00 2001 From: Dianna Ma Date: Thu, 10 Jul 2025 21:13:29 +0000 Subject: [PATCH 12/12] ce --- proposals/NNNN-environment-constrained-shared-libraries.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/proposals/NNNN-environment-constrained-shared-libraries.md b/proposals/NNNN-environment-constrained-shared-libraries.md index f2394d4d2f..9a170b0058 100644 --- a/proposals/NNNN-environment-constrained-shared-libraries.md +++ b/proposals/NNNN-environment-constrained-shared-libraries.md @@ -90,6 +90,7 @@ We will extend the `ArtifactsArchiveMetadata` schema to include a new `dynamicLi public enum ArtifactType: String, RawRepresentable, Decodable { case executable + case dynamicLibrary + case staticLibrary case swiftSDK } ``` @@ -126,7 +127,7 @@ Below is an example of the layout of an Artifact Bundle containing a single libr 📝 info.json ``` -A macOS Artifact Bundle would contain a `.dylib` instead of a `.so`. ECSLs will be supported on macOS, although we expect this will be an exceedingly rare use case. +A macOS Artifact Bundle would contain a `.dylib` instead of a `.so`. ECSLs will be supported on macOS, although we expect this will be an exceedingly rare use case, as this need is already well-served by the XCFramework. ## Security