Add ability to download extra packages when building a Swift SDK

[xtremekforever]

Right now, if you want to create a Swift SDK with other dependencies included, as shown in the README, you must use a container to install the dependency you need. However, this creates larger Swift SDKs, and is not always desirable.

I suggest adding a new option to download additional or extra packages when building a Swift SDK, like this:

swift run swift-sdk-generator make-linux-sdk --extra-packages "libsqlite3-dev"

To pass multiples, they could be separated by a space:

swift run swift-sdk-generator make-linux-sdk --extra-packages "libsqlite3-dev libsystemd-dev my-other-package"

There's a possibility this could also work for building Swift SDKs when using a container too, if an apt-get install <extra-packages> command is added to the container before exporting the files to generate the Swift SDK.

[euanh]

There's a possibility this could also work for building Swift SDKs when using a container too, if an apt-get install command is added to the container before exporting the files to generate the Swift SDK.

I'd prefer not to do this in the generator. The generator used to write a Dockerfile and then shell out to a container runtime to build it. It was possible to pass extra package names to the apt-get install line in the Dockerfile, but it was quite limited - often you also want to run other commands while building the container image, and that was not supported. In the general case we would need to accept any arbitrary Dockerfile and build it, but there's no particular benefit to the generator doing that - you could just build the container in a separate step.

We switched to a model where the generator expects to consume a container image which already exists. This gives you the full generality of writing your own Dockerfile to construct the image, and the generator is only responsible for extracting the relevant files and turning them into an SDK.

Since we changed to this model, we have also been reducing the need to start the container image as part of the SDK build process. If we can remove that completely, SDK builds will be faster and it will be possible to build an SDK on a machine with a different processor architecture without needing binfmt_misc or Rosetta to run foreign binaries.

Finally, I hope to remove the dependency on a container runtime entirely, by using some of the things we have learned from https://github.com/apple/swift-container-plugin to extract the container image from first principles in Swift.

[euanh]

Right now, if you want to create a Swift SDK with other dependencies included, as shown in the README, you must use a container to install the dependency you need. However, this creates larger Swift SDKs, and is not always desirable.

I suggest adding a new option to download additional or extra packages when building a Swift SDK, like this:

swift run swift-sdk-generator make-linux-sdk --extra-packages "libsqlite3-dev"

At the moment we don't have a .deb dependency solver, so the user would have to provide all the prerequisites in the --extra-packages list. This also doesn't help users who want to build SDKs from distributions using RPM or other packaging formats.

I think I prefer the idea of #186. Many distributions have some sort of sysroot creation tool, so we could expand our distribution support while also allowing the user fine-grained control over what goes into the SDK. This would be similar to the container-based model - build the sysroot separately, then use the generator to turn it into an SDK.

We could have tutorials or recipes in the documentation explaining how the process works for the major distribution types.

[xtremekforever]

At the moment we don't have a .deb dependency solver, so the user would have to provide all the prerequisites in the --extra-packages list. This also doesn't help users who want to build SDKs from distributions using RPM or other packaging formats.

Yeah that's the idea, where the user has the option of passing extra packages via that param. And it could even be done to where the packages can be installed in the container for RPM-based distros for you, so you don't have to build your own container if you're just wanting additional packages and no other customizations. But, just an idea...