Support building phobos against a system copy of zlib

[the-horo]

Many Linux distributions including Debian, Fedora, and, Gentoo, have a policy of unbundling 3rd party dependencies like zlib from software projects. This PR implements support for this so that distributions don't have to implement the same logic individually.

Currently Debian does remove the zlib code but they leave the static variants of libphobos with undefined symbols to zlib functions making -link-defaultlib-shared=false require the additional -lz switch when specified by users. Fedora tried a patch but it resulted in the build being broken so it has been reverted. Gentoo doesn't do anything currently but I am making this PR instead.

These changes are all behind the -DPHOBOS_SYSTEM_ZLIB=TRUE cmake argument so the current code will continue to behave the same and the dependency on zlib is strictly opt-in. A possible unwanted interaction is that, when cross-compiling, ldc2 will try to link -lz just like it would when building for the build host but zlib may not exist on the target system. The only solution for this is specifying -defaultlib= -L-l:libphobos2-ldc.a -L-l:libdruntime-ldc.a which is a mouthful but it is either this or complicating the linker code more.

An implementation downside of the code is that the phobos config option is placed inside the root CMakeLists.txt because the compiler needs to know whether it needs to links zlib when building statically and, since it is built with a custom_target its target compile arguments can not be modified through something like target_compile_definitions after the runtime subdirectory has been included.

What this PR doesn't support is MULTILIB=ON and PHOBOS_SYSTEM_ZLIB=TRUE simultaneously because cmake, or any other build system that I know of, doesn't allow to specify the ABI when looking for a package. The standard way of compiling for multiple ABIs is to setup a different build directory and configuring cmake for that ABI which is done by setting PKG_CONFIG_PATH to point to the correct libdir and setting environment variables for the toolchain similar to CC="gcc -m32" CXX="g++ -m32" DMD="ldmd -m32". The current cmake code worked for this with only minor changes but I didn't dive in too deeply yet.

I've made plenty of last minute improvements that ended up breaking something so it's fine if this PR waits for the new upcoming release to happen first and then be merged.

[the-horo]

I have now seen that #4716 already existed, but compared to it, this PR correctly handles linking to static phobos and it doesn't affect the druntime libraries at the cost of slightly changing the compiler linker code.

[the-horo]

I think it would be beneficial to add a changelog entry and a CI run for this option. Is circleci the preferred runner?

[kinke]

Thanks for doing this; LGTM after a superficial glance. Yeah some CI coverage would be great; CircleCI is just kept because it doesn't hurt/get in the way; Cirrus is unfortunately not a sufficient free option anymore, so in essence it's all GitHub's own CI now, they managed to kill all competitors (for open-source projects) unfortunately (edit: and with the help of crypto-kiddies). You could e.g. select one of the jobs in https://github.com/ldc-developers/ldc/blob/master/.github/workflows/supported_llvm_versions.yml; we use them for testing some CMake permutations too.

[kinke]

Ideally, we'd bake this into the ldc-build-runtime CMake invocations too, in

ldc/runtime/ldc-build-runtime.d.in

Lines 152 to 157 in 0cc531e

	string[] args = [
	"cmake",
	"-DLDC_EXE_FULL=" ~ config.ldcExecutable,
	"-DDMDFE_MINOR_VERSION=@DMDFE_MINOR_VERSION@",
	"-DDMDFE_PATCH_VERSION=@DMDFE_PATCH_VERSION@",
	];

[kinke]

When you're revising this PR, please add a trailing full stop here.

[kinke]

I'd probably slightly rephrase this, something like 'use system zlib instead of Phobos' vendored version'.

Note that the D bindings have hardcoded zlib version numbers which might be off then: https://github.com/dlang/phobos/blob/f7e523bc3c69506c3e7b4da7e78af93ed8547910/etc/c/zlib.d#L45-L47

The actual API has been pretty stable AFAIK, so I don't think this matters in practice.

[the-horo]

I've asked in the Gentoo irc rooms what would are the appropriate version of zlib that we can use given that we hardcode the headers to a certain version and the response was that, given it's zlib, so long as we link against libz.so.1 it should be fine.

If we want to restrict the version we can do that inside the find_package call so if we end up with an incompatible version of zlib in the future we can fail during configuration. For now I think that we shouldn't overly restrict the check unless we know for certain that a zlib version doesn't work.

[the-horo]

Rebased onto master to fix the macos xcode failure.

[schveiguy]

Just had a meeting about this with DLF. I think this doesn't go far enough.

If you call any etc.c.zlib functions and try to build with dynamically linked phobos (the default on these systems), then your binary will fail to link even though phobos has the indirect dependency on zlib.

I have been running into this problem for years from my raylib-d install script, without knowing the cause!

I would suggest that instead of linking against -lz only for static builds, you link against -lz always (when the -DPHOBOS_SYSTEM_ZLIB=TRUE switch is passed). The cost is almost nothing since libphobos depends on it anyway.

[kinke]

I would suggest that instead of linking against -lz only for static builds, you link against -lz always (when the -DPHOBOS_SYSTEM_ZLIB=TRUE switch is passed). The cost is almost nothing since libphobos depends on it anyway.

Yeah that should be doable. FWIW, I didn't think anyone would really use the etc.c.zlib C bindings directly, but the std.zlib wrappers.

[schveiguy]

FWIW, I didn't think anyone would really use the etc.c.zlib C bindings directly

std.zlib is not good for streaming.

e.g. in iopipe, I have a zip and unzip iopipe which compresses or decompresses as you go. std.zip cannot do this.

in other words, std.zip does not expose well enough the capabilities of zlib.

[the-horo]

I'll write here my concerns as well. If people want to use zlib and not std.zip then they should be linking -lz. Using a transitive link dependency in your code is simply bad practice and it should, at least, not be encouraged. The most I can agree with is that, given that projects have been working so far, we should avoid directly breaking them, but I think that this should be a temporary fix that gives developers time to update their projects and not a definitive decision on whether libphobos should be exposing the C api of its dependencies.

---

If it matters adding -lz is what gdc is doing and that can be another argument for choosing such a behavior

[schveiguy]

Using a transitive link dependency in your code is simply bad practice

There is no transitive link dependency -- official phobos currently includes zlib statically. That is, phobos defines these functions, and that is expected and supported.

The non-standard case here is the decision not to include zlib in phobos. If you are going to do this, the installation should compensate for it -- by adding the necessary link directives that you normally wouldn't need.

[the-horo]

Alright, neither of us seems to be willing to change their views. Your belief is that because phobos has been compiled so far in a way that embeds zlib then it should be guaranteed to all consumers that the C functions are available as well. It is my belief that the current way phobos embeds zlib is simply a happenstance and no one actually made a conscious decision from the angle of whether the C symbols would be available to consumers, whoever made the choice simply wanted to make sure that phobos could use zlib for std.zip regardless of OS, and the easiest way for that is to build and link it.

---

But whatever the actual reason is, is outside of our beliefs. Right now what needs to happen is for -lz to be added so that we get in a consistent state between distro and official packages, for all 3 compilers. After that the two possibilities that I see are:

1. Continue adding -lz
   Which I think is dumb because:
   a) It needlessly complicates the compiler code
   b) It's not phobos' concern to expose C function
   c) Why would -lz have the same importance to linking as libc
   d) Adding libs "z" to dub.sdl is trivial
   but it is consistent to the current behavior and doesn't break old projects
2. Stop adding -lz and force users to do so themselves (if they require it)
   It is a strictly better state from my point of view but it breaks existing projects, so It would require adequate deprecation

If it matters I can try to take a look at doing this for dmd some time later. gdc, like I said above, already handles it and ldc should be pretty easy to fix, is my guess.

[schveiguy]

It's not a matter of opinion or intent. It's a matter of facts. zlib is included with phobos, including linkable symbols from etc.c. This means that a valid D install should be able to link without explicit -lz parameters.

Note that I'm not really in favor of this method of distribution for zlib. It's just, that's what we have now.

Contrast that with std.net.curl, which has no statically linkable symbols. If you use std.net.curl without providing a curl library to run against, your code will fail. That is a very different experience from std.zip.

If we did not include zlib, then users of std.zip would be required to link against -lz. I'm perfectly fine if that's the way we want to go. I've even advocated for it! But what I don't agree with is a state where some compiler distributions have made it so you cannot link applications which build just fine with other distributions of the same compiler. This leads to very puzzling bug reports, without much evidence of why it is happening. That is my experience.

This entire PR is evidence of the requirement! Without this mechanism, you had to add -lz for static builds. Now, the compiler does it for you. Why can't the compiler do this for builds against dynamic phobos?

[the-horo]

Ok, I don't think it's a good use of both of our time so I'll try to refrain from commenting much:

zlib is included with phobos

phobos needs zlib => a way to solve this is embed zlib => zlib must be embedded (or anything that leads to the symbols being visible)

This means that a valid D install should be able to link without explicit -lz parameters.

Where is this documented?

If we did not include zlib, then users of std.zip would be required to link against -lz.

This is the whole discussion. You can use std.zip without seeing the symbols of zlib. We are not arguing about users being able to use std.zip, in all current and future approaches std.zip works and it will keep working without user intervention. Our whole discussion is about the 3rd party zlib library and the bindings in etc.

But what I don't agree with is a state where some compiler distributions have made it so you cannot link applications which build just fine with other distributions of the same compiler.

Yes, the lack of a clearly stated and written documentation about whether this is should be supported is a reason for this argument.

Why can't the compiler do this for builds against dynamic phobos?

It's not a matter of if it can, it's a matter of why should it (outside of staying consistent with previous behavior) when phobos works without it.

[schveiguy]

Well, I don't see the point of hiding the symbols on purpose, when you are including them. But in any case, there are better arguments to have. I think the best path forward is to add -lz to all these adjusted packages, either through the compiler or the .ini file.

[kinke]

IIUC, to enable what Steven wants, we'd just remove the !linkAgainstSharedDefaultLibs() condition here. Meaning that if the compiler was configured with PHOBOS_SYSTEM_ZLIB, and the user hasn't cleared the -defaultlib list, it implicitly adds -lz for linking on Posix.

On Windows, we'd actually have to export all zlib symbols from the Phobos DLL in order to enable user code to use the etc.c.zlib bindings directly when using -link-defaultlib-shared.

[the-horo]

#4963

[kinke]

FWIW, I didn't think anyone would really use the etc.c.zlib C bindings directly

std.zlib is not good for streaming.

e.g. in iopipe, I have a zip and unzip iopipe which compresses or decompresses as you go. std.zip cannot do this.

in other words, std.zip does not expose well enough the capabilities of zlib.

std.zip or std.zlib? The latter has 2 helper classes for chunked (de)compression, looking suitable for streams. If insufficient, I guess that module should be extended to accomodate for such usage, so that almost noone should have to resort to etc.c.zlib (with awesome docs: https://dlang.org/phobos/etc_c_zlib.html - only the Z_NULL symbol ;)).