Cross-compilation for FreeBSD doesn't detect crypt support

[rhelmot]

On FreeBSD, libcrypt is included in the base system, so perl should always be able to find and link against its implementation of crypt. The configure script reports figuring out that it can link with -lcrypt and should be looking for the crypt symbol, and the crypt symbol is in fact in that library, but it can't figure out that it HAS_CRYPT. Possibly to do with the fact that there is no crypt.h header, but rather crypt is defined in unistd.h?

I am able to build perl with this feature successfully by applying the -Dd_crypt configure flag, but I feel like this should be able to be discovered automatically since it only involves compiling and linking.

Build logs for the bad build (from nix) can be found in perl.log.gz.

[jkeenan]

Can you provide the program you used to try to do this cross-compilation, as well as how you called it on the command-line?

Also, can you provide the characteristics (uname -mrs) of the operating system on which you were attempting this?

Thanks.

[jkeenan]

I grepped the linked build log for crypt and came up with these results:

$ zgrep -inC1 crypt gh-22295.build.log.gz 
56-Checking whether to include <sys/bsdioctl.h> ... no
57:Checking whether to include <crypt.h> ... no
58-Checking whether to include <db.h> ... no
--
288-Checking for closedir ... found
289:Checking for crypt ... missing
290-Checking for ctermid ... found
--
696-Checking whether to include <sys/bsdioctl.h> ... no
697:Checking whether to include <crypt.h> ... no
698-Checking whether to include <db.h> ... yes
--
902-Deciding whether to use DynaLoader ... yes
903:Checking which libraries are available ... -lm -lcrypt -lpthread -ldl
904:Checking which libs to use for perl ... -lm -lcrypt -lpthread -ldl
905-Deciding how to name libperl ... libperl.a (static)
--
924-Checking for closedir ... found
925:Checking for crypt ... missing
926-Checking for ctermid ... missing
--
1310-Checking for ctime_r ... found, B_SB
1311:Checking for crypt_r ... unusable
1312-Checking for ctermid_r ... found, B_B
--
2153-./miniperl_top extlibs  > ext.libs
2154:x86_64-unknown-freebsd-clang -Wl,-E -o perl perlmain.o libperl.a  -lm -lcrypt -lpthread -ldl 
2155-cd ext/Errno/ && ../../miniperl_top -I../../lib Makefile.PL \

@rhelmot, is there anything else in this log relevant to your compilation failure that we should be looking at?

[jkeenan]

For reference, here's what I get with my regular configuration command for blead on FreeBSD-13.

$ zgrep -inC1 crypt 5f29d4af0d.configure.log.gz 
62-Checking for optional libraries...
63:What libraries to use? [-lpthread -lgdbm -ldl -lm -lcrypt -lutil]  
64-What optimizer/debugger flag should be used?
--
263-copysignl() found.
264:crypt() found.
265:<crypt.h> NOT found.
266:crypt_r() found.
267-Hmm.  Based on the hints in hints/freebsd.sh, 
268:the recommended value for $d_crypt_r_proto on this machine was "define"!
269-    Keep the recommended value? [y]  
--
851-cc -c -DPERL_CORE -DHAS_FPSETMASK -DHAS_FLOATINGPOINT_H -DNO_POSIX_2008_LOCALE -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -std=c99 -O2 -pipe -fstack-protector -fno-strict-aliasing -Wall -Werror=pointer-arith -Werror=vla -Wextra -Wno-long-long -Wno-declaration-after-statement -Wc++-compat -Wwrite-strings generate_uudmap.c
852:cc -o generate_uudmap -pthread -Wl,-E  -fstack-protector-strong -L/usr/local/lib generate_uudmap.o -lpthread -ldl -lm -lcrypt -lutil
853-./generate_uudmap uudmap.h bitcount.h mg_data.h

[rhelmot]

is there anything else in this log relevant to your compilation failure that we should be looking at?

I truly do not know. I was hoping you would be able to read the log better than I can. The only nix-specific thing which may be of interest to you is that it prints the configure flags:

configure flags: --prefix=/nix/store/9j3mry40z46577xv487c79n71f3dyj9h-perl-x86_64-unknown-freebsd-5.38.2 -Dlibpth=\"\" -Dglibpth=\"\" -Ddefault_inc_excludes_dot -Uinstallusrbinperl -Dinstallstyle=lib/perl5 -Dlocincpth=/nix/store/09ima5h6cj40fq0h5z3dbrxwbmsy3zxm-libc-x86_64-unknown-freebsd-14.0.0/include -Dloclibpth=/nix/store/09ima5h6cj40fq0h5z3dbrxwbmsy3zxm-libc-x86_64-unknown-freebsd-14.0.0/lib -Dusethreads --build=x86_64-unknown-linux-gnu --host=x86_64-unknown-freebsd --target=x86_64-unknown-freebsd

Also, can you provide the characteristics (uname -mrs) of the operating system on which you were attempting this?

Linux 6.8.0-31-generic x86_64

Can you provide the program you used to try to do this cross-compilation, as well as how you called it on the command-line?

Unfortunately, the best/simplest answer I have for you is nix build github:nixos/nixpkgs/62431fd38dc37ba6d42020eda3228a204835fdde#pkgsCross.x86_64-freebsd.perl. The commit hash of nixpkgs is the master branch at time of writing. If you run this, it will actually download a cached build that exhibits the bug:

$ strings ./result/bin/perl | grep 'crypt() function is unimplemented'
The crypt() function is unimplemented due to excessive paranoia.

If you want to instrument this build, the easiest way to do this is to clone nixpkgs and modify the build formula for perl pkgs/development/interpreters/perl/interpreter.nix, then change the command line argument talking about github to instead just be the root of nixpkgs on your filesystem.

However, installing and learning nix is kind of a lot for me to ask of an upstream maintainer, especially since imo I'm the one at fault for not knowing how to do this in a "normal" environment, so I'm totally willing to do remote debugging with you, or even to try to reverse engineer the build process myself if you can just explain how the process of defining HAS_CRYPT is supposed to work in theory.

[jkeenan]

is there anything else in this log relevant to your compilation failure that we should be looking at?

I truly do not know. I was hoping you would be able to read the log better than I can. The only nix-specific thing which may be of interest to you is that it prints the configure flags:

Is this what you are referring to as "nix"? https://nixos.org/

[rhelmot]

Yes. I am a package maintainer for nixpkgs, which is a canonical set of build instructions for nix/nixos.

[jkeenan]

Yes. I am a package maintainer for nixpkgs, which is a canonical set of build instructions for nix/nixos.

I'm unclear why you are attempting to use nix on Linux to cross-compile FreeBSD. Does nix not work on FreeBSD? Can you back up a minute and describe better what you're trying to accomplish?

[jkeenan]

Yes. I am a package maintainer for nixpkgs, which is a canonical set of build instructions for nix/nixos.

I'm unclear why you are attempting to use nix on Linux to cross-compile FreeBSD. Does nix not work on FreeBSD? Can you back up a minute and describe better what you're trying to accomplish?

I.e., is there a reasonable expectation that nix will be an adequate tool for cross-OS compilation?

[rhelmot]

Yes, the entirety of FreeBSD and many packages for it, including things like python and rust, compile cross from Linux to FreeBSD under nix. Nix's support for native FreeBSD->FreeBSD is ongoing, and the perl build does work there, but one of the main things that makes nix attractive to an end user is that you can run a single command to stand up an entire VM of a target system cross compiled from your host.

The goal here is that I was told by the other package maintainers to file an issue for the fact that the build system couldn't detect this value while cross compiling before I added the -Dd_crypt flag to our build.

[rhelmot]

Can you back up a minute and describe better what you're trying to accomplish?

On a less-meta level, the goal here is to build/manage FreeBSD VMs and physical machines with declarative configuration through nix, both native and cross. Perl is a helpful utility which lots of scripts are written in, and some of these scripts depend on crypt working when run on FreeBSD, e.g. to construct an /etc/shadow for a declaratively booting system. Currently the perl cross build requires a special case to build with crypt, and I'm sure you understand that package maintainers are allergic to special cases in their build configuration.

[Ericson2314]

@jkeenan Yeah for further context "cross compile from Linux -> FreeBSD" might sound really "wat! How could that ever work reliably!", but our general experience is that with the right foundation actually loads of stuff cross-compiles great, and relatively few packages need platform-specific hacks.

The specific failure struct me like as something indicating an upstream bug, even if it is a rather obscure. If there was a way to get a bit more debug log output (e.g. failing test program, if the configure script does that) then I think we could come up with a fix that looks good on its own terms, even if the original task here sounds like something too difficult to be worth upstream supporting.