Validate that all combinations of BUILDPLATFORM, TARGETPLATFORM, Dockerfile, and host CPU architecture produce correct executable

[rucoder]

With some combinations of BUILDPLATFORM, TARGETPLATFORM, Dockerfile, and host CPU architecture downstream rust applications are built for incorrect CPU architecture.

• Identify broken combinations fixed by Set default build target for native builds #10
• Add tests to validate it Add a placeholder for tests #9
• run tests on CI

[deitch]

@rucoder do you have an example to recreate it? I think @jsfakian also succeeded in creating the issue? I am not asking about the fix, just to recreate the problem.

[rucoder]

this is the example Ioannis sent me yesterday, but I have not tried it yet

# syntax=docker/dockerfile-upstream:1.5.0-rc2-labs
#FROM --platform=$BUILDPLATFORM jsfakian/eve-rust:1.80.1 AS rust-host
#ARG TARGETARCH
FROM jsfakian/eve-rust:1.80.1 AS rust


# map Docker's $TARGETARCH to Rust target
#FROM rust-host AS target-amd64
#ENV CARGO_BUILD_TARGET="x86_64-unknown-linux-musl"

#FROM rust-host AS target-arm64
#ENV CARGO_BUILD_TARGET="aarch64-unknown-linux-musl"

#FROM rust-host AS target-riscv64
#ENV CARGO_BUILD_TARGET="riscv64gc-unknown-linux-gnu"

#FROM target-$TARGETARCH AS rust

WORKDIR /usr/local/my-installer
COPY . .

# Add the vendor tar file
RUN tar xvf /usr/local/my-installer/vendor.tar.gz \
    && rm vendor.tar.gz

# tools already in place
RUN cargo build --release --verbose
RUN cargo sbom > sbom.spdx.json

RUN cp /usr/local/my-installer/target/$CARGO_BUILD_TARGET/release/installer /usr/local/my-installer/target/installer

FROM lfedge/eve-alpine:1f7685f95a475c6bbe682f0b976f12180b6c8726 AS build
SHELL ["/bin/ash", "-eo", "pipefail", "-c"]
ENV PKGS ncurses-dev smartmontools mtools strace util-linux e2fsprogs e2fsprogs-extra jq

RUN eve-alpine-deploy.sh

COPY --from=rust /usr/local/my-installer/target/installer /out/sbin/installer
COPY --from=rust /usr/local/my-installer/sbom.spdx.json /out/sbin/sbom.spdx.json

FROM scratch
COPY --from=build /out/ /
ENV RUST_BACKTRACE full
COPY run-installer.sh /sbin/run-installer.sh

WORKDIR /
ENTRYPOINT ["/sbin/run-installer.sh"]

and the error message

2.177 error: linking with `/usr/bin/clang` failed: exit status: 1
2.177   |
2.177   = note: LC_ALL="C" PATH="/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/bin:/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/bin:/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/bin:/usr/local/cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" VSLANG="1033" "/usr/bin/clang" "-Wl,--version-script=/tmp/rustc4y4cU5/list" "-Wl,--no-undefined-version" "-m64" "/tmp/rustc4y4cU5/symbols.o" "/usr/local/my-installer/target/release/deps/rustversion-6b2bbacd6ab242a4.rustversion.48f492ba9fdb42b3-cgu.0.rcgu.o" "/usr/local/my-installer/target/release/deps/rustversion-6b2bbacd6ab242a4.rustversion.48f492ba9fdb42b3-cgu.1.rcgu.o" "/usr/local/my-installer/target/release/deps/rustversion-6b2bbacd6ab242a4.rustversion.48f492ba9fdb42b3-cgu.2.rcgu.o" "/usr/local/my-installer/target/release/deps/rustversion-6b2bbacd6ab242a4.rustversion.48f492ba9fdb42b3-cgu.3.rcgu.o" "/usr/local/my-installer/target/release/deps/rustversion-6b2bbacd6ab242a4.rustversion.48f492ba9fdb42b3-cgu.4.rcgu.o" "/usr/local/my-installer/target/release/deps/rustversion-6b2bbacd6ab242a4.rustversion.48f492ba9fdb42b3-cgu.5.rcgu.o" "/usr/local/my-installer/target/release/deps/rustversion-6b2bbacd6ab242a4.rustversion.48f492ba9fdb42b3-cgu.6.rcgu.o" "/usr/local/my-installer/target/release/deps/rustversion-6b2bbacd6ab242a4.rustversion.48f492ba9fdb42b3-cgu.7.rcgu.o" "/usr/local/my-installer/target/release/deps/rustversion-6b2bbacd6ab242a4.1kzo4sn02q0yb1bqsb0prf8ua.rcgu.rmeta" "/usr/local/my-installer/target/release/deps/rustversion-6b2bbacd6ab242a4.72xrpt734hw9o2axvytp0dovp.rcgu.o" "-Wl,--as-needed" "-L" "/usr/local/my-installer/target/release/deps" "-L" "/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib" "-Wl,-Bstatic" "/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libproc_macro-bb90118a9d3a1120.rlib" "/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd-11278e6f292a83a2.rlib" "/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libpanic_unwind-1f0a0dc4faeec0a3.rlib" "/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libobject-aa008ac291cf6c9f.rlib" "/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libmemchr-222e4867c5d474c8.rlib" "/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libaddr2line-b729ba7443944882.rlib" "/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libgimli-59743343eaf025ae.rlib" "/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/librustc_demangle-cb5149dfbee56ccb.rlib" "/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd_detect-e57a974822694b25.rlib" "/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libhashbrown-a2863812bbdf5e6e.rlib" "/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/librustc_std_workspace_alloc-925e49df7b4ee2bb.rlib" "/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libminiz_oxide-fb6cd229e1eac996.rlib" "/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libadler-22b25a5ddfc02a07.rlib" "/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libunwind-bf17bea8f84cb8cb.rlib" "/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libcfg_if-218ac2e8053d7769.rlib" "/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/liblibc-c8a3bb72f2547d7e.rlib" "/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/liballoc-a6f924a0e16819b9.rlib" "/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/librustc_std_workspace_core-aedcb87a10c080a3.rlib" "/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libcore-c1cee64659224c4c.rlib" "/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libcompiler_builtins-b35caaccb3ca2afc.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib" "-o" "/usr/local/my-installer/target/release/deps/librustversion-6b2bbacd6ab242a4.so" "-Wl,--gc-sections" "-shared" "-Wl,-z,relro,-z,now" "-Wl,--strip-all" "-nodefaultlibs" "--ld-path=/usr/bin/mold" "--target=x86_64-unknown-linux-musl"
2.177   = note: mold: fatal: cannot open crtbeginS.o: No such file or directory
2.177           clang: error: linker command failed with exit code 1 (use -v to see invocation)
2.177
2.177
2.195    Compiling crossbeam-channel v0.5.13
2.195      Running `/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/bin/rustc --crate-name crossbeam_channel --edition=2021 /usr/local/my-installer/vendor/crossbeam-channel/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=z -C panic=abort -C linker-plugin-lto -C codegen-units=1 --warn=unexpected_cfgs --cfg 'feature="default"' --cfg 'feature="std"' --check-cfg 'cfg(docsrs)' --check-cfg 'cfg(feature, values("default", "std"))' -C metadata=3e219331d488bf74 -C extra-filename=-3e219331d488bf74 --out-dir /usr/local/my-installer/target/release/deps -C linker=/usr/bin/clang -C strip=symbols -L dependency=/usr/local/my-installer/target/release/deps --extern crossbeam_utils=/usr/local/my-installer/target/release/deps/libcrossbeam_utils-205a64670172d02c.rmeta --cap-lints allow -C link-arg=--ld-path=/usr/bin/mold -C link-arg=--target=x86_64-unknown-linux-musl`
2.222 error: could not compile `rustversion` (lib) due to 1 previous error
2.222
2.222 Caused by:
2.222   process didn't exit successfully: `/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/bin/rustc --crate-name rustversion --edition=2018 /usr/local/my-installer/vendor/rustversion/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C embed-bitcode=no -C debug-assertions=off --check-cfg 'cfg(docsrs)' --check-cfg 'cfg(feature, values())' -C metadata=6b2bbacd6ab242a4 -C extra-filename=-6b2bbacd6ab242a4 --out-dir /usr/local/my-installer/target/release/deps -C linker=/usr/bin/clang -C strip=symbols -L dependency=/usr/local/my-installer/target/release/deps --extern proc_macro --cap-lints allow -C link-arg=--ld-path=/usr/bin/mold -C link-arg=--target=x86_64-unknown-linux-musl --check-cfg 'cfg(cfg_macro_not_allowed)' --check-cfg 'cfg(host_os, values("windows"))'` (exit status: 1)
2.222 warning: build failed, waiting for other jobs to finish...

[deitch]

What platform was he running it on? How did he call it? And in what scenario does it work?

[rucoder]

What platform was he running it on? How did he call it? And in what scenario does it work?

both x86 linux and Mac. @jsfakian , I think I know why it failed for you. Did you build a multiplatfrom image for eve-rust and push it to your dockerhub or you built only a single platfrom?

[jsfakian]

I was running on x86 Linux and Mac. I built a multiplatform image for eve-rust and pushed it to my docker hub.

[deitch]

@jsfakian and I did a test. If you do a simple build, it fails. If you set the env var (like the Dockerfile), it fails. It fails on building proc-macro-error-attr. Here are the success and failure lines:

Success with env var export CARGO_BUILD_TARGET="x86_64-unknown-linux-musl":

/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/bin/rustc --crate-name proc_macro_error_attr --edition=2018 /usr/local/my-installer/vendor/proc-macro-error-attr/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --diagnostic-width=200 --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C embed-bitcode=no -C debug-assertions=off --check-cfg 'cfg(docsrs)' --check-cfg 'cfg(feature, values())' -C metadata=48656eceb2ec36da -C extra-filename=-48656eceb2ec36da --out-dir /usr/local/my-installer/target/release/deps -C linker=/usr/bin/clang -C strip=symbols -L dependency=/usr/local/my-installer/target/release/deps --extern proc_macro2=/usr/local/my-installer/target/release/deps/libproc_macro2-f6ff1dbe72cec79b.rlib --extern quote=/usr/local/my-installer/target/release/deps/libquote-4ff18852125cfc7a.rlib --extern proc_macro --cap-lints allow

Failure without env var:

/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/bin/rustc --crate-name proc_macro_error_attr --edition=2018 /usr/local/my-installer/vendor/proc-macro-error-attr/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C embed-bitcode=no -C debug-assertions=off --check-cfg 'cfg(docsrs)' --check-cfg 'cfg(feature, values())' -C metadata=48656eceb2ec36da -C extra-filename=-48656eceb2ec36da --out-dir /usr/local/my-installer/target/release/deps -C linker=/usr/bin/clang -C strip=symbols -L dependency=/usr/local/my-installer/target/release/deps --extern proc_macro2=/usr/local/my-installer/target/release/deps/libproc_macro2-f6ff1dbe72cec79b.rlib --extern quote=/usr/local/my-installer/target/release/deps/libquote-4ff18852125cfc7a.rlib --extern proc_macro --cap-lints allow -C link-arg=--ld-path=/usr/bin/mold -C link-arg=--target=x86_64-unknown-linux-musl

The only differences between the lines are:

• success has: --diagnostic-width=200 (we do not think that matters)
• failure has at the very end: -C link-arg=--ld-path=/usr/bin/mold -C link-arg=--target=x86_64-unknown-linux-musl

Same success if we do cargo build --target x86_64-unknown-linux-musl

We do not understand why explicitly setting CARGO_BUILD_TARGET causes that to happen. Since rust is on x86_64-unknown-linux-musl, wouldn't that be the default build target?

@rucoder thoughts?

[deitch]

More information. On failure, the linker says it is unable to find crtbeginS.o. This does exist on the platform, specifically at:

/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/<triple>/lib/self-contained/crtbeginS.o

E.g.:

/usr/local/my-installer # ls -l /usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/self-contained/
total 6084
-rw-rw-rw-    1 root     root          3688 Aug  8 20:02 Scrt1.o
-rw-rw-rw-    1 root     root          3688 Aug  8 20:02 crt1.o
-rw-rw-rw-    1 root     root          3624 Aug  8 20:02 crtbegin.o
-rw-rw-rw-    1 root     root          3624 Aug  8 20:02 crtbeginS.o
-rw-rw-rw-    1 root     root          1056 Aug  8 20:02 crtend.o
-rw-rw-rw-    1 root     root          1056 Aug  8 20:02 crtendS.o
-rw-rw-rw-    1 root     root          2656 Aug  8 20:02 crti.o
-rw-rw-rw-    1 root     root          2600 Aug  8 20:02 crtn.o
-rw-rw-rw-    1 root     root       5982382 Aug  8 20:02 libc.a
-rw-rw-rw-    1 root     root        204164 Aug  8 20:02 libunwind.a
-rw-rw-rw-    1 root     root          4864 Aug  8 20:02 rcrt1.o

It looks like the linker is not finding these libraries to link, except when CARGO_BUILD_TARGET is set (or CLI flag). We wonder if maybe there is a big somewhere where it sets the lib path using the target but misses it if not explicit?

[rucoder]

@jsfakian and I did a test. If you do a simple build, it fails. If you set the env var (like the Dockerfile), it fails. It fails on building proc-macro-error-attr. Here are the success and failure lines:

Success with env var export CARGO_BUILD_TARGET="x86_64-unknown-linux-musl":

/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/bin/rustc --crate-name proc_macro_error_attr --edition=2018 /usr/local/my-installer/vendor/proc-macro-error-attr/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --diagnostic-width=200 --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C embed-bitcode=no -C debug-assertions=off --check-cfg 'cfg(docsrs)' --check-cfg 'cfg(feature, values())' -C metadata=48656eceb2ec36da -C extra-filename=-48656eceb2ec36da --out-dir /usr/local/my-installer/target/release/deps -C linker=/usr/bin/clang -C strip=symbols -L dependency=/usr/local/my-installer/target/release/deps --extern proc_macro2=/usr/local/my-installer/target/release/deps/libproc_macro2-f6ff1dbe72cec79b.rlib --extern quote=/usr/local/my-installer/target/release/deps/libquote-4ff18852125cfc7a.rlib --extern proc_macro --cap-lints allow

Failure without env var:

/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/bin/rustc --crate-name proc_macro_error_attr --edition=2018 /usr/local/my-installer/vendor/proc-macro-error-attr/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C embed-bitcode=no -C debug-assertions=off --check-cfg 'cfg(docsrs)' --check-cfg 'cfg(feature, values())' -C metadata=48656eceb2ec36da -C extra-filename=-48656eceb2ec36da --out-dir /usr/local/my-installer/target/release/deps -C linker=/usr/bin/clang -C strip=symbols -L dependency=/usr/local/my-installer/target/release/deps --extern proc_macro2=/usr/local/my-installer/target/release/deps/libproc_macro2-f6ff1dbe72cec79b.rlib --extern quote=/usr/local/my-installer/target/release/deps/libquote-4ff18852125cfc7a.rlib --extern proc_macro --cap-lints allow -C link-arg=--ld-path=/usr/bin/mold -C link-arg=--target=x86_64-unknown-linux-musl

The only differences between the lines are:

* success has: `--diagnostic-width=200` (we do not think that matters)

* failure has at the very end: `-C link-arg=--ld-path=/usr/bin/mold -C link-arg=--target=x86_64-unknown-linux-musl`

Same success if we do cargo build --target x86_64-unknown-linux-musl

We do not understand why explicitly setting CARGO_BUILD_TARGET causes that to happen. Since rust is on x86_64-unknown-linux-musl, wouldn't that be the default build target?

@rucoder thoughts?

@deitch I think CARGO_BUILD_TARGET IS set but it is not -unknow-linux-musl because how would cargo know you want to build for musl? maybe try setting default toolchain via rustup?

e.g.

FROM tools-host AS target-arm64
RUN rustup default aarch64-unknown-linux-musl
ENV CARGO_BUILD_TARGET="aarch64-unknown-linux-musl"

UPDARE: no, toolchain has nothing todo with target. we MUST specify --target or CARGO_BUILD_TARGET to override the default target which matches the BUILD platform

[deitch]

@rucoder can you explain that?

The command that fails is:

 /usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/bin/rustc --crate-name proc_macro_error_attr --edition=2018 /installer/vendor/proc-macro-error-attr/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C embed-bitcode=no -C debug-assertions=off --check-cfg 'cfg(docsrs)' --check-cfg 'cfg(feature, values())' -C metadata=48656eceb2ec36da -C extra-filename=-48656eceb2ec36da --out-dir /installer/target/release/deps -C linker=/usr/bin/clang -C strip=symbols -L dependency=/installer/target/release/deps --extern proc_macro2=/installer/target/release/deps/libproc_macro2-f6ff1dbe72cec79b.rlib --extern quote=/installer/target/release/deps/libquote-4ff18852125cfc7a.rlib --extern proc_macro --cap-lints allow -C link-arg=--ld-path=/usr/bin/mold -C link-arg=--target=x86_64-unknown-linux-musl

That looks like it is building for the local target. But who knows?

I think it is fine if our default target is local platform, and we set it. So maybe the dockerfile to build should include that cross-stuff, so it always sets the default target, and you can override it? We already do it for building the cargo tools, maybe we should leave the ENV set? That would fix it.

[deitch]

@rucoder I think we can close this now?

[rucoder]

@rucoder I think we can close this now?

So we are not going to add tests/exaples ? 😉

[deitch]

Ok, fine.