Support for a v9 build-env #35

ydirson · 2025-06-26T13:17:58Z

No description provided.

XAPI build is not able to pass with a value of 1024. 2048 should still be low enough to leave container startup time reasonable. Signed-off-by: Yann Dirson <[email protected]>

Signed-off-by: Yann Dirson <[email protected]>

There does not seem to be a reason for not passing that flag when already on amd64. Will make it simpler to handle multiple arch. Signed-off-by: Yann Dirson <[email protected]>

dinhngtu · 2025-06-30T09:06:43Z

Many Alma 10 packages use the flat build directory structure, and so don't have the SPECS/SOURCES/etc. directories any more. Could you import detect spec file paths?

ydirson · 2025-06-30T16:19:56Z

Many Alma 10 packages use the flat build directory structure, and so don't have the SPECS/SOURCES/etc. directories any more. Could you import detect spec file paths?

Ah I had missed that one, looks like a good idea

files/init-container.sh

files/xcp-ng-8.99.repo

glehmann · 2025-07-01T08:28:04Z

build.sh

@@ -35,16 +35,19 @@ case "$1" in
    9.*)


a comment on the commit description (not sure why github doesn't allow to put a comment on the description itself).
Originally-by doesn't seem to be a standard git trailer; Co-authored-by seems more appropriate

Well that's more "started by one and continued by another", co-authoring to me rings more like pair-programming.

https://lore.kernel.org/git/[email protected]/ notes Based-on-patch-by and Original-patch-by and more used in the kernel. That's really free-text, and as I've been using that one a number of times already, I'm not sure changing now would bring much.

Github recognized Co-authored-by and properly displays the authors. I don't think it recognizes any other. Gitlab also recognizes Co-authored-by, not sure for gitea. It's also mentioned in the git documentation, although at a trailer specific of the git project.

https://git-scm.com/docs/SubmittingPatches

https://docs.github.com/en/pull-requests/committing-changes-to-your-project/creating-and-editing-commits/creating-a-commit-with-multiple-authors

https://gitlab.com/gitlab-org/gitlab-foss/-/issues/31640

I don't think that's mandatory, but it's a good practice to acknowledge the authorship with the tools we have IMO

Since init-container runs following the dockerfile, there seems to be no reason to rerun it with every run.py invocation. Signed-off-by: Yann Dirson <[email protected]>

Will help simplifying init-container.sh Signed-off-by: Yann Dirson <[email protected]>

Even after dropping --rebuild-srpm, adding new rpmbuild options had to be done on 2 commandlines. At the same time, using a subshell to deal with change of working directory makes the new code easier. Signed-off-by: Yann Dirson <[email protected]>

Not the same as running the whole run.py under `time` control, especially when using `--no-exit`. Also, does not include the build-env setup (which includes installation of the build-deps of SRPMS). Quite trivial now that we use a subshell for the build itself. Signed-off-by: Yann Dirson <[email protected]>

ydirson · 2025-07-09T14:07:11Z

Many Alma 10 packages use the flat build directory structure, and so don't have the SPECS/SOURCES/etc. directories any more. Could you import detect spec file paths?

Ah I had missed that one, looks like a good idea

Picked a slightly different logic for the flat-structure detection, separated support for source-fetching, and did the latter using Alma tools rather than the generic tool you used, which does not seem to properly support packages from Alma.

files/init-container.sh

ydirson · 2025-07-10T08:54:58Z

I dropped the --ccache support for now. I did manage to get it running yesterday, but the thing seems awfully brittle and refuses to do its job this morning.

dinhngtu

Worked with my shim-unsigned and grub2 packages using docker. Maybe @fallen can also give his opinion since he uses podman.

Dockerfile-9.x

Originally-by: Tu Dinh <[email protected]> This assumes the distro has already been bootstrapped by building xcp-ng-release and will not be usable for this one, the relevant RUN clause has to be commented out for this. Signed-off-by: Yann Dirson <[email protected]>

That could not be done in a subshell. Also, don't attempt to download users repo when there is none, 404 HTML contents are a bad fit for yum.repos.d Signed-off-by: Yann Dirson <[email protected]>

Originally-by: Tu Dinh <[email protected]> Signed-off-by: Yann Dirson <[email protected]>

Necessary with podman < v5.5.1 to workaround a bug. Signed-off-by: Yann Dirson <[email protected]>

Signed-off-by: Yann Dirson <[email protected]>

Note that with podman < 5.5.1 we cannot have both linux/amd64 and linux/amd64/v2. And using a completely different arch requires emulation. Signed-off-by: Yann Dirson <[email protected]>

Signed-off-by: Yann Dirson <[email protected]>

This allows to build packages with a flat (current Fedora/RHEL standard) source layout. Originally-by: Tu Dinh <[email protected]> Signed-off-by: Yann Dirson <[email protected]>

…ux bucket Signed-off-by: Yann Dirson <[email protected]>

Notably useful for -bp, to - quickly get a patched source tree - apply patches using quilt, in order to refresh them in 8.3 env, so they will not be rejected by the 9.0 one Signed-off-by: Yann Dirson <[email protected]>

It is highly disrupting to have to add `set -x` manually to the script and rebuild it, each time something goes wrong. Signed-off-by: Yann Dirson <[email protected]>

Signed-off-by: Yann Dirson <[email protected]>

fallen · 2025-07-21T12:15:52Z

I gave this a try with both docker and podman.
On docker it works well, I was able to generate the build image and then build @dinhngtu 's grub2 package.

On podman I was able to generate the build image, but I was not able to build grub2 package:

ysionneau@ysionneau-XPS-13-9350:~/dev/rpms/grub2$ XCPNG_OCI_RUNNER=podman ~/dev/xcp-ng-build-env/run.py -b 9.0 -l . -n --name grub2-9 -v ~/dev/sbkeys:/etc/pki/pesign-xcpng:ro --rm --enablerepo crb
Launching docker with args ['podman', 'run', '-i', '-t', '-u', 'builder', '--platform', 'linux/amd64/v2', '--userns=keep-id', '--rm=true', '-v', '/home/ysionneau/dev/rpms/grub2:/home/builder/rpmbuild', '-e', 'BUILD_LOCAL=1', '-e', 'NO_EXIT=1', '--name', 'grub2-9', '-v', '/home/ysionneau/dev/sbkeys:/etc/pki/pesign-xcpng:ro', '-e', 'ENABLEREPO=crb', '--ulimit', 'nofile=2048', 'xcp-ng/xcp-ng-build-env:9.0', '/usr/local/bin/init-container.sh']
WARNING: unknown release, not fetching user repo definitions
sudo: PAM account management error: Authentication service cannot retrieve authentication info
sudo: a password is required
sudo: PAM account management error: Authentication service cannot retrieve authentication info
sudo: a password is required
Found specfiles grub2.spec
INFO    : unifont-13.0.06.pcf.gz already exists and its checksum is correct
INFO    : theme.tar.bz2 already exists and its checksum is correct
INFO    : grub-2.12.tar.xz already exists and its checksum is correct
INFO    : gnulib-9f48fb992a3d7e96610c4ce8be969cff2d61a01b.tar.gz already exists and its checksum is correct
sudo: PAM account management error: Authentication service cannot retrieve authentication info
sudo: a password is required
Building target platforms: x86_64_v2
Building for target x86_64_v2
warning: Macro expanded in comment on line 167: %global legacy_target_cpu_name i386

warning: Macro expanded in comment on line 168: %global legacy_package_arch pc

warning: Macro expanded in comment on line 169: %global platform pc

warning: Macro expanded in comment on line 44: %{x86_64} aarch64 ppc64le

warning: Macro expanded in comment on line 45: %define sb_ca		%{Source1000}

warning: Macro expanded in comment on line 46: %define sb_cer		%{Source1001}

warning: Macro expanded in comment on line 55: %define pe_signing_token NSS Certificate DB

warning: Macro expanded in comment on line 56: %define pe_signing_cert /etc/pki/pesign-xcpng

setting SOURCE_DATE_EPOCH=1749513600
error: Failed build dependencies:
	autoconf is needed by grub2-1:2.12-20.xcpng8.99.alma.2.x86_64_v2
	automake is needed by grub2-1:2.12-20.xcpng8.99.alma.2.x86_64_v2
	bison is needed by grub2-1:2.12-20.xcpng8.99.alma.2.x86_64_v2
	bzip2-devel is needed by grub2-1:2.12-20.xcpng8.99.alma.2.x86_64_v2
	dejavu-sans-fonts is needed by grub2-1:2.12-20.xcpng8.99.alma.2.x86_64_v2
	device-mapper-devel is needed by grub2-1:2.12-20.xcpng8.99.alma.2.x86_64_v2
	flex is needed by grub2-1:2.12-20.xcpng8.99.alma.2.x86_64_v2
	freetype-devel is needed by grub2-1:2.12-20.xcpng8.99.alma.2.x86_64_v2
	fuse-devel is needed by grub2-1:2.12-20.xcpng8.99.alma.2.x86_64_v2
	gettext-devel is needed by grub2-1:2.12-20.xcpng8.99.alma.2.x86_64_v2
	help2man is needed by grub2-1:2.12-20.xcpng8.99.alma.2.x86_64_v2
	ncurses-devel is needed by grub2-1:2.12-20.xcpng8.99.alma.2.x86_64_v2
	pesign >= 0.99-8 is needed by grub2-1:2.12-20.xcpng8.99.alma.2.x86_64_v2
	rpm-devel is needed by grub2-1:2.12-20.xcpng8.99.alma.2.x86_64_v2
	squashfs-tools is needed by grub2-1:2.12-20.xcpng8.99.alma.2.x86_64_v2
	texinfo is needed by grub2-1:2.12-20.xcpng8.99.alma.2.x86_64_v2
	xz-devel is needed by grub2-1:2.12-20.xcpng8.99.alma.2.x86_64_v2

RPM build warnings:
    Macro expanded in comment on line 167: %global legacy_target_cpu_name i386

    Macro expanded in comment on line 168: %global legacy_package_arch pc

    Macro expanded in comment on line 169: %global platform pc

    Macro expanded in comment on line 44: %{x86_64} aarch64 ppc64le

    Macro expanded in comment on line 45: %define sb_ca		%{Source1000}

    Macro expanded in comment on line 46: %define sb_cer		%{Source1001}

    Macro expanded in comment on line 55: %define pe_signing_token NSS Certificate DB

    Macro expanded in comment on line 56: %define pe_signing_cert /etc/pki/pesign-xcpng


real	0m0.201s
user	0m0.126s
sys	0m0.075s
[builder@08db6f1d34db ~]$ 
logout

glehmann · 2025-07-21T13:34:08Z

run.py

@@ -15,10 +15,10 @@
 import uuid

 CONTAINER_PREFIX = "xcp-ng/xcp-ng-build-env"


Suggested change

CONTAINER_PREFIX = "xcp-ng/xcp-ng-build-env"

CONTAINER_PREFIX = "localhost/xcp-ng/xcp-ng-build-env"

I had to do that to make podman not propose to interactively select a registry:

? Please select an image: ▸ registry.fedoraproject.org/xcp-ng/xcp-ng-build-env:9.0 registry.access.redhat.com/xcp-ng/xcp-ng-build-env:9.0 docker.io/xcp-ng/xcp-ng-build-env:9.0

it doesn't have that behavior without the --platform option

It may be interesting to have the images pre-built in ghcr.io, and use ghcr.io in the image name, but I keep that for another PR :)

I had to do that to make podman not propose to interactively select a registry:

I remember having to do that in one early test, but did not need to do that anymore recently 🤔

It may be interesting to have the images pre-built in ghcr.io, and use ghcr.io in the image name, but I keep that for another PR :)

IIRC @psafont mentionned he has plans for something like this :)

Yeah, it would be good to upload the images so they are accessible to mock and avoid reinstalling packages to set the base :)

I had to do that to make podman not propose to interactively select a registry:

I remember having to do that in one early test, but did not need to do that anymore recently 🤔

@glehmann actually when I add localhost/, using podman 4.3.1, it fails in a very strange way, but that's apparently "just because" I omitted the --platform linux/amd64 workaround needed for that old version (podman < v5.5.1 has a bug where the container is not registed as linux/amd64/v2).

The thing is, when the localhost/ prefix is not used the error is pretty easy to link with the bug when one knows about it:

Error: short-name "xcp-ng/xcp-ng-build-env:9.0" did not resolve to an alias and no unqualified-search registries are defined in "/etc/containers/registries.conf"

But with the prefix:

... WARN[0002] Failed, retrying in 1s ... (3/3). Error: initializing source docker://localhost/xcp-ng/xcp-ng-build-env:9.0: pinging container registry localhost: Get "https://localhost/v2/": dial tcp [::1]:443: connect: connection refused Error: initializing source docker://localhost/xcp-ng/xcp-ng-build-env:9.0: pinging container registry localhost: Get "https://localhost/v2/": dial tcp [::1]:443: connect: connection refused

We'll need to raise awareness of this behaviour, or some of us may get trapped :/

ydirson · 2025-07-21T14:05:23Z

I gave this a try with both docker and podman. On docker it works well, I was able to generate the build image and then build @dinhngtu 's grub2 package.

On podman I was able to generate the build image, but I was not able to build grub2 package:
error: Failed build dependencies:
	autoconf is needed by grub2-1:2.12-20.xcpng8.99.alma.2.x86_64_v2

This is likely due to packages not in base, many packages have breq's in crb, you can pass --enablerepo=crb. I had not enabled it by default to get an idea of who needs it, but given the number of them it makes sense to enable that repo by default.

fallen · 2025-07-21T14:16:20Z

I gave this a try with both docker and podman. On docker it works well, I was able to generate the build image and then build @dinhngtu 's grub2 package.
On podman I was able to generate the build image, but I was not able to build grub2 package:
error: Failed build dependencies:
	autoconf is needed by grub2-1:2.12-20.xcpng8.99.alma.2.x86_64_v2
This is likely due to packages not in base, many packages have breq's in crb, you can pass --enablerepo=crb. I had not enabled it by default to get an idea of who needs it, but given the number of them it makes sense to enable that repo by default.

I used the same command line as for docker:

XCPNG_OCI_RUNNER=podman ~/dev/xcp-ng-build-env/run.py -b 9.0 -l . -n --name grub2-9 -v ~/dev/sbkeys:/etc/pki/pesign-xcpng:ro --rm --enablerepo crb

But it did not work.

ydirson added 4 commits June 27, 2025 15:35

run.py: raise the ulimit nofile value

e987f73

XAPI build is not able to pass with a value of 1024. 2048 should still be low enough to leave container startup time reasonable. Signed-off-by: Yann Dirson <[email protected]>

build.sh: handle more relevant versions first

ec70301

Signed-off-by: Yann Dirson <[email protected]>

init-container: fix "run.py --define=... --rebuild-srpm=..."

e8cfd2f

Signed-off-by: Yann Dirson <[email protected]>

Make --platform flag systematic

3f7be42

There does not seem to be a reason for not passing that flag when already on amd64. Will make it simpler to handle multiple arch. Signed-off-by: Yann Dirson <[email protected]>

ydirson force-pushed the ydi/9 branch 2 times, most recently from 02bcee8 to adce911 Compare June 27, 2025 15:12

ydirson marked this pull request as ready for review June 27, 2025 15:13

ydirson requested review from glehmann, stormi and dinhngtu June 27, 2025 15:13

stormi approved these changes Jun 27, 2025

View reviewed changes

glehmann reviewed Jul 1, 2025

View reviewed changes

ydirson added 5 commits July 9, 2025 09:49

Move yum clean all from init-container to Dockerfile

f8a1aed

Since init-container runs following the dockerfile, there seems to be no reason to rerun it with every run.py invocation. Signed-off-by: Yann Dirson <[email protected]>

run.py: drop unused --package flag

6c35da2

Will help simplifying init-container.sh Signed-off-by: Yann Dirson <[email protected]>

run.py: drop unused --srpm and --rebuild-srpm flags

2dd739b

Will help simplifying init-container.sh Signed-off-by: Yann Dirson <[email protected]>

init-container: avoid copypasta

a894ec4

Even after dropping --rebuild-srpm, adding new rpmbuild options had to be done on 2 commandlines. At the same time, using a subshell to deal with change of working directory makes the new code easier. Signed-off-by: Yann Dirson <[email protected]>

ydirson force-pushed the ydi/9 branch from adce911 to 754feb9 Compare July 9, 2025 13:53

ydirson requested a review from psafont July 9, 2025 13:53

ydirson force-pushed the ydi/9 branch from 754feb9 to b87efd8 Compare July 9, 2025 14:04

ydirson requested review from glehmann and stormi July 9, 2025 14:08

ydirson mentioned this pull request Jul 9, 2025

Properly package the python script #36

Open

dinhngtu requested changes Jul 9, 2025

View reviewed changes

files/init-container.sh Outdated Show resolved Hide resolved

ydirson force-pushed the ydi/9 branch from 0d0b822 to a9c4989 Compare July 10, 2025 08:49

ydirson force-pushed the ydi/9 branch from a9c4989 to 822e2fe Compare July 10, 2025 09:05

ydirson requested a review from dinhngtu July 10, 2025 09:06

dinhngtu approved these changes Jul 10, 2025

View reviewed changes

Dockerfile-9.x Outdated Show resolved Hide resolved

ydirson added 11 commits July 11, 2025 10:22

init-container: make OS_RELEASE available for setting other vars

eee9aac

That could not be done in a subshell. Also, don't attempt to download users repo when there is none, 404 HTML contents are a bad fit for yum.repos.d Signed-off-by: Yann Dirson <[email protected]>

Build 9.0 for x86_64_v2

f2b471e

Originally-by: Tu Dinh <[email protected]> Signed-off-by: Yann Dirson <[email protected]>

run.py: allow selection of container platform

66b8f46

Necessary with podman < v5.5.1 to workaround a bug. Signed-off-by: Yann Dirson <[email protected]>

build.sh: add support for options

1ea107c

Signed-off-by: Yann Dirson <[email protected]>

build.sh: allow selection of container platform

d7a972d

Note that with podman < 5.5.1 we cannot have both linux/amd64 and linux/amd64/v2. And using a completely different arch requires emulation. Signed-off-by: Yann Dirson <[email protected]>

run.py: add --rpmbuild-opts to help use "rpmbuild --without xxx" etc

005b021

Signed-off-by: Yann Dirson <[email protected]>

run.py --local-build: detect spec file paths

e89c1d7

This allows to build packages with a flat (current Fedora/RHEL standard) source layout. Originally-by: Tu Dinh <[email protected]> Signed-off-by: Yann Dirson <[email protected]>

run.py --local-build: teach fetching source tarballs from the Almalin…

15de92e

…ux bucket Signed-off-by: Yann Dirson <[email protected]>

run.sh: --rpmbuild-stage to override "-ba"

1dc59ef

Notably useful for -bp, to - quickly get a patched source tree - apply patches using quilt, in order to refresh them in 8.3 env, so they will not be rejected by the 9.0 one Signed-off-by: Yann Dirson <[email protected]>

run.py: new --debug flag

d5d1920

It is highly disrupting to have to add `set -x` manually to the script and rebuild it, each time something goes wrong. Signed-off-by: Yann Dirson <[email protected]>

ydirson force-pushed the ydi/9 branch from 822e2fe to 9d87cb7 Compare July 11, 2025 08:23

README: provide an example to refresh patches

1ec6108

Signed-off-by: Yann Dirson <[email protected]>

ydirson force-pushed the ydi/9 branch from 9d87cb7 to 1ec6108 Compare July 11, 2025 08:48

ydirson removed the request for review from psafont July 21, 2025 09:34

glehmann reviewed Jul 21, 2025

View reviewed changes

		@@ -15,10 +15,10 @@
		import uuid

		CONTAINER_PREFIX = "xcp-ng/xcp-ng-build-env"

	CONTAINER_PREFIX = "xcp-ng/xcp-ng-build-env"
	CONTAINER_PREFIX = "localhost/xcp-ng/xcp-ng-build-env"

Support for a v9 build-env #35

Are you sure you want to change the base?

Support for a v9 build-env #35

Uh oh!

Conversation

ydirson commented Jun 26, 2025

Uh oh!

dinhngtu commented Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ydirson commented Jun 30, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ydirson commented Jul 9, 2025

Uh oh!

Uh oh!

ydirson commented Jul 10, 2025

Uh oh!

dinhngtu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

fallen commented Jul 21, 2025

Uh oh!

glehmann Jul 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ydirson commented Jul 21, 2025

Uh oh!

fallen commented Jul 21, 2025

Uh oh!

Uh oh!

dinhngtu commented Jun 30, 2025 •

edited

Loading

glehmann Jul 21, 2025 •

edited

Loading