Skip to content

Properly package the python script #36

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# Distribution / packaging
build/
*.egg-info/
__pycache__/
95 changes: 31 additions & 64 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,88 +22,55 @@ If you have both installed, docker will be used by default. If you
want to use a specific container runtime, set `XCPNG_OCI_RUNNER` to
the docker-compatible command to use (typically `podman` or `docker`).

You'll need to install git-lfs to be able to download the source tarballs from
git, otherwise when running xcp-ng-dev, it won't be able to extract the sources.

## Installation

Clone this repository and install the `xcp-ng-dev` package:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not fully convinced by the xcp-ng-dev name for a command. It probably lacks an action verb for me. I also find xcp-ng-dev-build ambiguous because when we run the build env, that's also for building stuff usually. And we "build" software a lot more often than we "build" the container. Not that the old names are better, sure :).

Something that is not quite right at the moment is also the fact that we have a single multipurpose run.py command. How about adding a mandatory parameter to it that says what action it's supposed to do? start to start the container without automatically starting a build, and build to start the container, install the deps, and build. These are the two main uses of the container at the moment. And, translated as installed commands from the package, this would give:

  • xcp-ng-build-env-run = start the container (== run.py start)
  • xcp-ng-dev-build = start the container, install the deps, do the build (== run.py build)
  • xcp-ng-build-env-create = avoiding the word "build" here, but that's actually build.sh that could probably rename to create.sh.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe shell would be a more descriptive action than start in this context? (xcp-ng-build-env-shell?) (OK it's not an action verb, but maybe still descriptive enough?)

xcp-ng-dev-build = start the container, install the deps, do the build (== run.py build)

did you mean xcp-ng-build-env-build?

In any case, using separate entrypoints would require some surgery in the previously-run.py, and #35 already touches quite a bit, maybe this idea can be implemented after that other PR?

Copy link
Member

@stormi stormi Jul 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

xcp-ng-build-env-build felt awkward so I figured it was not so important to have the same prefix for every command and we could have a shorter one just for the one we use most. And it does perform a build in the context of development. Looks like this idea hasn't seduced you :)

maybe this idea can be implemented after that other PR

Yes, but I think we should at least group "packaging the scripts" and "creating separate entry points". So maybe both after #35?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

xcp-ng-build-env-build felt awkward

Maybe xcpng-buildenv-build would feel less so (I'm not a big fan of too many hyphens 😉)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not fully convinced by the xcp-ng-dev name for a command. It probably lacks an action verb for me.

It's missing it on purpose: the command can do several things: drop into a shell, or do a local build, for example.

How about adding a mandatory parameter to it that says what action it's supposed to do?

That's what I wanted to explore next, but wanted to restrict the scope of the PR because I haven't used the tool too much.

In general I would like the command to be just one, and feed it actions:xcp-ng-dev shell, xcp-ng-dev build

xcp-ng-build-env-create = avoiding the word "build" here, but that's actually build.sh that could probably rename to create.sh.

Yes, it's awkward as it is, using env-create might be better, but I do want to keep the common prefix, it's different from the other commands as they have their action separate from the command. So I'll go with xcp-ng-dev-env-create.

I do think that #35 should be merged before this PR, and others that change the interface.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

xcp-ng-dev shell, xcp-ng-dev build and xcp-ng-dev-env-create look like good names to me.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've renamed build.sh to xcp-ng-dev-env-create. We can rework xcp-ng-dev later

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Personnally, I'd prefer one single big change in how the build env is supposed to be used rather than two successive changes, as in each case people will have to change their habits.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll wait to do the rework until the PR for xcp 9 is merged, then.


```bash
git clone xcp-ng/xcp-ng-build-env
cd xcp-ng-build-env
uv tool install --editable .
```

If `uv` is not available you can use other tools to install python packages,
like `pipx install --editable .`

Using the `--editable` flag will allow you to update the cli tools just by
updating the contents of the git repository, without reinstalling.
If you do not want this behaviour, use: `uv tool install --from . xcp-ng-dev`
or `pipx install .`

After this, two new commands will be available: `xcp-ng-dev-env-create` and
`xcp-ng-dev`.

## Building the container image(s)

You need one container image per target version of XCP-ng.

Clone this repository (outside any container), then use `build.sh` to
Clone this repository (outside any container), then use `xcp-ng-dev-env-create` to
generate the images for the wanted releases of XCP-ng.
Note that Docker and Podman store container images separately.

```
Usage: ./build.sh {version_of_XCP_ng}
Usage: xcp-ng-dev-env-create {version_of_XCP_ng}
... where {version_of_XCP_ng} is a 'x.y' version such as 8.0.
```

## Using the container

Use the `run.py` script. It accepts a variety of parameters allowing for different uses:
Use `xcp-ng-dev`. It accepts a variety of parameters allowing for different uses:
* rebuild an existing source RPM (with automated installation of the build dependencies)
* build a package from an already extracted source RPM (sources and spec file), or from a directory that follows the rpmbuild convention (a `SOURCES/` directory and a `SPECS/` directory). Most useful for building packages from XCP-ng's git repositories of RPM sources: https://github.com/xcp-ng-rpms.
* or simply start a shell in the build environment, with the appropriate CentOS, EPEL and XCP-ng yum repositories enabled.

```sh
usage: run.py [-h] [-b BRANCH] [-l BUILD_LOCAL] [--define DEFINE]
[-r REBUILD_SRPM] [-o OUTPUT_DIR] [-n] [-p PACKAGE] [-s SRPM]
[-d DIR] [-e ENV] [-v VOLUME] [--rm] [--syslog] [--name NAME]
[-a ENABLEREPO] [--fail-on-error]
...

positional arguments:
command Command to run inside the prepared container

optional arguments:
-h, --help show this help message and exit
-b BRANCH, --branch BRANCH
XCP-ng version: 7.6, 8.0, etc. If not set, will
default to 8.0.
-l BUILD_LOCAL, --build-local BUILD_LOCAL
Install dependencies for the spec file(s) found in the
SPECS/ subdirectory of the directory passed as
parameter, then build the RPM(s). Built RPMs and SRPMs
will be in RPMS/ and SRPMS/ subdirectories. Any
preexisting BUILD, BUILDROOT, RPMS or SRPMS
directories will be removed first. If --output-dir is
set, the RPMS and SRPMS directories will be copied to
it after the build.
--define DEFINE Definitions to be passed to rpmbuild (if --build-local
or --rebuild-srpm are passed too). Example: --define
'xcp_ng_section extras', for building the 'extras'
version of a package which exists in both 'base' and
'extras' versions.
-r REBUILD_SRPM, --rebuild-srpm REBUILD_SRPM
Install dependencies for the SRPM passed as parameter,
then build it. Requires the --output-dir parameter to
be set.
-o OUTPUT_DIR, --output-dir OUTPUT_DIR
Output directory for --rebuild-srpm and --build-local.
-n, --no-exit After executing either an automated build or a custom
command passed as parameter, drop user into a shell
-p PACKAGE, --package PACKAGE
Packages for which dependencies will be installed
-s SRPM, --srpm SRPM SRPMs for which dependencies will be installed
-d DIR, --dir DIR Local dir to mount in the image. Will be mounted at
/external/<dirname>
-e ENV, --env ENV Environment variables passed directly to docker -e
-v VOLUME, --volume VOLUME
Volume mounts passed directly to docker -v
--rm Destroy the container on exit
--syslog Enable syslog to host by mounting in /dev/log
--name NAME Assign a name to the container
-a ENABLEREPO, --enablerepo ENABLEREPO
additional repositories to enable before installing
build dependencies. Same syntax as yum's --enablerepo
parameter. Available additional repositories: xcp-ng-
updates_testing, xcp-ng-extras, xcp-ng-extras_testing.
--fail-on-error If container initialisation fails, exit rather than
dropping the user into a command shell
```

**Examples**

Rebuild an existing source RPM (with automated installation of the build dependencies)
```sh
./run.py -b 8.0 --rebuild-srpm /path/to/some-source-rpm.src.rpm --output-dir /path/to/output/directory --rm
xcp-ng-dev -b 8.2 --rebuild-srpm /path/to/some-source-rpm.src.rpm --output-dir /path/to/output/directory --rm
```

Build from git (and put the result into RPMS/ and SRPMS/ subdirectories)
Expand All @@ -116,7 +83,7 @@ git clone https://github.com/xcp-ng-rpms/xapi.git
# ... Here add your patches ...

# Build.
/path/to/run.py -b 8.0 --build-local xapi/ --rm
xcp-ng-dev -b 8.2 --build-local xapi/ --rm
```

**Important switches**
Expand Down Expand Up @@ -157,11 +124,11 @@ make

If you'd like to develop using the tools on your host and preserve the changes
to source and revision control but still use the container for building, you
can do using by mouning a volume in the container, using the `-v` option to mount
can do so by mounting a volume in the container, using the `-v` option to mount
a directory from your host to a suitable point inside the container. For
example, if I clone some repos into a directory on my host, say `/work/code/`,
then I can mount it inside the container as follows:

```sh
./run.py -b 8.0 -v /work/code:/mnt/repos
xcp-ng-dev -b 8.2 -v /work/code:/mnt/repos
```
21 changes: 21 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
[project]
name = "xcp-ng-build"
description = "A tool to create XCP-ng-based build environments"
readme = "README.md"
requires-python = ">=3.9"
license = "MIT"
license-files = ["LICENSE"]
dynamic = ["version"]

[project.scripts]
xcp-ng-dev = "xcp_ng_dev.cli:main"
xcp-ng-dev-env-create = "xcp_ng_dev.cli:build"

[project.urls]
Homepage = "https://github.com/xcp-ng/xcp-ng-build-env/"

[build-system]
requires = ["setuptools >= 77.0.3", "setuptools-scm>=8"]
build-backend = "setuptools.build_meta"

[tool.setuptools_scm]
Empty file added src/xcp_ng_dev/__init__.py
Empty file.
8 changes: 5 additions & 3 deletions build.sh → src/xcp_ng_dev/build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,10 @@

set -e

SELF_NAME="xcp-ng-dev-env-create"

if [ -z "$1" ]; then
echo "Usage: $0 {version}"
echo "Usage: $SELF_NAME {version}"
echo "... where {version} is a 'x.y' version such as 8.0."
exit
fi
Expand Down Expand Up @@ -32,12 +34,12 @@ CUSTOM_ARGS=()
case "$1" in
7.*)
REPO_FILE=files/xcp-ng.repo.7.x.in
DOCKERFILE=Dockerfile-7.x
DOCKERFILE=files/Dockerfile-7.x
CENTOS_VERSION=7.2.1511
;;
8.*)
REPO_FILE=files/xcp-ng.repo.8.x.in
DOCKERFILE=Dockerfile-8.x
DOCKERFILE=files/Dockerfile-8.x
CENTOS_VERSION=7.5.1804
;;
*)
Expand Down
8 changes: 6 additions & 2 deletions run.py → src/xcp_ng_dev/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
SRPMS_MOUNT_ROOT = "/tmp/docker-SRPMS"

DEFAULT_BRANCH = '8.3'
DEFAULT_ULIMIT_NOFILE = 1024
DEFAULT_ULIMIT_NOFILE = 2048

RUNNER = os.getenv("XCPNG_OCI_RUNNER")
if RUNNER is None:
Expand Down Expand Up @@ -125,7 +125,7 @@ def main():
if args.command != []:
docker_args += ["-e", "COMMAND=%s" % ' '.join(args.command)]
if args.build_local:
docker_args += ["-v", "%s:/home/builder/rpmbuild" %
docker_args += ["-v", "%s:/home/builder/rpmbuild:z" %
os.path.abspath(args.build_local)]
docker_args += ["-e", "BUILD_LOCAL=1"]
if args.define:
Expand Down Expand Up @@ -204,6 +204,10 @@ def main():

sys.exit(return_code)

def build():
bargs = [os.path.join(os.path.dirname(__file__ ), 'build.sh')] + sys.argv[1:]
return_code = subprocess.call(bargs)
sys.exit(return_code)

if __name__ == "__main__":
main()
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.