Skip to content
Draft
Show file tree
Hide file tree
Changes from 28 commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
4e86b2c
Start by moving cross-compile section out of KB
mgorny Dec 8, 2025
6dbd8fa
Rewrite the introduction to be standalone
mgorny Dec 8, 2025
d7ea6cb
Restructure the headings
mgorny Dec 8, 2025
954380e
Replace the SO reference with inline explanation
mgorny Dec 8, 2025
32f9d0c
Add a note about misleading terms
mgorny Dec 8, 2025
1580579
Fix links
mgorny Dec 8, 2025
a06ed74
Clarify the relation between build/host nad prefixes
mgorny Dec 8, 2025
5cbac59
Try to make build_platform/target_platform clearer
mgorny Dec 8, 2025
a9f36a6
Remove incorrect make dep from meson snippet
mgorny Dec 8, 2025
5d52068
Split examples into subsections
mgorny Dec 8, 2025
eda4d9b
Update autotools/cmake/meson examples
mgorny Dec 8, 2025
a0271d3
Fix NumPy doc link
mgorny Dec 8, 2025
8ddf602
Shift selectors left in the MPI example
mgorny Dec 8, 2025
1884b9a
Add missing sub-heading for other examples
mgorny Dec 8, 2025
8147f2b
Make the NumPy/CMake example more concise
mgorny Dec 8, 2025
9eedc3a
Add stdlib('c') to numpy
mgorny Dec 8, 2025
a646d29
Use tabs for v0/v1 recipes
mgorny Dec 8, 2025
65cf675
Fix KB URL
mgorny Dec 8, 2025
f7ff02f
Remove obsolete anchors
mgorny Dec 8, 2025
2d18757
Provide RecipeTabs syntactic sugar
jaimergp Dec 9, 2025
0235757
update link
jaimergp Dec 9, 2025
ae09b5e
update again
jaimergp Dec 9, 2025
2aa709b
Remove the new cross-compile page from old sidebar
mgorny Dec 9, 2025
810b8d6
Fix missing $ in v1 variable ref
mgorny Dec 9, 2025
866b799
Talk of "platform" rather than "machine"
mgorny Dec 9, 2025
22d52d3
Rewrite "enabling" to be less target-centric
mgorny Dec 9, 2025
e578b7c
Update v1 examples to use `host_platform`
mgorny Dec 9, 2025
3a94764
Fix Python cross-compilation link
mgorny Dec 9, 2025
1d9f8ab
Apply suggestions from code review
mgorny Dec 9, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/_sidebar_diataxis.json
Original file line number Diff line number Diff line change
Expand Up @@ -100,9 +100,9 @@
"id": "how-to/advanced/talk-to-the-bots"
},
{
"type": "link",
"type": "doc",
"label": "Cross-compile",
"href": "/docs/maintainer/knowledge_base/#cross-compilation"
"id": "how-to/advanced/cross-compilation"
},
{
"type": "link",
Expand Down
382 changes: 382 additions & 0 deletions docs/how-to/advanced/cross-compilation.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,382 @@
---
title: 'Cross-compilation'
---

import { RecipeTabs } from "@site/src/components/RecipeTabs";

Cross-compiling means building a package for a different architecture or a different operating
system than the one the build process is running on. It is a common way of obtaining packages for an
architecture that conda-forge does not provide any runners for (the other available technique is
[emulation](/docs/maintainer/knowledge_base/#emulated-builds)). Given how abundant x86_64 runners
are, most common cross-compilation setups will target non-x86_64 architectures from x86_64 runners.

## Terminology

Cross-compilation terminology usually distinguishes between two types of platform:

- Build: The platform running the building process.
- Host: The platform we are building packages for.

:::note

Some cross-compilation documentation might also distinguish between a third type of platform, the
target platform. This is used primarily when building cross-compilers, and indicates the platform
for which the built package will generate code for. For the purposes of this documentation, we'll
consider this to be irrelevant and the target platform to be the same as the host.

Note that some resources may use the term "host" to refer to the build platform, and the term
"target" to refer to the host platform. This is usually the case if they are not referring to
building compilers.

:::

## How to enable cross-compilation

By default, the build scripts only enable building for platforms that feature native conda-forge
runners. To enable cross-compilation, you need to extend the
[build_platform](/docs/maintainer/conda_forge_yml/#build-platform) mapping in `conda-forge.yml`
that specifies which build platform to use to cross-compile for a specific platform.
For example, to build a `linux_aarch64` package on `linux_64` host, you would set:

```yaml
build_platform:
linux_aarch64: linux_64
```

Then [rerender](/docs/how-to/basics/rerender/) the feedstock. This will generate the appropriate CI workflows and
conda-build input metadata. See also [test](/docs/maintainer/conda_forge_yml/#test) for how to skip the test phase when
cross-compiling. Provided the requirements metadata and build scripts are written correctly, the
package should just work. However, in some cases, it'll need some adjustments; see examples below
for some common cases.

The used platforms are exposed in recipes as selectors and in the build scripts as environment
variables. For v1 recipes, the following variables are used:

- `build_platform`: The platform on which `conda-build` is running, corresponding to the `build`
environment that is made available in `$BUILD_PREFIX`.
- `host_platform`: The platform on which the package will be installed, corresponding to the `host`
environment that is made available in `$PREFIX`. For native builds, matches `build_platform`.

In v0 recipes, `target_platform` is used in place of `host_platform`.

:::note

Many existing v1 recipes are using `target_platform` instead of `host_platform`. This works because
target platform is almost always the same as host platform, though it is technically incorrect.

:::

In addition to these two variables, there are some more environment variables that are set by
conda-forge's automation (e.g. `conda-forge-ci-setup`, compiler activation packages, etc) that
can aid in cross-compilation setups:

- `CONDA_BUILD_CROSS_COMPILATION`: set to `1` when `build_platform` and `target_platform`
differ.
- `CONDA_TOOLCHAIN_BUILD`: the autoconf triplet expected for build platform.
- `CONDA_TOOLCHAIN_HOST`: the autoconf triplet expected for host platform.
- `CMAKE_ARGS`: arguments needed to cross-compile with CMake. Pass it to `cmake` in your build
script.
- `MESON_ARGS`: arguments needed to cross-compile with Meson. Pass it to `meson` in your build
script. Note a [cross build definition file](https://mesonbuild.com/Cross-compilation.html) is
automatically created for you too.
- `CC_FOR_BUILD`: C compilers targeting the build platform.
- `CXX_FOR_BUILD`: C++ compilers targeting the build platform.
- `CROSSCOMPILING_EMULATOR`: Path to the `qemu` binary for the host platform. Useful for running
tests when cross-compiling.

This is all supported by two main conda-build features introduced in version 3:

- How [requirements metadata](https://docs.conda.io/projects/conda-build/en/latest/resources/define-metadata.html#requirements-section)
is expressed in `meta.yaml`, which distinguishes between `build` and `host` platforms.
- The `compiler()` Jinja function and underlying [conventions for the compiler packages](https://docs.conda.io/projects/conda-build/en/latest/resources/compiler-tools.html).

## Placing requirements in build or host

The rule of the thumb is:

- If it needs to run during the build, it goes in `build`.
- If it needs to be available on the target host, it goes in `host`.
- If both conditions are true, it belongs in both.

However, there are some exceptions to this rule; most notably [Python cross-compilation](#details-about-cross-compiled-python-packages).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you also note that python needs to be in host even if not cross-compiled because it hasn't historically supported cross-compilation?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be honest, I don't think that's the right place to say that. I mean, people who need to depend on Python while not cross-compiling aren't going to exactly look for where to place it in docs about cross-compiling.


## Cross-compilation examples

A package needs to make a few changes in their recipe to be compatible with cross-compilation. Here
are a few examples.

### Autotools

A simple C library using autotools for cross-compilation might look like this:

<RecipeTabs>

```yaml
requirements:
build:
- {{ compiler("c") }}
- {{ stdlib("c") }}
- make
- pkg-config
- gnuconfig
host:
- libogg
```

```yaml
requirements:
build:
- ${{ compiler("c") }}
- ${{ stdlib("c") }}
- make
- pkg-config
- gnuconfig
host:
- libogg
```

</RecipeTabs>

In the build script, it would need to update the config files and guard any tests when
cross-compiling:

```bash
# Get an updated config.sub and config.guess
cp $BUILD_PREFIX/share/gnuconfig/config.* .

./configure
make -j${CPU_COUNT}

# Skip ``make check`` when cross-compiling
if [[ "${CONDA_BUILD_CROSS_COMPILATION:-}" != "1" || "${CROSSCOMPILING_EMULATOR:-}" != "" ]]; then
make check -j${CPU_COUNT}
fi
```

### CMake

A simple C++ library using CMake for cross-compilation might look like this:

<RecipeTabs>

```yaml
requirements:
build:
- {{ compiler("cxx") }}
- {{ stdlib("c") }}
- cmake
- ninja
host:
- libboost-devel
```

```yaml
requirements:
build:
- ${{ compiler("cxx") }}
- ${{ stdlib("c") }}
- cmake
- ninja
host:
- libboost-devel
```

</RecipeTabs>

In the build script, it would need to update `cmake` call and guard any tests when cross-compiling:

```bash
# Pass ``CMAKE_ARGS`` to ``cmake``
cmake ${CMAKE_ARGS} -G Ninja ..
cmake --build .

# Skip ``ctest`` when cross-compiling
if [[ "${CONDA_BUILD_CROSS_COMPILATION:-}" != "1" || "${CROSSCOMPILING_EMULATOR:-}" != "" ]]; then
ctest
fi
```

### Meson

Similarly, with Meson, the `meta.yaml` needs:

<RecipeTabs>

```yaml
requirements:
build:
- {{ compiler("c") }}
- {{ compiler("cxx") }}
- {{ stdlib("c") }}
- meson
- pkg-config
host:
- libogg
```

```yaml
requirements:
build:
- ${{ compiler("c") }}
- ${{ compiler("cxx") }}
- ${{ stdlib("c") }}
- meson
- pkg-config
host:
- libogg
```

</RecipeTabs>

And this in `build.sh`:

```bash
# Pass ``MESON_ARGS`` to ``meson``
meson setup ${MESON_ARGS} ..
meson compile
```

### Python

A simple Python extension using Cython and NumPy's C API would look like so:

<RecipeTabs>

```yaml
requirements:
build:
- {{ compiler("c") }}
- {{ stdlib("c") }}
- cross-python_{{ target_platform }} # [build_platform != target_platform]
- python # [build_platform != target_platform]
- cython # [build_platform != target_platform]
- numpy # [build_platform != target_platform]
host:
- python
- pip
- cython
- numpy
run:
- python
```

```yaml
requirements:
build:
- ${{ compiler("c") }}
- ${{ stdlib("c") }}
- if: build_platform != host_platform
then:
- cross-python_${{ host_platform }}
- python
- cython
- numpy
host:
- python
- pip
- cython
- numpy
run:
- python
```

</RecipeTabs>

For more details about NumPy see [Building against NumPy](/docs/maintainer/knowledge_base/#building-against-numpy).

### MPI

With MPI, openmpi is required for the build platform as the compiler wrappers are binaries, but mpich is not required as the compiler wrappers are scripts (see [example](https://github.com/conda-forge/mpi4py-feedstock/blob/743d379c4a04/recipe/meta.yaml#L37)):

<RecipeTabs>

```yaml
requirements:
build:
- {{ mpi }} # [build_platform != target_platform and mpi == "openmpi"]
host:
- {{ mpi }}
run:
- {{ mpi }}
```

```yaml
requirements:
build:
- if: build_platform != host_platform and mpi == "openmpi"
then: ${{ mpi }}
host:
- ${{ mpi }}
run:
- ${{ mpi }}
```

</RecipeTabs>

In the build script, openmpi compiler wrappers can use host libraries by setting the environmental variable `OPAL_PREFIX` to `$PREFIX`.

```sh
if [[ "$CONDA_BUILD_CROSS_COMPILATION" == "1" && "${mpi}" == "openmpi" ]]; then
export OPAL_PREFIX="$PREFIX"
fi
```

### Other examples

There are more variations of this approach in the wild. So this is not meant to be exhaustive,
but merely to provide a starting point with some guidelines. Please look at [other recipes for more examples](https://github.com/search?q=org%3Aconda-forge+path%3Arecipe%2Fmeta.yaml+%22%5Bbuild_platform+%21%3D+target_platform%5D%22&type=code).

## Finding NumPy in cross-compiled Python packages using CMake

If you are building a Python extension via CMake with NumPy and you want it to work in
cross-compilation, you need to prepend to the CMake invocation in your build script the following
lines:

```sh
Python_INCLUDE_DIR="$(python -c 'import sysconfig; print(sysconfig.get_path("include"))')"
Python_NumPy_INCLUDE_DIR="$(python -c 'import numpy; print(numpy.get_include())')"
# usually either Python_* or Python3_* lines are sufficient
CMAKE_ARGS+=" -DPython_EXECUTABLE:PATH=${PYTHON}"
CMAKE_ARGS+=" -DPython_INCLUDE_DIR:PATH=${Python_INCLUDE_DIR}"
CMAKE_ARGS+=" -DPython_NumPy_INCLUDE_DIR=${Python_NumPy_INCLUDE_DIR}"
CMAKE_ARGS+=" -DPython3_EXECUTABLE:PATH=${PYTHON}"
CMAKE_ARGS+=" -DPython3_INCLUDE_DIR:PATH=${Python_INCLUDE_DIR}"
CMAKE_ARGS+=" -DPython3_NumPy_INCLUDE_DIR=${Python_NumPy_INCLUDE_DIR}"
```

## Details about cross-compiled Python packages

Cross-compiling Python packages is a bit more involved than other packages. The main pain point is
that we need an executable Python interpreter (i.e. `python` in `build`) that knows how to
provide accurate information about the target platform. Since this is not officially supported, a
series of workarounds are required to make it work. Refer to [PEP720](https://peps.python.org/pep-0720/) or [the discussion in this issue](https://github.com/conda-forge/conda-forge.github.io/issues/1841) for more information.

In practical terms, for conda-forge, this results into two extra metadata bits that are needed in
`meta.yaml`:

- Adding `cross-python_{{ target_platform }}` in `build` requirements, provided by the
[cross-python-feedstock](https://github.com/conda-forge/cross-python-feedstock). This is a
wrapper for the `crossenv` Python interpreters with [some activation logic that adjust some of
the crossenv workarounds](https://github.com/conda-forge/cross-python-feedstock/blob/main/recipe/activate-cross-python.sh)
so they work better with the conda-build setup.
- Copying some Python-related packages from `host` to `build` with a `[build_platform !=
target_platform]` selector:
- `python` itself, to support `crossenv`.
- Non-pure Python packages (i.e. they ship compiled libraries) that need to be present while the
package is being built, like `cython` and `numpy`.

In the terms of the [PEP720](https://peps.python.org/pep-0720/), the conda-forge setup
implements the "faking the target environment" approach. More specifically, this will result in the
following changes before the builds scripts run:

- A modified `crossenv` installation in `$BUILD_PREFIX/venv`, mimicking the architecture of
`$PREFIX`.
- Forwarder binaries in `$BUILD_PREFIX/bin` that point to the `crossenv` installation.
- Symlinks that expose the `$BUILD_PREFIX` site-packages in the `crossenv` installation, which
is also included in `$PYTHONPATH`.
- A copy of all `$PREFIX` site-packages to `$BUILD_PREFIX` (except the compiled libraries).

All in all, this results in a setup where `conda-build` can run a `$BUILD_PREFIX`-architecture
`python` interpreter that can see the packages in `$PREFIX` (with the compiled bits provided by
their corresponding counterparts in `$BUILD_PREFIX`) and sufficiently mimic that target
architecture.
Loading
Loading