mxdev [mɪks dɛv] is a utility that makes it easy to work with Python projects containing lots of packages, of which you only want to develop some.
It builds on top of the idea to have stable version constraints and then develop from a VCS on top of it.
As part of the above use-case sometimes versions of the stable constraints need an override with a different (i.e. newer) version. Other software follow the same idea are mr.developer for Python's zc.buildout or mrs-developer for NPM packages.
mxdev procedure is:
- Configuration is read,
- Requirements and constraints (given in the configuration) are read.
- Sources from VCS are fetched into a target directory,
- Modified constraints (handled packages commented, overridden versions replaced) and requirements (handled packages as editable from sources) are written.
mxdev will not run pip for you!
pip install mxdevmxdev >=4.0 needs pip version 23 at minimum to work properly
- Create
mx.iniconfiguration file:
[settings]
requirements-in = requirements.txt
requirements-out = requirements-mxdev.txt
constraints-out = constraints-mxdev.txt
# Custom variables for reuse
github = git+ssh://[email protected]/
[mypackage]
url = ${settings:github}myorg/mypackage.git
branch = main
extras = test- Run mxdev to fetch sources and generate files:
mxdev- Install with pip using the generated files:
pip install -r requirements-mxdev.txtFor more examples see the example/ directory.
Configuration is done in an INI file (default: mx.ini) using configparser.ExtendedInterpolation syntax.
The main section must be called [settings], even if kept empty.
| Option | Description | Default |
|---|---|---|
requirements-in |
Input requirements file (can be URL). Empty value = generate from INI only | requirements.txt |
requirements-out |
Output requirements with development sources as -e entries |
requirements-mxdev.txt |
constraints-out |
Output constraints (developed packages commented out) | constraints-mxdev.txt |
| Option | Description | Default |
|---|---|---|
default-target |
Target directory for VCS checkouts | ./sources |
threads |
Number of parallel threads for fetching sources | 4 |
smart-threading |
Process HTTPS packages serially to avoid overlapping credential prompts (see below) | True |
offline |
Skip all VCS fetch operations (handy for offline work) | False |
default-install-mode |
Default install-mode for packages: direct or skip |
direct |
default-update |
Default update behavior: yes or no |
yes |
default-use |
Default use behavior (when false, sources not checked out) | True |
When smart-threading is enabled (default), mxdev uses a two-phase approach to prevent credential prompts from overlapping:
- Phase 1: HTTPS packages are processed serially (one at a time) to ensure clean, visible credential prompts
- Phase 2: Remaining packages (SSH, local) are processed in parallel for speed
This solves the problem where parallel git operations would cause multiple credential prompts to overlap, making it confusing which package needs credentials.
When to disable: Set smart-threading = false if you have git credential helpers configured (e.g., credential cache, credential store) and never see prompts.
Override package versions which are already defined in a dependent constraints file.
I.e. an upstream constraints.txt contains already somefancypackage==2.0.3.
Given that, for some reason (like with my further developed sources), we need version 3.0.0 of the above package.
Then in this section, this can be defined as:
[settings]
version-overrides =
somefancypackage==3.0.0
otherpackage==33.12.1It is possible to add as many overrides as needed. When writing the constraints-out, the new version will be taken into account. If there is a source section defined for the same package, the source will be used and entries here are ignored.
Note: When using uv pip install the version overrides here are not needed, since it supports overrides natively.
With uv it is recommended to create an overrides.txt file with the version overrides and use uv pip install --override overrides.txt [..] to install the packages.
Ignore packages that are already defined in a dependent constraints file.
No new version will be provided.
This is specifically handy if a package is going to be installed editable from local file system (like -e .), but was already pinned in an upstream constraints file.
This can be defined as:
[settings]
ignores =
somefancypackage
otherpackagemxdev can handle one Python package as main package directly via ini config. If defined, it will be added as last entry in the resulting requirements out file.
This can be defined as:
[settings]
main-package = -e .[test]If the main package is defined in a dependent constraint file, its name must be added to ignores.
Include one or more other INI files.
The included file is read before the main file, so the main file overrides included settings. Included files may include other files. Innermost inclusions are read first.
If an included file is an HTTP-URL, it is loaded from there.
If the included file is a relative path, it is loaded relative to the parent's directory or URL.
Default: empty
mxdev provides a default setting containing the current working directory which can be used inside package or custom sections:
[sectionname]
param = ${settings:directory}/some/pathAdditionally, custom variables can be defined as key = value pair.
Those can be referenced in other values as ${settings:key} and will be expanded there.
[settings]
github = git+ssh://[email protected]/
gitlab = git+ssh://[email protected]/Sections other than [settings] can define:
- Package sources:
[PACKAGENAME]- VCS sources to checkout and develop - Hook configuration:
[hookname-section]- Settings for mxdev extensions (see EXTENDING.md)
For package sources, the section name is the package name: [PACKAGENAME]
| Option | Type | Description | Default |
|---|---|---|---|
url |
required | VCS checkout URL | — |
vcs |
optional | Version control system: git, fs, svn, gitsvn, hg, bzr, darcs |
git |
branch |
optional | Branch name or tag to checkout | main |
extras |
optional | Comma-separated package extras (e.g., test,dev) |
empty |
subdirectory |
optional | Path to Python package when not in repository root | empty |
target |
optional | Custom target directory (overrides default-target) |
default-target |
pushurl |
optional | Writable URL for pushes (not applied after initial checkout) | — |
VCS Support Status:
git(stable, tested)fs(stable, tested) - local directory pseudo-VCSsvn,gitsvn,hg,bzr,darcs(unstable, tests need rewrite)
| Option | Description | Default |
|---|---|---|
install-mode |
direct: Install with pip -e PACKAGEPATHskip: Only clone, don't install |
default-install-mode |
use |
When false, source is not checked out and version not overridden |
default-use |
| Option | Description | Default |
|---|---|---|
depth |
Git clone depth (shallow clone). Set GIT_CLONE_DEPTH=1 env var for global default |
full clone |
submodules |
Submodule handling: always, checkout, recursive (see below) |
always |
always(default): Git submodules will always be checked out, updated if already presentcheckout: Submodules only fetched during checkout, existing submodules stay untouchedrecursive: Fetches submodules recursively, results ingit clone --recurse-submoduleson checkout andsubmodule update --init --recursiveon update
Run mxdev (for more options run mxdev --help).
Mxdev will
- read the configuration from
mx.ini, - fetch the packages defined in the config file and
- write a requirements and constraints file.
Now, use the generated requirements and constraints files with i.e. pip install -r requirements-mxdev.txt.
This looks like so:
[settings]
requirements-in = requirements.txt
requirements-out = requirements-mxdev.txt
constraints-out = constraints-mxdev.txt
version-overrides =
baz.baaz==1.9.32
ignores =
my.ignoredpackage
# custom variables
github = git+ssh://[email protected]/
mygit = git+ssh://[email protected]/
[foo.bar]
url = ${settings:github}orga/foo.bar.git
branch = fix99
extras = test,baz
[kup.fancyproject]
url = ${settings:mygit}customers/fancycorp/kup.fancyproject.git
branch = fix99
extras = test,bazFor comprehensive examples demonstrating all features, see the example/ directory.
The functionality of mxdev can be extended by hooks. This is useful to generate additional scripts or files or automate any other setup steps related to mxdev's domain.
See EXTENDING.md for complete documentation on creating mxdev extensions.
There is a constraint file like -c constraints.txt with a package foo.bar with a version pin.
Then it is not possible to install this package in a requirements file editable like -r requirements.txt with -e git+ssh://[email protected]/orga/foo.bar.git@fix-99.
Neither it is possible to override inherited version constraints with custom ones.
A pre-processor fetches (as this can be an URL) and expands all -c SOMEOTHER_FILE_OR_URL and -r SOMEOTHER_FILE_OR_URL files into one, filtering out all packages given in a configuration file.
For each of those packages, a -e ... entry is generated instead and written to a new TARGET.txt.
Same is true for version overrides: a new entry is written to the resulting constraints file while the original version is disabled.
The configuration is read from a file mx.ini in ExtendedInterpolation INI syntax (YAML would be nice, but the package must have as less dependencies as possible to other packages).
Mx (generally pronounced like mix [mɪks], or [məks] in the UK) is meant to be a gender-neutral alternative to the titles Mr. and Ms. but also associates with the word "mix".
The VCS-related code is taken from mr.developer.
Thanks to Florian Schulze and Contributors.