Make Windows runner subsystem production ready

.github/workflows/tests.yaml

@@ -49,8 +49,8 @@ jobs:
     - name: Run tests
       # FIXME: Can this be run without elevated privileges?
       run: |
-        sudo $(command -v pytest)
-        coverage xml
+        sudo --preserve-env $(command -v pytest)
+        coverage xml --omit postroj/winrunner.py
 
     - name: Upload coverage results to Codecov
       uses: codecov/codecov-action@v3

CHANGES.rst

@@ -11,6 +11,22 @@ in progress
 - Improve platform guards and naming things
 - Improve central command invocation function
 - Improve documentation
+- Make Windows runner subsystem production ready.
+- Add support for Windows Server Core 2019 and friends, like
+  ``windows/servercore:ltsc2019``, ``windows/nanoserver:1809``, or
+  ``eclipse-temurin:17-jdk``.
+- Add support for Windows Server Core 2016, 2022 and friends, like
+  ``windows/servercore:ltsc2016``, ``windows/servercore:ltsc2022``, or
+  ``windows/nanoserver:ltsc2022``.
+- Improve documentation about the Windows backend
+- Rename environment variables used to control the Windows Docker Machine
+  subsystem. The new names are ``RACKER_WDM_VCPUS``, ``RACKER_WDM_MEMORY``,
+  and ``RACKER_WDM_MACHINE``.
+- Add environment variable ``RACKER_WDM_PROVIDER`` to reconfigure the
+  Vagrant virtualization backend differently than VirtualBox.
+- Documentation: Add use case how to build a Python package within a
+  Windows environment, using Microsoft Visual C++ Build Tools 2015 and
+  Anaconda, both installed using Chocolatey, and ``cibuildwheel``.
 
 
 2022-05-20 0.2.0

README.rst

@@ -109,6 +109,13 @@ another one for Windows.
   `systemd-nspawn`_. Provisioning of additional software is performed using the
   native package manager of the corresponding Linux distribution.
 
+- For running Windows operating systems containers, Racker uses `Vagrant`_,
+  `Docker`_, and `Windows Docker Machine`_. The virtual machine base image is
+  acquired from `Vagrant Cloud`_, container images are acquired from the
+  `Microsoft Container Registry`_. For provisioning additional software, the
+  `Chocolatey`_ package manager is used. All of cmd, PowerShell and Bash are
+  pre-installed on the container images.
+
 
 Operating system coverage
 -------------------------
@@ -132,6 +139,11 @@ Linux
 - SUSE SLES 15 and BCI:latest
 - Ubuntu LTS 20 and 22 (focal, jammy)
 
+Windows
+.......
+- Windows Server Core LTSC 2016, 2019, and 2022
+- Windows Nano Server 1809 and LTSC 2022
+
 
 Prior art
 ---------
@@ -212,6 +224,8 @@ Racker
 The ``racker`` program aims to resemble the semantics of Docker by providing a
 command line interface compatible with the ``docker`` command.
 
+Linux
+-----
 ::
 
     # Invoke the vanilla Docker `hello-world` image.
@@ -244,6 +258,20 @@ command line interface compatible with the ``docker`` command.
     time echo "hello world" | racker run -it --rm fedora:37 cat /dev/stdin > hello
     cat hello
 
+Windows
+-------
+
+An example of a basic command line invocation should get you started,
+especially if you are familiar with the ``docker`` command::
+
+    racker --verbose run --rm --platform=windows/amd64 mcr.microsoft.com/windows/servercore:ltsc2022 -- wmic os get caption
+
+    Caption
+    Microsoft Windows Server 2022 Datacenter
+
+More extensive information, including many examples, can be found at the
+`Racker Windows backend`_ documentation.
+
 
 Postroj
 =======
@@ -429,6 +457,7 @@ Troubleshooting
 .. _Packer: https://www.packer.io/
 .. _Podman: https://podman.io/
 .. _Racker sandbox installation: https://github.com/cicerops/racker/blob/main/doc/sandbox.rst
+.. _Racker Windows backend: https://github.com/cicerops/racker/blob/main/doc/winrunner.rst
 .. _skopeo: https://github.com/containers/skopeo
 .. _systemd: https://www.freedesktop.org/wiki/Software/systemd/
 .. _systemd-nspawn: https://www.freedesktop.org/software/systemd/man/systemd-nspawn.html

doc/cratedb.rst

@@ -1,10 +1,53 @@
-#########################
-Using postroj for CrateDB
-#########################
+#######################################
+Using Racker and Postroj for CrateDB CI
+#######################################
 
-.. note::
 
-    This is still a work in progress.
+**********
+racker run
+**********
+
+Purpose: Invoke programs in a Java/OpenJDK environment, within a
+virtualized/dockerized, volatile/ephemeral Windows environment.
+
+Run the CrateDB test suite on OpenJDK 18 (Eclipse Temurin)::
+
+    time racker --verbose run --rm --platform=windows/amd64 eclipse-temurin:18-jdk \
+        "sh -c 'mkdir /c/src; cd /c/src; git clone https://github.com/crate/crate --depth=1; cd crate; ./gradlew --no-daemon --parallel -PtestForks=2 :server:test -Dtests.crate.run-windows-incompatible=false --stacktrace'"
+
+Use the same image, but select a specific operating system version::
+
+    export RACKER_WDM_MACHINE=2019-box
+    racker --verbose run --rm --platform=windows/amd64 eclipse-temurin:18-jdk -- wmic os get caption
+
+Invoke a Java command prompt (JShell) with OpenJDK 18::
+
+    racker --verbose run -it --rm --platform=windows/amd64 eclipse-temurin:18-jdk jshell
+    System.out.println("OS: " + System.getProperty("os.name") + ", version " + System.getProperty("os.version"))
+    System.out.println("Java: " + System.getProperty("java.vendor") + ", version " + System.getProperty("java.version"))
+
+Build CrateDB from source, extract Zip archive, and invoke available programs::
+
+    # Spawn a Windows environment with `cmd` shell.
+    # TODO: Bind-mounting not possible via command line yet, need to touch the code for this.
+    racker --verbose run --rm -it --platform=windows/amd64 eclipse-temurin:18-jdk -- cmd
+
+    # Build CrateDB.
+    cd \crate
+    gradlew clean distZip
+
+    # Extract zip archive.
+    mkdir \tmp
+    cd \tmp
+    unzip \crate\app\build\distributions\crate-5.3.0-SNAPSHOT-327070e3fe.zip
+    cd crate-5.3.0-SNAPSHOT-327070e3fe
+
+    # Run CrateDB and tools.
+    bin\crate
+    # Submit <Ctrl+C> to stop
+
+    bin\crate-node fix-metadata
+    # Send y to the prompt
 
 
 ****************

doc/use-cases/python-on-windows.rst

@@ -0,0 +1,74 @@
+###############################
+Use cases for Python on Windows
+###############################
+
+
+*************************
+Build wheels for PyTables
+*************************
+
+About
+=====
+
+DIY, without a hosted CI provider.
+
+How to build a Python wheel package, here PyTables, within a Windows
+environment, using Microsoft Visual C++ Build Tools 2015 and Anaconda, both
+installed using Chocolatey, and ``cibuildwheel``.
+
+References
+==========
+
+- https://github.com/PyTables/PyTables/pull/872#issuecomment-773535041
+- https://github.com/PyTables/PyTables/blob/master/.github/workflows/wheels.yml
+
+Synopsis
+========
+
+.. note::
+
+    The ``windows-pytables-wheel.sh`` program is part of this repository. You
+    will only find it at the designated location when running ``racker`` from
+    the working tree of its Git repository.
+
+    You still can get hold of the program and invoke it, by downloading it from
+    `windows-pytables-wheel.sh`_.
+
+So, let's start by defining the download URL to that file::
+
+    export PAYLOAD_URL=https://raw.githubusercontent.com/cicerops/racker/windows/doc/use-cases/windows-pytables-wheel.sh
+
+Unattended::
+
+    time racker --verbose run --rm --platform=windows/amd64 python:3.9 -- \
+        "sh -c 'wget ${PAYLOAD_URL}; sh windows-pytables-wheel.sh'"
+
+Or, interactively::
+
+    racker --verbose run -it --rm --platform=windows/amd64 python:3.9 -- bash
+    wget ${PAYLOAD_URL}
+    sh windows-pytables-wheel.sh
+
+
+Future
+======
+
+See https://github.com/cicerops/racker/issues/8.
+
+When working on the code base, you can invoke the program directly from
+the repository, after the ``--volume`` option got implemented::
+
+    # Unattended.
+    time racker --verbose run --rm \
+        --volume=C:/Users/amo/dev/cicerops-foss/sources/postroj:C:/racker \
+        --platform=windows/amd64 python:3.9 -- \
+        sh /c/racker/doc/use-cases/windows-pytables-wheel.sh
+
+    # Interactively.
+    racker --verbose run -it --rm \
+        --volume=C:/Users/amo/dev/cicerops-foss/sources/postroj:C:/racker \
+        --platform=windows/amd64 python:3.9 -- bash
+    /c/racker/doc/use-cases/windows-pytables-wheel.sh
+
+
+.. _windows-pytables-wheel.sh: https://raw.githubusercontent.com/cicerops/racker/main/doc/use-cases/windows-pytables-wheel.sh

doc/use-cases/windows-pytables-wheel.sh

@@ -0,0 +1,75 @@
+#!/bin/bash
+#
+# Build wheels for PyTables on Windows. DIY, without a hosted CI provider.
+# https://github.com/cicerops/racker/blob/main/doc/use-cases/python-on-windows.rst
+#
+# Synopsis::
+#
+#   racker --verbose run -it --rm --platform=windows/amd64 python:3.9 -- bash
+#   /c/racker/doc/use-cases/windows-pytables-wheel.sh
+#
+set -e
+
+# Install prerequisites.
+
+# Miniconda - A minimal installer for Anaconda.
+# https://conda.io/miniconda.html
+scoop bucket add extras
+scoop install miniconda3
+#export PATH="$PATH:/c/Users/ContainerAdministrator/scoop/apps/miniconda3/current/condabin"
+#alias conda=conda.bat
+#/c/Users/ContainerAdministrator/scoop/apps/miniconda3/current/condabin/conda.bat init bash
+source /c/Users/ContainerAdministrator/scoop/apps/miniconda3/current/Scripts/activate
+conda --version
+
+# /c/Users/ContainerAdministrator/scoop/apps/miniconda3/4.12.0/Scripts/conda.exe
+
+# TODO: `/AddToPath:1` seems to not work, so adjust `$PATH` manually.
+#export PATH="$PATH:/c/Tools/miniconda3/condabin"
+
+# TODO: At least within Bash, just addressing `conda` does not work.
+#export conda="conda.bat"
+
+
+# Check prerequisites.
+#echo $PATH
+#$conda --version
+# cibuildwheel --version
+
+# Pretend to be on a build matrix.
+export MATRIX_ARCH=win_amd64      # win32
+export MATRIX_ARCH_SUBDIR=win-64  # win-32
+
+# Activate and prepare Anaconda environment for building.
+conda create --yes --name=build
+conda activate build
+conda config --env --set subdir ${MATRIX_ARCH_SUBDIR}
+
+# Install needed libraries.
+conda install --yes blosc bzip2 hdf5 lz4 lzo snappy zstd zlib
+
+# Install cibuildwheel.
+# Build Python wheels for all the platforms on CI with minimal configuration.
+# https://cibuildwheel.readthedocs.io/
+pip install --upgrade cibuildwheel
+
+# Configure cibuildwheel.
+#export CIBW_BUILD="cp36-${MATRIX_ARCH} cp37-${MATRIX_ARCH} cp38-${MATRIX_ARCH} cp39-${MATRIX_ARCH} cp310-${MATRIX_ARCH}"
+export CIBW_BUILD="cp39-${MATRIX_ARCH}"
+#export CIBW_BEFORE_ALL_WINDOWS="conda create --yes --name=build && conda activate build && conda config --env --set subdir ${MATRIX_ARCH_SUBDIR} && conda install --yes blosc bzip2 hdf5 lz4 lzo snappy zstd zlib"
+#export CIBW_ENVIRONMENT_WINDOWS='CONDA_PREFIX="C:\\Miniconda\\envs\\build" PATH="$PATH;C:\\Miniconda\\envs\\build\\Library\\bin"'
+export CIBW_ENVIRONMENT="PYTABLES_NO_EMBEDDED_LIBS=true DISABLE_AVX2=true"
+export CIBW_BEFORE_BUILD="echo $PATH; pip install -r requirements.txt cython>=0.29.21 delvewheel"
+export CIBW_REPAIR_WHEEL_COMMAND_WINDOWS="delvewheel repair -w {dest_dir} {wheel}"
+
+# Debugging.
+# env
+
+# Acquire sources.
+mkdir -p /c/src
+cd /c/src
+test ! -d PyTables && git clone https://github.com/PyTables/PyTables --recursive --depth=1
+cd PyTables
+
+# Build wheel.
+cibuildwheel --platform=windows --output-dir=wheelhouse