devcontainer fails to run postscript

[nzolman]

Received the following with a fresh VS Code devcontainer on a new machine.

Running the postCreateCommand from devcontainer.json...

[1554250 ms] Start: Run in container: /bin/sh -c source /home/firedrake/firedrake/bin/activate && pip install -e .
/bin/sh: 1: source: not found
[1554352 ms] postCreateCommand failed with exit code 127. Skipping any further user-provided commands.

I think this is related to this issue for running source in Docker containers.

The container otherwise builds fine, so it seems plausible to build everything from scratch in terminal afterwards. However, because the python venv is not successfully activated, it doesn't install hydrogym and both python and pip will point to the wrong python inside the container. When I manually source activate ... and pip install ., I get the following:

Building wheels for collected packages: hydrogym
  Building editable for hydrogym (pyproject.toml) ... done
  Created wheel for hydrogym: filename=hydrogym-0.1.2.3-py3-none-any.whl size=4657 sha256=cc5962681edbc48ff3038ada26605462eebfce4437ce5206736a09386004e72f
  Stored in directory: /tmp/pip-ephem-wheel-cache-omv0052m/wheels/29/0c/01/4594b1f70030c37e0a25dfd9f9ac190400a1c0186343cefd3a
Successfully built hydrogym
Installing collected packages: gym-notices, gmsh, networkx, fsspec, cloudpickle, gym, dmsuite, control, hydrogym
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
icepack 1.1.0 requires geojson, which is not installed.
icepack 1.1.0 requires geopandas, which is not installed.
icepack 1.1.0 requires meshio>=3.3.1, which is not installed.
icepack 1.1.0 requires MeshPy, which is not installed.
icepack 1.1.0 requires pooch, which is not installed.
icepack 1.1.0 requires pygmsh<=6.1.1, which is not installed.
icepack 1.1.0 requires rasterio, which is not installed.
icepack 1.1.0 requires ROL@ git+https://github.com/icepack/pyrol, which is not installed.
icepack 1.1.0 requires roltrilinos@ git+https://github.com/icepack/Trilinos, which is not installed.
icepack 1.1.0 requires shapely, which is not installed.
icepack 1.1.0 requires tqdm, which is not installed.
icepack 1.1.0 requires xarray, which is not installed.
Successfully installed cloudpickle-3.1.0 control-0.9.4 dmsuite-0.1.1 fsspec-2024.10.0 gmsh-4.13.1 gym-0.26.2 gym-notices-0.0.8 hydrogym-0.1.2.3 networkx-3.4.2

This implies that there are a number of dependencies that are not accounted for during the pip installation. After manually installing, I can import hydrogym, etc.

[ludgerpaehler]

There is now a complete rewrite of the dev container on the lp/bug-hunting branch, which I just linked against this issue. It is already working but has a slight issue with the handling of user IDs which I am hoping to fix tomorrow.

Let me know if it already works for you.

With the last fixed kink, I'd be inclined to close this issue with the merging of the PR if that is okay with you?

[nzolman]

I probably don't have time to try and build a fresh install anytime soon, but if someone else can confirm the new devcontainer build works, I'd be fine with closing this once merged. A quick question, though:

From a brief glance it looks like instead of using a prebuilt hydrogym image, you're pointing the devcontainer to a Dockerfile for the build [1] and the latest official firedrake image [2]? Does that risk future Hydrogym users experiencing issues if the latest Firedrake has breaking changes (like it did a couple years ago)? I only have a basic understanding of docker and devcontainers, so please feel free to ignore if not relevant.

[1]

hydrogym/.devcontainer/devcontainer.json

Line 9 in 2fc21e3

"dockerfile": "Dockerfile"

[2]

hydrogym/.devcontainer/Dockerfile

Line 1 in 2fc21e3

From firedrakeproject/firedrake:latest

[ludgerpaehler]

That reading is correct, for now it binds to the upstream Firedrake container. But through the Docker image tag :latest, we could bind to a specific version again.

The thinking here is that Firedrake is currently redoing its packaging getting closer and closer to a pure pip install. So for now it seems to be the best of course action to stay in line with the upstream development. That might obviously change down the line..

The Firedrake issues from back then are only a small part of the issues that arose I would argue, rllib drastically changed its interfaces, as well as gymnasium. Most the versions in the project description are written as >= i.e. only define a minimum version, but no upper bound. So I think we will always have to keep updating dependencies, and the interaction with them, outside of narrowing the range of versions HydroGym accepts which will fail sooner or later, as all these dependencies interact in their own way...

[ludgerpaehler]

(Still have to write up the description of the PR, but I will write everything down in there in case anyone looks back at what happened there)

[ludgerpaehler]

Currently also writing a page in the docs for it, so that should hopefully help anyone using the Decontainer in the future as there are some constraints Devcontainers imposes in our specific case here.

To be specific, Ray doesn't obtain as much virtual memory as it'd like as Docker containers always have limited memory and as such one needs to limit the task-parallelism when running Ray in the dev container.

[ludgerpaehler]

Last issue with the user IDs fixed just now, so the dev container on the lp/bug-hunting branch is fully functional.