pyeddl cannot be installed in Google Colab

[jonandergomez]

We are thinking in using Google Colab for the WinterSchool, but I tried the following commands from three different notebook cells, in order to follow the indications in the pyeddl documentation about installation, and it fails.

!python3 -m pip install --upgrade setuptools pip

!python3 -m pip install --upgrade numpy 'pybind11<2.6' pytest

!(export EDDL_WITH_CUDA="true" ; python3 -m pip install pyeddl)

I wonder if it would be possible to use Colab for the Winter School or we have to search for another option. Anyway, the installation of the pyeddl with pip is difficult in general, I cannot be successful in other Linux machines I administer.

Here the output of the three commands to see the failures:

================================================
================================================
!python3 -m pip install --upgrade setuptools pip
================================================
Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (57.4.0)
Collecting setuptools
  Downloading setuptools-58.5.3-py3-none-any.whl (946 kB)
     |████████████████████████████████| 946 kB 5.1 MB/s 
Requirement already satisfied: pip in /usr/local/lib/python3.7/dist-packages (21.1.3)
Collecting pip
  Downloading pip-21.3.1-py3-none-any.whl (1.7 MB)
     |████████████████████████████████| 1.7 MB 43.7 MB/s 
Installing collected packages: setuptools, pip
  Attempting uninstall: setuptools
    Found existing installation: setuptools 57.4.0
    Uninstalling setuptools-57.4.0:
      Successfully uninstalled setuptools-57.4.0
  Attempting uninstall: pip
    Found existing installation: pip 21.1.3
    Uninstalling pip-21.1.3:
      Successfully uninstalled pip-21.1.3
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
datascience 0.10.6 requires folium==0.2.1, but you have folium 0.8.3 which is incompatible.
Successfully installed pip-21.3.1 setuptools-58.5.3
================================================
================================================
!python3 -m pip install --upgrade numpy 'pybind11<2.6' pytest
================================================
Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (1.19.5)
Collecting numpy
  Downloading numpy-1.21.4-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.7 MB)
     |████████████████████████████████| 15.7 MB 57 kB/s             
Collecting pybind11<2.6
  Downloading pybind11-2.5.0-py2.py3-none-any.whl (296 kB)
     |████████████████████████████████| 296 kB 48.8 MB/s            
Requirement already satisfied: pytest in /usr/local/lib/python3.7/dist-packages (3.6.4)
Collecting pytest
  Downloading pytest-6.2.5-py3-none-any.whl (280 kB)
     |████████████████████████████████| 280 kB 45.8 MB/s            
Requirement already satisfied: iniconfig in /usr/local/lib/python3.7/dist-packages (from pytest) (1.1.1)
Collecting pluggy<2.0,>=0.12
  Downloading pluggy-1.0.0-py2.py3-none-any.whl (13 kB)
Requirement already satisfied: py>=1.8.2 in /usr/local/lib/python3.7/dist-packages (from pytest) (1.11.0)
Requirement already satisfied: attrs>=19.2.0 in /usr/local/lib/python3.7/dist-packages (from pytest) (21.2.0)
Requirement already satisfied: importlib-metadata>=0.12 in /usr/local/lib/python3.7/dist-packages (from pytest) (4.8.2)
Requirement already satisfied: packaging in /usr/local/lib/python3.7/dist-packages (from pytest) (21.2)
Requirement already satisfied: toml in /usr/local/lib/python3.7/dist-packages (from pytest) (0.10.2)
Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=0.12->pytest) (3.6.0)
Requirement already satisfied: typing-extensions>=3.6.4 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=0.12->pytest) (3.10.0.2)
Requirement already satisfied: pyparsing<3,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging->pytest) (2.4.7)
Installing collected packages: pluggy, pytest, pybind11, numpy
  Attempting uninstall: pluggy
    Found existing installation: pluggy 0.7.1
    Uninstalling pluggy-0.7.1:
      Successfully uninstalled pluggy-0.7.1
  Attempting uninstall: pytest
    Found existing installation: pytest 3.6.4
    Uninstalling pytest-3.6.4:
      Successfully uninstalled pytest-3.6.4
  Attempting uninstall: numpy
    Found existing installation: numpy 1.19.5
    Uninstalling numpy-1.19.5:
      Successfully uninstalled numpy-1.19.5
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
datascience 0.10.6 requires folium==0.2.1, but you have folium 0.8.3 which is incompatible.
albumentations 0.1.12 requires imgaug<0.2.7,>=0.2.5, but you have imgaug 0.2.9 which is incompatible.
Successfully installed numpy-1.21.4 pluggy-1.0.0 pybind11-2.5.0 pytest-6.2.5
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
================================================
================================================
!(export EDDL_WITH_CUDA="true" ; python3 -m pip install pyeddl)
================================================
Collecting pyeddl
  Downloading pyeddl-1.1.0.tar.gz (109 kB)
     |████████████████████████████████| 109 kB 5.1 MB/s            
  Preparing metadata (setup.py) ... done
Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from pyeddl) (58.5.3)
Requirement already satisfied: pybind11<2.6 in /usr/local/lib/python3.7/dist-packages (from pyeddl) (2.5.0)
Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from pyeddl) (1.21.4)
Building wheels for collected packages: pyeddl
  Building wheel for pyeddl (setup.py) ... error
  ERROR: Failed building wheel for pyeddl
  Running setup.py clean for pyeddl
Failed to build pyeddl
Installing collected packages: pyeddl
    Running setup.py install for pyeddl ... error
ERROR: Command errored out with exit status 1: /usr/bin/python3 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-h7aadw_j/pyeddl_e328db526b654086ac67053f3daec44c/setup.py'"'"'; __file__='"'"'/tmp/pip-install-h7aadw_j/pyeddl_e328db526b654086ac67053f3daec44c/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-_w6deco8/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python3.7/pyeddl Check the logs for full command output.
================================================
================================================

[simleo]

Hi Jon. I'm not sure what happened exactly. It says "Check the logs for full command output" in the end, but it looks like you haven't posted them. Did you install EDDL in a standard system location before running the above commands? That's required before trying installation with pip. Alternatively, a fully automated, easy installation is available via Conda. Full instructions are available at https://github.com/deephealthproject/conda_builds, but here is what you need for PyEDDL:

(example for python 3.7 and ecvl compiled for GPU):

channel configuration:

conda config --add channels dhealth
conda config --add channels bioconda
conda config --add channels conda-forge
conda config --set channel_priority strict

env creation and installation:

conda create -y -n dh_toolkit
conda activate dh_toolkit
conda install -y python=3.7 pyecvl-gpu

Packages are also available Python 3.6, 3.7 and 3.8, and compiled for CPU, GPU and GPU + CUDNN.

[RParedesPalacios]

But this doesn't work on Google Collab where the python is just running and we run the commands from the python in shell mode. In Google Collab like in other environments is the pip install method the one that is always well supported, but unfortunately pyeddl does not provide pip install.

Anyway, perhaps you can try it in Google Collab and if you get it you can send us the notebook to use it in the Winter School.

[simleo]

I managed to install it via a package called condacolab. I pushed the notebook with the installation steps to this repo as docs/ColabInstall.ipynb.

[RParedesPalacios]

In the case of CUDNN it seems that something failed (cudatoolkit?):

!conda install pyeddl-cudnn

Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed

SpecsConfigurationConflictError: Requested specs conflict with configured specs.
requested specs:
- pyeddl-cudnn
pinned specs:
- cudatoolkit=11.1
Use 'conda config --show-sources' to look for 'pinned_specs' and 'track_features'
configuration parameters. Pinned specs may also be defined in the file
/usr/local/conda-meta/pinned.

[RParedesPalacios]

The same for CUDA:

Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed

SpecsConfigurationConflictError: Requested specs conflict with configured specs.
requested specs:
- pyeddl-gpu
pinned specs:
- cudatoolkit=11.1
Use 'conda config --show-sources' to look for 'pinned_specs' and 'track_features'
configuration parameters. Pinned specs may also be defined in the file
/usr/local/conda-meta/pinned.

[simleo]

It looks like the Colab backend is currently using CUDA 11. condacolab tries to be conservative and pins the cudatoolkit version according to the detected CUDA version to avoid compatibility problems. The conda packages require cudatoolkit version 10, so the installation fails. However, we have been running EDDL compiled for CUDA 10 on a machine with CUDA 11 for a long time and it works fine. So the trick is to remove the cudatoolkit pin from /usr/local/conda-meta/pinned and then installation succeeds. I've pushed the notebook with the steps for GPU installation to GitHub. Note that the GPU runtime needs to be explicitly enabled before running cells.