SwanProjects extension for jupyter lab 3

README.md

@@ -12,4 +12,5 @@ Repository that stores all the Jupyter extensions for SWAN.
 * [SwanNotebookViewer](SwanNotebookViewer) - Read-only mode for opening notebooks (used from the Sharing interface) inside Jupyter Notebooks
 * [SwanNotifications](SwanNotifications) - Extension to display notifications to users
 * [SwanOauthRenew](SwanOauthRenew) - Extension that fetches the latest oAuth tokens from JupyterHub and writes to the file observed by EOS
-* [SwanShare](SwanShare) - Jupyter Notebooks/CERNBox sharing integration used by SwanContents
+* [SwanShare](SwanShare) - Jupyter Notebooks/CERNBox sharing integration used by SwanContents
+* [SwanProjects](SwanProjects) - Jupyter Lab extension with backend and dialogs to Create/Edit projects, also have a customized KerneSpecManager to handle kernels in multiple environments.

SwanProjects/.bumpversion.cfg

@@ -0,0 +1,9 @@
+[bumpversion]
+current_version = 0.1.0
+commit = True
+tag = True
+tag_name = SwanProjects/v{new_version}
+message = SwanProjects v{new_version}
+
+[bumpversion:file:package.json]
+

SwanProjects/MANIFEST.in

@@ -0,0 +1,28 @@
+include LICENSE
+include README.md
+include pyproject.toml
+recursive-include jupyter-config *.json
+
+include swanprojects/kernelmanager/resources/*
+include swanprojects/stacks/*/*
+include swanprojects/static/index.html
+include package.json
+include install.json
+include ts*.json
+include yarn.lock
+
+graft swanprojects/labextension
+
+# Javascript files
+graft src
+graft style
+prune **/node_modules
+prune lib
+prune binder
+
+# Patterns to exclude from any directory
+global-exclude *~
+global-exclude *.pyc
+global-exclude *.pyo
+global-exclude .git
+global-exclude .ipynb_checkpoints

SwanProjects/README.md

@@ -0,0 +1,34 @@
+# SwanProjects
+
+Server and Lab extension that provides:
+* In the backend, the endpoints to:
+  * Create and edit projects
+  * Get project information
+  * Get software stack information
+  * A customized Kernel Spec Manager to handle kernel metadata
+* In the Lab extension:
+  * React dialogs to create and edit projects
+  * LabIcons required for the dialogs
+
+## Requirements
+
+JupyterLab~=3.0 and SwanContents
+
+## Install
+
+Install the package and the lab extension:
+
+```bash
+pip install swanprojects
+```
+
+To replace the default Jupyter Contents Manager and Kernel Spec Manager in the JupyterLab Notebook configuration (i.e in `jupyter_notebook_config.py`), set the following:
+
+```python
+c.NotebookApp.contents_manager_class = 'swancontents.filemanager.swanfilemanager.SwanFileManager'
+c.NotebookApp.kernel_spec_manager_class = 'swanprojects.kernelmanager.kernelspecmanager.SwanKernelSpecManager'
+c.KernelSpecManager.ensure_native_kernel = False
+
+c.SwanConfig.stacks_path=path_to_stacks_folder
+c.SwanConfig.kernel_resources=path_to_native_kernel_resources
+```

SwanProjects/bin/swan_bash

@@ -0,0 +1,51 @@
+#!/bin/bash -i
+#Author [email protected] 2021
+###
+# This script allows to start a bash interpreter inside a project environment.
+# This is called from the extension SwanTerminal and it requires like a parameter the project name to
+# build the path to the project with $HOME/SWAN_projects/$PROJECT to load the project environment.
+# When the bash session is started we modify the prompt indicating in which project the terminal is located.
+# Aditionally bash run completed isolated from other possible environments then we pass like parameters to the isolated environment some
+# required basic environment variables such as:
+#
+# * HOME with path to the user home
+# * PATH with default paths for the system
+# * OAUTH2_TOKEN required by EOS storage
+#
+# swan_env will load the other variables from the stack, inside the isolated enviroment.
+###
+
+clear
+if ! [ -x "$(command -v jq)" ]; then
+  echo 'Error: jq is not installed.' >&2
+  sleep 60
+  exit 1
+fi
+if [[ $# -gt 1 ]] ; then
+    PROJECT_PATH=$1
+    PROJECT_NAME=`IFS='/'; ARR=($PROJECT_PATH);echo "${ARR[-1]}"`
+    PROJECT_FILE="$PROJECT_PATH/.swanproject"
+
+    STACKS_PATH="$2"
+
+    if [ -d "$PROJECT_PATH" ]
+    then
+      STACK=`jq '.stack' "$PROJECT_FILE"`
+      RELEASE=`jq '.release' "$PROJECT_FILE"`
+      PLATFORM=`jq '.platform' "$PROJECT_FILE"`
+      USER_SCRIPT="$PROJECT_PATH/.userscript"
+      echo "Loading $RELEASE with plafortm $PLATFORM "
+      # FIXME: this have to be removed when environment isolation is not needed anymore, it's only temporary
+      env -i HOME=$HOME \
+             OAUTH2_TOKEN=$OAUTH2_TOKEN \
+             PROJECT="$PROJECT_NAME" \
+             PROJECT_PATH="$PROJECT_PATH" PS1="$PS1" \
+             bash -c "swan_env \"$PROJECT_PATH\" \"$STACKS_PATH\" \"$PROJECT_PATH\" bash --rcfile  <(echo 'PS1=\"($PROJECT_NAME) $PS1 \"') "
+    else
+      echo "Error: project $PROJECT_PATH doesn't exist" >&2
+      # JupyterLab closes the terminal window immediately after the process ends
+      # this sleep is to allow the user to see the message
+      sleep 60
+      exit 1
+    fi
+fi

SwanProjects/bin/swan_env

@@ -0,0 +1,92 @@
+#!/bin/bash
+#Author [email protected] 2021
+###
+# This script allows to execute code inside the project environment.
+#
+# The parameters are:
+# * Project name: Name of the project.
+# * Cwd: Current working directory, where the command will be executed.
+# * Command: command to execute inside the project environment.
+#
+#
+# With the project name we search the project located at $HOME/SWAN_projects/
+# reading the configuration to load the environment and the bash script provided by the user.
+#
+# This command is called from SwanKernelSpecManager to start the kernel and swan_bash to start a bash session inside the project environment.
+##
+PARAMETERS=("$@")
+# Check if jq is installed
+if ! [ -x "$(command -v jq)" ]; then
+  echo 'Error: jq is not installed.' >&2
+  # JupyterLab closes the terminal window immediately after the process ends
+  # this sleep is to allow the user to see the message
+  sleep 60
+  exit 1
+fi
+
+# Check arguments
+if [[ $# -lt 3 ]] ; then
+  echo 'Error: project name, cwd and commands required.' >&2
+  echo 'Format is: swan_env myproject cwd command command_options' >&2
+  echo 'Example: swan_env myproject . python --version (shows version for the python inside the project environment)' >&2
+  # JupyterLab closes the terminal window immediately after the process ends
+  # this sleep is to allow the user to see the message
+  sleep 60
+  exit 1
+fi
+
+###
+# In the next block I made the path to the project to read the information from .swanproject file such as 
+# stack, release and platform.
+# I also create the path to use bash user script to source it.
+###
+PROJECT_PATH=$1
+PROJECT_NAME=`IFS='/'; ARR=($PROJECT_PATH);echo "${ARR[-1]}"`
+PROJECT_FILE="$PROJECT_PATH/.swanproject"
+STACK=`jq -r '.stack' "$PROJECT_FILE"`
+RELEASE=`jq -r '.release' "$PROJECT_FILE"`
+PLATFORM=`jq -r '.platform' "$PROJECT_FILE"`
+USER_SCRIPT="$PROJECT_PATH/.userscript"
+
+# Path to the stack where the setup.sh for this project is located.
+# The available stackas are provided in the extension with --SwanProjects.stacks_path
+STACKS_PATH="$2"
+STACK_PATH="$STACKS_PATH/$STACK"
+STACK_SETUP="$STACK_PATH/setup.sh"
+
+# Working directory parameter
+CWD="$3"
+i=2
+
+for j in $(seq 0 1 $((i)));do
+    unset PARAMETERS[$j]
+done
+# After project name and working directory the rest of the options passed to this script is the command to be execute inside the environment.
+COMMAND=${PARAMETERS[@]}
+
+if [ ! -d $STACK_PATH ]; then
+  echo "Error sourcing the environment, the stack $STACK was not found in stacks path $STACKS_PATH" >&2
+  echo "project $PROJECT_NAME can not be loeaded." >&2
+  exit 1
+fi
+
+if [ -f $STACK_SETUP ]; then
+  . $STACK_SETUP
+else
+   echo "Error loading stack setup.sh on $STACK_SETUP, file doesn't exists." >&2
+   exit 1
+fi
+
+# The next variables allows the user to know in the environment basic information about the project such as stack, name and path.
+export SWAN_STACK="$RELEASE($PLATFORM)"
+export SWAN_PROJECT_NAME=$PROJECT_NAME
+export SWAN_PROJECT_PATH=$PROJECT_PATH
+
+if [ "$USER_SCRIPT" != "" ] && [ -f "$USER_SCRIPT" ]; then
+    . "${USER_SCRIPT}"
+fi
+
+# The command is executed inside the $CWD folder.
+# if the command produces output it will be there.
+cd "$CWD"
+$COMMAND

SwanProjects/bin/swan_kmspecs

@@ -0,0 +1,136 @@
+#!/usr/bin/env python
+# Copyright (c) SWAN Development Team.
+# Author: [email protected] 2021
+
+"""
+This script allows to find the kernels for python2/3 and kernel spec paths for our kernel spec manager.
+
+The script run a subprocess inside the project environment trying to find the package ipykernel
+and run jupyter_path('kernels') to get the list of available paths for the differents kernels inside the environment as well.
+
+This is execute in the Create/Edit project handlers and the stdout is captured, getting the json information printed in the function generate_ksminfo()
+"""
+import argparse
+import json
+import os
+import pprint
+import subprocess
+import sys
+from distutils.spawn import find_executable
+from shutil import rmtree
+try:
+    from jupyter_core.paths import jupyter_path, get_home_dir
+except ImportError:
+    print("{'status':0, 'msg':'Package jupyter_core not found in the environment, kernel paths can not be found.'}")
+    sys.exit(1)
+
+
+def checkipykernel(python_interpreter):
+    """
+    Checks if ipykernel is available for the given python interpreter,
+    this function have to be executed inside the project environment.
+
+    Parameters
+    ----------
+    python_interpreter : str
+        full path to python interpreter.
+
+    Returns
+    -------
+    int
+        zero if ipykernel was found.
+    """
+    python_code = 'import ipykernel'
+    command = [python_interpreter, "-c", python_code]
+    proc = subprocess.Popen(command, stdout=subprocess.PIPE)
+    proc.wait()
+    proc.communicate()
+    return proc.returncode
+
+
+def check_native_kernel():
+    """
+    Checks if ipykernel is available for python2 and python3.
+
+    This routine is called inside the project environment,
+    It checks if python2/3 are available to check if the package ipykernel is installed,
+    if the package was found the results is saved in the dictionary with the results.
+
+    Why? kernel.json is not always available in the software stacks but the package ipykernel is there,
+    then we can create the json file locally in the project to put it work, because our kernel spec manager requires
+    the kernel path with the kernel.json file.
+
+    Returns
+    -------
+    dict
+        information about python version found and if ipykernel was found for the python version.
+    """
+    project_data = {}
+    # checking if python2 is found
+    python2 = find_executable("python2")
+    if python2 is not None:
+        project_data["python2"] = {"found": True, "path": python2}
+        # checking is ipython is found for python2
+        rcode = checkipykernel(python2)
+        if rcode == 0:
+            project_data["python2"]["ipykernel"] = True
+        else:
+            project_data["python2"]["ipykernel"] = False
+    else:
+        project_data["python2"]["found"] = False
+
+    # checking if python2 is found
+    python3 = find_executable("python3")
+    if python3 is not None:
+        project_data["python3"] = {"found": True, "path": python3}
+        # checking is ipython is found for python3
+        rcode = checkipykernel(python3)
+        if rcode == 0:
+            project_data["python3"]["ipykernel"] = True
+        else:
+            project_data["python3"]["ipykernel"] = False
+    else:
+        project_data["python3"]["found"] = False
+    return project_data
+
+
+def get_kernel_paths():
+    """
+    Allows to find kernel paths inside the environment
+
+    Returns
+    -------
+    list
+        kernel paths found inside the environment.
+    """
+    kernels_blacklist_paths = [os.path.join(
+        get_home_dir(), '.local/share/jupyter/kernels'), '/usr/local/share/jupyter/kernels', '/usr/share/jupyter/kernels']
+    tmp_paths = jupyter_path('kernels')
+    if "kernels" in tmp_paths:
+        tmp_paths.remove("kernels")
+    paths = []
+    for path in tmp_paths:
+        found = False
+        for bl_path in kernels_blacklist_paths:
+            if bl_path in path:
+                found = True
+        if not found:
+            paths.append(path)
+    return paths
+
+
+def generate_ksminfo():
+    """
+    Function to generated all the kernel spec manager info,
+    calling the function check_native_kernel and  get_kernel_paths.
+    """
+    ksminfo = check_native_kernel()
+    ksminfo["kernel_dirs"] = get_kernel_paths()
+    ksminfo["status"] = 1
+    print(json.dumps(ksminfo, indent=4))
+
+if __name__ == '__main__':
+    """
+    Entry point to use this script, calls generate_ksminfo routine to generate the kernel info.
+    """
+    generate_ksminfo()

SwanProjects/install.json

@@ -0,0 +1,5 @@
+{
+  "packageManager": "python",
+  "packageName": "swanprojects",
+  "uninstallInstructions": "Use your Python package manager (pip, conda, etc.) to uninstall the package swanprojects"
+}

SwanProjects/jupyter-config/nb-config/swanprojects.json

@@ -0,0 +1,7 @@
+{
+  "NotebookApp": {
+    "nbserver_extensions": {
+      "swanprojects": true
+    }
+  }
+}

SwanProjects/jupyter-config/server-config/swanprojects.json

@@ -0,0 +1,7 @@
+{
+  "ServerApp": {
+    "jpserver_extensions": {
+      "swanprojects": true
+    }
+  }
+}