Managing Python environments and Jupyter kernels

Info

Please visit the Conda App Page for more detailed documentation and usage information.

Many projects that use Python code require careful management of the respective Python environments. Rapid changes in package dependencies, package version conflicts, deprecation of APIs (function calls) by individual projects, and obsolescence of system drivers and libraries make it virtually impossible to use an arbitrary set of packages or create one all-encompassing environment that will serve everyone's needs over long periods of time. The high velocity of changes in the popular ML/DL frameworks and packages and GPU computing exacerbates the problem.

Setup a Jupyter Kernel for Your Environment

Often, we want to use the environment in a Jupyter notebook. To do that, we can create our own Jupyter Kernel.

Add the jupyterlab package

In order to use an environment in Jupyter, we need to make sure we install the jupyterlab package in the environment:

mamba install jupyterlab

Copy the template_kernel folder to your path

On HiPerGator, Jupyter looks in two places for kernels when you launch a notebook:

1. /apps/jupyterhub/kernels/ for the globally available kernels that all users can use. (Also a good place to look for troubleshooting getting your own kernel going)

2. ~/.local/share/jupyter/kernels for each user. (Again, your home directory and the .local folder is hidden since it starts with a dot)

Make the Jupyter kernels directory:

mkdir -p ~/.local/share/jupyter/kernels

Copy the template_kernal folder into your kernels directory:

cp -r /apps/jupyterhub/template_kernel/ ~/.local/share/jupyter/kernels/hfrl

Note: This also renames the folder in the copy. It is important that the directory names be distinct in both your directory and the global /apps/jupyterhub/kernels/directory.

Edit the template_kernel files

The template_kernel directory has four files: the run.sh and kernel.json files will need to be edited in a text editor. We will use nano in this tutorial. The logo-64X64.png and logo-32X32.png are icons for your kernel to help visually distinguish it from others. You can upload icons of those dimensions to replace the files, but they need to be named with those names.

Edit the kernel.json file

Let's start editing the kernel.json file. As an example, we can use:

nano ~/.local/share/jupyter/kernels/hfrl/kernel.json

The template has most of the information and notes on what needs to be updated. Edit the file to look like:

{
 "language": "python",
 "display_name": "HF_Deep_RL",
 "argv": [
  "~/.local/share/jupyter/kernels/hfrl/run.sh",
  "-f",
  "{connection_file}"
 ]
}

Edit the run.sh file

The run.sh file needs the path to the python application that is in our environment. The easiest way to get that is to make sure the environment is activated and run the command: which python

The path it outputs should look something like:

/blue/group/user/conda/envs/hfrl/bin/python

Copy that path.

Edit the run.sh file with nano:

nano ~/.local/share/jupyter/kernels/hfrl/run.sh

The file should looks like this, but with your path:

#!/usr/bin/bash

exec /blue/ufhpc/magitz/conda/envs/hfrl/bin/python -m ipykernel "$@"

If you are doing this in a Jupyter session, refresh your page. If not, launch Jupyter.

Your kernel should be available in the default kernel list ready for you to use!