Skip to content

Using Pixi for Package Management on HiPerGator

Pixi is a modern package management tool that provides a faster, more user-friendly alternative to conda for managing project environments. While pixi works out of the box, please configure your cache location to avoid storage issues on HiPerGator.

Advantages:

  • Project-local environments: Environment files are stored in your project directory (.pixi/). (pixi does use a global cache, which you should configure.)

  • Reproducible environments: Uses pixi.toml for declarative environment specification

  • Faster installations: Pixi uses an efficient solver and caching system

  • Multi-language support: Works seamlessly with Python, R, Rust, C++, and other languages

  • Cross-platform compatibility: Works consistently across different operating systems

Usage

pixi is available in the conda module. To use pixi, first load the conda module. 

ml conda

Cache Configuration

HiPerGator home directories have limited storage (40GB), so configure pixi to use your /blue storage for caching.

Why does pixi need a cache? While pixi keeps environment-specific files in your project directory (.pixi/), it uses a global cache for downloaded packages, metadata, and solve results. This cache can be shared across projects, making subsequent installations much faster and reducing storage usage overall.

We recommend using a symlink to your allocation directory since this will also take care of other applications that use the ~/.cache directory.

Move your cache to your allocation and create a symlink:

# Assume your group storage is /blue/dr-florida and your username is allie.gator

# Move your existing cache if it's there
mv ~/.cache/ /blue/dr-florida/allie.gator/.cache

# Create a symlink
ln -s /blue/dr-florida/allie.gator/.cache ~/.cache

Info

The pixi cache directory can alternatively be set via the PIXI_CACHE_DIR environment variable, but we recommend symlinking the default location instead.

Basic Usage

Initialize a Project

Using a terminal, create a new project directory and initialize pixi:

mkdir my-project
cd my-project
pixi init

This creates a pixi.toml file that defines your project configuration. More information about the pixi.toml format can be found in the official documentation.

Activate the Environment

Activate the pixi shell environment:

pixi shell

This is similar to conda activate (conda) or source venv/bin/activate (Python's built-in venv module). You'll see the environment name in your prompt.

Deactivate with:

exit

Add Packages

Add packages to your environment:

# Add Python packages
pixi add python==3.10.* numpy pandas matplotlib

# Add packages from specific channels
pixi add pytorch --channel pytorch
pixi add cuda --channel nvidia/label/cuda-12.4.0

# Add development dependencies
pixi add --dev pytest black

# Add system-level packages
pixi add cmake gcc

Install from Existing pixi.toml

If you have an existing pixi.toml file:

pixi install

Python-Specific Features

Using pyproject.toml

Pixi supports managing Python projects using pyproject.toml files, which is a popular standard for Python project configuration:

pixi init --format pyproject
pixi add python=3.10

Installing PyPI Packages

Add Python packages from PyPI (AKA like pip):

pixi add --pypi requests flask[async] fastapi

You can also specify version constraints:

pixi add --pypi "numpy>=1.24.0" "pandas<2.0.0"

Installing from Git Repositories

Install packages directly from Git:

pixi add --pypi "clip @ git+https://github.com/openai/CLIP.git"

Multi-Language Support

Pixi supports multiple programming languages in the same environment:

R Support

pixi add r-base r-ggplot2 r-dplyr

Rust Support

pixi add rust

C/C++ Development

pixi add cmake gcc gxx make

Notebook Integration

Creating Jupyter Kernels for Jupyter Notebooks (Python)

  1. Initialize your project and add Python:
pixi init
pixi add python ipykernel
  1. Install the kernel:
pixi run python -m ipykernel install --user --name "my-project" --display-name "My Project"

VS Code Integration

In VS Code:

  1. Open your project directory and .ipynb notebook file
  2. In the kernel selector (top right), choose "Select Another Kernel"
  3. Select the Python interpreter from the pixi environment (it should be called 'default' and live at '.pixi/envs/default/bin/python')

Installing Packages directly from Notebook Cells

Install packages directly from notebook cells:

# Install packages from within the notebook
!pixi add numpy pandas matplotlib

# Now you can import them
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

This approach updates your pixi.toml file automatically.

Environment Management

List Environments

View all pixi environments:

pixi list

Remove Packages

Remove packages from your environment:

pixi remove numpy pandas

Clean Cache

Clean the pixi cache (use carefully; make sure you've done your cache configuration correctly):

pixi clean cache

Update Packages

Update all packages:

pixi update

Update specific packages:

pixi update numpy pandas

Using Pixi in HPG Scripts

Command Line Usage

Run commands in your pixi environment without activating:

pixi run python my_script.py
pixi run jupyter notebook

If you activate your env with pixi shell, you can run commands directly:

python my_script.py

SLURM Job Scripts

Use pixi in SLURM job scripts:

#!/bin/bash
#SBATCH --job-name=pixi-job
#SBATCH --output=./logs/job-%j.log
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --mem=16gb
#SBATCH --time=24:00:00
#SBATCH --account=your-account
#SBATCH --qos=your-qos

ml conda
cd /full/path/to/your/project

# Run your script
pixi run python train_model.py --epochs 100

Getting Help