Skip to content

GPU Access on HiPerGator

GPU Resources

See the GPU Resources On HiPerGator for the description of GPU hardware and the information on acquiring an allocation.

There is no burst QOS for GPUs

Quickstart for Requesting GPUs from the Scheduler

To request a GPU in a scheduled job or session, first decide on the type of GPU to be requested.

L4 GPUs are integrated into the hybrid HPG4 nodes available both for cpu-only and gpu jobs for general computation and do not require the partition to be specified whereas RTX6000 and B200 GPUs are only available in respective partitions and should only be used for jobs requiring extreme performance and VRAM (gpu memory) such as large model training.

You must request at least one CPU core per GPU or the job will be rejected by the scheduler.

  • L4 GPUs: specify the gpu quantity with --gpus=NUMBER in a job script or OOD session form alongside the same or larger number of CPU cores. No other specifications are necessary.

  • RTX6000 GPUs: specify --partition=hpg-rtx6000 in addition to the gpu quantity specification i.e. both --partition=hpg-rtx6000 and --gpus=NUMBER OR --gres=gpu:NUMBER are required. The GRES string for this GPU type is rtx-pro-6000.

  • B200 GPUs: specify --partition=hpg-b200 in addition to the gpu quantity specification i.e. both --partition=hpg-b200 and --gpus=NUMBER OR --gres=gpu:NUMBER are required. The GRES string for this GPU type is b200.

Request format

--gpus=NUMBER

Example for one GPU:

SBATCH --gpus=1

Example for 3 GPUs:

SBATCH --gpus=3

GRES format can also be used both in job scripts and in OOD session forms.

--gres=gpu:NUMBER

or

gpu:NUMBER

Interactive Sessions

Shell

Interactive sessions are limited to 12 hours. In order to request interactive command line access to a GPU under SLURM, use commands similar to these:

  • To request access to an L4 GPU for a session with the default duration of 10 minutes:

Note: sruni command is in the ufrc environment module.

sruni --gpus=1

or

sruni --partition=hpg-rtx6000 --gpus=1

  • To request access to two B200 GPUs on a single node for a 3-hour session with 300gb RAM:

sruni --partition=hpg-b200 --gpus=2 --mem=300gb --time=3:00:00 --ntasks=1

Open On Demand

Note

Interactive Open OnDemand Jobs in the GPU partition are limited to 12 hrs. Computational GPU jobs are limited to 14 days. Each GPU job requires at least one CPU core.

To access GPUs using Open OnDemand, you need to set the partition and a GRES (generic resource) with the number and (optionally) type of GPU.

If your application supports multiple GPU types, choose the GPU partition and specify number of GPUs and type:

  • To request one L4 GPU, select the cluster partition 'hpg-turin' and use this gres string:

gpu:1

  • To request one RTX6000 GPU, select the cluster partition 'hpg-rtx6000' and use this gres string:

gpu:1

  • To request one B200 GPU, select the cluster partition 'hpg-b200' and use this gres string:

gpu:1

  • To request multiple GPUs (of any type, use this gres string were n is the number of GPUs you need):

gpu:n

Scheduled Jobs

To request GPU resources for scheduled jobs you generally need to specify the partition and gpu number.

Examples:

  • Request two L4 GPUs on a single server (--nodes defaults to "1"):
#SBATCH --partition=hpg-turin
#SBATCH --gpus=2
  • Request two RTX6000 GPUs on a single server (--nodes defaults to "1") will be allocated to the job:
#SBATCH --partition=hpg-rtx6000
#SBATCH --gpus=2`
  • In this example, two B200 GPUs on a single server (--nodes defaults to "1") will be allocated to the job:
#SBATCH --partition=hpg-b200
#SBATCH --gpus=2

Alternatively, use '--gres=gpu:1' or '--gres=gpu:b200:1' format. Note, if '--gpus=' format is used SLURM will not provide the data on GPU usage to slurmInfo and those GPUs will not be shown in slurmInfo output.

If no GPUs are available, your request will be queued and your connection established once the next GPU becomes available. Otherwise, you may cancel your job and try lowering requested resources. If you have requested a longer time than is needed, please be sure to end your session so that the GPU will be available for other users.

Job Script Example

This is a sample script for MPI parallel VASP job requesting and using GPUs under SLURM:

Expand to view a sample script 
#!/bin/bash
#SBATCH --job-name=vasptest
#SBATCH --output=vasp_%j.out
#SBATCH --error=vasp_%j.err
#SBATCH --mail-type=END,FAIL
#SBATCH --mail-user=email@ufl.edu
#SBATCH --nodes=1
#SBATCH --ntasks=8
#SBATCH --cpus-per-task=1
#SBATCH --ntasks-per-node=8
#SBATCH --ntasks-per-socket=4
#SBATCH --mem-per-cpu=7000mb
#SBATCH --distribution=cyclic:cyclic
#SBATCH --partition=hpg-turin
#SBATCH --gpus=4
#SBATCH --time=00:30:00

echo "Date      = $(date)"
echo "host      = $(hostname -s)"
echo "Directory = $(pwd)"

module purge
module load cuda  intel  openmpi vasp # You may want to specify versions
                # Use `module spider vasp` to see available versions of VASP
                # And `module spider vasp/### with the version number to see options 
                # for cuda, intel, and openmpi versions. 

T1=$(date +%s)
srun --mpi=${HPC_PMIX} vasp_gpu
T2=$(date +%s)

ELAPSED=$((T2 - T1))
echo "Elapsed Time = $ELAPSED"

Hardware Accelerated GUI

GPUs in these servers are used to accelerate rendering for graphical applications. These servers are in the SLURM "hwgui" partition.

There are several preset applications available in the Open OnDemand drop-down list (e.g. Freeview, Unreal Engine). You can run additional GUI applications by starting a Console or HiPerGator Desktop session, loading the application module and running the application.

To do this:

  • Select the 'hwgui' partition for an Open OnDemand Console or HiPerGator Desktop Application. See Open OnDemand for details on using OOD.
  • Once connected to the session, open a terminal and load the appropriate environment module and launch the application in question.

GPU Use by Applications

See the Using GPUs from Applications documentation for details.