Skip to content

Scheduling

Scheduling computational jobs

Quick notes

Most computation on HiPerGator is scheduled through the SLURM scheduler.

  • Use sbatch to submit a script with the resource request.
  • Use squeue or slurmInfo to check on your jobs and group resources.
  • Jobs use your primary group's resource by default, the --account and --qos flags can be used to specify a secondary group if needed.

UFIT-RC uses the Simple Linux Utility for Resource Management, or SLURM, to allocate resources and schedule jobs (View the official SLURM documentation). Users can create SLURM job scripts to submit jobs, or schedule interactive jobs from the command line or one of several web interfaces.

The key resource requests that are pertinent on HiPerGator are:

Resource SLURM Flag Open on Demand Field Description
Cores --nodes
--tasks
--cpus-per-task		 Number of CPU cores requested per MPI task The --cpus-per-task flag will specify how many cores you want.
  • For the vast majority of users, the main flag to set is --cpus-per-task. Unless you are using MPI applications (see this page for MPI information), you want a single node (server, --nodes=1), one task (MPI rank, --tasks=1), and one or more cores.
  • For many applications, only a single core is needed. (e.g. --cpus-per-task=1).
  • If your application can use multiple cores, modify the number for the number of cores you want.
Memory --mem Maximum memory requested for this job in Gigabytes The amount of memory (RAM) you want available for your application must be specified. In most cases, the --mem flag is the best method to use, e.g. for 1GB of RAM, --mem=1gb (but see this page for MPI applications).

If your application uses more memory than requested, your job will be killed. The error typically shows and oom event (Out of Memory).
GPUs --gres Generic Resource Request GPUs on HiPerGator a requested with the --gres flag, e.g. --gres=gpu:1. You will also need to specify a partition with GPUs.
Time --time Time Requested for this job in hours The amount of time you want the resources, e.g. --time=1:00:00. This can be specified in a number of formats, HH:MM:SS or just a number of minutes are commonly used. If your job is still running at the end of the requested time, it will be canceled.
Tip

It is important to know what resources (CPUs, memory and GPUs) your job needs and for how long it will use those resources.

  • CPUs: Can your job use multiple CPU cores? Many applications cannot, and requesting multiple cores will not make them run faster, but rather waste resources. Even applications that can use multiple cores do not always scale well, or only scale to a certain point. Experimenting with your application and understanding its details will vastly improve your ability to run it efficiently.

  • Memory: How much memory, or RAM, will your application use? Your application will have access to the full amount requested. While you want to leave some buffer (15-20%), requesting much more RAM than will be used is wasteful of resources and will not speed up computation. If your application attempts to use more memory than was requested, the SLURM scheduler will cancel the job.

  • GPUs: Does your application use GPUs? If so, can it use only one, or can it use multiple GPUs?

  • Time: How long will your application take to run? The scheduler will reserve the resources for the time requested. If your application finishes, the job ends and the resources are released. If your application is still running at the end of the requested time, the scheduler will cancel the job. In general request 15-20% more time than you expect to use.

Submit scripts

Submit scripts should be edited to control several aspects of your job, like resource allocation, email notifications, or an output destination.

To submit a job script from one of the login nodes accessed via hpg.rc.ufl.edu, use the following command (substituting the name of your script):

sbatch your_job_script.sh

To check the status of submitted jobs, use the following command (substituting your username):

squeue -u username

View SLURM Commands for more useful SLURM commands.

Managing Cores and Memory

See Account and QOS limits under SLURM for the main documentation on efficient management of computational resources and an extensive explanation of QOS and SLURM account use.

The amount of resources within an investment is calculated in NCU (Normalized Computing Units), which correspond to 1 CPU core and about 7.8GB of memory for each NCU purchased. CPUs (cores) and RAM are allocated to jobs independently as requested by your job script.

  • Your group's investment can run out of cores (SLURM may show QOSGrpCpuLimit in the reason column of the squeue command) OR memory (SLURM may show QOSGrpMemLimit in the reason column) depending on the current use by running jobs.

The majority of HiPerGator nodes have the same ratio of about 8 GB of RAM per core, which, after accounting for the operating system and system services, leaves about 7.8 GB usable for jobs; hence the ratio of 1 core and 7.8GB RAM per NCU.

Many HiPerGator nodes have 128 CPU cores and 1000GB of RAM. The bigmem nodes go up to 4TB of available memory. See Available Node Features for the exact data on resources available on all types of nodes on HiPerGator.

You must specify both the number of cores and the amount of RAM needed in the job script for SLURM with the --mem (total job memory) or --mem-per-cpu (per-core memory) options. Otherwise, the job will be assigned the default 600mb of memory.

Using secondary group resources

By default, SLURM will use your primary group's resources. If you are in multiple groups and want to use resources from a secondary group, you must specify both the account and qos of the secondary group.

For example if you want to use the other_group resources, you would use:

#SBATCH --account=other_group
#SBATCH --qos=other_group

If you are using OOD, the Account and QOS fields in the form can be used.

Monitoring Your Workloads

You can see presently running workloads with the squeue command, e.g., squeue -u username.

Open OnDemand offers a method to monitor jobs using the Jobs menu in the upper toolbar on your dashboard. This will show your current running, pending, and recently completed jobs. Select: Jobs -> Active Jobs from the upper dashboard menu.

We provide a number of helpful commands in the UFRC module. The ufrc module is loaded by default at login, but you can also load the ufrc module with the following command: module load ufrc.

Examples of commands for SLURM or HiPerGator specific UFRC environment module (see the full list of ufrc module commands):

Command Description
slurmInfo Displays resource usage for your group. Be sure to check out the -h help for other flags, like slurmInfo -p to see use by partition.
showQos Displays your available QoS
home_quota Displays your /home quota
blue_quota Displays your /blue quota
orange_quota Displays your /orange quota
sacct Displays job ID and state of your recent workloads
nodeInfo Displays partitions by node types, showing total RAM and other features
sinfo -p partition Displays the status of nodes in a partition
jobhtop Displays CPU and RAM resource usage for running jobs
jobnvtop Displays GPU resource usage for running GPU jobs
which python Displays the path to the Python install of the environment modules you have loaded