Skip to content

New User Training

This page mirrors and expands upon the content provided in the HiPerGator User Training in Canvas.

HiPerGator User Training Registration

The HiPerGator User Training is offered through UF’s Professional and Workforce Development site.

REGISTER HERE: https://go.ufl.edu/hpg-training.

The course is free.

Expand for full registration details

After navigating to https://go.ufl.edu/hpg-training

  • Click the + to expand the only section of the course, and click “Add to Cart.”
  • In your cart, click “Checkout.”
  • When signing in, users with a GatorLink account should sign in with their GatorLink. Federated users may select from the other options to sign in.
  • After completing the checkout process (no payment will be required), users will get two confirmation emails from Matt Gitzendanner (magitz@ufl.edu).
  • For UF users, the course should be available in your Canvas dashboard when logging in using the orange "GatorLink" button.

  • For Federated Users:

    1. Follow this link: https://elearning.ufl.edu/
    2. Click the blue "External ID" button
    3. Login with the same account you used to register.

A more in-depth guide to accesing courses can be found here

Taking the Course is Required

While this page mirrors the content, to receive credit for the training, you must complete it and pass the final quiz in the Canvas course.

Training Objectives

  1. Recognize the role of UFIT Research Computing.
  2. Understand UFIT Research Computing's investment model for resource allocation.
  3. Understand different mechanisms to access HiPerGator.
  4. Describe the appropriate use of the login servers and how to request resources for work beyond those limits.
  5. Understand the primary resources tracked by the Slurm scheduler.
  6. Understand how to log in to HiPerGator using an SSH client.
  7. Understand several common mistakes HiPerGator users make and how to avoid them.

Module 1: Introduction to Research Computing and HiPerGator

HiPerGator

  • About 60,000 CPU cores Racks of the HiPerGator 4th Generation B200 Servers
  • 504 B200, 360 RTX Pro 6000 Blackwell Server Edition, and 600 L4 GPUs
  • 12 Petabytes of all-flash, high-performance parallel storage (Blue storage)
  • 32 Petabytes of capacity storage (Orange storage)

The HiPerGator page has updated information on the current hardware, and the History page provides historical information.

Summary

HiPerGator is a large, high-performance compute cluster capable of tackling some of the largest computational challenges, but users need to understand how to use the resources responsibly and efficiently.

Investor Supported

HiPerGator is heavily subsidized by the university, but researchers need to make investments for access to resources. UFIT Research Computing sells three main products:

  1. Compute: NCUs (Normalized Compute Units) An image highlighting a CPU, a DDN storage cabinet and an NVIDIA DGX Server faceplate
    • 1 CPU core and 7.8 GB of RAM (as of Jan 2021)
  2. Storage:
    • Blue: High-performance storage for most data during analyses. Filesystem: /blue
      • Note: Since all data being analyzed on HiPerGator should be stored on the /blue filesystem, all groups are required to have at least 1TB of Blue storage.
    • Orange: Intended for long-term storage, raw data, and archival use. Not intended for regular job input/output (I/O). Filesystem: /orange
  3. GPUs: NGUs (Normalized Graphic Units)
    • Sold in units of GPU cards
    • An NCU investment is required to make use of GPU(s)

Hardware vs. Services

Investments can be categorized as hardware investments, with a fixed 5-year term and no indirect costs (IDC), or service investments, with flexible investment durations of 1 month or longer and IDC. Both are purchasing access to the same resources. Investors do not own any particular server or piece of hardware, but are purchasing priority access to the corresponding resources.

Module 2: Accessing HiPerGator and Running Jobs

Cluster Components

Accessing HiPerGator

Proper use of Login Nodes

  • Generally speaking, interactive work other than managing jobs and data is discouraged on the login nodes.
  • Login nodes are intended for file and job management, and short-duration testing and development. See more information here

Login server limits

UFIT-RC uses various methods to limit user processes on login servers. Exact limits are dynamically set, but are about 4 cores and 30 GB RAM.

  • User processes attempting to use more CPU cores than the limit will compete for resources and slow down.
  • User processes attempting to use more RAM (memory) than the limit will be automatically killed. Do not restart killed processes. Submit a job requesting appropriate resources from the scheduler.

For users of VSCode and other IDEs with SSH connections

As outlined in the Common mistakes section below, many popular IDEs have SSH extensions that will connect to a login server.

Most of these will consume large amounts of CPU resources and violate acceptable use of the login servers.

Similarly, Agentic coding applications tend to spawn many processes and will perform poorly on login servers.

It is generally not recommended to use these applications on a login server.

Resources for Scheduling a Job

For use beyond what is acceptable on the login servers, you can request resources on development servers, GPUs servers, through JupyterHub, Graphical User Interface servers via Open onDemand or submit batch jobs. All of these services work with the scheduler to allocate your requested resources, ensuring that your computations run efficiently and do not impact other users.

Scheduling a Job

  1. Understand the resources that your analysis will use:
    • CPUs: Can your job use multiple CPU cores? Does it scale?
    • Memory: How much RAM will it use? Requesting more will not make your job run faster!
    • GPUs: Does your application use GPUs?
    • Time: How long will it run?
  2. Request those resources:
    • Sample Slurm Scripts
    • Watch the HiPerGator: Slurm Submission Scripts training video. This video is approximately 35 minutes long and includes a demonstration Play Video
    • Watch the HiPerGator: Slurm Submission Scripts for MPI Jobs training video. This video is approximately 25 minutes long and includes a demonstration Play Video
    • Open onDemand and JupyterHub have other mechanisms to request resources as Slurm needs this information to schedule your job.
    • Open onDemand
    • JupyterHub
  3. Submit the Job
    • Either using the sbatch JOB_SCRIPT command or through on of the interfaces
    • Once your job is submitted, Slurm will check that there are resources available in your group's allocation and schedule the job to run
  4. Run
    • Slurm will work through the queue and run your job

Locations for Storage

The storage systems are reviewed on the storage page. Photo of the Blue storage racks

Note

In the examples below, the text in all-caps (e.g. USER) indicates example text for user-specific information (e.g. /home/albertgator)

  1. Home Storage: /home/USER
    • Each user has 40GB of space.
    • Good for scripts, code and compiled applications.
    • Do not use for job input/output (I/O).
    • Snapshots are available for a month.
  2. Blue Storage: /blue/GROUP
    • Our highest-performance filesystem.
    • All I/O from jobs should go here.
  3. Orange Storage: /orange/GROUP
    • Slower than /blue
    • Not intended for large I/O for jobs.
    • Primarily for archival purposes.

Backup and Quotas

No Storage Backup by Default

No backups graphic Unless purchased separately, nothing is backed up on HiPerGator!!

Without backup, accidental file erasure is unrecoverable. People can and do lose valuable research data. You are responsible for protecting your data.

  • The Data Protection page describes the tape backup options and costs.
  • Orange and Blue storage quotas are at the group level and based on investment.
  • The blue_quota and orange_quota commands will show your group's current quota and use.
  • The home_quota command will show your home directory quota and use.

Automounting of Directories

Directory Automounting

  • Directories on the Orange and Blue filesystems are automounted--they are added only when accessed.
  • Your group's directory may not appear until you access it.

A GIF displaying how a directory is automounted upon access

As you can see in the GIF above, an ls in /blue does not show the group directory for ufhpc. After a cd into the directory, and back out, it does show. This shows how the directory is automounted on access. Of course, it would be easier to just go directly without stopping in /blue first.

Remember

  • Directories may not show up in an ls of /blue or /orange
  • If you cd to /blue and type ls, you will likely not see your group directory. You also cannot tab-complete the path to your group's directory. However, if you add the name of the group directory (e.g., cd /blue/GROUP/ the directory becomes available and tab-completion functions.
  • Of course, there is no need to change directories one step at a time...cd /blue/GROUP, will get there in one command.
  • You may need to type the path in SFTP clients or Globus for your group directory to appear
  • You cannot always use tab completion in the shell

Important user profile files

  • Do not remove files necessary for your account to function normally, such as the ~/.ssh directory or your entire /home directory.
  • Keep permissions on your home and ~/.ssh directories as they are.
    • Making your ~/.ssh directory readable to others can prevent user login.
    • Making your home directory accessible to others is a bad security practice.

Environment Modules System

  • HiPerGator uses the Lmod Environment Modules System to hide application installation complexity and make it easy to use the installed applications.
  • For applications, compilers or interpreters, load the corresponding module
  • See a list of all Installed Applications

Module 3: Common Mistakes and Getting Support

Common Mistakes

  1. Running resource intensive applications on the login nodes
  2. Using IDE SSH connections
    • VSCode, Spyder, and PyCharm all have SSH extensions. These will connect you to a login server, not a job running on HiPerGator.
    • Running applications using these often violates the acceptable use of login nodes.
    • Opening a folder with many files starts a recurring process to check for file modifications. With millions of files to check, this can quickly consume large amounts of CPU resources and impact the filesystem.
    • Use the VSCode Remote Tunnel extension as documented here.
  3. Writing to /home or /orange during batch job execution
  4. Wasting resources
    • Understand CPU and memory needs of your application
    • Over-requesting resources generally does not make your application run faster--it prevents other users from accessing resources.
  5. Blindly copying scripts from colleagues
    • Make sure you understand what borrowed scripts do
    • Many users copy previous labmate scripts, but do not understand the details
    • This often leads to wasted resources
  6. Misunderstanding the group investment limits and the burst QOS
    • Each group has specific limits
    • Burst jobs are run as idle resources are available
      • The slurmInfo command can show your group's investment and current use.
      • The showQos command can show the resource limits for a QOS. See qos limits for more information.
  7. When using our Python environment modules, attempting to install new Python packages may not work because incompatible packages may get installed into the ~/.local folder and result in errors at run time. If you need to install packages, create a personal or project-specific Conda environment or request the addition of new packages in existing environment modules via the RC Support System.

See the FAQ page for these and more hints, answers, and potential pitfalls that you may want to avoid.

How to get Help

See the Get Help page for more details. In general,

  • Submit support requests via the UFRC Support System
  • For problems with running jobs, provide:
    • JobID number(s)
    • Filesystem path(s) to job scripts and job logs
    • As much detailed information as you can about your problem
  • For requests to install an application, provide:
    • Name of application
    • URL to download the application

Module 4:

Have you ever sat through a training and thought, "This section exists because someone really messed up"? Well, welcome to that section — where we cover important policies and gently remind you that common sense isn't always as common as we'd hope.

Prohibition on Large Language Models from Countries of Concern

Florida state law prohibits the use of software, including large language models (LLMs), originating from countries of concern (CoC). You can find the current list of designated countries here, but for LLMs, this primarily means China. The university's official position is as follows:

The University strictly prohibits the deployment or utilization of large‑language models that are developed, operated, or otherwise hosted by organizations located in, or controlled by, a foreign country of concern. Accordingly, any use of such models on university computing resources, or the processing of any university data through them, is not permitted.

The UFIT Information Security Office maintains a list of Fast Path Solutions (requires UF login), with a filter for “Not Permitted” items, including the Prohibited use of Large Language Models. Do not work with language models developed in countries of concern on HiPerGator or other UF computing resources.

Important notes:

  • This prohibition applies to all UF computing resources, including HiPerGator, and covers inference, research, and any other form of use.
  • This prohibition applies to all users, whether or not they are affiliated with UF or located within the state of Florida.
  • We recognize that some may have legitimate research interests in these models, but there are no exceptions.
  • Finetuning, LoRA and other post-training modifications of LLMs from CoC do not make them acceptable. When the base model, trained with the original data, is used, no amount of post-training, optimization or re-releases by 3rd parties removes the concerns associated with these models.
  • First offenses typically result in an account suspension with a warning, and a meeting with UFIT staff is required for account reactivation.
  • Any subsequent violation will result in a permanent ban from HiPerGator.

Any questions? Please open a support request and ask rather than hoping you are OK.

Circumventing Idle GPU Monitoring

UFIT Research Computing policy prohibits allocating a GPU and leaving it idle. We actively enforce this. Running fake or non-productive workloads solely to get around our idle-detection tools is a violation of this policy — and we will notice.

You Are Responsible for Your Agentic AI Tools

If you deploy agentic AI tools, you own everything they do. If your agent runs a prohibited LLM, or figures out a creative way to game the idle GPU policy, that's on you — not the agent. We strongly encourage users to retain approval for all agent actions.

Bonus module for students using the cluster for courses

  • Course rosters are used to automatically create accounts after add/drop. Do not use the account request form for courses.
  • Course group names will be in the format pre1234 with the 3-letter department prefix and 4-digits course code.
  • All sections of a course will typically use the same group name, so the name may be different from the specific course you are enrolled in.
  • In the documentation below, substitute the generic pre1234 with your particular group name.
  • Students who do not have a HiPerGator Account will have one created for them.

  • Students who already have a HiPerGator Account will be added to the course group (as a secondary group).

    • You can create a folder in the class's /blue/pre1234 folder with your GatorLink username.

    • To use the resources of the class rather than your primary group, use the --account and --qos flags, in the submit script, in the sbatch command or in the boxes in the Open onDemand interface.

      1. In your submit script, add these lines: 

        #SBATCH --account=pre1234
#SBATCH --qos=pre1234

      2. In the sbatch command: sbatch --account=pre1234 --qos=pre1234 my_script.sh

      3. Using Open onDemand: Open OnDemand Account and QOS session configuration page

      JupyterHub Primary Group Only

      JupyterHub can only use your primary group's resources and cannot be used for accessing secondary group resources. To use Jupyter using your secondary group, please use Open onDemand.

  • Using your account implies agreeing to the Acceptable Use Policy.

  • Students understand that no restricted data should be used on HiPerGator.
  • Classes are typically allocated 32-cores, 250GB RAM and 2TB of storage
    • Instructors should keep this in mind when designing exercises and assignments.
    • Students should understand that these are shared resources: use them efficiently, share them fairly and know that if everyone waits until the last minute, there may not be enough resources to run all jobs.
  • All storage should be used for research and coursework only.
  • Accounts created for the class and the contents of the class /blue/pre1234 folder will be deleted at the end of the semester. Please copy anything you want to keep off of the cluster before the end of the semester.
  • Students should consult with their professor or TA rather than opening a support request.
    • Only the professor or TA should open support requests if needed.