Data Management

To perform research analyses, you need to upload and manage data. Note that misuse of the storage systems is the second main reason for account suspension after running analyses on login nodes.

Transferring Data

If you need to transfer datasets to or from HiPerGator and your local computer or another external location, you have to pick the appropriate transfer mechanism.

• For small or medium file transfers, use sftp, scp, or rsync to login to sftp.rc.ufl.edu, hpg.rc.ufl.edu, or rsync.rc.ufl.edu.

• For large file transfers or transfers with many small files, use our Globus service.

For more in-depth information, see Data Transfer Overview.

Small/Medium File Transfer

SFTPRSync

SFTP, or secure file transfer, works well for small to medium data transfers and is appropriate for both small and large data files.

If you would like to use a Graphical User Interface secure file transfer client, we recommend:

• Tabby for Windows, Linux, or MacOS.
• WinSCP for Windows.
• Cyberduck for MacOS X and Windows.

After you have chosen and downloaded a client, configure the client to connect to hpg.rc.ufl.edu, specifying port number 22. Use your username and password to log in.

If you prefer to use the command-line or to get maximum efficiency from your data transfers, Rsync, which is an incremental file transfer utility that minimizes network usage, is a good choice. It does so by transmitting only the differences between local and remote files rather than transmitting complete files every time a sync is run, as SFTP does. Rsync is best used for tasks like synchronizing files stored across multiple subdirectories or updating large datasets. It works well for both small and large files.

• See the Rsync page for instructions on using rsync.

Globus: Large File Transfer

Globus is a high-performance mechanism for file transfer. Globus works especially well for transferring large files or datasets.

• See the Globus page for setup and configuration information.

Samba

Samba service, also known as a 'network share' or 'mapped drive', provides you with the ability to connect to some HiPerGator filesystems as locally mapped drives (or mount points on Linux or MacOS X). Once you are connected to a share, this mechanism provides you with a file transfer option that allows you to use your client computer's native file manager to access and manage your files.

UFRC Samba setup does not provide high performance, so try to use it sparingly and for smaller files, like job scripts or analysis reports to be copied to your local system. You must be connected to the UF network (either on-campus or through the UF VPN) to connect to Samba shares.

• See the page on accessing Samba for setup information specific to your computer's operating system.

Automounted Paths

Note: NFS-based storage on our systems is typically automounted, which means they are dynamically mounted only when users are actually accessing them. For example, if you have an invested folder as /orange/smith, to access it, you will have to specifically type in the full path of /orange/smith to be able to see the contents and access them. Directly browsing /orange will not show the smith sub-folder unless someone else is using it at that moment. Automounted folders are very common on our systems, and include /blue, /orange, /bio, /rlts, and even /home etc.