Transfer Data¶

This section describes transferring data between a local computer (client) and HiPerGator (HPG). For file sharing on the cluster see Sharing Files

Tools¶

There are a variety of command-line, GUI, and even web-based tools available for transferring data to or from HiPerGator. Command-line tools include, for example, cp, mv, scp, rsync, sftp, wget, curl, or ncftp. GUI tools like Cyberduck, WinSCP, BitVise SFTP and similar tools can be used on your local computer. Globus data transfer tool is available via a web interface in addition to command-line and GUI versions that can be run on your local computer. Please visit the HPG how-to video series on Data Transfer for more details.

General guidelines for selecting a tool¶

Method	Best for	Considerations	Resume interrupted transfers?	Speed
Globus	Large datasets	Requires Globus on both ends of transfer	Auto-resumes	Fastest
rsync	Most use cases	A good option, but command line only	Manual resume	Fast
SFTP	Small to medium sized transfers	Convenient CLI and GUI options	No	Fast
Samba	Browsing or small transfers	Convenient, with folders mounted on local desktop, but has issues	No	Slow
OOD Files menu	Small uploads/downloads	Doesn't handle large files or folders well	No	Medium
Jupyter upload	Single small files	Cannot upload/download folder, limited to <2GB	No	Slow

Transferring Data between a local computer and HiPerGator¶

Login servers on HiPergator (hpg.rc.ufl.edu) can be used for rsync, copying files with cp or mv depending on the permissions or filesystem ACLs (extended permissions).

Globus¶

If your data files are large (hundreds of megabytes or gigabytes) then try Globus first.

SFTP/Rsync¶

For smaller file sizes, or if Globus is not an option, use SFTP/rsync/scp by connecting to the hpg.rc.ufl.edu. Set port to 22 if you have to specify it. Use your gatorlink credentials to connect. Make sure MFA (multi-factor authentication) is taken into account to avoid having to go through MFA for every file transferred. You can also use GUI interfaces like Cyberduck (Mac, Windows, Linux), WinSCP (Windows) or Bitvise (Windows).

Note: If you are using Cyberduck on a MacOS computer and notice that it is using the old password resulting in a security lockout uncheck 'Use Keychain' in 'Preferences > General'.

Note

Check out the How-To Video: Transferring data with SFTP

Do not use FileZilla

We do not recommend using FileZilla. It does not work well with the MFA setup on HiPerGator.

Alternative UF Fast Path Solution approved SFTP Applications include:

Cyberduck (MacOS, Windows, Linux)
WinSCP (Windows)
BitVise (Windows)

Open OnDemand Files Tab¶

Open OnDemand has a menu for Files, where you can navigate your home, Blue, and if present, Orange folders. You can upload, download, view and edit files in this interface. Howeve, we do not recommend this tool for managing large numbers of files, or nested folder structures. Uploading or downloading files over a couple of gigabytes will be slow and may fail.

JupyterHub/JupyterLab¶

The easiest way to upload data using JypyterHub is by using the Upload Files button included in the File Browser (Ctrl+Shift+F) menu.

As with the Open OnDemand Files menu, we do not recommend this tool for managing large numbers of files, or nested folder structures. Uploading or downloading files over a couple of gigabytes will be slow and may fail.

Transferring Data From HiPerGator to a remote system/site¶

If you are logged into HiPerGator and need to transfer data to or from a remote system or a site use the login nodes. Transfers can also be made from within developmental sessions. You can use ftp (lftp command), sftp, scp, rsync, or Globus to transfer data out.

Downloading Data from an External Server or Website¶

Most "Download" buttons on external websites have the option to right-click and select an option similar to "Copy link address". Once the link address is copied in the clipboard, you can paste it in your HPG terminal following your "wget" or "curl" commands, such as:

$ wget <https://github.com/author/software/main.zip>`

Internet Browser from HPG Host¶

Another option is to use the Firefox or Chrome browsers via Open OnDemand HiPerGator Desktop session to a website, then download files into a target directory.

Start a HiPerGator Desktop session on Open OnDemand:

From a HPG node, launch the module ubuntu, which contains a version of the Google Chrome browser.
```
   $ module load ubuntu
   $ chrome (or firefox)
   The Internet browser should launch.
```
Navigate and login to the third-party service, such as Dropbox or Google Drive using your gatorlink account.
Edit the browser's download settings to change the Download location (path) to your preferred /blue directory.
Navigate to the target directory in your Drive account using the browser and select files or folders to download.

The files should download directly to your selected /blue path.

Download data to personal computer first¶

Alternatively, users can download files to their local computers and upload them into HPG using one of the methods described in the "Between a local computer and HiPerGator" section above.

Providing access to your data¶

If you need to share data with another RC user or publicly for collaboration, view instructions at Providing Access To Data