Temporary Directories

The /tmp use is prohibited

Warning

Do not use /tmp directory for any analyses

The /tmp directories on compute nodes are very small and are not suitable for use in jobs. Any /tmp use may result in job suspension or cancellation as full /tmp partitions cause the compute nodes to become unusable.

Job Temporary Directory

Here is the solution for the temporary directory needs in jobs. When a SLURM job starts, the scheduler creates a custom temporary directory for the job on the compute node's fast flash-based local hard drive. This directory also works great for jobs that need to use or generate a large number of small files, as the main RC parallel filesystem (blue storage) is optimized for large file streaming and is less suitable for small files.

The job temporary directory is owned by the user running the job. At the end of the job, the temporary directory is automatically removed unless the job fails in which case it's temporarily retained for troubleshooting.

This directory is accessible within the job via the $SLURM_TMPDIR or $TMPDIR environment variables. You can use the ${SLURM_TMPDIR} variable as a filesystem path in the shell script portion of the job script as an argument for any commands you run in a job.

You can also use the ${SLURM_TMPDIR} variable in shell commands at the beginning of the job to copy data to the temporary job directory (staging out) and copy results back in to blue at the end of the job script.

Default Temporary Path

Many applications and programming languages use the $TMPDIR environment variable, if available, as the default temporary directory path. If this variable is not set, the applications will default to using the /tmp directory, which is not feasible. SLURM will set $TMPDIR to the same value as $SLURM_TMPDIR unless $TMPDIR has already been set, in which case it will be ignored. Check your job script(s) and shell initialization files like .bashrc and .bash_profile to make sure you do not have $TMPDIR set.

If a personal Apptainer or Singularity container is used, make sure that the $APPTAINERENV_TMPDIR or $SINGULARITYENV_TMPDIR variable is set within the job to export the local scratch location into the Singularity container.