Tips

On this page

Job Arrays

Job arrays provide a means of submitting your job to multiple nodes at once. The only difference being an array index, that is incremented for each job, which can be used in your batch script or codes as a variable to change different parameters, data input/output files, or the flow of your code.

To submit an array job use the #SBATCH --array=i-n directive:

#!/bin/sh

#SBATCH --mail-type=ALL
#SBATCH --mail-user=uniqname@umich.edu
#SBATCH --time=1-0
#SBATCH --job-name=array_job_test
#SBATCH --array=1-10

Rscript --vanilla ./script.R

This batch script will submit job 10 jobs with names array_job_test[1] … array_job_test[10] incrementing the $SLURM_ARRAY_TASK_ID environment variable with each job submission. This environment variable can be access directly in your code, for example in R, like this

# grab the array id value from the environment variable passed from sbatch
slurm_arrayid <- Sys.getenv('SLURM_ARRAY_TASK_ID')

# coerce the value to an integer
n <- as.numeric(slurm_arrayid)

Job Dependencies

It is possible to delay the start of a job until a specified dependency has been satisfied. For example, suppose you have multiple jobs running concurrently that each generate separate data sets that need to be aggregated as the final step. You could just wait for all the jobs to complete and download the results and combine on your local system. Or you could create another job to aggregate your results and set it dependent on all your jobs. Examples can be found on the department's GitHub page.

See the SLURM sbatch documentation for more details.

Job Performance Factors

For most work, performance differences between similar jobs do not have an impact on the results of the research. Jobs are submitted and results are received when the job completes. However, some workflows may be sensitive to this, like writing new methods or packages and then comparing the speed to generally available packages.

Differences in job performance for the same code are caused mainly by three factors:

  • the computers the job runs on
  • the geometry of the resources allocated within the computer
  • how heavily the computer is utilized

The following suggestions should only be implemented if your work is dependent on consistent performance. These suggestions can cause longer queue times and lesser job throughput.

The computers the job runs on

Jobs on the cluster run on one of the many compute nodes which were added piecewise over the lifetime of the cluster beginning in 2010. Currently there are 9 different hardware configurations in the cluster. As you can imagine, not all of these configurations are equally performant. The older computers tend to have slower processors and lower clocked memory than the newer ones. When doing performance sensitive work it is critical that you run your jobs on the same hardware configuration.

To enable this, all hardware types have unique "features" defined with the workload manager Slurm. You should not request these features if your jobs do not need consistent hardware. When you request a feature, your job will only run on a node with that feature. If all nodes with that feature are busy, your job will wait in the queue until nodes with that feature are free - even if other nodes are idle. The hardware configurations and their corresponding features may be found on the Hardware Overview page.

To request a given feature, add this line to your batch scripts: --constraint=<feature>

For example, to run on a PowerEdge R430 with Intel Xeon E5-2650 v4 @ 2.20GHz use:#SBATCH --constraint=E5-2650v4

You may want to consider including the hardware specifications that your jobs ran on in your papers. For example "This work was performed on a Dell PowerEdge R430 with dual Intel Xeon E5-2650 v4 @ 2.20GHz and 256 GB of Memory".

Utilization of the Computer

The frequency of a CPU core changes depending on the amount of work being done by the processor. This is due to a feature called turboboost. When a processor is heavily utilized, the individual CPU cores operate at the advertised base frequency. When the processor is not doing much work, the frequency of the individual cores increases up to the potential maximum called the turbo boost maximum. Turbo boost is great for getting more work done on a processor with low utilization, but it makes it difficult to have predictable and consistent performance on a compute node which is being used by multiple jobs. In order to minimize the effects of turboboost, you should submit your jobs with the exclusive flag #SBATCH --exclusive. This will guarantee that your job is the only job running on the node. Jobs with the exclusive flag are often in queue for a longer time as they must wait for a node to become idle before they can start.

NUMA

Requesting exclusive jobs also mitigates the effects of Non-Uniform Memory Architecture (NUMA). The compute nodes are all dual-socket, meaning they have 2 processors with memory channels attached to each processor. Jobs running on one CPU core on processor 0 can access the memory on both socket 0 and 1, however accessing the memory on socket 1 is slower. With only one job on a node you will get a consistent NUMA arrangement. Please only use the exclusive flag when you really need it. Use of this parameter can decrease cluster throughput dramatically.