The Slurm workload scheduler is used to manage the compute nodes on the cluster. Jobs must be submitted through the scheduler to have access to compute resources on the system.There are two groups of partitions (aka "queues") currently configured on the system. One group is specifically designated for CPU jobs, i.e., non-GPU jobs: defq* default compute nodes, CPU only, highThru high throughput or highMem high memory. Consisting of a 192gb, 384gb or 3tb of memory respectively. NOTE: Running jobs on the login nodes is prohibited and may result in account suspension.
- highThru – Nodes in this category have large memory (384gb), high CPU frequency nodes with a low core count. These nodes are used for single threaded jobs that cannot take advantage of multiple cores.
- highMem –Nodes in this category have large memory – 3tb and are for jobs that require more memory intensive jobs.
- debug-cpu - please see DebugPartition for more details
- short - Designed for quicker turnaround of shorter running jobs – 1 day. Some nodes here overlap with defq.
- tiny -Designed for even quicker turnaround of shorter running jobs – 8 hours. Some nodes here overlap with defq.
The other is specifically for jobs requiring GPU resources and should not be used for jobs that do not require GPU resources:
- small-gpu – has access to the GPU nodes, each has two NVIDIA NVIDIA Tesla V100 GPU.
- large-gpu – has access to four Nvidia Tesla V100 SXM2 16GB GPUs with NVLink enabled
- debug-gpu - please see DebugPartition for more details.
Note that you must set a time limit (with the -t flag, for example -t 1:00:00 would set a limit of one hour) for your jobs when submitting, otherwise they will be immediately rejected. This allows the Slurm scheduler to keep the system busy by backfilling when trying to allocate resources for larger jobs. Currently there is no maximum time limit, but you are encouraged to keep jobs limited to a day - longer running processes should checkpoint and restart to avoid losing significant amounts of outstanding work if there is a problem with the hardware or cluster configuration. The maximum time-limit for any job is 14 days. We will not under any circumstances increase time limits for jobs already begun. Please estimate your required time carefully and request a time limit accordingly. We will provide guidance to help you better understand your job's requirements if you are unsure. The time-node limit for a single job may not exceed 224 node*days which prevents any single job from allocating more the 16 nodes for 14 days.
As an example of a simple MPI job script, which could be submitted as
#!/bin/sh # one hour timelimit: #SBATCH --time 1:00:00 # default queue, 32 processors (two nodes worth) #SBATCH -p defq -n 32 module load openmpi mpirun ./test
The Slurm documentation has further documentation of some of the advanced features. The Slurm Quick-Start User Guide provides a good overview. The use of Job Arrays (Job Array Support) is mandatory for people submitting a large quantity of similar jobs.
#!/bin/bash # set output and error output filenames, %j will be replaced by Slurm with the jobid #SBATCH -o testing%j.out #SBATCH -e testing%j.err # single node in the "short" partition #SBATCH -N 1 #SBATCH -p short # half hour timelimit #SBATCH -t 0:30:00 module load matlab ssh login4 -L 27000:188.8.131.52:27000 -L 27001:184.108.40.206:27001 -N & export LM_LICENSE_FILE="27000@localhost" # test.m is your matlab code matlab -nodesktop < test.m