Slurm's Configuration and Usage

• Job Scheduling: Queues and schedules jobs based on priorities, resource availability, and user limits.
• Resource Allocation: Handles requests for specific hardware like GPUs, which is crucial for AI training.
• Fault Tolerance: Manages node failures and job restarts.
• Scalability: Supports clusters from a few nodes to thousands, making it ideal for distributed AI training (e.g., data parallelism or model parallelism).
• Plugins and Extensibility: Integrates with tools like MPI for multi-node jobs.

Slurm is widely used in supercomputing centers (e.g., on the Top500 list) and is free under the GPL license. It's maintained by SchedMD.

AI models, especially large language models or computer vision tasks, require significant computational resources. Slurm allows you to:

• Request multiple GPUs across nodes.
• Run jobs non-interactively (batch mode) for long training sessions.
• Monitor and manage resource usage to avoid overloading the cluster.
• Scale training with distributed frameworks like Horovod or PyTorch Distributed.

Without Slurm, you'd manually manage resources, which is inefficient on shared clusters.

Run sinfo to view cluster information. If it's not found, Slurm isn't installed or not in your PATH.

Example output:

PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
gpu          up   infinite      4   idle gpu[1-4]
cpu          up   infinite     10   idle cpu[1-10]

This shows partitions (queues) like "gpu" for GPU jobs.

• Job: A unit of work, like running a Python script for model training.
• Partition: A queue where jobs are submitted (e.g., "gpu" for GPU-enabled nodes).
• Node: A physical or virtual machine in the cluster.
• Task: A process within a job (e.g., one task per GPU).
• Allocation: The resources granted to your job (e.g., 2 GPUs for 4 hours).
• Account and QoS: User groups and quality-of-service levels for fair sharing.

• sinfo: Shows partitions, nodes, and their states. Example: sinfo -p gpu (view only GPU partition).
• squeue: Lists running and queued jobs. Example: squeue -u yourusername (your jobs only). Output columns: JOBID, PARTITION, NAME, USER, ST (state: R=running, PD=pending), TIME, NODES, NODELIST.

There are two main ways: interactive (srun) for testing, and batch (sbatch) for production.

• Interactive Job: srun Useful for quick tests or debugging AI code. Example: srun --partition=gpu --gres=gpu:1 python train.py This requests 1 GPU and runs train.py interactively.
• Batch Job: sbatch Submit a script for non-interactive execution. Example: sbatch myslurmscript.slurm Returns a JOBID for tracking.

• scancel JOBID: Cancel a specific job. Example: scancel 12345

• sacct: View accounting info for completed jobs (e.g., sacct -j JOBID for CPU/GPU usage).
• scontrol: Advanced control, like scontrol show job JOBID for details.
• salloc: Allocate resources for an interactive session (similar to srun but without running a command immediately).

Batch scripts are shell scripts with #SBATCH directives at the top. These specify resource requests.

Scripts typically start with #!/bin/bash, followed by #SBATCH lines, then your commands.

Example for AI training (save as train.slurm):

#!/bin/bash
#SBATCH --job-name=AI_Training     # Job name
#SBATCH --partition=gpu            # Partition (queue)
#SBATCH --nodes=1                  # Number of nodes
#SBATCH --ntasks=1                 # Number of tasks (processes)
#SBATCH --cpus-per-task=4          # CPUs per task
#SBATCH --gres=gpu:2               # GPUs per node (e.g., 2 GPUs)
#SBATCH --mem=16G                  # Memory per node
#SBATCH --time=04:00:00            # Time limit (HH:MM:SS)
#SBATCH --output=train_%j.out      # Stdout file (%j = JOBID)
#SBATCH --error=train_%j.err       # Stderr file
#SBATCH --mail-type=END,FAIL       # Email notifications
#SBATCH --mail-user=your@email.com # Your email

# Load environment (e.g., modules for Python/PyTorch)
module load python/3.10
module load cuda/11.8
module load pytorch/2.0

# Activate virtual environment if needed
source ~/venv/bin/activate

# Run your AI training script
python train_model.py --epochs 50 --batch-size 32

# Optional: Post-processing
echo "Training complete!"

• –gres=gpu:N: Request N GPUs per node. Check available with sinfo.
• –nodes=M: For multi-node training (e.g., distributed data parallel).
• –ntasks-per-node=K: Tasks (e.g., one per GPU).
• –time: Max runtime; jobs are killed if exceeded.
• –mem: Total memory; use –mem-per-cpu for per-CPU.
• –account: If your cluster uses accounts for billing.
• –constraint: Specify hardware features, e.g., –constraint="volta" for GPU type.

For AI, ensure your script handles GPU visibility (e.g., via CUDA_VISIBLE_DEVICES, but Slurm sets it automatically with –gres).

• Submit: sbatch train.slurm
• Monitor: squeue -j JOBID
• View logs: Check train_JOBID.out and .err files.

Use --gres=gpu:4 for 4 GPUs. In PyTorch, use torch.nn.DataParallel or DistributedDataParallel.

Example addition to script:

srun python train.py --num_gpus 4

(Use srun inside batch scripts for MPI-like launching.)

For large models:

• Request --nodes=2 --ntasks-per-node=1 --gres=gpu:4 (8 GPUs total).
• Use MPI or PyTorch DDP.
• Launch with srun --mpi=pmix python train.py (assuming MPI module loaded).

• Use modules: module load for software like CUDA, PyTorch.
• Containers: Slurm supports Singularity/Apptainer for Docker-like images. Example: srun singularity exec --nv myimage.sif python train.py (–nv for GPU).

• Mount shared storage (e.g., /scratch) for datasets.
• Use --chdir=/path/to/data to set working directory.

• Submit dependent jobs: sbatch --dependency=afterok:JOBID nextjob.slurm

Ideal for AI hyperparameter search. Example script:

#SBATCH --array=0-9
PARAMS=(0.001 0.01 0.1 ...)  # Array of learning rates
python train.py --lr ${PARAMS[$SLURM_ARRAY_TASK_ID]}

• Check with sacctmgr or admin for QoS.
• Fairshare: Jobs from heavy users may have lower priority.

For reproducible AI environments:

• Build a Singularity image from Docker: singularity build myimage.sif docker://pytorch/pytorch
• Run: sbatch with singularity exec --nv.