domains / families / devops / mcore-run-on-slurm

mcore-run-on-slurm

separate

How to launch distributed Megatron-LM training jobs on a SLURM cluster. Covers a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis.

When to use

• Submitting a SLURM job
• Writing or debugging an sbatch script
• Configuring multi-node distributed training
• Setting MASTER_ADDR / MASTER_PORT / WORLD_SIZE
• Diagnosing a SLURM job failure
• Setting CUDA_DEVICE_MAX_CONNECTIONS
• Monitoring SLURM jobs
• Per-rank failure diagnosis

Skill page

Source