domains / families / devops / nemo-mbridge-multi-node-slurm

nemo-mbridge-multi-node-slurm

separate

Convert single-node scripts to multi-node Slurm sbatch jobs and debug common multi-node failures. Covers srun-native vs uv run torch.distributed approaches, container setup, NCCL timeouts, OOM sizing for MoE models, and interactive allocation.

When to use

• Writing or converting Slurm sbatch scripts
• Scaling to multiple nodes
• Debugging NCCL/launch failures
• Investigating multi-node training failures
• srun-native vs uv run torch.distributed approaches
• Container setup
• NCCL timeouts
• OOM sizing for MoE models

Skill page

Source