domains / families / science / nemo-automodel-distributed-training

nemo-automodel-distributed-training

separate

Guide for selecting and configuring distributed training strategies in NeMo AutoModel, including FSDP2, Megatron FSDP, DDP, and parallelism settings.

When to use

• NeMo AutoModel uses PyTorch-native distributed training. All parallelism is orchestrated through a single `MeshContext` object that holds device meshes, strategy configs, and axis names. <!-- NVSkills catalog signing requested after PR #29…

When not to use

• Model onboarding
• Recipes
• Slurm
• SkyPilot
• Checkpointing

Skill page

Source