domains / families / science / nemo-mbridge-perf-megatron-fsdp

nemo-mbridge-perf-megatron-fsdp

separate

Operational guide for enabling Megatron FSDP in Megatron-Bridge, including config knobs, code anchors, pitfalls, and verification.

When to use

• enabling Megatron FSDP
• using FSDP-based data parallelism instead of DDP
• tracing an OOM or regression to a FSDP config change
• configuring use_megatron_fsdp
• configuring data_parallel_sharding_strategy
• sharded data parallel setup
• Megatron FSDP verification

When not to use

• using torch FSDP2
• using use_tp_pp_dp_mapping
• using torch_dist checkpoint format

Skill page

Source