domains / families / data / nemo-mbridge-resiliency

nemo-mbridge-resiliency

separate

Resiliency features in Megatron Bridge including fault tolerance, straggler detection, in-process restart, preemption, and re-run state machine.

When to use

• Enabling resiliency features
• investigating a commit that caused training hangs
• straggler detection failures
• broken restarts
• fault tolerance
• straggler detection
• hang detection
• automatic restart

Skill page

Source