domains / families / science / nemo-mbridge-perf-hierarchical-context-parallel

nemo-mbridge-perf-hierarchical-context-parallel

separate

Operational guide for enabling hierarchical context parallelism in Megatron-Bridge, including config knobs, code anchors, pitfalls, and verification.

When to use

• Scaling context parallelism beyond KV heads
• Investigating a commit that changed CP config and caused OOM or a regression
• Configuring hierarchical_context_parallel_sizes
• Using a2a+p2p cp comm type
• Multi-level CP configuration

When not to use

• Tasks not related to hierarchical context parallelism

Skill page

Source