domains / families / science / nemo-mbridge-perf-activation-recompute

nemo-mbridge-perf-activation-recompute

separate

Validate and use selective and full activation recompute in Megatron Bridge to reduce GPU memory usage at the cost of extra compute. Use for activation memory OOMs or regressions involving recompute_granularity, recompute_num_layers, recompute_modules, recompute_method, selective recompute, full recompute, or activation checkpointing.

When to use

• activation memory OOMs
• regressions involving recompute_granularity
• regressions involving recompute_num_layers
• regressions involving recompute_modules
• regressions involving recompute_method
• selective recompute
• full recompute
• activation checkpointing

When not to use

• allocator fragmentation
• pipeline parallelism greater than one with cpu_offloading

Skill page

Source