domains / families / science / nemo-mbridge-perf-cpu-offloading

nemo-mbridge-perf-cpu-offloading

separate

Validate and use CPU offloading in Megatron Bridge, including layer-level activation offloading and fractional optimizer state offloading with HybridDeviceOptimizer.

When to use

• Enabling CPU offload to reduce GPU memory
• investigating a commit that changed CPU offloading config and caused OOM or a crash
• cpu_offloading
• optimizer_cpu_offload
• optimizer_offload_fraction
• HybridDeviceOptimizer
• move optimizer to CPU

When not to use

• Throughput is top priority
• CUDA graphs are needed with activation offloading

Skill page

Source