domains / families / data / megatron-core

megatron-core

separate

Trains large language models (2B-462B parameters) using NVIDIA Megatron-Core with advanced parallelism strategies. Use when training models >1B parameters, need maximum GPU efficiency (47% MFU on H100), or require tensor/pipeline/sequence/context/expert parallelism. Production-ready framework used for Nemotron, LLaMA, DeepSeek.

When to use

• Training models >10B parameters
• Need maximum efficiency (target >40% MFU)
• Using NVIDIA GPUs (A100, H100)
• Production training at scale
• Want fine-grained parallelism control
• **PyTorch FSDP**: Models <70B, simpler API, PyTorch native
• **DeepSpeed**: Easier setup, good for <100B models
• **HuggingFace Accelerate**: Prototyping, simpler workflows

Skill page

Source