domains / families / research / torchtitan

torchtitan

separate

Provides PyTorch-native distributed LLM pretraining using torchtitan with 4D parallelism (FSDP2, TP, PP, CP). Use when pretraining Llama 3.1, DeepSeek V3, or custom models at scale from 8 to 512+ GPUs with Float8, torch.compile, and distributed checkpointing.

When to use

• Pretraining LLMs from scratch (8B to 405B+)
• Need PyTorch-native solution without third-party dependencies
• Require composable 4D parallelism (FSDP2, TP, PP, CP)
• Training on H100s with Float8 support
• Want interoperable checkpoints with torchtune/HuggingFace
• **Megatron-LM**: Maximum performance for NVIDIA-only deployments
• **DeepSpeed**: Broader ZeRO optimization ecosystem, inference support
• **Axolotl/TRL**: Fine-tuning rather than pretraining

Skill page

Source