domains / families / research / trl-fine-tuning

trl-fine-tuning

separate

Fine-tune LLMs using reinforcement learning with TRL - SFT for instruction tuning, DPO for preference alignment, PPO/GRPO for reward optimization, and reward model training. Use when need RLHF, align model with preferences, or train from human feedback. Works with HuggingFace Transformers.

When to use

• Need to align model with human preferences
• Have preference data (chosen/rejected pairs)
• Want to use reinforcement learning (PPO, GRPO)
• Need reward model training
• Doing RLHF (full pipeline)
• **SFT**: Have prompt-completion pairs, want basic instruction following
• **DPO**: Have preferences, want simple alignment (no reward model needed)
• **PPO**: Have reward model, need maximum control over RL

Skill page

Source