domains / families / research / knowledge-distillation

knowledge-distillation

separate

Compress large language models using knowledge distillation from teacher to student models. Use when deploying smaller models with retained performance, transferring GPT-4 capabilities to open-source models, or reducing inference costs. Covers temperature scaling, soft targets, reverse KLD, logit distillation, and MiniLLM training strategies.

When to use

• **Compress models** from 70B → 7B while retaining 90%+ performance
• **Transfer capabilities** from proprietary models (GPT-4) to open-source (LLaMA, Mistral)
• **Reduce inference costs** by deploying smaller student models
• **Create specialized models** by distilling domain-specific knowledge
• **Improve small models** using synthetic data from large teachers

Skill page

Source