domains / families / data / lm-evaluation-harness

lm-evaluation-harness

separate

Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.

When to use

• Benchmarking models for academic papers
• Comparing model quality across standard tasks
• Tracking training progress
• Reporting standardized metrics (everyone uses same prompts)
• Need reproducible evaluation
• **HELM** (Stanford): Broader evaluation (fairness, efficiency, calibration)
• **AlpacaEval**: Instruction-following evaluation with LLM judges
• **MT-Bench**: Conversational multi-turn evaluation

Skill page

Source