domains / families / data / bigcode-evaluation-harness

bigcode-evaluation-harness

separate

Evaluates code generation models across HumanEval, MBPP, MultiPL-E, and 15+ benchmarks with pass@k metrics. Use when benchmarking code models, comparing coding abilities, testing multi-language support, or measuring code generation quality. Industry standard from BigCode Project used by HuggingFace leaderboards.

When to use

• Evaluating **code generation** models specifically
• Need **multi-language** evaluation (18 languages via MultiPL-E)
• Testing **functional correctness** with unit tests (pass@k)
• Benchmarking for **BigCode/HuggingFace leaderboards**
• Evaluating **fill-in-the-middle** (FIM) capabilities
• **lm-evaluation-harness**: General LLM benchmarks (MMLU, GSM8K, HellaSwag)
• **EvalPlus**: Stricter HumanEval+/MBPP+ with more test cases
• **SWE-bench**: Real-world GitHub issue resolution

Skill page

Source