domains / families / data / nemo-evaluator

nemo-evaluator

separate

Evaluates LLMs across 100+ benchmarks from 18+ harnesses (MMLU, HumanEval, GSM8K, safety, VLM) with multi-backend execution. Use when needing scalable evaluation on local Docker, Slurm HPC, or cloud platforms. NVIDIA's enterprise-grade platform with container-first architecture for reproducible benchmarking.

When to use

• Need **100+ benchmarks** from 18+ harnesses in one platform
• Running evaluations on **Slurm HPC clusters** or cloud
• Requiring **reproducible** containerized evaluation
• Evaluating against **OpenAI-compatible APIs** (vLLM, TRT-LLM, NIMs)
• Need **enterprise-grade** evaluation with result export (MLflow, W&B)
• **lm-evaluation-harness**: Simpler setup for quick local evaluation
• **bigcode-evaluation-harness**: Focused only on code benchmarks
• **HELM**: Stanford's broader evaluation (fairness, efficiency)

Skill page

Source