domains / families / research / eval-audit

eval-audit

authentic

Audit an LLM eval pipeline and surface problems: missing error analysis, unvalidated judges, vanity metrics, etc. Use when inheriting an eval system, when unsure whether evals are trustworthy, or as a starting point when no eval infrastructure exists. Do NOT use when the goal is to build a new evaluator from scratch (use error-discovery, write-judge-prompt, or validate-evaluator instead).

When to use

• inheriting an eval system
• assessing whether evals are trustworthy
• starting point when no eval infrastructure exists
• inspecting LLM eval pipelines
• producing prioritized lists of problems

When not to use

• building a new evaluator from scratch
• using error-discovery
• using write-judge-prompt
• using validate-evaluator

Skill page

Source