domains / families / data / validate-evaluator

validate-evaluator

authentic

Calibrate an LLM judge against human labels using data splits, TPR/TNR, and bias correction. Use after writing a judge prompt (write-judge-prompt) when you need to verify alignment before trusting its outputs. Do NOT use for code-based evaluators (those are deterministic; test with unit tests per `write-code-eval`).

When to use

• Calibrating LLM judges
• Measuring TPR and TNR
• Applying bias correction
• Verifying judge alignment
• Inspecting judge disagreements
• Estimating true success rate

When not to use

• Code-based evaluators
• Deterministic tests
• Unit testing

Skill page

Source