domains / families / agent-tooling / agent-evaluation
agent-evaluation
authentic
Evaluate agent behavior with versioned cases and explicit verifiers. Use when comparing agent or prompt changes, reproducing failures, or running agent regression tests.
When to use
- Use when comparing a changed agent, prompt or tool configuration, reproducing an observed failure, or estimating reliability on a declared task distribution. Do not infer product readiness from a public benchmark percentage or a generic sc…
When not to use
- Illustrative 80/90% thresholds and score weights in the architecture sketches are not universal merge/deploy rules; define project-specific criteria and keep critical failures separate.
- A small-sample chi-squared comparison or absence of significance does not prove equivalence; use a method suited to counts, pairing and multiple comparisons.
- Exceptions are not automatic safe rejections, and test retries must not erase the first failure.
- Similarity to a retrieved answer may be legitimate RAG behavior; leakage depends on what the evaluation permits the agent to know.
- LLM judges do not substitute for real user feedback, and output truncation does not remove private data. Use synthetic or authorized redacted inputs with bounded retention.
Skill page
Source