domains / agent-tooling / agent-evaluation / evaluation

evaluation

Skill card

combined

The second copy is a direct mirror of the first (it cites the first repository as its source), but it is not a faithful reproduction. It introduces specific, conflicting performance metrics (the '95% Finding' with 80/10/5 variance splits) that the first copy explicitly avoids by using qualitative descriptors ('Primary', 'Secondary'). It also includes specific model recommendations (Claude Sonnet 4.5, GPT-5.2) and a 'risk: safe' metadata field absent from the original. Because the copies disagree on the quantitative basis for evaluation design, crowning one would hide a real alternative interpretation of the underlying research.

Repositories

Other copies

Combined skill

---
name: evaluation
description: "Build evaluation frameworks for agent systems: deterministic checks, regression suites, multi-dimensional rubrics, and production monitoring."
---

# Evaluation Methods for Agent Systems

Evaluate agent systems by focusing on outcomes rather than execution paths, as agents are non-deterministic and may take multiple valid routes to a goal.

## When to Activate
- Testing agent performance systematically
- Validating context engineering choices
- Catching regressions before deployment
- Building quality gates for agent pipelines

## Core Concepts

### Performance Drivers
When designing evaluation budgets, account for the primary drivers of performance variance:
- **Token usage**: The primary driver. More exploration can improve performance until cost or context quality collapses.
- **Tool calls**: Secondary driver. More tool use helps only when calls retrieve useful evidence.
- **Model choice**: Secondary but multiplicative. Better models often use tokens and tools more efficiently.

**Implications for Design:**
- Set realistic token budgets that reflect production constraints.
- Compare model upgrades against token increases to determine the most efficient path to improvement.
- Validate multi-agent architectures against single-agent baselines to ensure the added token/tool overhead is justified.

### Evaluation Challenges
- **Non-Determinism**: Define outcome criteria (correctness, completeness) rather than checking for specific steps.
- **Context-Dependent Failures**: Include simple, medium, complex, and very complex test cases to surface failures that emerge as context accumulates.
- **Composite Quality**: Score dimensions independently to avoid averaging away critical signals.

## Rubric Design

Build multi-dimensional rubrics with descriptive levels (excellent to failed) for:
1. **Factual accuracy**: Claims match ground truth.
2. **Completeness**: Output covers requested aspects.
3. **Citation accuracy**: Citations match claimed sources.
4. **Source quality**: Uses appropriate primary sources.
5. **Tool efficiency**: Uses right tools a reasonable number of times.

**Scoring:**
- Map assessments to numeric scores (0.0 to 1.0).
- Apply per-dimension weights based on use-case priorities.
- Set passing thresholds (e.g., 0.7 for general, 0.9 for high-stakes).

## Methodologies

1. **Deterministic Validation**: Run machine-checkable checks (schema validity, duplicate keys, manifest sync) before LLM judgment to fail fast.
2. **LLM-as-Judge**: Use for scalable evaluation once deterministic checks are stable. Provide clear task descriptions, ground truth, and structured judgment scales.
3. **Human Evaluation**: Use to catch edge cases, hallucinations on unusual queries, and subtle biases that automation misses.

## Test Set Design
- Start with small samples during early development to reveal large effects.
- Sample from real usage patterns and include known edge cases.
- Ensure coverage across all complexity levels.