domains / families / agent-tooling / create-skill-test

create-skill-test

separate

Scaffolds eval.yaml evaluation specs for skills, custom agents, and redistributable gh-aw workflow packages in the dotnet/skills repository. Use when creating skill or workflow-package tests, writing evaluation stimuli, defining graders and rubrics, sizing an eval for statistical power, or setting up test fixture files. Handles the Vally eval.yaml schema, fixture organization, and overfitting avoidance. Do not use for running or debugging existing evals (use improve-skill-quality) nor for skills authoring (use create-skill).

When to use

• Creating a new `eval.yaml` for a skill, agent, or workflow package
• Adding stimuli to an existing eval
• Sizing an eval so the pass gate can actually be reached
• Setting up or repairing fixture files alongside an eval
• Reviewing whether rubric items and graders risk overfitting

When not to use

• Diagnosing a failing or regressed eval — use `improve-skill-quality`
• Modifying the skill-validator or the evaluation workflows
• Creating or editing `SKILL.md` files — use `create-skill`

Skill page

Source