domains / families / software-engineering / ai-prompt-regression-testing

ai-prompt-regression-testing

authentic

Use when comparing prompt changes against a fixed evaluation set: record actual model settings, required checks, missing results, judge limitations and case-level regression evidence.

When to use

• Compare a prompt change with an observed baseline on a fixed evaluation set.
• Detect schema, factuality, refusal, tool-selection or task-completion regressions.
• Record a reviewable decision when results are noisy, missing or contradictory.

When not to use

• No RPC, relayer, signing key or blockchain deployment is required for this
• Exact text match is unsuitable for many valid open-ended outputs; embeddings
• A golden set must be reviewed/versioned; it is not created merely by naming it.
• This entry does not include a provider adapter, judge, dataset or runnable suite.
• A finite test set cannot prove future prompt behavior or absence of injection.

Skill page

Source