domains / families / design / a-b-test-design

a-b-test-design

separate

Design an A/B experiment — hypothesis, variants, primary metric, and sample size. Use when a change can be measured quantitatively at scale. For observing behaviour qualitatively, use `test-scenario`.

When to use

• Designing A/B experiments
• Defining hypotheses
• Setting up controlled variants
• Selecting primary metrics
• Calculating sample size
• Determining test duration

When not to use

• Observing behaviour qualitatively
• Very low traffic scenarios
• Ethical concerns with withholding improvement
• Foundational changes affecting everything
• When qualitative insight is more valuable

Skill page

Source