domains / marketing / ab-test-setup / ab-test-setup

ab-test-setup

Skill card

combined

The LeoYeAI copies (openclaw-marketing-skills and openclaw-master-skills) are nearly identical, providing a comprehensive guide with specific sample size tables, traffic allocation strategies, and implementation details (client vs. server-side). The sickn33 copy is a more rigorous, gate-based procedure that adds critical steps missing from the LeoYeAI versions, specifically a detailed 'Tracking Verification' checklist (event firing, variant attribution, de-duplication, randomization checks) and 'Refusal Conditions' for when a test cannot be validly run. Because the sickn33 version contains material safety and verification steps that the LeoYeAI versions lack, and the LeoYeAI versions contain specific reference data (sample size tables) that the sickn33 version omits, crowning one would hide a real alternative approach to the task.

Repositories

Other copies

Combined skill

# A/B Test Setup

## 1. Initial Assessment & Context
1. Check for product marketing context (e.g., `.agents/product-marketing-context.md`) if available.
2. Understand Test Context, Current State (baseline rate, traffic volume), and Constraints.

## 2. Hypothesis Lock (Hard Gate)
Before designing variants, define a specific hypothesis:
- **Structure**: Because [observation], we believe [change] will cause [outcome] for [audience].
- **Requirements**: Must include target audience, primary metric, expected direction, and Minimum Detectable Effect (MDE).
- **Refusal Condition**: If baseline rate is unknown or traffic is insufficient to detect the MDE, refuse to proceed.

## 3. Test Design
1. **Test Type**: Choose the simplest valid test (A/B, A/B/n, MVT, or Split URL). Default to A/B.
2. **Metrics**:
   - **Primary**: Single metric tied to business value.
   - **Secondary**: Contextual metrics.
   - **Guardrail**: Metrics that must not degrade.
3. **Sample Size**: Calculate required sample size per variant based on baseline, MDE, alpha (0.05), and power (80%).
   - *Reference*: Use calculators or tables (e.g., 1% baseline, 10% lift requires ~150k/variant).
4. **Traffic Allocation**: Decide split (50/50 standard, 90/10 conservative).

## 4. Tracking Verification (Mandatory)
Before launch, verify:
1. **Event Firing**: Confirm events arrive in analytics with expected latency.
2. **Variant Attribution**: Ensure variant ID is attached to all relevant events.
3. **De-duplication**: Confirm reloading does not double-count events.
4. **Randomization**: Check for sample-ratio mismatch and assignment stability.
5. **Guardrails**: Ensure guardrail metrics have working dashboards/alerts.

## 5. Execution Readiness Gate
Proceed only if:
- Hypothesis is locked.
- Primary metric is frozen.
- Sample size is calculated.
- Tracking is verified.
- QA is completed.

## 6. Running the Test
- **DO**: Monitor technical health, document external factors.
- **DON'T**: Peek at results, change variants, add new traffic sources, or redefine success criteria.

## 7. Analysis & Documentation
1. Check if sample size was reached.
2. Evaluate statistical significance (p < 0.05) and effect size.
3. Review guardrail metrics.
4. Document hypothesis, results, decision, and learnings in a shared location.