A/B Testing Planner

Design rigorous A/B tests that produce statistically valid, actionable results. Covers hypothesis formation, variant design, sample sizing, and result interpretation.

Usage

Describe what you want to test, your current metrics, and traffic volume. The skill produces:

• Hypothesis: Structured "If we change X, then Y will happen because Z"
• Variants: Control and treatment designs with specific changes
• Primary Metric: The one metric that determines winner
• Guardrail Metrics: Metrics that must not degrade
• Sample Size: Required visitors/users for statistical significance
• Test Duration: Estimated runtime based on your traffic
• Analysis Plan: How to interpret results, including edge cases

Examples

1. Landing Page: "Plan an A/B test for our pricing page. Hypothesis: showing annual pricing first (vs monthly) increases plan selection by 15%. Current: 3% conversion, 10K monthly visitors."

1. Email Subject Lines: "Design an A/B test for our weekly newsletter subject lines. Testing emoji vs no emoji. List size: 50K, current open rate: 22%."

1. Onboarding Flow: "Plan a test for our app onboarding. Control: 5-step wizard. Variant: 3-step simplified flow. Measuring 7-day retention. 500 new users/week."

1. Checkout Page: "Test adding trust badges to our checkout page. Current cart abandonment: 68%. 2K monthly transactions."

Guidelines

• Test one variable at a time — multivariate tests need much larger sample sizes
• Calculate required sample size BEFORE starting the test, not after
• Use 95% confidence level and 80% statistical power as defaults
• Run tests for full weeks to account for day-of-week effects
• Never peek at results and stop early — this inflates false positive rates
• Define the minimum detectable effect upfront (what improvement would be meaningful?)
• Document and share results regardless of outcome — failed tests are valuable learning