Experiment Design

Design controlled experiments that support valid causal inference. Covers randomization, control groups, blinding, confound identification, and validity threats with practical solutions.

Usage

Describe the causal question you want to answer, your population, available resources, and ethical constraints. The designer creates an experiment plan that maximizes internal validity while remaining practical.

Parameters

• Design: True experiment, Quasi-experiment, Natural experiment, or A/B test
• Setting: Laboratory, Field, Online, or Clinical
• Constraints: Ethical limitations, Budget, Timeline, or Sample access
• Outcome: Single primary, Multiple primary, or Exploratory

Examples

1. Software A/B Test: Design an A/B test for a pricing page — randomization unit, sample size calculation, duration estimation, novelty effects, and guardrail metrics to prevent harm.

1. Classroom Intervention: Quasi-experimental design for a teaching method study where random assignment isn't possible — matched control groups, difference-in-differences, and regression discontinuity options.

1. Lab Experiment: Controlled laboratory study of decision-making under time pressure — within-subjects vs. between-subjects tradeoffs, counterbalancing, and demand characteristics mitigation.

1. Multi-Arm Bandit: Adaptive experimental design that adjusts allocation during the experiment — Thompson sampling, regret minimization, and when adaptive designs are appropriate vs. fixed allocation.

Guidelines

• Randomization procedures are specified (simple, stratified, cluster, or blocked)
• Control conditions are designed to isolate the specific mechanism being tested
• Blinding is implemented where possible (single, double, or triple blind)
• Confounding variables are identified and controlled through design or statistical adjustment
• Pre-registration commits to hypotheses and analysis plans before data collection
• Intention-to-treat analysis preserves the benefits of randomization
• Manipulation checks verify that the experimental treatment worked as intended
• Replication considerations are built into the design from the start
• Ethical review requirements are identified and addressed proactively
• Pilot runs test procedures and measurement instruments before the full study