Experiment designer

--- name: experiment-designer description: "Design statistically rigorous A/B tests and interpret experiment results. Use this when asked to design an experiment, run an A/B test, calculate sample size, interpret test results, or assess whether an experiment was successful. It produces a complete experiment design that includes a hypothesis, sample size, run time, success criteria, and risk flags:or an interpretation of the results with a recommendation to ship, iterate, or kill." --- # Experiment Designer Skill Produce rigorous experiment designs based on product hypotheses, and interpret results with statistical and practical significance:so you can defend every decision to a skeptical engineering lead or data scientist. ## Required Inputs Ask the user for these if not provided: For experiment design: - Hypothesis (what change, what metric, what expected movement) - Current baseline metric value - Minimum detectable effect (MDE) : the smallest lift worth caring about - Available daily sample size For results interpretation: - Control and variant results (raw numbers or percentages) - P-value or confidence interval - Run duration (days) - Any anomalies observed during the test ## Two-Phase Process ### Phase 1: Experiment Design 1. Restate the hypothesis as: “If we [change], we expect [metric] to [change by X%] because [reason]” 2. Clearly define the control and variant 3. Select a primary metric (only one) and secondary guardrail metrics (2:3 max) 4. Calculate the required sample size based on MDE and the baseline 5. Estimate the run duration in days 6. Set predefined success criteria before the test begins:no changing the rules midway 7. Identify design risks: novelty effects, seasonal confounds, multiple testing issues, network effects, sample ratio mismatch ### Phase 2: Interpretation of Results 1. Assess statistical significance (p < 0.05 threshold) 2. Assess practical significance: Was the lift meaningful for the business, not just statistically significant? 3. Interpret confidence intervals 4. Investigate confounding factors 5. Recommend: Launch / Iterate / Cancel / Run a follow-up test 6. Validate : Confirm that the test ran for the full planned duration. Flag if it was stopped early (peeking problem). Confirm that no sample ratio mismatch occurred. ## Output Structure [Design or Results header based on phase] Hypothesis: "If we [change], we expect [metric] to [change by X%] because [reason]" Primary metric: [One metric only] Guardrail metrics: [2:3 max] Required sample size: [n per variant] Estimated run time: [days] Pre-defined success threshold: [specific number] Design risk flags: [any concerns] Results (Phase 2 only): Statistical significance: [p-value and conclusion] Practical significance: [lift size vs. business threshold] Recommendation: Ship / Iterate / Kill / Follow-up : [rationale] ## Quality Checks - [ ] The hypothesis specifies the change, the metric, the direction, and the reason - [ ] The primary metric is singular : guardrail metrics are secondary - [ ] Success criteria are defined before the test launches (not after seeing results) - [ ] The test was not stopped early (or clearly flagged if it was) - [ ] Practical significance is assessed separately from statistical significance - [ ] Sample ratio mismatch is checked when interpreting results ## Anti-Patterns - [ ] Do not define success criteria after seeing preliminary results : post-hoc success definitions constitute HARKing (Hypothesizing After Results are Known) and invalidate the experiment - [ ] Do not stop a test early simply because the result appears significant:early stopping dramatically inflates false positive rates; the test must run until the planned sample size is reached - [ ] Do not equate statistical significance with practical significance:a p < 0.05 result with a 0.1% lift is real but may not be worth implementing - [ ] Do not run the same experiment on the same population multiple times without correction : multiple testing proportionally increases the chance of a false positive - [ ] Do not use more than one primary metric : multiple primary metrics require multiple hypothesis corrections and make the decision to ship or kill ambiguous