A/B testing

How this question breaks down

1. 1DesignCore24 minWrite the hypothesis and the decision before the data exists, pick the unit that gets randomised, hash it into stable buckets, analyse only the people the change could reach, and ramp traffic in steps that trade risk against power.
2. 2SignificanceCore17 minSize the test from the smallest effect worth shipping and the metric's noise, read the result as an interval rather than a verdict, and use pre-period data, the right variance formula and multiple-testing corrections so the interval is both narrow and honest.
3. 3PitfallsCore19 minMost wrong experiment results come from three places, effects that change over time, users who affect each other, and people who check the result too often. Each one leaves a signature in the data and has a design that avoids it.

Work through the parts in order and you’ve covered the full design. Each part is one topic.