Success metrics

How this question breaks down

1. 1Online vs offlineCore18 minOffline metrics grade a model on logged data in minutes; online metrics grade the product on live users over weeks. The skill is knowing when the first predicts the second, and building the rungs between them.
2. 2GuardrailsCore17 minThe metrics a launch must not break (latency, crashes, opt-outs, other surfaces, the experiment's own integrity), how to set a margin for each, and how to power a test so "no harm" means something.
3. 3ProxiesCore17 minWhy every metric you can read in two weeks stands in for one you can't, how to check that the stand-in moves with the goal across experiments, and the ways optimising it breaks the link.

Work through the parts in order and you’ve covered the full design. Each part is one topic.