Serving architectures

How this question breaks down

1. 1Batch vs onlineCore21 minCompute a prediction before anyone asks (batch), when they ask (online), or when its inputs change (streaming), and pick by what you know in advance, how fresh the answer must be and how many answers will ever be read.
2. 2FunnelsCore19 minCheap models look at many items and expensive models at a few. How retrieval, filtering, light ranking, heavy ranking and re-ranking split the work, how to choose each stage's cut, and why every stage can only lose items the next one would have wanted.
3. 3LatencyCore23 minTurn one end-to-end p99 target into a time slice per stage, let each stage size its work to its slice, and decide in advance what the request does when a stage runs over.

Work through the parts in order and you’ve covered the full design. Each part is one topic.