Evaluation and operations · 3 parts
Cost and latency
Budgets, caching and model routing.
How this question breaks down
“Design a cost and latency”The question
Work through the parts in order and you’ve covered the full design. Each part is one topic.