LLM inference

How this question breaks down

1. 1KV cacheCore25 minWhy a language model keeps every past token's keys and values in GPU memory, how big that gets, and how paging, sharing and smaller heads let one GPU hold hundreds of conversations instead of dozens.
2. 2BatchingCore22 minWhy serving a language model one request at a time wastes the GPU, why fixed batches waste it differently, and how rebuilding the batch on every step, with chunked prefills and admission by free memory, keeps it full without stalling anyone.
3. 3SpeculativeCore21 minLet a cheap guesser propose several tokens and the big model check them all in one pass, keep the ones it agrees with, and get the same output distribution in fewer slow steps, when the load is light enough to have compute to spare.

Work through the parts in order and you’ve covered the full design. Each part is one topic.