Trace an LLM Request Through a Paged-KV Inference Engine
AMDSoftware Engineer · OnsiteLast updated
Trace LLM inference from request admission through prefill and decoding, explaining paged KV block tables, continuous batching, cache lifetime, and latency tradeoffs.
Trace a prompt through an LLM inference engine, using a vLLM-style paged-KV architecture as the reference. Explain request admission, prefill, decoding, PagedAttention, the KV-cache block table, and continuous batching.
Constraints & Assumptions
Discuss the architecture and invariants rather than a particular library release or internal class name. State model, context-length, latency, and memory assumptions when they affect the design.
Clarifying Questions
How long are prompts and outputs? What does the latency target measure? Can prefixes be shared? What happens when KV-cache capacity is exhausted? Is the request streamed?
Follow-up questions
- Why can contiguous per-request KV allocation waste memory? How do block tables enable noncontiguous storage? What changes when requests finish at different times or long prompts enter an active batch?
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.