Use the Roofline Model to Analyze Prefill and Decoding
AMDSoftware Engineer · OnsiteLast updated
Apply roofline analysis to LLM prefill and decoding using arithmetic intensity, sustainable bandwidth, batching, KV-cache traffic, and measured kernel behavior.
Use the roofline model to reason about an LLM inference workload. How do you determine whether a kernel is memory-bound or compute-bound, and how do prefill and decoding differ?
Constraints & Assumptions
Specify the memory level and operation-count convention used in arithmetic intensity. Treat prefill/decoding bottlenecks as workload-dependent rather than universal labels.
Clarifying Questions
What are the batch size, sequence length, precision, hardware compute ceiling, and sustainable memory bandwidth? Are measurements kernel-local or end-to-end? Which bytes are actually transferred?
Follow-up questions
- Can a kernel sit far below both roofs? How do kernel launches, synchronization, and small shapes affect the diagnosis? What changes when decode batch size or context length grows?
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.