Large language model inference is increasingly becoming a systems problem. As models grow, context windows expand, and inference architectures become more distributed, delivering better performance is no longer simply a matter of deploying faster GPUs. The infrastructure surrounding those GPUs increasingly determines how efficiently they can serve tokens.
One of the most important parts of infrastructure optimization is Key-Value (KV) cache management. This white paper examines how KV-cache size, movement, and reuse affect inference performance.
Many techniques exist, but three performance drivers matter most:
- Shrink it: Reduce the KV-cache footprint
- Move it: Transfer KV state efficiently
- Reuse it: Avoid unnecessary recomputation
