The Hidden Cost of Moving Your KV Cache

Getting your Trinity Audio player ready...

To run efficiently, inference clusters use disaggregated nodes to separate the two main phases of text generation. “Prefill” processes incoming prompts using heavy parallel compute, while “Decode” generates the text using high memory bandwidth. Running both on the same GPU is highly inefficient, so disaggregation cleanly splits these tasks into dedicated nodes.

But long context comes at a cost. These prompts generate large KV cache during the prefill stage and moving that data from the prefill node to the decode node becomes a serious bottleneck as workloads scale across more GPU nodes. This blog introduces KVeXpress, our data transfer framework that speeds up the handoff with three core elements: Just-in-Time (JIT) Decode Allocation, Vectorized KV Transfer, and Decode-Driven Movement. Together, these three elements deliver the required performance improvements.

The Cost of Idle VRAM

One of the most significant bottlenecks in current disaggregated LLM workloads is critical timing and control mismatch. Traditional transfer methods keep the prefill node involved in the data movement path and force the decode node to reserve memory before prefill work begins, leaving costly decode VRAM sitting idle.

To understand why this happens, we need to look at how memory is managed. LLM serving engines like SGLang do not store each request’s KV cache as one contiguous memory range. In an 8-GPU tensor-parallel decode group, the KV cache is sharded across GPUs and managed through fixed-size pages in each GPU’s KV memory pool. For a single 32K-token request on a 671B model, this can add up to roughly 50GB of KV cache across the node, represented as many page mappings as possible rather than one contiguous block.

Moving this fragmented data between nodes takes coordination. Modern transfer frameworks like Mooncake handle the transfer with a push-based design built on one-sided RDMA WRITE operations, where the prefill node writes the KV cache directly into the decode node’s memory.

In this push-based design, the prefill node initiates and executes the actual data transfer. Before any data moves, the two nodes exchange metadata about the cache layout, and the prefill node drives the kernel launches and control notifications that set up the write. Because a one-sided write needs a destination that is already reserved, the decode node has to allocate its VRAM in advance. It reserves that memory long before the prefill node finishes computing and the data actually arrives, so the decode memory sits idle, waiting for the transfer. By forcing decode memory to be reserved up front, this idle VRAM window reduces cluster concurrency, leaves requests waiting longer before prefill can finish, and increases Time to First Token in prefill-heavy multi-prefill topology. This is exactly the inefficiency KVeXpress was built to solve.

The Three Integrated Elements of KVeXpress

Our team bypassed this coordination bottleneck by building KVeXpress, an alternative transfer framework to Mooncake that rethinks how data moves between nodes. It works through three main changes:

JIT Decode Allocation

Under KVeXpress, we implement smarter memory allocation in SGLang’s disaggregated KV cache transfer path. This technique delays reserving VRAM on the decode node until the prefill node has completely finished processing the input prompt. This ensures that decode VRAM remains completely unallocated and available for other active requests while the prefill node is performing its heavy computation.

Vectorized KV Transfer

Paged KV memory layouts are highly fragmented and inefficient for network transfers. To solve this, KVeXpress introduces a KV Transfer Packing stage that analyzes the page index maps from both prefill and decode. This stage checks the source and destination layouts to find physically adjacent pages on both servers. It then groups these neighboring pages into large blocks before starting the RDMA transfer. This reduces the number of small network transactions and lets the network interface card (NIC) move data in larger blocks while maintaining the underlying paged memory model.

Decode-Driven Movement

To maximize the efficiency of the compute-heavy prefill node, we shifted the network data transfer to a pull-based model. Instead of forcing the prefill node to orchestrate the pushing of data, the transfer responsibility moves entirely to the decode node. The prefill node finishes its computation and moves on to the next prompt almost immediately, because decode work is short compared to prefill. The decode node manages its own queues and actively pulls the data over the network only when it is ready.

Performance Data from a Benchmark Sweep

We evaluated KVeXpress on a 3-node AMD MI355X cluster using a prefill-heavy long-context workload: 64K input tokens and 1K output tokens. The comparison used a 2P1D topology, with two prefill nodes and one decode node, and measured KVeXpress against the strongest Mooncake baseline observed across the same three-node setup:

Concurrency	TTFT (ms) ↓	Inter-Token Latency (ms) ↓	TPOT (ms) ↓	E2E Latency (ms) ↓	Output Throughput (tok/s) ↑	Total Throughput (tok/s) ↑	Actual mean concurrency
Conc.8	-15.0%	+1.3%	+1.2%	-4.1%	+5.0%	+5.0%	+0.7%
Conc.16	-24.9%	+5.6%	+5.6%	-12.6%	+15.6%	+15.6%	+1.0%
Conc.32	-5.6%	+11.1%	+11.1%	-1.4%	+5.2%	+5.2%	+3.7%
Conc.48	-8.0%	+16.4%	+16.3%	-4.2%	+6.3%	+6.3%	+1.9%
Average	-13.4%	+8.6%	+8.5%	-5.6%	+8.0%	+8.0%	+1.8%
↓ lower is better	↑ higher is better

 

This data presents a clear narrative:

• Faster Total Completion is achieved because the input context is large. SGLang reduces Time to First Token by an average of 13.4 percent.
• Slower Token Pacing occurs after text generation begins, meaning individual tokens are delivered at a slower pace. This is represented by an 8.5 percent slower TPOT and an 8.6 percent slower ITL. This occurs because delaying VRAM allocation allows SGLang to run more concurrent user requests on the active GPUs, which distributes physical compute resources.
• Overall latency is reduced because the upfront time savings absorb the sequential token-pacing delay over a 1K token generation loop. Ultimately, the entire request finishes faster, delivering a 5.6 percent reduction in End-to-End Latency alongside an 8% boost in overall cluster throughput.

Maximizing Your AI Cluster Investment

As AI workloads grow larger and more complex, one truth becomes clear. AI workload performance is a recipe of many ingredients, both software and hardware, and getting that “sugar, spice, and everything nice” into perfect balance takes far more than throwing raw compute at the problem. It demands a holistic, deep understanding of every layer of the AI pipeline, from how memory is orchestrated to how data physically moves across the cluster.

This is exactly where the DriveNets Full-Stack approach provides unique value. By leveraging our vast experience in high-performance networking and deep cooperation with silicon leaders like AMD, we deliver bottom-up tuning from the network OS all the way to the inference runtime.

KVeXpress is a prime example of this out-of-the-box engineering philosophy. We identified a software bottleneck that holds back the long context use case in disaggregated inference, and we re-engineered it. By eliminating idle VRAM windows and shifting to smarter memory allocation, we ensure AI workloads are never held back by the software surrounding them.

Key takeaways

• • The Idle VRAM Bottleneck: Traditional push-based frameworks force decode nodes to reserve memory before prefill work finishes, leaving expensive GPU VRAM sitting idle and lowering overall cluster concurrency.

• • Pull-Based Orchestration: KVeXpress shifts the data transfer responsibility from compute-heavy prefill nodes to a pull-based model managed by the decode node, allowing prefill servers to immediately move to the next prompt.

• • Vectorized Packing: To combat memory fragmentation (where a single 32K token request can span 50GB of non-contiguous pages across an 8-GPU group), KVeXpress groups adjacent pages into large blocks prior to RDMA transfer.

• • Proven Performance Metrics: Benchmarks on a 3-node AMD MI355X cluster show that KVeXpress decreases Time to First Token (TTFT) by an average of 13.4% and boots total cluster throughput by 8.0% over the Mooncake baseline.

Frequently Asked Questions

What is KVeXpress and how does it optimize disaggregated LLM inference?

KVeXpress is a data transfer framework that optimizes disaggregated LLM inference by accelerating the KV cache handoff between prefill and decode nodes. The framework utilizes Just-in-Time (JIT) Decode Allocation, Vectorized KV Transfer, and a pull-based Decode-Driven Movement model. These integrated elements eliminate idle VRAM windows, reducing coordination bottlenecks across GPU nodes.

How does JIT Decode Allocation solve the idle VRAM problem in LLM clusters?

JIT Decode Allocation solves idle VRAM inefficiencies by delaying memory reservation on the decode node until the prefill node completely finishes processing input prompts. This technique ensures decode VRAM remains unallocated and available for other active requests during heavy parallel compute phases. By eliminating the upfront reservation required by traditional push-based frameworks, cluster concurrency improves significantly.

What performance gains does KVeXpress deliver compared to the Mooncake baseline?

KVeXpress delivers a 13.4% average reduction in Time to First Token (TTFT) compared to Mooncake baselines. Tested on a three-node AMD MI355X cluster with 64K input tokens, the framework reduces end-to-end latency by 5.6% and increases total cluster throughput by 8.0%. These improvements are achieved despite a minor 8.5% slowdown in token-pacing metrics.

Related content for AI networking infrastructure

DriveNets AI Networking Solution

Latest Resources on AI Networking: Videos, White Papers, etc