How to upgrade GQA models to MLA for massive KV Cache Savings

Getting your Trinity Audio player ready...

Memory has become equally critical. Currently, TSMC’s CoWoS packaging is severely constrained, sending supply shockwaves throughout the entire hardware ecosystem.

It will be a long time before we can meet customer demand.
C.C. Wei, Chairman and CEO of TSMC

As Broadcom’s President Charlie Kawwas recently highlighted at the RAISE Summit 2026 in Paris, the ever-growing memory demands of Tier-1 AI infrastructure are creating a massive supply crunch even outside the AI business.

Simply put, the industry is hitting an AI Memory Wall. Expensive GPUs are spending far too much time sitting idle, waiting for massive volumes of historical conversation data to move in and out of their VRAM. To break through this wall, we do not just need faster hardware. We need smarter data compression.

In this blog, we will break down the memory wall and explore practical ways to shrink an AI workload’s KV cache. To keep things accessible, we will cover the underlying linear algebra at a high, easy-to-understand level.

The KV Cache problem

During new text generation, a Large Language Model (LLM) must process every previous token in the sequence to predict the next word. To avoid recomputing tokens at every single step, the inference engine saves the calculated Key and Value matrices in a dedicated memory buffer known as the Key-Value (KV) Cache.

As agentic workloads become dominant, prompts grow longer and tasks become more iterative and complex, sometimes stretching to 128K tokens or more, causing this KV Cache to explode.

The VRAM fills up instantly, which limits how many user sessions a GPU can handle concurrently and can force the system to recompute previously calculated data simply because sufficient fast, accessible storage is not available. When a GPU hits its memory limit, it starts throwing away data only to recalculate it later. This wasted effort slows everything down and keeps users waiting for a response. To stop this cycle, engineers have spent years completely rethinking how AI manages its memory.

The evolution of attention from MHA to MLA

To understand how the KV cache is generated and stored, it helps to understand the mechanisms that produce it. These are the different types of attention:

• Multi-Head Attention (MHA) was the original standard, powering early models like the first ChatGPT. In this design, the model looks at the text through dozens of parallel attention heads, and every single Query head gets its own dedicated Key and Value memory slot. It is a huge memory hog, making it too expensive for real-world long conversations.
• Multi-Query Attention (MQA) was the first big memory saver. Instead of giving every Query head its own memory, all of them share a single Key and Value record, and the model barely notices. The cache shrinks dramatically, but the cut is lossy and too aggressive for some tasks.
• Grouped Query Attention (GQA) is used by popular models like LLaMA and GPT-oss. It takes the middle road between MHA and MQA. The Query heads are split into groups, and each group shares a single Key and Value record, cutting the cache by the group size. This saves a lot of memory, but it is a brute-force cut that can hurt the AI’s reasoning and fine details, and the group count is locked in at training time.
• Multi-Head Latent Attention (MLA) broke the compromise between memory and quality. DeepSeek designed MLA to act like a smart zip file. Instead of storing massive sparse raw memory vectors for every single word, MLA uses linear algebra to compress the data down to a dense vector (c^KV). Just as MP3 audio compression removes sounds humans cannot hear, MLA spots the overlap inside the model’s Key and Value memory and rewrites it into a new dense vector space. This brings back the extreme accuracy while shrinking the KV Cache footprint by up to 93%. As a result, a single GPU can handle more users in the exact same VRAM space.

Solving the retraining problem

Most of the popular models in production today run on GQA, including LLaMA, Mistral, Qwen, Gemma, and GPT-oss. Meanwhile, models built on far more efficient attention designs keep arriving, from the above MLA to Kimi K3’s KDA. So why isn’t everyone simply switching?

The hesitation is not due to a lack of value. MLA’s compressed KV cache can benefit almost anyone serving AI models in production, since squeezing more user sessions out of the same VRAM translates directly into better ROI. The problem is adoption, as it usually requires throwing away existing models and spending millions of dollars training new ones from scratch.

To bridge this gap, we based our efforts on the TransMLA research paper. This paper introduced a training-free mathematical pipeline capable of refactoring standard GQA models directly into MLA-compatible architectures.

By building directly upon the foundational mathematical framework laid out in the TransMLA paper, our team turned that conversion process into an automated, enterprise-ready pipeline and improved its performance.

Under the hood: how offline conversion works

TransMLA acts as an offline matrix encoder. Before deploying a model, an offline script transforms the static weight matrices through five core stages:

• Merge: The separate Key (W^K) and Value (W^V) projection matrices from each group are stacked into a joint projection matrix (W^DKV).
• RoRoPE: Rotary Position Embedding (RoPE) information normally scatters across all attention heads. RoRoPE rotates the joint matrix so most of the positional information is isolated into the leading attention head, leaving the remaining heads purely semantic. This mirrors MLA’s own design, where part of the vector carries positional information, and the rest does not.
• FreqFold: Because adjacent RoPE frequencies share nearly identical rotation angles (θ_l ≈ θ_l+1), FreqFold groups adjacent frequency bands together into shared subspaces, protecting positional semantics during compression.
• BKV Normalization and PCA: Key vector magnitudes are scaled to match Value magnitudes (BKV), preventing greedy compression algorithms from erasing Value data. Principal Component Analysis (PCA) then extracts the core mathematical variance via eigen-decomposition of the covariance matrix.
• Decompression Multiplex: Because matrix multiplication is associative, the key-up projection matrix can be folded directly into the model’s Query weights (W^Q) ahead of time. This offline multiplication means the keys never need to be decompressed at inference at all – the query is projected into the compressed latent space instead, so the KV cache stays small throughout and keeps memory and data-movement costs low during decoding.

Together, these five stages compress the Key-Value representation for deployment without losing accuracy. At runtime the GPUs simply read this ultra-compact cache natively without experiencing any decompression overhead.

Real-world results

We kept the performance comparison simple and fair. Both models ran under identical conditions.

Setup	Details
Model	GPT-oss-120B: vanilla GQA vs. our MLA-converted version
Hardware	One server with 8X AMD MI355X GPUs
Software	Identical serving stack for both models
GPU memory	Exact same memory budget for both models
Workload	Long prompts, 60K tokens per request
Method	Raise concurrent requests until throughput collapses (the capacity cliff)

The results, shown in the chart below, speak for themselves. The vanilla GQA model hits its cliff at 300 concurrent requests. The MLA-converted model keeps serving until 900, roughly 3X the capacity from the exact same hardware and memory. That improvement is directly from the cache reduction. Every token’s KV entry shrinks from 2K bytes to 768 bytes, a 62.5% saving, so the same memory pool holds almost three times as many requests.

The obvious question is what the compression costs in quality, so we measured it. To analyze the tradeoff, we used the MHA2MLA and TransMLA papers’ accuracy benchmark across a seven-task evaluation suite (MMLU, ARC-easy/challenge, PIQA, HellaSwag, OpenBookQA, and WinoGrande). The MLA-converted model scores on par with the vanilla GQA baseline at the same compressed cache budget used above. No retraining and no accuracy tax.

DriveNets edge

Getting the maximum performance out of an AI cluster requires a holistic view of every single layer. That is exactly how DriveNets operates. We deliver industry-leading cluster performance by tuning the network, the GPU collectives (xCCL), the KV-cache transport, and even the attention mechanism of the model itself.

At the model layer, our first step was to take the conversion process from the research paper and enhance it into a fully automated and enterprise-ready pipeline. Our goal is to allow any model operator to convert an existing GQA model into an MLA architecture without the massive training investment that is usually required.

For the next phase, we are investigating new algorithms tailored to specific use cases. One major enhancement is called Spectral Energy-Based Rank Selection. The original TransMLA method applies the exact same compression limit across every layer of the model. This approach is inefficient because it completely ignores how information is actually distributed throughout the network.

To fix this, researchers at DriveNets AI Labs analyze the mathematical variance of each layer by measuring its information density (eigenvalues). We call this the layer’s spectral energy.

By extending the mathematical foundation of the TransMLA research paper with dynamic Spectral Energy-Based Rank Selection, we can dramatically lower VRAM requirements for post-training. Keep an eye out for our upcoming white paper, where we will dive much deeper into the exact mechanics of this innovative approach.

Conclusion

The AI Memory Wall does not have to be a permanent roadblock. By rethinking how models handle attention and applying smart mathematical conversions, we can shrink the KV cache footprint by 62.5% and triple the capacity for concurrent requests on the exact same hardware. Most importantly, these massive gains are achieved without the enormous cost of retraining or any accuracy tax.

Key Takeaways

• GPU performance is increasingly bottlenecked by memory capacity and hardware constraints (such as TSMC’s CoWoS packaging) rather than pure compute, as expanding context lengths cause Key-Value (KV) Caches to consume vast amounts of VRAM.
• Using an automated pipeline built on the TransMLA research paper, existing Grouped Query Attention (GQA) models can be mathematically converted to Multi-Head Latent Attention (MLA) offline, eliminating the need for expensive scratch retraining.
• Benchmark tests on GPT-oss-120B demonstrated a 62.5% reduction in per-request KV cache size (from 2K to 768 bytes per token) and tripled max concurrent request capacity from 300 to 900 on the same GPU budget.
• The offline compression technique preserves key semantic and positional data, achieving massive memory savings without incurring an accuracy penalty on standard evaluation suites like MMLU and HellaSwag.
• DriveNets is expanding on TransMLA by introducing Spectral Energy-Based Rank Selection, which measures layer-by-layer information density (eigenvalues) to apply dynamic compression ratios instead of uniform limits across all layers.

Frequently Asked Questions

How does upgrading GQA models to MLA improve LLM inference performance?

Upgrading Grouped Query Attention (GQA) to Multi-Head Latent Attention (MLA) reduces the KV cache footprint by 62.5% per request. By shrinking token KV entries from 2K bytes to 768 bytes, an 8x AMD MI355X GPU cluster tripled concurrent request capacity from 300 to 900 without accuracy loss.

How can GQA models be converted to MLA without retraining from scratch?

GQA models convert to MLA without retraining using an automated offline matrix encoding pipeline based on the TransMLA framework. This process refactors static weight matrices through five core stages—Merge, RoRoPE, FreqFold, BKV Normalization/PCA, and Decompression Multiplex—allowing GPUs to natively read compact latent cache at runtime.

Does converting a model to MLA negatively impact LLM accuracy?

Converting GQA models to MLA does not penalize benchmark accuracy when tuned to optimal rank budgets. Across a seven-task evaluation suite including MMLU, ARC, and HellaSwag, the converted GPT-oss-120B model matched the vanilla GQA baseline score while achieving a 62.5% reduction in KV cache memory.