MIT researchers cut LLM memory usage 50x with seconds-fast compression technique

Quick Facts

• MIT’s Attention Matching technique compresses LLM memory by 50x in seconds without quality loss
• Method eliminates hours of GPU training required by previous compression approaches
• Researchers tested successfully on 60,000-token medical datasets and advanced math problems

MIT researchers developed a breakthrough technique called Attention Matching that compresses large language model memory by 50 times without accuracy loss. The method works in seconds rather than the hours of GPU training required by previous approaches.

“In practice, KV cache memory is the biggest bottleneck to serving models at ultra-long context,” Adam Zweiger, co-author of the paper, told VentureBeat. The KV cache stores key-value pairs that help AI models remember previous parts of conversations or documents.

The technique preserves two critical mathematical properties when compressing memory vectors. It maintains the “attention output” – the actual information AI extracts when querying memory – using simple algebraic methods instead of compute-heavy optimization.

“Attention Matching is, in some ways, the ‘correct’ objective for doing latent context compaction in that it directly targets preserving the behavior of each attention head after compaction,” Zweiger said.

Researchers tested the method on popular models including Llama 3.1 and Qwen-3 using enterprise datasets. They used QuALITY, a reading comprehension benchmark with 5,000 to 8,000-word documents, and LongHealth, containing 60,000-token medical records.

In stress tests on advanced math problems, the system compressed working memory by 50 percent up to six consecutive times mid-calculation. The AI still solved problems successfully, matching performance of models with unlimited memory.

The breakthrough addresses major cost challenges. A single 128K context prompt on Llama 3.1-70B consumes about 40GB of high bandwidth memory just for the KV cache. A 70B model with 8K context requires roughly 640GB for a batch of 32 requests.

However, enterprise adoption faces hurdles. “This is not currently a simple plug-and-play software update,” Zweiger notes. Companies using closed APIs cannot implement the technique themselves – they need access to model weights through open-weight models.

The method joins other recent advances in KV cache optimization. Microsoft’s FastGen reduces memory demands by up to 50 percent, while KVTC achieves up to 20x compression for specific use cases. LMCACHE delivers up to 15x higher throughput when combined with inference frameworks like vLLM.

“The main practical tradeoff is that if you are trying to preserve nearly everything in-context on highly information-dense tasks, you generally need a milder compaction ratio to retain strong accuracy,” Zweiger explains.

Integration into existing commercial inference engines requires significant engineering work. Modern AI infrastructure uses complex optimizations like prefix caching and variable-length memory packing that must accommodate the new compression technique.

Read more: New KV cache compaction technique cuts LLM memory 50x without accuracy loss