NYU Research Team Cuts LLM Input 16x With New Context Compression Method

Quick Facts

• Research team from six universities developed Latent Context Language Models that compress LLM inputs up to 16x while maintaining 75% accuracy
• At 4x compression, the method achieves 91.76% accuracy compared to 94.41% with no compression – less than 3 percentage points drop
• Models are open-sourced on HuggingFace and work with standard LLM inference engines without modification

A research team from NYU, Columbia, Princeton, University of Maryland, Harvard and Lawrence Livermore National Laboratory developed a breakthrough method that compresses large language model input by up to 16 times without destroying performance.

The team’s Latent Context Language Models (LCLMs) compress input token sequences before decoder processing. This produces output 8.8 times faster than traditional KV cache methods on the RULER long-context benchmark at 16x compression ratios.

At 4x compression, LCLMs achieve 91.76% accuracy on the RULER benchmark compared to 94.41% with no compression. Even at extreme 16x compression – where 93.75% of input tokens are removed – accuracy only drops to 75.06%.

Technical Architecture

The system uses a 0.6B-parameter encoder paired with a 4B-parameter decoder. The encoder compresses blocks of input tokens into shorter sequences of latent embeddings. The decoder then processes these compressed representations instead of original tokens.

The team trained models on over 350 billion tokens each at compression ratios of 1:4, 1:8, and 1:16. This continual pre-training approach solved a key tradeoff where earlier compression methods sacrificed general task performance to maintain reconstruction accuracy.

“These ballooning contexts take up memory and compute, and they are becoming a computational bottleneck for LLMs,” said Micah Goldblum, co-lead advisor on the project at Columbia University. “If you can make such a language model, everything becomes cheaper and faster.”

Commercial Impact

The breakthrough addresses critical cost concerns in LLM deployment. Models that perform well on academic benchmarks often become commercially unfeasible due to excessive compute costs per unit of inference.

Self-hosted deployments can save up to 78% compared to pay-per-token services for high-volume workloads. Context compression technologies could expand these savings further by reducing the computational overhead of processing long inputs.

The models are production-ready and available on HuggingFace under the latent-context organization. The compression approach works with standard LLM inference engines and supports parallelization without requiring infrastructure changes.

This research could reshape LLM economics by making inference more affordable. The combination of high compression ratios with minimal accuracy loss removes a major scalability bottleneck that has limited commercial LLM adoption across industries.

Read more: Context compression finally works in production: new research cuts LLM input 16x without the accuracy hit