IndexCache Delivers 1.82x Faster Inference for Long-Context AI Models

Quick Facts

• IndexCache cuts up to 75% of redundant computation in sparse attention models, delivering 1.82x faster time-to-first-token at 200K context lengths
• The training-free technique reduces deployment costs by at least 20% for long-context workloads like RAG and document analysis
• Open-source patches are available for major serving engines including vLLM and SGLang

Researchers at Tsinghua University and Z.ai have developed IndexCache, a sparse attention optimizer that delivers significant speed improvements for long-context AI models. The technique eliminates redundant computation by reusing indices across layers rather than compressing memory.

IndexCache achieved a 1.82x speedup in prefill latency at 200K context length, reducing processing time from 19.5 seconds to 10.7 seconds. During decoding, the system boosted per-request throughput from 58 tokens per second to 86 tokens per second, delivering a 1.48x improvement.

“IndexCache is not a traditional KV cache compression or sharing technique,” said Yushi Bai, co-author and research lead. “It eliminates this redundancy by reusing indices across layers, thereby reducing computation rather than just memory footprint.”

The system works by partitioning layers into Full layers that run their own indexers and Shared layers that reuse cached data from nearby Full layers. This approach cuts computational overhead while preserving model accuracy.

Testing on a 30B parameter model showed the optimized version matched baseline performance on long-context benchmarks, scoring 49.9 versus the original 50.2. On the AIME 2025 math reasoning benchmark, the IndexCache model outperformed the baseline with a score of 92.6 compared to 91.0.

The technique applies to models using DeepSeek Sparse Attention architecture, including the latest DeepSeek and GLM families. Preliminary tests on the 744-billion-parameter GLM-5 model confirmed the performance gains scale to production environments.

“In terms of ROI, IndexCache provides consistent benefits across scenarios, but the gains are most noticeable in long-context workloads such as RAG, document analysis, and agentic pipelines,” Bai said. “In these cases, we observe at least an approximate 20% reduction in deployment cost.”

The researchers offer two deployment approaches. The training-free method uses a greedy search algorithm to select which layers retain indexers without requiring weight updates. The training-aware method introduces multi-layer distillation loss for even greater efficiency.

Open-source patches are available on GitHub for SGLang and vLLM serving engines. Integration requires minimal configuration changes to existing inference stacks.

Read more: IndexCache, a new sparse attention optimizer, delivers 1.82x faster inference on long-context AI models