AT&T Cuts AI Costs by 90% Processing 8 Billion Daily Tokens Through Multi-Agent Architecture

Quick Facts

• AT&T processes 8 billion tokens daily while achieving 90% cost savings through multi-agent AI architecture
• The company deployed 410+ generative AI agents across 100,000+ employees using LangChain and Microsoft Azure
• AT&T targets $2 billion in AI-driven cost savings by mid-2026 as part of broader efficiency goals

AT&T transformed its AI operations to handle massive scale while dramatically cutting costs. The telecom giant now processes 8 billion tokens daily through a multi-agent architecture that delivers 90% cost savings compared to traditional large language model approaches.

Chief Data Officer Andy Markus led the overhaul when AT&T’s token usage reached unsustainable levels. The company rebuilt its orchestration layer around small language models instead of pushing everything through expensive large reasoning models.

‘I believe the future of agentic AI is many, many, many small language models,’ Markus said. ‘We find small language models to be just about as accurate, if not as accurate, as a large language model on a given domain area.’

The new system uses LangChain as its core framework. Large language model ‘super agents’ direct smaller ‘worker’ agents that perform specific tasks. This approach improved latency and response times while slashing costs.

AT&T deployed its Ask AT&T personal assistant to over 100,000 employees. The platform generates 2 billion tokens daily across applications ranging from automated call summaries to database queries in plain English.

The company now operates more than 410 generative AI agents in production. These agents handle customer service updates, network troubleshooting, and code debugging. Human oversight remains central to operations, with all agent actions logged and role-based access controls enforced.

AT&T’s AI investments target significant financial returns. The company expects AI and generative AI efforts to contribute substantially to its goal of $2 billion in run-rate cost savings by mid-2026. Existing AI applications already save millions annually, including $7 million from predictive maintenance and over $10 million from route optimization.

The platform serves 100,000 users across AT&T with 750 million API calls into production systems. Recent achievements include top rankings on industry AI benchmarks for text-to-SQL accuracy and Open-Telco LLM performance.

Markus contrasts AT&T’s success with broader industry struggles. ‘We love our story because it’s so counter to the MIT study that just came out that concluded that 95% of companies are getting zero return from their AI investment,’ he said.

The company follows three core principles for AI deployment: accuracy, cost, and tool responsiveness. Markus advises against over-engineering solutions, emphasizing that not every tool needs to be agentic.

AT&T’s latest innovation, Ask AT&T Workflows, provides a drag-and-drop interface for employees to build automation agents. The tool integrates with proprietary AT&T systems for document processing, natural language-to-SQL conversion, and image analysis.

Read more: 8 billion tokens a day forced AT&T to rethink AI orchestration — and cut costs by 90%