PayPal Doubled AI Accuracy to 95% by Building Around the Model, Not Changing It

Quick Facts

• PayPal's internal AI assistant went from 43% to 95% accuracy by adding golden queries and analytical playbooks, not by changing the LLM.
• The company's Cosmos LLM gateway now handles 200 billion tokens per day across more than 100 models and serves over 12,000 developers.
• PayPal has built multi-agent workflows of up to 18 agents to handle security reviews across more than 33,000 services.

PayPal's internal data science assistant nearly doubled its accuracy rate without a single change to its underlying language model, according to remarks made by company executives at VentureBeat's AI Impact Series in October 2025. The result is a case study in what enterprise AI teams can achieve by rebuilding the systems around a model rather than the model itself.

Ren Zhang, PayPal's VP and global head of AI product, said the assistant already had access to vetted, high-quality data before the accuracy jump. The missing piece was context: the definitions, query patterns, and analytical procedures PayPal's employees use to turn raw data into business answers.

The Knowledge Layer

PayPal's team solved the problem by building what Zhang calls a knowledge layer, kept separate from the LLM. The layer has two parts. The first is a "golden query," a pre-built specification for how to retrieve data for a given purpose. The second is an analytical playbook that defines how to investigate a question, including which segments, products, or behaviors to examine.

"The key thing is, we believe our IP is truly to build that knowledge layer," Zhang said. "The LLM is still the same."

By storing that IP outside the model, PayPal can swap to a new LLM release without rebuilding its institutional knowledge. Zhang said she deliberately designed the assistant to be model-agnostic, with the guiding principle: "I should not change my knowledge."

That architecture also changed how the team thinks about fine-tuning. Fine-tuning requires specialized expertise, and each new model release creates pressure to repeat the process. PayPal reduced its reliance on it without abandoning it entirely. For a separate customer-facing commerce agent, the team tested four or five models and found that larger models introduced roughly 15 seconds of latency. Switching to an 8-billion-parameter model cut that lag but hurt accuracy. Zhang said fine-tuning the smaller model on commerce tasks improved accuracy but still fell short of targets.

Productivity and Scale

The internal assistant, designed to answer questions that would otherwise go to PayPal's data science team, logged more than 1,000 questions and saved an estimated 1,600 hours over 10 weeks. Zhang cautioned that question volume alone is not a reliable measure of productivity gains.

The assistant runs on top of a much larger infrastructure stack. PayPal's Cosmos LLM Gateway, built on an Envoy-based proxy layer, processes roughly 200 billion tokens per day across more than 100 models. It operates at 99.999% availability, adds a median latency overhead of 4 milliseconds, and supports more than 12,000 developers. Prompt-cache hit rates reach as high as 96% on the best workloads.

Multi-Agent Security Workflows

PayPal is extending its agentic infrastructure into security operations. Deepika Khera, PayPal's senior director of AI, said at VentureBeat's Oct. 28 Agentic Infrastructure Stack Summit that the company built a seven-agent workflow to investigate alerts across more than 33,000 services. A separate security review workflow uses as many as 18 agents.

Khera noted that memory access cannot always be left to an agent's discretion. In a software development workflow, the team required memory to be consulted at a specific step rather than letting the agent decide. For other tool interactions, PayPal uses Model Context Protocol when agents need to reason about tool selection.

Srini Venkatesan of PayPal summed up the governance posture: "We still have humans in the loop," he said, "and in a system moving trillions, that is critical."

Governance and Payments Scale

PayPal processes roughly $1.8 trillion in payments annually. The company's 2025 annual report describes a new Enterprise AI Governance Framework adopted that year, formalized through an Enterprise AI Policy and Standard and an AI Governance Charter that assigns executive leadership, operational oversight, and policy and risk functions to dedicated governance bodies.

PayPal is also building toward agentic commerce using open standards including Model Context Protocol and Google's Agent Payments Protocol to keep agent-led transactions secure and auditable. The company has enabled more than 10,000 engineers to build with agentic AI tools.