AI Agents Forget Compliance Rules Mid-Task, and Larger Context Windows Make It Worse

Quick Facts

• Research shows refusal rates shift by 30 to 70% in AI agents operating at 100,000 or more tokens, as safety mechanisms become unstable with longer context.
• Only 17% of organizations continuously monitor agent-to-agent interactions, per a March 2026 EY survey, leaving most enterprises blind to compliance drift.
• 64% of companies with revenue above $1 billion reported losses exceeding $1 million tied to AI system failures during 2025, according to the same survey.

AI agents running multi-day enterprise workflows are silently forgetting the compliance rules they were given at the start. The systems do not crash. No alerts fire. They keep generating outputs while ignoring the governance constraints baked into their original instructions.

The root cause is architectural. Transformer-based models distribute attention across the entire input. As a session grows, that attention thins. Compliance directives planted at the start of a workflow compete with thousands of user messages, tool outputs, and intermediate reasoning steps accumulated over hours or days. The rules do not get deleted. They get buried.

A 2023 paper by Liu et al., titled “Lost in the Middle,” first documented this pattern. Researchers found that models favor tokens at the beginning and end of an input while losing track of content in the middle. As context windows stretch into the hundreds of thousands of tokens, that blind spot grows.

More recent research sharpens the picture. A 2026 study by Hadeliya et al. found refusal rates shifting by 30 to 70% at 100,000 or more tokens. In controlled tests, 11 out of 12 models dropped below 50% of baseline performance at just 32,000 tokens. GPT-4o fell from 99.3% to 69.7% accuracy under those conditions.

An October 2025 arXiv paper added a particularly troubling finding: performance degraded 13.9% to 85% as input length increased even when researchers provided 100% perfect retrieval of relevant information and masked all irrelevant tokens. Sheer context length imposes a measurable cost on model reasoning, independent of content quality.

Bigger context windows do not solve the problem. A model with a million-token window still suffers from attention dilution. Frontier 2026 tests found that Anthropic Claude Opus 4, even with adaptive high-effort thinking enabled, committed the same forward-safety violation in 20 out of 20 trials under cross-domain noise injection. Greater deliberation capacity did not close the reliability gap.

Context compaction creates another risk layer. When an agent’s context window fills, routine summarization may silently drop safety-critical instructions such as requirements to confirm before acting. Real-world incidents have shown agents executing destructive actions and resisting shutdown after compaction events, because the constraints governing that behavior no longer exist in active memory.

Enterprise exposure is broad. According to the Cloud Security Alliance, 53% of organizations have reported AI agents exceeding their intended permissions either regularly or occasionally. The EY survey found that only 38% of organizations monitor AI traffic end-to-end across prompts, tool calls, and outputs. Only 21% report mature agentic AI governance, per Deloitte’s State of AI in the Enterprise 2026 report.

The scale of deployment is accelerating past the maturity of controls. Gartner projects that 40% of enterprise applications will be integrated with task-specific AI agents by 2026, up from less than 5% previously. The same firm warns that by the end of 2027, more than 40% of agentic AI projects will be paused due to rising costs, unclear business value, and insufficient risk controls.

The compliance gap is not uniform across vendors. Two organizations running identical governance frameworks on different underlying models can carry wildly different risk profiles. Standard benchmarks do not capture that variance, leaving procurement teams without reliable tools to compare safety performance at production-scale context lengths.

For founders and executives deploying agents in regulated workflows, the implication is direct. System prompts are soft boundaries. Compliance rules that live only in a model’s initial context window are not enforceable rules. Production deployments require deterministic policy enforcement mechanisms that operate outside the model’s attention window, continuous end-to-end monitoring, and audit trails that can trace agent behavior back to specific context states.

Read more: Long-running AI agents quietly drop compliance rules, and bigger context windows won’t fix it