Google, Microsoft, OpenAI and Anthropic Back Single Robots.txt Token for AI Training Opt-Out

Quick Facts

• Google, Microsoft, OpenAI and Anthropic announced on October 30, 2026, that they will honor a shared robots.txt token called __usage_ai-training that lets publishers opt out of AI training data collection with a single entry.
• Implementation timelines vary: Anthropic has already begun, OpenAI plans to comply within months, Microsoft sets February 1, 2027, and Google targets sometime in 2027.
• The token controls training use only, not crawling itself, and carries no enforcement mechanism beyond voluntary compliance.

Google, Microsoft, OpenAI and Anthropic announced plans to honor a shared website crawling standard that gives publishers a single opt-out signal for AI training data collection. The announcement came October 30, 2026.

Today, a publisher that wants to block AI training scrapers must add a separate robots.txt entry for every provider and every bot that provider runs. The new standard replaces that process with one line.

How It Works

The draft standard, an individual IETF Internet-Draft titled "Machine-Readable Training Preference Signal," was authored by Anthropic's Kevin Kelley and dated October 30, 2026. The document defines a cross-vendor policy token: __usage_ai-training.

A publisher adds User-Agent: __usage_ai-training paired with Disallow: / to block all participating AI companies from using site content for generative model training. Specific paths can still be permitted. The draft's example pairs a blanket disallow with Allow: /press-kit/, letting AI companies use press materials while blocking the rest.

One detail matters for publishers: the token controls training, not fetching. Bots may still retrieve pages. The standard covers training data collection only, leaving search indexing and real-time retrieval under separate rules.

Rollout Timelines

Each company is moving at its own pace. Anthropic says support is already live. OpenAI says it will comply within the coming months. Microsoft has set a hard date of February 1, 2027. Google plans to follow sometime in 2027.

Microsoft added a second control called AI Grounding, designed for situations where a model accesses a page to answer a user question rather than to collect training data. That setting will also apply to third-party AI tools built on Microsoft technology. Publishers currently using NoArchive or NoCache to signal training preferences to Microsoft must switch to the new token by March 1, 2027, after which Microsoft will stop reading the older tags as training signals.

Google said it will treat an opt-out expressed through the shared token as equivalent to a direct disallow of Google-Extended, unless a site addresses Google-Extended separately. Existing provider-specific tokens, including GPTBot, Google-Extended and ClaudeBot, remain in force alongside the new standard.

Meta expressed support for the concept but has not committed to implementing it.

Standards Process and Limits

The IETF notes that an individual Internet-Draft has no formal standing in the IETF standards process and is not endorsed by the organization. The AIPREF working group, chaired by Mark Nottingham of Cloudflare and Suresh Krishnan of Cisco, is overseeing the proposal. The draft expires May 3, 2027.

No enforcement mechanism accompanies the announcement. The four companies are making voluntary commitments. There is no penalty for violations and no independent body to verify compliance.

The draft also allows private agreements between a site and a provider to override the expressed preference, which limits the signal's reliability for publishers without direct commercial relationships with AI companies.

A History of Non-Compliance

Voluntary standards have a poor track record in this space. A Tollbit report found AI bot scrapes bypassing robots.txt grew more than 40% between Q3 and Q4 of 2024. Publisher Ziff Davis alleged in a lawsuit that GPTBot continued scraping its sites even after the company implemented OpenAI's stated opt-out instructions.

Several major publishers, including The New York Times, have active lawsuits against OpenAI over training data use. The new standard does not address those disputes and does not resolve the underlying question of whether past scraping violated copyright law.

What Publishers Should Do Now

Adding the token is a one-line change to an existing robots.txt file. Publishers that want to opt out can add it today, but enforcement will be uneven through most of 2027 as each provider rolls out support on its own schedule. Existing provider-specific disallow entries remain the more reliable near-term tool until all four companies have completed their implementations.