BMShop Products
← All articles
Four AI giants agree on a shared robots.txt signal to block AI training
AI News

Four AI giants agree on a shared robots.txt signal to block AI training

A single line in robots.txt can now tell OpenAI, Google, Microsoft and Anthropic that a site's content must not be used for AI training.

Ammar Rahimi · 1 November 2026 · 4 min read

On 30 October 2026, the four largest AI companies announced they will honor a shared signal in robots.txt. Website owners can use it to keep their content out of AI model training with one rule, instead of writing separate rules for each company. The signal comes from an IETF draft submitted by Kevin Kelley of Anthropic.

1. What was announced

OpenAI, Google, Microsoft and Anthropic all say they will honor a shared policy token named __usage_ai-training in robots.txt. That file has long existed and is used by websites to tell crawlers what they may collect.

The proposal is still an early draft at the IETF. The IETF datatracker states that an individual draft like this is not endorsed by the IETF and has no formal standing in the standards process. The draft is dated 30 October 2026 and expires on 3 May 2027.

2. How it works

A site owner adds one group to robots.txt with the line User-Agent: __usage_ai-training followed by Disallow: /. They can also allow specific folders with an Allow line, for example a press kit folder.

The draft introduces no new directives and no new grammar. Existing robots.txt parsers will simply ignore the group they do not recognize, so nothing breaks for parsers that have not implemented the signal yet.

3. When each company starts

The rollout schedule differs by company:

  • Anthropic: implementation work has already started.
  • Google: support is targeted for 2027, according to the Google Search Central blog.
  • Microsoft: will honor the new preferences from 1 February 2027, replacing the NoArchive and NoCache tags it has used since 2023.
  • OpenAI: aims to implement the signal "in the coming months", with no firm date.

Google added its own detail. If a site uses the token but does not explicitly mention Google-Extended, Google plans to treat it as if the publisher had disallowed Google-Extended. Sites that specifically allow Google-Extended remain eligible to appear in Gemini Apps answers and receive clicks.

4. What it does not do

The signal only restricts AI training, not page fetching. Bots can still pull the page, the draft says. A site's position in search results such as Bing and Google is unaffected.

It also does not pull back content that was already collected. Data already inside training sets stays there. OpenAI itself calls the mechanism a coordination tool rather than a security control, so scrapers that ignore the rules can disregard it.

5. Where other players stand

Meta voiced support for the shared signal. Andy Stone, Meta's head of communications, said a common signal that conveys preferences to multiple providers at once is an important step. He did not say whether Meta will honor it.

Other companies are free to ignore the token. It is voluntary, and a private agreement between a site and an AI company can override it, according to the draft and reporting from The Next Web.

What it means for you

If you run a website or blog built on your own content, this turns a decision that used to require a long list of per-crawler rules into one clear step.

  1. One line for everyone: Add the __usage_ai-training group to your robots.txt and your choice reaches the four major providers at once.
  2. Keep your old rules for now: Since Google targets 2027 and OpenAI has no date yet, do not remove per-crawler rules such as GPTBot and CCBot yet.
  3. Old content is not protected: The signal only limits future collection. Data already used in training cannot be pulled back.
  4. Take the middle path: Allow lines let you block AI training while keeping key pages open to readers and search, such as a contact page.
  5. It is not a guarantee: The token is voluntary. If strong protection matters to you, combine it with server-side controls such as crawler blocking at your CDN.

Sources