AI training opt-out in robots.txt: __usage_ai-training token backed by Google, Microsoft, OpenAI

On 30 October 2026 Google, Microsoft and OpenAI said they will honour a single robots.txt signal, the policy token __usage_ai-training, for publishers who do not want their pages used to train AI models, and The Verge reports that Anthropic is on board too. The token comes from an Internet-Draft by Kevin Kelley of Anthropic and is not yet an official standard. Nothing is switched on today: Microsoft starts on 1 February 2027, Google says early 2027, and OpenAI says "coming months". Site owners can add the group to robots.txt now, though, and it costs nothing.

Key points

• The token is __usage_ai-training, written on a User-Agent line in an ordinary robots.txt group; it adds no new directives, according to the draft draft-kelley-aipref-training-preference-signal-00, dated 30 October 2026.
• A User-Agent: * group does not match the token, so a blanket wildcard rule says nothing about AI training, and a file without a token group expresses no preference, per the draft.
• Google says it intends to support the signal in early 2027, and plans to treat an opt-out as if the site had disallowed Google-Extended unless Google-Extended is set explicitly; Microsoft begins honouring it on 1 February 2027 (Search Engine Roundtable); OpenAI gives no date beyond "coming months" (The Verge).
• The token is not a crawling directive: to stop a crawler fetching pages you still name that crawler in robots.txt.
• The draft expires on 3 May 2027 unless it is replaced, and the IETF itself warns that drafts should only be cited as "work in progress".

What to do

If you run a website and do not want your content used to train generative AI models:

1. Add a group to the robots.txt at the root of your domain. The draft's own example opts out everywhere except a press kit: User-Agent: __usage_ai-training, then Disallow: /, then Allow: /press-kit/.
2. Do not rely on User-Agent: *. The draft says wildcard groups "MUST NOT be interpreted as matching the policy token".
3. Keep your existing crawler rules (GPTBot, Google-Extended and so on). Google says company-specific controls "will continue to function alongside" the new signal, and the draft lets providers treat their own tokens as more specific than the shared one.
4. If you want to block fetching rather than training, name the crawler in its own group; the token does not do that.
5. If you want your pages to appear in Gemini's responses, note Google's statement that sites which explicitly allow Google-Extended are eligible to appear in grounded answers in Gemini Apps.
6. To check whether you are affected: open https://yourdomain/robots.txt. If there is no __usage_ai-training line, no preference has been expressed under this scheme, whatever else the file says.

What happened

The IETF's AI Preferences (aipref) working group has spent more than a year working on a vocabulary for how publishers can state what AI systems may do with their content. On 30 October 2026 Kelley, who works at Anthropic, posted an individual Internet-Draft, draft-kelley-aipref-training-preference-signal-00, that takes one piece of that work, the definition of "AI training", and binds it to robots.txt using a "policy token".

Within hours, the major AI and search companies said they would adopt it voluntarily. Google wrote on its Search Central blog that the change "will help websites express their AI training preferences across multiple AI providers in a single step", and added a note to its crawler documentation that it intends to support user-agent: __usage_ai-training in early 2027. Microsoft announced it was joining the shared signal and going further (see below). The Verge reported that OpenAI, Google, Microsoft and Anthropic said they will adopt the standard and restrict their AI crawlers from sites that use the token, with Anthropic having already adopted it, Google and Microsoft planning to support it next year, and OpenAI implementing it in the "coming months". Sources differ slightly on Anthropic: Search Engine Roundtable also lists it among the adopters, but we could not find an Anthropic announcement of its own to confirm timing, and one other write-up we read says it found none. Meta's communications head Andy Stone said the company supports a common signal but, in The Verge's words, did not say whether Meta would honour it.

Background

robots.txt dates from the 1990s and was standardised as RFC 9309. It tells crawlers which paths they may fetch, and was built for search indexing, not for saying how fetched content may be used. As AI crawlers multiplied, each company published its own token: Google-Extended for Google's AI training, separate crawler names for others. A publisher who wanted a blanket opt-out had to track and list each one. The Internet Architecture Board's AI-CONTROL workshop in September 2024 and the IETF's aipref working group grew out of that problem.

Other approaches exist alongside it. The Verge points to the Really Simple Licensing (RSL) standard, which lets publishers charge crawlers, and to Cloudflare blocking AI crawlers by default. We have also covered how fast the rules are shifting on the commercial side, for example when Meta removed an AI-training sentence from its new business terms after a scraping backlash.

Google also tied the move to regulation: its post says the proposal is consistent with the EU Code of Practice for General Purpose AI Model Providers under the EU AI Act, which recognises robots.txt as a primary method for publishers to express machine-readable preferences.

How it works

The draft deliberately changes as little as possible. Section 4 sets the rules:

• Matching. Consumers match the token exactly (apart from case). The __ prefix is meant to make accidental matches unlikely. One group can name several policy tokens on consecutive User-Agent lines.
• Reading a preference. The consumer looks up the path for the token as RFC 9309 specifies. If the path is disallowed, a preference against AI training has been expressed for it; if allowed, a preference in favour. If no group contains the token, "no preference has been expressed".
• Wildcards excluded. User-Agent: * groups never match, which avoids a blanket search-engine rule being misread as an AI training opt-out (and vice versa).
• Independent of crawler rules. An AI crawler still checks for rules under its own name exactly as before. The draft says "this policy token is not a directive for crawling behavior", though a consumer that crawls only for training may choose not to fetch disallowed pages.
• Precedence. Specific expressions should win over general ones, and providers may document that their own proprietary tokens are more specific. Private agreements may override the preference.

Parsers that do not know the token simply ignore the group, as they would a rule for any unknown bot, so adding it should not change how search engines read the rest of the file.

"AI training" is defined in the draft by quoting the working group's vocabulary: using an asset to modify the learned parameters of a generative AI model. Training models used only for non-generative tasks, such as classification or ranking, is not included.

What it means

For publishers, the immediate benefit is one line instead of a list, but only once providers actually honour it, and the dates are 2027. Until then, existing per-company controls remain the only enforced mechanism.

For AI companies, adoption is voluntary and, in the draft's words, preferences are not enforcement: nothing in the text stops a non-participating crawler. Google's comment that "other AI providers can handle it similarly" points to the real test, which is whether smaller labs and Meta follow.

For site owners who want visibility in AI answers, Microsoft's version separates the two questions. According to Search Engine Roundtable's summary of Microsoft's announcement, a publisher can block training while still allowing Copilot to cite its pages, and can block use of its content for AI answers separately, with traditional Bing rankings unaffected. Microsoft's dates: 15 December 2026, temporary grounding controls in Bing Webmaster Tools for sites using the older NoArchive or NoCache directives; 1 February 2027, Microsoft begins honouring the new preferences as it crawls; 1 March 2027, deadline for publishers on the old directives to configure the new ones to keep their restrictions; 30 September 2027, the temporary grounding preferences expire.

What we don't know yet

• Whether Anthropic and OpenAI will name exact dates, and how each will treat sites that set both the shared token and its own crawler rules.
• Whether Meta, Apple, xAI, Mistral and the Chinese labs will honour the token. Only Meta has commented, and only in support of a common signal.
• Whether the working group will adopt the draft as written. It is an individual submission, expires 3 May 2027, and is marked as work in progress.
• How the signal will interact with legal obligations and with licensing deals, which the draft says may override it.

What to watch next

• Microsoft's 15 December 2026 and 1 February 2027 milestones, and Google's "early 2027" support.
• OpenAI's implementation date and crawler documentation updates.
• Revisions of the draft and the working group's discussion on the ai-control mailing list.
• Whether robots.txt checkers and CMS plugins add the token.

Sources

• IETF Internet-Draft: Machine-Readable Training Preference Signal (draft-kelley-aipref-training-preference-signal-00)
• Google Search Central Blog: Supporting a shared AI training control for websites
• Search Engine Roundtable: Google Adopts New AI Training Control
• Search Engine Roundtable: Microsoft Bing, New AI Controls With AI Training From AI Search Grounding
• The Verge: The biggest names in AI are backing a new way to opt out of scraping
• OpenAI: Making it easier for websites to express AI training preferences (page needs JavaScript; we did not read it directly)