Last updated: November 4, 2026. This article is an independent explainer based on publicly reported announcements and company documentation. The proposal is still a draft and details may change, so confirm current syntax and dates in each company’s own documentation before editing your site. This guide is not affiliated with Google, Microsoft, OpenAI or Anthropic and is not legal advice.
On October 30, 2026, several major AI and search companies announced support for a single way to tell AI developers whether your content may be used to train their models. Instead of listing a different crawler name for every provider in your robots.txt file, a site could use one shared “policy token.” If you publish a blog, a news site or any business website, this is worth understanding now, even though most of it will not take effect until 2027.
The Short Answer
- What it is: a proposed robots.txt convention, from the IETF’s AI Preferences (AIPREF) working group, that lets a site state one AI training preference using the token
__usage_ai-training. - Who backs it: Google, Microsoft, OpenAI and Anthropic have announced related support, according to industry coverage. Meta, Amazon and Apple have made no such commitment, as reported.
- When: Google’s crawler documentation says it intends to support the signal in early 2027. Microsoft’s dates are listed below. OpenAI and Anthropic have given no firm dates.
- Does it affect search rankings? No. Google says Google-Extended does not affect inclusion in Google Search, and blocking Googlebot or Bingbot would be a separate and much bigger decision.
- Do you need to act today? Not urgently. You can decide on your policy now and prepare the line you would add.
How the Shared Token Works
Today, a publisher who wants to limit AI training has to know and list each provider’s control token, such as Google-Extended, and keep up as new ones appear. The AIPREF draft takes a different approach. You add a robots.txt group whose “user agent” is the policy token rather than a crawler name:
User-agent: __usage_ai-training Disallow: /
As described in reports on the draft:
Disallow: /expresses a preference against AI training.Allowlines can carve out exceptions for specific paths, and normal robots.txt matching rules apply.- A wildcard group (
User-agent: *) does not count as a shared training preference. A site that blocks all bots with a wildcard has not opted out of AI training under this scheme and needs the specific token. - The token covers use, not crawling. It does not stop a bot from fetching your pages.
- “Training” refers to modifying a generative model’s learned parameters. Retrieval at answer time, and non-generative systems such as ranking or classification, are outside the definition.
What Each Company Has Said
Google’s documentation describes Google-Extended as a standalone product token, not a separate crawler, that publishers can use to manage whether crawled content may be used for future Gemini model training and grounding features. It says Google-Extended does not affect a site’s inclusion in Google Search and is not a ranking signal. Google says it intends to support the AIPREF signal in early 2027. According to Search Engine Roundtable’s report, if a site uses the shared token to opt out of training and does not explicitly address Google-Extended, Google will treat it as if the site had disallowed Google-Extended. Company-specific controls keep working alongside the shared signal, and a site that explicitly allows Google-Extended can still appear in Gemini Apps responses and receive clicks through grounding.
Microsoft
Microsoft is adding separate controls for AI training and AI grounding, with page-level settings in addition to robots.txt, and says these do not change traditional Bing search eligibility. Reported dates:
| Date | What happens |
|---|---|
| Oct 30, 2026 | Announcements and draft submission |
| Dec 15, 2026 | Bing Webmaster Tools opens temporary grounding preferences for sites that use NoArchive or NoCache |
| Feb 1, 2027 | Microsoft begins honoring the new preferences as pages are crawled |
| Mar 1, 2027 | Microsoft stops reading NoArchive and NoCache as training or grounding signals |
If your site uses NoArchive or NoCache today to limit how Microsoft’s AI features use your pages, the March 1, 2027 change matters: those directives stop working as AI controls, so plan to migrate.
OpenAI and Anthropic
Both are reported to support the approach, and Anthropic’s Kevin Kelley is reported to have written the IETF draft. OpenAI’s wording refers to “foundational” training, which may be narrower than the draft’s definition, and neither company has announced a firm implementation date.
The Trade-Offs Publishers Should Weigh
- It is voluntary. The draft has no enforcement mechanism. It works only for companies that choose to honor it.
- It is a draft. The proposal is an individual submission that the IETF has not endorsed, and it expires on May 3, 2027 unless revised, as reported. Terms could change.
- Training and grounding are not the same at every company. At Google, per reports, the shared opt-out is treated like a Google-Extended disallow, which also limits grounding in Gemini Apps. Microsoft’s version treats training and grounding separately, and warns that restricting grounding can reduce brand visibility in its AI experiences.
- No guarantee either way. Opting in does not guarantee traffic or payment, and opting out does not remove content already collected.
- Licensing deals can override a preference.
For many ad-supported publishers, the real question is whether AI answers send readers back. Our breakdown of what four studies say AI Overviews did to clicks is a good starting point for that decision.
A Practical Checklist for Site Owners
- Open your robots.txt (yourdomain.com/robots.txt) and note what is there. Many WordPress sites have a generated file created by an SEO plugin.
- Look for existing AI rules such as Google-Extended, and for page-level NoArchive or NoCache usage, especially if you target Bing.
- Write down your policy in plain words: allow AI training, block it, or allow with exceptions (for example, block a premium section).
- Do not rely on a wildcard block. If you want to express an AI training preference, you will need the specific token once providers support it.
- Do not block Googlebot or Bingbot unless you intend to leave search results. That is different from AI training controls.
- Wait for the providers’ own documentation before adding the token, then test the file and confirm the change in each provider’s tools. Keep a dated note of the change.
- Review again in early 2027, when Google and Microsoft say they will start honoring the signals.
What About Other Search Changes?
This is a content-use control, not a ranking change. Industry trackers noted on November 2 that Google’s history still listed September’s spam update as its latest confirmed update. If you advertise on Google, see our guide to Google Ads automated promotions and how to opt out, and for campaigns adapting to AI features, read about the AI Max upgrade for search campaigns.
Frequently Asked Questions
Will opting out of AI training hurt my Google rankings?
Google states that Google-Extended does not affect a site’s inclusion in Google Search and is not a ranking signal. Blocking Googlebot is a different matter and would remove pages from Search.
Is the new robots.txt token live now?
No. Google’s documentation says support is planned for early 2027, and Microsoft’s preferences begin to be honored on February 1, 2027.
Does it stop AI companies from crawling my site?
No. The token states a preference about use for training. It does not block fetching.
Does my existing “User-agent: *” block count?
Not for this purpose. According to reports on the draft, a wildcard group does not express the shared training preference.
Should small blogs act now?
There is no deadline for most sites. Decide your policy, check your current robots.txt, and review again when providers publish final instructions.
Bottom Line
A single robots.txt line for AI training preferences would simplify life for publishers, but it is still a voluntary draft with provider-specific differences. Decide where you stand, audit your current file and watch the dates: December 15, 2026 and February 1, 2027 for Microsoft, and early 2027 for Google.
Sources: Google Search Central, “Google’s common crawlers” documentation (developers.google.com); Search Engine Roundtable, report on Google’s planned shared AI training control (October 30, 2026); PPC Land, report on the Google, Microsoft, OpenAI and Anthropic announcement (October 30, 2026); Digital Hitmen, SEO and digital marketing brief (November 2, 2026). Details come from the cited reporting and may change as the IETF draft evolves.
