Google, OpenAI and Microsoft Back One Rule to Block AI Training

Website owners will soon be able to tell several AI companies “don’t train on my site” with one rule instead of one rule per company. In a post on the Google Search Central blog on October 30, Google said it plans to support a shared AI training control in robots.txt, the small text file that tells web crawlers what they can and can’t do on a site. OpenAI, Microsoft and Anthropic backed the same signal on the same day.

1. What problem this solves

The old way: today, a site that doesn’t want its pages used to train AI has to find each AI company’s crawler name and block it separately in robots.txt. Every new AI company means another rule, and it is easy to miss one.

The new way: the shared signal uses one special name, called a policy token, in robots.txt. Instead of naming a crawler, it names the purpose: AI training. Any AI company that honors the token reads the same rule.

2. How the rule works

The token: the proposal, written by Anthropic’s Kevin Kelley and submitted to the IETF as part of its AI preferences work, names the token __usage_ai-training. A site adds a normal robots.txt group with that name, then uses Disallow and Allow lines to choose which parts of the site are off limits for training.

Watch the wildcard: a catch-all “User-Agent: *” rule does not count. The draft says only a group that names the token itself expresses an AI training preference.

3. Who supports it, and when

Search is not affected: Microsoft says sites will still show up in Bing search whatever they choose, and Google’s existing AI controls already work without changing how a site appears in Google Search.

4. What it means for Google and Gemini

Google-Extended: since 2023, Google has offered a robots.txt control called Google-Extended that lets sites keep their content out of training for Google’s AI models such as Gemini. Google says the new shared opt-out will act like a Google-Extended block, unless the site has its own Google-Extended rule. A site-specific rule wins, so owners can opt out everywhere but still allow Google, or the other way round.

A possible trade-off: Google’s post also notes that sites which specifically allow Google-Extended are eligible to appear in answers in Gemini Apps and get clicks back through grounding (when Gemini pulls in and links to live web pages). So a blanket opt-out could mean fewer chances to show up in Gemini answers. Site owners will want to weigh that before flipping the switch.

Our take

This won’t change anything on your Chromebook or in Chrome. It matters if you run a blog, a school site or a small business website, because it turns a messy list of crawler rules into one line. The bigger point is who signed on: the companies behind Gemini, ChatGPT, Copilot and Claude agreeing on one switch is rare. The catch is timing, since Google is only aiming to support it in 2027, so if you already block Google-Extended, keep that rule in place for now. With Google shipping new models like Gemini 4 Carbon at a fast pace, simple controls like this are overdue.