Google, Microsoft, Anthropic and OpenAI back a single robots.txt rule to block AI training crawlers

Google, Microsoft, Anthropic and OpenAI said on October 30 that they will support an upcoming website crawling standard that lets site operators block AI developers from using their content to train models. According to SiliconANGLE's report on the announcement, the four companies have already submitted a draft of the proposal to the Internet Engineering Task Force, the nonprofit that maintains the Robots Exclusion Protocol.

The mechanism builds on robots.txt, the text file websites have used for decades to tell crawlers what they may access, what they may not, and how often they may visit. That same file now carries a second job: governing the scrapers AI providers use to assemble training data.

The difficulty is granularity. A publisher that wants its pages kept out of training sets currently has to write a separate entry for every AI provider, and a separate entry for every scraper that provider operates. As new crawlers appear, that list grows longer and easier to get wrong.

The proposed standard collapses the work into a single declaration. One robots.txt entry would block all crawlers collecting training data for generative AI models, without disturbing other access rules. The draft also draws a line: the setting would not apply to models that perform exclusively non-generative tasks.

Microsoft said it will pair the standard with an additional crawl control called AI Grounding, aimed at a different kind of request, where a model fetches a page to answer a user's question rather than to build a training corpus. That control would extend to third-party AI tools that use Microsoft technology to process prompts. OpenAI said it plans similar controls of its own.

Timelines differ across the group. Microsoft and OpenAI intend to add support in the coming months, while Google will follow sometime next year. Meta said it supports the idea behind the technology but has not committed to implementing it.

The significance is that training and retrieval are being separated into distinct switches, giving publishers a clearer way to state what they will and will not allow than the current patchwork of per-provider entries. But robots.txt remains a voluntary convention, so the standard's practical weight depends on whether crawlers comply and whether violations carry consequences. The IETF review, Meta's next move, and how the non-generative carve-out is drawn in practice are the things to watch.