🔍 Intelligence Synthesis JP/EN
Ethics · Regulation · Technology — Pharma Practice Notes

AI Tech Flash ── 2026-11-02

Four AI Giants Back One-Line Training Opt-Out

Google and three others back a single robots.txt entry to refuse AI training

What changes

Website operators may soon be able to refuse use of their content for AI training with one entry. Until now, they had to add separate entries for each AI provider and each crawler, SiliconANGLE reported.

Start dates differ by company, though. According to PPC Land, no company has yet said it reads the new entry.

Why it matters

According to SiliconANGLE, opting out of training means adding a separate entry for every AI provider. If a provider runs several crawlers, each one needs its own entry.

Google, Microsoft, OpenAI and Anthropic said they will support one shared method. According to PPC Land, all four gave reducing the burden on site operators as their reason.

What it is

The target is robots.txt, a text file on a website that tells crawlers (automated programs that visit pages) what content they may access.

Per PPC Land, the proposal is the Internet-Draft draft-kelley-aipref-training-preference-signal-00. A site names a label called a policy token, "__usage_ai-training". That signals a preference against use of its pages for generative AI training.

According to PPC Land, it is a draft submitted to the Internet Engineering Task Force (IETF), not an official standard.

Who and where

According to PPC Land, the draft was written by Kevin Kelley of Anthropic and sent to the mailing list of the IETF's AIPREF working group.

Google, Microsoft and OpenAI also announced support on October 30, in their own blog or policy posts. SiliconANGLE reported that Meta backed the idea but has not committed to implementing it.

Why it was built

According to SiliconANGLE, blocking training-related scraping means adding entries to robots.txt for each AI provider. A provider with several scrapers needs an entry for each one.

PPC Land reported that Google wants to end "managing separate instructions for every AI developer." OpenAI cited the burden of maintaining separate rules for every provider.

How it works

A site operator names the policy token where a crawler name would normally go. In the example PPC Land quotes, "User-Agent: __usage_ai-training" is followed by "Disallow: /" and "Allow: /press-kit/". That expresses a preference against training on every path except /press-kit/.

Per PPC Land, the draft adds no new directives or grammar. It reuses the path matching and longest-match rule of RFC 9309, the existing robots.txt specification.

The draft states that the wildcard "User-Agent: *" does not match the token. According to SiliconANGLE, the setting does not apply to models that perform only non-generative tasks.

Key points

  1. Google, Microsoft, OpenAI and Anthropic back one robots.txt entry to refuse generative AI training.
  2. According to PPC Land, the proposal is still a draft. Microsoft says it starts February 1, 2027, and Google plans early 2027.
  3. According to PPC Land, "User-Agent: *" does not match the token, so sites must name it explicitly to opt out.

Conclusion

The four companies said they will read the same signal. That could cut the work for site operators.

But it is still a draft, and PPC Land says no company has yet announced that it reads it. Microsoft gives February 1, 2027, and Google early 2027. OpenAI and Anthropic have given no dates.

Quotes and sources

“It would allow website operators to block all scrapers that collect training data for generative AI models with a single robots.txt entry.”
── SiliconANGLE, 2026-10-31
“It introduces no new directive names and no new grammar: the path language, longest-match precedence, and group structure of robots.txt are not changed”
── PPC Land, 2026-10-31
“wildcard groups (i.e., User-Agent: *) MUST NOT be interpreted as matching the policy token”
── PPC Land, 2026-10-31

Source articles

Written by AI (Claude) from the cited articles, then checked by Jev (TypeSafe's judgement model), a Gemini fact-check and a verbatim check of every quote.