Summary: Cloudflare announced Disallow AI Training on September 15, 2026 to separate search indexing from AI training for mixed-use crawlers. Apple, Google, and Microsoft honor the setting or have committed to honor it, while Bing still lacks robots.txt support targeted for early 2027. The article also reports that fewer than 1% of sites block Search bots, but 17% use some training-block mechanism.
Two uses, one crawler: why the choice was binary
For years, getting a site out of AI training left owners with a single blunt move: block the crawler. The trouble is that the largest crawlers — Applebot, Bingbot, Googlebot — handle search and training in one pass. Refuse one and the other goes with it. Cloudflare calls these mixed-use crawlers, and they were the hard part of the training question.
Cloudflare’s answer, announced on September 15, 2026, is a Disallow AI Training setting that pulls the two apart. It works by publishing a robots.txt preference on your behalf, paired with network-level identification of who is crawling and why. Accountable mixed-use crawlers remain allowed for search; every other training crawler gets blocked. Apple, Google, and Microsoft honor the setting, or have committed to honor it.
What the September 15 change actually does
Block and “Block on pages with ads” now cover mixed-use crawlers. That is a genuine behavior change: pick Block and you halt Applebot, Bingbot, and Googlebot completely, search included. To stop training while keeping search, Disallow AI Training is the setting to reach for.
“Block AI Bots” is deprecated, replaced by the separate Search, Training, and Agent controls. Managed Robots.txt gives way to Bot Preference Sync, and existing customers migrate on their own. Cloudflare says most accounts have nothing to do.
One caveat deserves a close read: Bingbot does not yet honor a no-training preference in robots.txt. Microsoft is building that mechanism, aimed at early 2027. Until it ships, Disallow AI Training does not carry the preference to Bing automatically — practically the same outcome as the older Training Block setting. For Bing today, the supplied material points to the NOARCHIVE meta tag or the Block URLs and Content Removal tools.
Why robots.txt alone was never enough
A robots.txt directive can be published by anyone. It cannot tell who is crawling, work out why, or halt a crawler that ignores it. Cloudflare’s case is that a network can: publish the preference, classify the crawler’s behavior, block the ones that ignore it, and report each operator’s conduct on Radar.
That distinction matters when you decide where to enforce policy. With a text file as your only lever, you are trusting compliance. With a lever in the request path, you are enforcing it. The same divide appears in agent-facing retrieval, where the question is whether a crawler fetches on behalf of a human or harvests at scale — see this earlier post on choosing a web search API for agents for how that retrieval task shapes the tooling decision.
The numbers behind the granularity
Cloudflare reports that under 1% of its sites choose to block Search bots, while 17% switch on some mechanism to block training. That gap is the design rationale in full: site owners broadly want to be found, and a meaningful minority want training refused. A one-size-fits-all “Block AI” cannot express both preferences at once.
For new domains, Cloudflare now offers two presets keyed to whether the site earns money from advertising. Ad-supported sites get the more restrictive configuration, on the reasoning that ad revenue depends on a human actually seeing the page, whereas training replaces that visit with an answer. Either preset can be edited during onboarding or at any later point.
What is still unresolved
AI summaries are the next target. Cloudflare argues that a site-wide yes or no is too blunt there, since how much of your content appears in a summary matters as much as whether it appears. An opt-out for AI summaries is already a stated requirement for mixed-use crawler operators, and the goal is to let site owners control how much content is included from one place by early next year.
Agents still have no Disallow setting. Cloudflare notes that agents do not create the same search-discoverability tradeoff, and no well-established directive exists for expressing preferences to them. That is a gap to watch if your product depends on user-directed agents reaching your pages.
The practical takeaway: audit your current setting before assuming it means what it used to. Block now stops search as well, and Disallow AI Training is the only option that keeps you indexed while refusing training — with Bing as a known exception until Microsoft ships its robots.txt support.