The tradeoff that forced a binary choice
For years, a site owner who wanted out of AI training had one blunt instrument: block the crawler. But the biggest crawlers — Applebot, Bingbot, Googlebot — do search and training in the same pass. Refuse one, and you refuse the other. Cloudflare calls these mixed-use crawlers, and they were the hard part of the training question.
On September 15, 2026, Cloudflare announced a Disallow AI Training setting that splits the two. The mechanism is a robots.txt preference published on your behalf, combined with network-level identification of who is crawling and why. Accountable mixed-use crawlers stay allowed for search; every other training crawler is blocked. Apple, Google, and Microsoft honor or have committed to honor the setting.
What actually changed on September 15
Block and “Block on pages with ads” now apply to mixed-use crawlers. That is a real behavior change: if you select Block, you stop Applebot, Bingbot, and Googlebot entirely, search included. If you want training stopped but search kept, Disallow AI Training is the setting to use.
“Block AI Bots” is deprecated in favor of separate Search, Training, and Agent controls. Managed Robots.txt is deprecated in favor of Bot Preference Sync, and existing customers migrate automatically. For most accounts, Cloudflare says there is nothing to do.
One caveat worth reading closely: Bingbot does not yet respect a no-training preference in robots.txt. Microsoft is building that mechanism, targeted for early 2027. Until then, Disallow AI Training does not automatically convey the preference to Bing — the same practical behavior as the old Training Block setting. For Bing today, the supplied material points to the NOARCHIVE meta tag or the Block URLs and Content Removal tools.
Why robots.txt alone was never enough
Anyone can publish a robots.txt directive. It cannot identify who is crawling, determine why, or stop a crawler that ignores it. Cloudflare’s argument is that a network can: publish the preference, classify the crawler’s behavior, block the ones that ignore it, and report what each operator does on Radar.
That distinction matters when you are choosing where to enforce policy. If your only lever is a text file, you are trusting compliance. If your lever sits in the request path, you are enforcing it. The same split shows up in agent-facing retrieval work, where the question is whether a crawler is fetching on behalf of a human or harvesting at scale — see this earlier post on choosing a web search API for agents for how that retrieval task shapes the tooling decision.
The numbers behind the granularity
Cloudflare reports that less than 1% of its sites choose to block Search bots, while 17% enable some mechanism to block training. That gap is the whole design rationale: site owners broadly want to be found, and a meaningful minority want training refused. A one-size-fits-all “Block AI” cannot express both preferences at once.
For new domains, Cloudflare now offers two presets depending on whether the site earns money from advertising. Ad-supported sites get the more restrictive configuration, on the reasoning that ad revenue depends on a human actually seeing the page, while training replaces that visit with an answer. Both presets are editable during onboarding or later.
What is still unresolved
AI summaries are the next target. Cloudflare argues a site-wide yes or no is too blunt there, because how much of your content appears in a summary matters as much as whether it appears. An opt-out for AI summaries is already a stated requirement for mixed-use crawler operators, and the goal is to let site owners control how much content is included from one place by early next year.
Agents have no Disallow setting yet. Cloudflare notes that agents do not create the same search-discoverability tradeoff, and no well-established directive exists for expressing preferences to them. That is a gap to watch if your product depends on user-directed agents reaching your pages.
The practical takeaway: audit your current setting before assuming it means what it used to. Block now stops search too, and Disallow AI Training is the only option that keeps you indexed while refusing training — with Bing as a known exception until Microsoft ships its robots.txt support.
Sources
AI-assisted summary compiled from the sources above, reviewed by a human before publishing.