Summary: Cloudflare announced Bot Preference Sync on August 21, 2026 to write dashboard AI bot preferences into robots.txt automatically. The feature is available from Free to Enterprise and covers Search, Agent, and Training categories, with publisher defaults that disallow training. Mixed-use crawlers must provide transparency before receiving the benefit of remaining searchable when training is disallowed.
What Changed: Keeping Policy and Enforcement in Step
Cloudflare announced Bot Preference Sync on August 21, 2026. The feature writes the AI bot preferences you set in the Cloudflare dashboard straight into your robots.txt file, keeping it current on its own. It is available to all customers from Free to Enterprise, and the pain point it addresses is a familiar one: when your robots.txt says one thing while your enforcement rules do another, some crawlers may ignore your stated preferences or try to bypass your rules. Holding the two in sync removes that inconsistency.
Managing a static robots.txt separately from your edge enforcement rules used to be the norm. Now Cloudflare generates or updates your robots.txt from your configuration, so what you publish is what you enforce. You can switch the feature on or off at any time, and for new customers it is on by default.
How It Works: Category-Wide Policies
Bot Preference Sync builds on the AI traffic controls Cloudflare launched on July 1, 2026. Those controls cover three categories of AI traffic—Search, Agent, and Training—each with its own preference. For Search and Agent you choose among Allow, Block on pages that serve ads, or Block everywhere. For Training, Disallow is also available, and it writes a “no training” preference to your robots.txt.
Enabling sync makes Cloudflare prepend the generated directives to your existing robots.txt, so manual Disallow rules you already have survive untouched. Which bots get included follows Cloudflare’s BotBase directory and is refreshed periodically. The verified bots classified as Search, Agent, and Training can be found in the public bots directory.
Here is a shortened, anonymized example of the prepended block:
# BEGIN Cloudflare Bot Preference Sync
User-agent: TrainingBot1
User-agent: TrainingBot2
User-agent: TrainingBot3
User-agent: MixedUseBot-Extended
Disallow: /
...
# END Cloudflare Bot Preference SyncBear in mind that Bot Preference Sync works at the category level rather than on individual custom rules. If a special arrangement with a specific company means you need to grant an exception, turn off the sync and manage your robots.txt manually.
Publisher-Friendly Defaults
Cloudflare is introducing a new default aimed at publishers and ad-supported sites as well. During onboarding, those customers can select “I monetize from pages with ads on this domain,” which sets Training to Disallow out of the box. Content then stays out of model training, while search engines that send readers to your pages keep their access. The setting can be changed at any time.
Non-publisher new customers get no blocks or disallows by default. The decision is left to you: whether to block Search, Agent, or Training is something you can settle later, and the starting point has nothing blocked on your behalf.
Transparency as the Price of Admission
Cloudflare is also tightening requirements for mixed-use crawlers—bots that combine search, agent use, and training behind a single user agent. To avoid being blocked when you set “Disallow Training,” these bot operators must provide additional information:
- Respect a “no training” preference in robots.txt through any mechanism they choose.
- Give site owners a way to opt out of AI summaries.
- Provide URL-level visibility into which pages were made available for training, plus metrics on search results.
- Show publicly that disallowing training does not hurt traditional search results.
The AI bot transparency section of Cloudflare Radar tracks the bots that meet these criteria, showing both best practices honored and violations. A crawler that offers no transparency gets no benefit of the doubt—it is still blocked when you disallow training. Transparency thereby becomes a requirement for access.
Limitations and Trade-offs
Bot Preference Sync simplifies policy management, but it is not for everyone. Category-wide decisions are what it was designed for, not case-by-case exceptions. With complex custom rules, or special agreements with specific crawlers, turning off the sync and maintaining your robots.txt manually is the sensible move. The sync also touches nothing beyond the robots.txt file; your edge enforcement rules stay as they are, so blocking traffic at the edge still means configuring those separately.
One more consideration: the list of bots in robots.txt comes from Cloudflare’s tracking, which may not include every crawler, so it pays to verify that the bots you care about are covered. And since the feature rolls out over the coming week, existing customers on the legacy managed robots.txt will be prompted to review and confirm their preferences.
Takeaway: Align Your Stated Preferences with Reality
Bot Preference Sync is a practical step toward narrowing the gap between what you declare and what you enforce. For most site owners it is a low-effort way to keep AI bot policies consistent. Where you need fine-grained control or have special arrangements, opting out and managing your own robots.txt remains an option. The key is to choose deliberately: either let Cloudflare sync your preferences, or take manual control—but never leave your settings inconsistent, because that is exactly what some crawlers will exploit.