AGENTIC COMMONSAI industry briefings

繁中EN

TopicAI Agents & WorkflowsPublished

All English articles

BotBase for Operators: Cloudflare Gives Bot Operators a Clearer Path to the Directory

On this page7 sections
  1. The Shift: Out of the Black Box and Into Submission History
  2. Editing, Canceling, and Keeping Details Current
  3. Behavior, Content Use, and Operator Type: A Three-Part Taxonomy
  4. Review That Scales: Automated and More Consistent
  5. Trade-offs and Limits
  6. The Takeaway: Transparency as a Workable Model
  7. Sources

Summary: Cloudflare released BotBase for Operators on August 28, 2026 to replace one-way bot directory submissions with visible statuses, rejection reasons, in-place edits, and cancellation. Operators now describe behavior, content use, and whether they run directly or as an intermediary, aligning bot declarations with owners’ Content Signals. Automated checks validate duplicate entries, user-agent precision, IP lists, reverse DNS, or Web Bot Auth, though directory acceptance still does not require site owners to allow a bot.

The Shift: Out of the Black Box and Into Submission History

Getting a bot into Cloudflare’s BotBase directory used to feel, for the operator, like mailing a letter into empty space. Once you pressed submit there was no status to check, no explanation behind a rejection, and no way to revise what you had filed; emailing support was the only recourse left. Cloudflare shipped BotBase for Operators on August 28, 2026 to close that gap, opening the submission process up so operators can actually see what is happening.

The submission form now sits somewhere far more sensible in the Cloudflare dashboard: Protect & Connect → Application Security → BotBase. It previously lived under Manage Account → Configurations, a spot that linked a bot to your account but left it disconnected from the wider bots ecosystem. BotBase for Operators now breaks into three parts: Bots directory for browsing and searching known bots, Submission form for filing a new one, and Submission history for tracking everything you have sent in.

The headline addition is the Submission history tab. Every submission attached to your account now carries an unambiguous status: Waiting for review, Accepted, or Rejected. A rejection arrives with the actual reason spelled out and concrete steps you can take to fix it. Where a submission was accepted but Cloudflare adjusted how the bot is classified, you can see what moved. This answers the complaint Cloudflare kept hearing from operators: that submitting a bot felt like a black box.

Editing, Canceling, and Keeping Details Current

The identifying details of a bot drift over time. A website rebuild might relocate an IP list to a different endpoint, or you might step away from an IP allowlist and begin signing traffic with Web Bot Auth. Reacting to changes like these once meant filling out the entire form over again and ending up with a brand-new entry. Neither is necessary now: a submission you already made can be edited in place, and one still waiting for review can be canceled.

Cloudflare makes the point that accurate details are what let a bot earn and hold onto Verified status. Verified bots tend to be allowed by site owners on the basis of behavior, though the call always rests with the individual site owner. Keeping a bot’s record current is a good deal easier than it was.

Behavior, Content Use, and Operator Type: A Three-Part Taxonomy

Rather than forcing every bot into one label, the new submission form asks for a three-part description that reflects what bots genuinely do. It also lines up with the behavior and content use model Cloudflare introduced on July 1, 2026.

First, what your bot does. Several behaviors can be selected at once: indexing pages for search, acting on a user’s behalf, collecting data, training models, supporting SEO tools, and others besides. You are no longer reduced to picking whichever single label comes closest.

Second, how it uses what it reads. A crawler that skims a page to build a search snippet is a different animal from one that files the page away to train a model. Here you declare how much content use your bot needs, expressed through the very same Content Signals model website owners apply when setting rules in robots.txt. A site might, for instance, declare Content-Signal: search=yes, ai-train=no, use=reference — indexing for search and holding a reference are fine, training a model is not. What your bot declares about content use gets measured against preferences of exactly that shape.

Third, who’s actually running it. The choice is between direct and intermediary. Direct means you run the bot off your own infrastructure, as a search engine does when it builds its own index. Intermediary means you provide a platform that other companies build on, carrying their traffic without being the party that decided to send it. A general-purpose AI assistant operator, for example, may supply the infrastructure while a separate company’s app, built on that API, is what actually triggers the requests.

Set out this way, the description hands website owners a sharper picture of your bot, which lets them decide more accurately how it ought to be treated.

Review That Scales: Automated and More Consistent

The flow of new bot submissions has risen steeply — about 7 times since 2023 — and reviewing them by hand does not scale. Cloudflare rebuilt the process so that it runs on its own. A submission is put through a series of automated checks: Does it duplicate an entry that already exists? Is the user-agent pattern narrow enough to avoid overlap? Most importantly, does the claimed verification method hold up? The system pulls your IP list, confirms reverse DNS, or validates Web Bot Auth signatures automatically — the manual checks are gone.

When all of it passes, your bot becomes trackable straight away. When something warrants a closer look, it is routed to the team with the specific reason already flagged, rather than arriving as a blank row in a queue. For operators, the upshot is that most submissions clear faster than before.

Trade-offs and Limits

BotBase for Operators is a real step forward, but it is worth being clear about what it does not do. For one thing, acceptance into the directory promises nothing about whether any given website will allow your bot; site owners keep full control over the traffic they permit. For another, automated review will not catch every nuance, and certain submissions will still need a human to look at them, which can take longer. The new taxonomy, flexible as it is, also depends on you describing yourself truthfully, and misrepresentation could mean rejection or the loss of Verified status. Finally, as things stand these features are reachable only through the Cloudflare dashboard — no public API for submissions or status checks has been mentioned.

The Takeaway: Transparency as a Workable Model

What BotBase for Operators shows, concretely, is how a one-way submission can become an ongoing relationship. For product builders the lesson worth taking is how much transparency is worth in automated traffic management. Status tracking, entries you can edit, and a structured taxonomy together cut friction for bot operators while giving website owners better grounds for their decisions.

If crawlers, agents, or automated traffic figure anywhere in your product, it is worth asking how these principles translate to your case: show operators where their submission stands, let them correct their information without friction, and classify by structured, behavior-based categories rather than loose labels. Done well, this sharpens the operator’s experience and builds trust on both sides of the ecosystem.

On Cloudflare’s roadmap sit ownership and observability — claiming bots, managing live directory entries, understanding how sites treat your bot — and, further out, conversation: a sustainable route for operators to request access by demonstrating value. For the moment, BotBase for Operators stands as a solid first step toward a bot ecosystem that works more collaboratively.

Sources

AGENTIC COMMONSOperated by PHLEGON LABS
ShareXEmail
Support us

Related reading

  1. Cloudflare and OpenAI Pilot: Discovery Signals Reshape AI Crawling

    Cloudflare and OpenAI's joint pilot helped ChatGPT discover and index hundreds of millions of previously unknown pages.

    Cloudflare

  2. Bot Preference Sync: Cloudflare Syncs Your AI Bot Policies to robots.txt

    Cloudflare's Bot Preference Sync writes your AI bot settings to robots.txt, keeping stated preferences and enforced rules aligned. Learn how it works, publisher defaults, and…

    Cloudflare

  3. B2B Prospecting: Five Prompts Backed by Fresh Web Data

    Firecrawl's five-step prospecting guide pairs copy-paste AI prompts with /search, /scrape, and Alexandria provider data.

    Firecrawl