Summary: Cloudflare and OpenAI ran a research pilot — launched in July 2026 — to test whether signals from Cloudflare’s network could help ChatGPT discover more useful web content. Using a participating cohort of about 1% of Cloudflare’s network, the pilot brought hundreds of millions of new pages into OpenAI’s search index. If you build or run a website, this changes how your content gets found and how often crawlers hit your servers.
Why discovery needed help
The web’s most useful pages are often its most obscure ones. A washing machine manual for one model, current passport fees, tonight’s restaurant menu — these matter enormously to the person who needs them, even when few other sites link to them. In the old model, discoverability meant ranking on a search results page and letting the human do the rest. As the Cloudflare announcement puts it, AI systems now need to find the right sources, judge what’s useful, and connect people to it — through an answer, a citation, or a visit.
That’s the gap the pilot targets. Cloudflare’s network supplies signals that help OpenAI’s search crawler discover more useful pages, while OpenAI independently evaluates and ranks them. The result, per Cloudflare’s early report: hundreds of millions of newly indexed pages, with ChatGPT Search linking back to the originals.
What the crawler learned to skip
The practical insight for site operators is what the signals filter out. Automated traffic can make nonexistent pages look interesting — bots repeatedly requesting files that aren’t there, sites redirecting almost any address to the homepage, or many addresses serving copies of the same content. Recognizing these patterns lets the crawler avoid wasted fetches, which is good for everyone: fewer pointless requests against your origin, and more of OpenAI’s resources spent on pages that actually help people.
The pilot is also testing signals that flag potentially harmful pages. A legitimate site can host a single compromised page — a fake login form, say — and the approach is to flag that page rather than penalize the whole domain. Early results suggest these signals can identify pages that should stay out of an index.
The load tradeoff you should expect
There’s a wrinkle worth knowing before you assume crawling goes down. Discovering new pages can initially increase crawling while those pages get evaluated and indexed. The stated goal is to keep discovery and freshness up while reducing wasted work over time — filtering nonexistent or redirecting pages, and using change signals to decide when a page is worth another visit. I covered a related piece of this stack in Cloudflare Markdown for Agents, which tackles the other half of the problem: serving AI systems a clean version of your pages once they arrive.
Where the pilot goes next
Cloudflare says the next phase will improve page selection, test faster change signals, and — notably — measure what happens after content is discovered, including whether better site discovery translates into real business outcomes. That last part is the one to watch: indexing is nice, but referrals and conversions are what make participation worthwhile.
Two boundaries matter. First, a signal about a page doesn’t grant permission to access it — domain owners’ choices still govern. Second, this research uses metadata for search and discovery, not page content for model training. If you want to see and control how AI systems access your site, Cloudflare points to its AI Crawl Control tooling, and I’ve written about how crawl policies are configured elsewhere on this blog.
The grounded takeaway: if your site has deep, link-poor pages that AI search engines can’t find, discovery infrastructure is being rebuilt around network-level signals rather than backlinks. Whether that traffic converts is still being measured — but the crawl efficiency gains alone make it worth keeping an eye on your own logs.