Cloudflare will block AI crawlers on ad pages by default: what it means for AEO

On 1 July 2026, Cloudflare announced that from 15 September it will block a whole class of AI crawlers by default on pages that carry ads. It is the clearest sign yet that the open web AI trained on for free is being fenced off - and it changes the ground rules for anyone who wants to be found and cited by an answer engine.

If you have spent the last year worrying about whether ChatGPT and Perplexity name your brand, this is the other side of the same coin. Answer engines can only cite what their crawlers are allowed to read. When the company that sits in front of roughly a fifth of the web changes what its crawlers are allowed to read by default, that is worth understanding before it lands on your site without you noticing.

What Cloudflare actually announced

Cloudflare framed the move as "Content Independence Day." The mechanics matter more than the branding, so here is the plain version. From 15 September 2026, for new domains onboarding to Cloudflare and for existing free customers, the default behaviour on ad-monetized pages will be to block training and agent bots while still allowing search bots. The reasoning Cloudflare gave is simple: an ad is a signal that a site owner meant a person to land on that page, so human attention takes priority on monetized content.

The categories are the part to hold onto:

• Search crawlers - the ones that index pages to answer queries, including the crawlers behind Google AI Overviews. Still allowed by default.
• Training crawlers - the ones that harvest content to train models. Blocked by default on ad pages.
• Agent crawlers - the ones that fetch a page on behalf of a user's AI agent in real time. Also blocked by default on ad pages.

Know the bots that are actually knocking

To make a good decision, it helps to know the names on the door. The AI crawlers hitting your site fall into a few buckets, and they are not interchangeable. Training crawlers - OpenAI's GPTBot, Anthropic's ClaudeBot, Common Crawl's CCBot, and Google-Extended - collect content to train or improve models. Search and answer crawlers - OAI-SearchBot, PerplexityBot, and the Google and Bing crawlers that assemble live answers - fetch pages to build the response a user sees right now. Agent fetchers retrieve a single page in real time because a user's assistant is acting on their behalf.

The distinction matters because your goals differ by bucket. You may be genuinely indifferent to whether your marketing copy trains the next model. You are almost certainly not indifferent to whether the crawler that feeds a live ChatGPT or Perplexity answer can reach you - that one is the difference between being cited and being invisible. A blanket "block AI" toggle collapses all of these into one decision, and that is precisely the mistake to avoid.

The catch: multi-purpose crawlers

Here is where it gets sharp for AEO. Some of the biggest crawlers do more than one job. Cloudflare's own examples include Googlebot, Applebot and Bingbot, which combine search with training. Cloudflare has said these multi-purpose bots will be "allowed or blocked according to all of their behaviours," enforced by the most restrictive applicable rule.

Translated: if a site owner blocks training crawlers, a multi-purpose bot that also powers search can get blocked entirely, because one of its behaviours trips the block. The intent might be to stop model training, but the collateral effect can be losing the crawl that fed your presence in an AI answer. That is the risk to watch, and it is the reason "just block the AI bots" is not the safe default it sounds like.

"Answer engines can only cite what their crawlers are allowed to read."

From pay-per-crawl to pay-per-use

This did not come from nowhere. Cloudflare has been building toward a paid model for AI access, starting with a pay-per-crawl marketplace where AI companies pay to fetch content, and signalling a broader shift toward paying publishers when their content creates value - for instance, when it shows up in an AI-generated result. The direction of travel is unmistakable: the era of AI crawling the whole web for free is closing, and access is becoming something negotiated rather than assumed.

For large publishers, that is a revenue story. For most brands, it is a visibility story. Your marketing site is not trying to charge OpenAI to read it - you want to be read, because being read is how you get cited. The danger is not the paywall; it is getting swept up in a default that was designed for publishers, not for you.

What this means for your brand

Three practical takeaways.

1. Do not accept the default blindly. If your site is on Cloudflare, the bot-management settings are now a decision, not a background detail. Know which crawler categories you are allowing, and choose deliberately rather than inheriting a policy tuned for ad-heavy publishers.
2. Keep search and answer-engine crawlers allowed. The crawlers that feed answer engines are the ones you want reading you. Blocking training while preserving search access is possible, but only if you configure it on purpose and account for the multi-purpose-bot trap.
3. Watch your citations, not just your settings. A configuration change is invisible until it shows up as a drop in how often you are named. The only way to know the change did not cost you is to measure whether you are still being cited across the engines after 15 September.

What to do, by the kind of site you run

The right setting is not the same for everyone, because "an ad on the page" means very different things depending on your business.

• Marketing sites and product pages. You want to be read by search and answer crawlers, and you rarely run display ads on these pages, so the new default should not touch you. The action is to confirm that - check that nothing is inadvertently blocking the answer-engine crawlers, and keep it that way. Being cited is the whole point of the page.
• Content and media sites with ads. This is who the default was built for, and it is a genuine judgment call. Blocking training crawlers protects your work from feeding a model for free; but be deliberate about not blocking the search and answer crawlers along with them, or you trade a training concern for a visibility loss. The multi-purpose-bot trap below is aimed squarely at you.
• Documentation and developer sites. You almost always want to be maximally readable - your users increasingly reach your docs through an assistant, and being the source an AI quotes is a distribution channel, not a threat. Lean open.
• E-commerce. As AI shopping grows, being readable to the crawlers that power product answers is becoming a sales channel. A reflexive block here can quietly remove you from the set of products an assistant will recommend.

How to actually check your settings

This does not have to be abstract. If your site is on Cloudflare, you can see and set the relevant controls in the dashboard's bot and AI-crawler settings, where the crawler categories map to the buckets above. Two practical habits make the difference:

1. Audit before the deadline, not after. Look at what you are currently allowing and blocking now, so the 15 September default does not change your posture without you noticing. A setting you inherited is still a setting you own.
2. Confirm your robots.txt and CDN rules agree. It is common to have a robots.txt directive that says one thing and a CDN rule that says another. When they conflict, the crawler's experience is whatever the most restrictive layer enforces - and that is often not what you intended. Make the two layers tell the same story.

How we got here: from a 1994 handshake to a 2026 wall

This blocking-by-default move did not appear out of nowhere. It is the end of a thirty-year arc that started with a gentleman's agreement. In 1994, a software engineer named Martijn Koster proposed robots.txt: a plain text file at the root of a site that tells crawlers where they may and may not go. It was never enforceable. It worked because everyone agreed to honor it, and for nearly three decades the major search crawlers did.

The AI training era broke that handshake. Once content became fuel for models, the incentive to comply evaporated for some players. Large-scale measurement studies of crawler behaviour have found that scrapers respect robots.txt only selectively - honouring blocks on some pages while ignoring them on others, and in some cases disguising their user-agent to slip past. Perplexity was publicly accused in 2024 of fetching pages it had been told not to, and the disputes escalated from there into the Cloudflare versus Perplexity standoff that made headlines.

The courts have not rescued the old model either. In the litigation flowing from The New York Times v. OpenAI and Microsoft, filed in December 2023 and still in discovery in 2026, a judge in a related case reportedly treated robots.txt as "mere requests" rather than a real access control. Read those two threads together and Cloudflare's announcement makes sense: if a polite request cannot stop a crawler and a lawsuit takes years, the only lever left is the network itself. Blocking at the edge is what you reach for when asking nicely has stopped working.

Publishers and brands are not solving the same problem

The default Cloudflare shipped was designed with one group in mind, and it is not most of the companies reading this. It helps to be honest about which side of the line you sit on, because the right instinct for one is the wrong instinct for the other.

• Publishers are protecting an asset. If your content is the product - news, research, investigations, a paid archive - then a crawler reading you for free is lost leverage. Blocking training bots and pushing toward a paid arrangement is a rational revenue play. The thing to watch is that you do not throw away referral traffic and citations while defending the training rights, because a live answer that names you still sends readers your way.
• Brands are chasing distribution. If your content exists to sell something else - a product, a service, a demo - then being read is not a cost, it is the entire point. You are not trying to charge OpenAI to quote your pricing page. You want it quoted. For you, the danger is inheriting a publisher's defensive default and quietly removing yourself from the answers where buyers are looking.

The tell is simple: ask whether you would rather be paid for a crawl or be named in the answer. Publishers can reasonably want the first. Almost every brand wants the second, and should configure for it on purpose.

How to catch a citation drop after 15 September

A settings change is silent. You will not get an alert that says "you have been dropped from Perplexity's answer for your best query." The only way to know the new default did not cost you is to have measured your baseline before it landed and to keep watching after. Practical steps:

1. Capture a baseline now. Before the deadline, record how often you are cited across ChatGPT, Perplexity, Google AI Overviews and the others for your priority questions. Without a before, you cannot prove an after.
2. Re-run the same prompts on a schedule. Use a fixed set of buyer questions and check them weekly through mid-September and beyond. You are looking for the moment your name stops appearing, not a vague sense that things feel quieter.
3. Watch your server logs for the crawlers that matter. If PerplexityBot, OAI-SearchBot or the Google answer crawlers suddenly stop hitting your pages, that is your early warning, and it shows up in logs days before it shows up in a lost citation.
4. Separate a config problem from a content problem. If a citation disappears and the crawler is still fetching you fine, the answer engine simply chose a better source, and that is a content fix. If the crawler stopped reaching you, that is an access problem you created, and it is reversible in the dashboard.

The bigger pattern

Cloudflare's move is one instance of a broader tightening: the web is learning to charge for, throttle and gate AI access, and every gate is a place your brand can quietly disappear from an answer. The brands that stay visible will be the ones treating crawler access and AI citations as a monitored channel - not a thing they set up once and forgot. The open web that made AEO easy is getting more complicated, and the response is not panic; it is attention.

Related: How to get cited by ChatGPT, Perplexity & Gemini →