Cloudflare now gives website owners separate controls for search crawlers, model-training crawlers and AI agents. Its September 15 rollout offers new domains two different starting configurations. An ad-supported site is offered Search: Allow, Training: Disallow AI Training and Agent: Block on pages with ads. A site that does not monetize with ads is offered Allow for all three. Owners can change the recommendations during onboarding or later.

Existing Cloudflare domains follow their previous choices. A site that had left the old “Block AI Bots” switch off moves to Allow for Search, Training and Agent. One that had selected either legacy blocking option moves to Search: Allow, Training: Disallow AI Training and Agent: Block on pages with ads. Domains that had already configured the three controls retain the practical effect of those choices under Cloudflare’s migration table. The controls are available on every plan, including Free; plan eligibility does not tell an owner which policy their domain currently has.

What the three categories cover

Cloudflare’s July changelog defines Search as crawling that indexes content to answer later queries. Agent covers real-time activity on a person’s behalf, including chat fetch bots and browser-use agents. Training covers collection for model training or fine-tuning. A crawler can have more than one purpose. Cloudflare says each blocking option covers verified bots in its classification and additional unverified bots classified the same way. That is a classification-based control, not a claim that every automated browser or unidentified request will be caught.

For a domain using the ad-supported recommendation, “Block on pages with ads” applies to the Agent category on pages Cloudflare detects as serving ads. It does not describe a whole-site agent ban. Search remains allowed by that preset. The Training setting is different: September’s Disallow AI Training publishes a no-training preference through Bot Preference Sync and blocks training crawlers other than mixed-use crawlers Cloudflare calls Accountable. Those Accountable crawlers can continue search crawling while honoring the training preference. Cloudflare says the setting covers training-only crawlers separately, so blocking those need not shut off search.

This September arrangement refines the July documentation, which still describes a future default that blocks both Training and Agent on ad pages. The later release announcement and Cloudflare’s detailed September migration and recommendation tables specify Disallow AI Training for an ad-supported new domain, with the ad-page block applied to Agent. The older description should not be read as a universal policy for existing customers.

“Disallow” and “Block” also have different effects on a crawler used for both search and training. Cloudflare says mixed-use crawlers account for 36.6% of verified crawler traffic on its network, though it gives no comparable figure here for individual sites. A full Training block now includes mixed-use crawlers such as Googlebot, Bingbot and Applebot, which can remove their search access too. Disallow AI Training is intended to retain search access for Accountable mixed-use crawlers while expressing the training restriction. Cloudflare notes a present limit: Microsoft has not yet shipped support for a no-training preference in robots.txt for Bingbot, so selecting Disallow AI Training does not automatically convey that preference to Bing today.

What a site owner can check

In the Cloudflare dashboard, open the domain’s Security Settings → Configure AI bot policies and inspect the current Search, Training and Agent selections. Cloudflare’s configuration documentation describes Allow, Block and Block on pages with ads; its September account explains the additional Disallow AI Training choice and the migration of legacy settings. If the site relies on ad impressions, check whether the domain uses the ad-supported preset and whether Agent is limited to detected ad pages. If preserving conventional search access matters, check the Training choice before selecting Block for mixed-use crawlers.

These are Cloudflare’s documented controls and rollout rules. BIG CHANGE did not change a Cloudflare account or measure bot traffic on a live site.

The big change

Cloudflare has separated three uses of automated access into domain-level policies. For new ad-supported domains, its September recommendation keeps search open, expresses a no-training preference and blocks classified agents on detected ad pages. Existing owners need to read their own migrated settings before drawing conclusions about access.

Sources & further reading