Cloudflare's Sept. 15 AI Crawler Deadline: What Recipe Publishers Should Check

Tomorrow, September 15, is the deadline Cloudflare gave AI companies in July. After that date, Cloudflare's default settings block "mixed-use" crawlers, bots that fetch pages for search as well as for AI training or agents, from pages that host ads. For publishers whose business depends on search visibility and ad revenue, recipe and lifestyle sites in particular, it is worth understanding what changes and what does not.

What Cloudflare announced

On July 1, as TechCrunch reported, Cloudflare gave AI companies until September 15 to separate crawlers used for search from those used for AI training and agents. After the deadline, its default settings block mixed-use crawlers from pages that carry advertising. The new defaults apply to new customers, new sites added by existing customers and free-plan users.

Cloudflare paired the change with a new model it calls Pay Per Use. Where its earlier Pay Per Crawl, introduced in July 2025, charged AI companies when a bot fetched a page, Pay Per Use aims to compensate publishers when their content actually appears in an AI product's answer. The initial partners are Ceramic.ai and You.com. Cloudflare also said more than half of AI crawler traffic involves re-fetching pages that have not changed.

Cloudflare CEO Matthew Prince framed the move this way: "Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge."

Why ad-supported pages are the line

Tying the default to pages that host ads is deliberate. An ad-supported page makes money when a person visits it. When an AI system reads that page and answers the question elsewhere, the publisher bears the cost of producing and serving the content while the visit, and the ad impression, never happens. Recipe sites have felt this sharply: a recipe is structured, factual and easy to summarize, which makes it ideal material for an AI answer and a poor candidate for a click-through.

What changes and what does not

• Existing configurations are not automatically rewritten. The new defaults target new customers, new sites and free plans. If you are an existing paying customer, your current bot settings remain yours to manage. Check them rather than assuming.
• Search crawling is not the target. The policy aims at bots that mix purposes. Crawlers used purely for search indexing are a separate category.
• Google is a special case. Google offers Google-Extended, a control that lets sites opt out of certain AI training uses without affecting Search ranking. But Googlebot itself crawls for Search, and Search now includes AI Overviews. Blocking Googlebot to keep content out of AI features would also remove you from Search, which few ad-supported publishers can afford.

A checklist for recipe and lifestyle publishers

1. Review your bot settings. If you use Cloudflare, look at your AI crawler controls and decide deliberately which bots can access which sections. If you use another CDN or host, check what similar controls it offers.
2. Set a clear robots.txt policy. List the AI-related user agents you allow or disallow, including Google-Extended. Robots.txt is voluntary, but major crawlers state that they honor it, and it documents your intent.
3. Measure bot traffic against ad traffic. Compare server-side request logs with ad impressions. A large gap between requests and human pageviews is a cost with no revenue attached.
4. Keep bots out of the auction. Make sure known bots and crawlers are not triggering ad requests. Invalid traffic in your bid stream hurts how buyers value all of your inventory.
5. Look at your referral mix. Track sessions from traditional search, AI assistants, social and direct separately. Understanding which sources still send readers tells you where to invest.
6. Evaluate licensing and pay-per-use options on the numbers. If AI companies offer to pay for access or usage, compare the income with realistic estimates of the traffic you would keep or lose. Neither blanket blocking nor blanket allowing is automatically right.

Questions to settle internally

Before changing settings, agree on a few decisions as a team. Which sections of the site are you willing to let AI systems read, and which carry the most value to protect? Is being cited in AI answers worth more to your brand than the content costs to serve? Who owns bot policy: engineering, editorial, revenue or legal? Settling these first keeps crawler rules from being set by whoever happens to log in to the CDN dashboard.

Build for readers who do click

Crawler controls can slow the extraction of your content, but they cannot bring back every lost click. The publishers holding up best are building direct relationships: newsletters, saved-recipe accounts, apps and communities that bring readers back without a search engine in between. Those visits also tend to monetize better, with longer sessions, more pages and, with consent, first-party data that buyers value.

On the ad side, the job is to make each human visit count: fast pages, sensible ad density, well-tuned header bidding and direct or private marketplace deals with brands that want food and lifestyle context. Recipe audiences remain attractive to grocery, kitchenware, home and retail advertisers, especially heading into the fourth quarter.

The bigger shift

September 15 is one date in a much longer negotiation over who gets paid when content feeds AI systems. Infrastructure providers, publishers, AI companies and regulators are all moving on it at once. For ad-supported publishers, the practical stance is the same regardless of how that negotiation ends: know who is reading your pages, decide on purpose who gets access and make every real visit as valuable as possible. That last part is where HBDR spends its time with publishers every day.