A Cloudflare change taking effect today, 15 September 2026, means some website owners could unintentionally block Googlebot while trying to prevent their content being used for AI training.
Cloudflare has changed the way it classifies automated crawlers, splitting them into Search, Agent and Training categories.
For new domains, Training and Agent crawlers can now be blocked by default on pages displaying advertising, while Search crawlers remain allowed.
The bigger SEO issue is how Cloudflare handles crawlers that serve more than one purpose. Cloudflare says mixed-purpose crawlers are now subject to the most restrictive rule applied to them. That includes crawlers such as Googlebot, Bingbot and Applebot.
Core Web UK — 3-minute check
- Check Cloudflare: review Search, Agent and Training crawler settings.
- Test Googlebot: use Google Search Console’s Test Live URL on an important page.
- Check crawl errors: look for sudden 403 responses, blocked pages or crawling failures.
If Google can fetch the page normally, you probably do not need to change anything.
Cloudflare confirmed that the new defaults take effect on 15 September 2026. Existing Free-plan customers who had not previously changed their AI crawler settings may also be affected by the revised defaults.
What has Cloudflare changed?
Until recently, Cloudflare offered a relatively simple option to block AI bots. The new system gives website owners more control by separating automated crawlers according to what they are doing.
Search crawlers collect and index pages so they can appear in search results.
Agent crawlers access websites in real time on behalf of a user, such as an AI assistant retrieving information.
Training crawlers collect information that may be used to train or improve AI models.
Website owners can decide which categories to allow or block. The complication comes when one crawler performs more than one role. Cloudflare says mixed-purpose crawlers are governed by the strictest applicable rule.
Why this matters: a setting designed to block AI training traffic can potentially affect ordinary search crawling too. That turns what looks like an AI privacy setting into a technical SEO issue.
Could Cloudflare stop Google indexing a website?
Yes. Cloudflare sits between the visitor and your web server. If Cloudflare rejects a crawler before the request reaches WordPress or your hosting server, Google may receive an error response instead of the webpage.
Google’s basic indexing requirements include being able to access a page and receiving a successful HTTP response. If Googlebot starts receiving a 403 Forbidden response, crawling can be affected.
That does not normally mean pages disappear from Google immediately. Previously indexed content can remain visible for some time. The bigger risk comes when Google repeatedly tries to recrawl pages and cannot access them.
New pages may also take longer to be discovered or indexed. For a service business, ecommerce store or regularly updated website, that can become a significant SEO problem.
The Cloudflare setting to check
If your website uses Cloudflare, open the relevant domain in the Cloudflare dashboard and review the site’s AI crawler controls.
- Search
- Agent
- Training
Do not simply switch everything to Allow. Publishers and website owners may have legitimate reasons for restricting AI training.
The important question is whether the rule you have chosen also affects a crawler your business depends on for search visibility. Businesses should be especially careful if they previously enabled Cloudflare’s older Block AI Bots setting and have not reviewed it recently.
How to test whether Google can access your website
The quickest reliable check is Google Search Console. Open URL Inspection, enter one of your important pages, then choose Test Live URL.
Google will attempt to fetch the current version of the page. If the result shows that the URL is available to Google, the page is currently accessible. If it cannot be fetched, review the Page Availability information.
Problems can include blocked requests, server errors, DNS failures, firewall restrictions, 403 responses or inaccessible resources.
For a site-wide issue, also check Search Console → Settings → Crawl Stats. A sudden increase in failed crawl requests or 403 responses deserves investigation.
Core Web UK tip: test more than just your homepage. Check an important service page, a recent article and a key commercial landing page. One accessible URL does not prove the whole website is crawlable.
Do not test Googlebot by changing your browser user agent
Some website owners try to test crawler access by changing their browser’s user-agent string to Googlebot. That can give misleading results.
Cloudflare and other security platforms can verify whether traffic claiming to be Googlebot is actually coming from Google’s infrastructure. A fake Googlebot request may therefore be blocked even though the genuine crawler is allowed.
For most businesses, Search Console’s Live URL test is a much safer first check.
WordPress websites can be affected too
This is not specifically a WordPress problem. But many WordPress websites use Cloudflare for speed, caching, DNS or security.
That means your WordPress dashboard can look completely normal while Cloudflare blocks a request before it ever reaches WordPress.
In that situation, updating WordPress, clearing Elementor cache, resaving permalinks, disabling plugins or regenerating CSS will not fix the underlying problem. The Cloudflare or firewall rule must be corrected.
That distinction can save hours of unnecessary WordPress troubleshooting.
What if Search Console shows 403 errors?
A 403 status code means the server or security layer understood the request but refused access.
If you start seeing them after changing Cloudflare settings, check AI crawler policies, Cloudflare WAF rules, Bot Fight Mode, custom firewall rules, verified-bot restrictions, user-agent rules and hosting-level firewalls.
Once the rule has been corrected, run Test Live URL again. Only request indexing after Google can successfully access the page.
If your traffic dropped suddenly: check whether anything changed in Cloudflare, your hosting firewall or bot protection around the same time. Sudden technical SEO problems often start outside WordPress itself.
What if Googlebot was already blocked?
A short interruption may cause little or no noticeable damage. Longer blocking can have a bigger effect.
- slower crawling
- delayed indexing
- old page versions remaining in search
- temporary ranking losses
- pages eventually dropping from results
- reduced visibility in Google Images, News or Discover
Once access is restored, Google still needs to crawl the affected pages again. Recovery therefore may not be immediate.
For an important website, monitor Crawl Stats and indexing reports over the following days.
Should businesses block AI training crawlers?
There is no universal answer. A news publisher protecting original journalism may make a different decision from a plumber, retailer, solicitor or local service business.
The important distinction is between being discoverable and allowing content to be used for other purposes.
Many businesses actively want their services to appear in Google, Bing, AI search engines, recommendation systems and AI assistants. Others may want traditional search visibility while restricting model-training access.
Cloudflare’s new controls are intended to make that choice easier. The difficulty is that crawler purposes are not always completely separated. That is why businesses should avoid treating AI crawler controls as a simple on/off switch.
What about ChatGPT and AI search visibility?
Blocking every AI crawler may also affect how easily some AI systems can retrieve current information about your business.
Traditional Google SEO is no longer the only discovery channel. People increasingly use AI assistants to research local businesses, suppliers, products, professional services, software, reviews and prices.
That does not mean every crawler should automatically be allowed. It means crawler management is becoming part of a wider search and AI visibility strategy.
What UK small businesses should do now
For most businesses that rely on organic search, the priorities are simple: keep legitimate search crawlers accessible, avoid broad bot-blocking rules you do not understand, test important URLs after changing Cloudflare or firewall settings, and watch Search Console for unusual crawling errors.
Keep a record of major security changes too. If traffic drops two days after someone changes Cloudflare settings, knowing exactly what changed makes troubleshooting much easier.
If you manage a WordPress site and want ongoing help with uptime, security, updates and technical checks, see our Website Maintenance & Support service. If your site is already returning errors or has suddenly stopped working correctly, our Emergency Website Fixes service covers urgent technical problems.
Cloudflare’s AI change is now an SEO issue
Cloudflare’s crawler changes are designed to give website owners greater control over how automated systems use their content. But the change also creates a technical SEO consideration.
A rule intended to restrict AI training can potentially affect the crawlers responsible for discovering and refreshing pages in traditional search engines.
For businesses relying on Google traffic, the sensible response is not panic. It is verification.
Check your Cloudflare crawler settings. Run a live Search Console test. Check Crawl Stats for errors. If Google can access your important pages normally, leave a working configuration alone. If it cannot, investigate the Cloudflare and firewall rules before making unnecessary changes inside WordPress.
Sources: Cloudflare developer documentation and Google Search Central guidance on crawling, Googlebot and URL inspection.