AI crawlers have created a new tension between website publishers and technology companies because the old web bargain is changing. Search engines traditionally crawled content and sent users back to the source. Generative AI systems can also summarise, answer and compare information directly, which raises new questions about traffic, attribution, control and commercial value.
For publishers and businesses, the practical issue is not whether AI exists. It is deciding which systems may access content, how that content can be reused and whether the website still receives enough value in return.
Why AI crawling is different from traditional search crawling
Traditional search crawling is mainly designed to discover and index pages so they can appear in search results.
AI systems may use web content for several different purposes, including search grounding, answer generation, model improvement or other AI features. Those purposes are not always controlled by the same crawler or directive.
That is why publishers need to check the documentation for each platform instead of assuming one robots.txt rule applies to every AI use.
Why publishers are concerned
The concern is economic as much as technical.
Publishers invest in reporting, research, specialist knowledge, photography, product testing and editorial staff. If an AI system can answer a user’s question without a meaningful visit to the source, the publisher may lose advertising revenue, subscriptions, leads or sales.
At the same time, appearing as a cited or supporting source inside an AI answer may create new visibility. The difficult part is that the value exchange is still evolving.
AI search does not mean websites are irrelevant
Google’s current guidance for AI features in Search says AI Overviews and AI Mode surface links to relevant web content and that normal SEO fundamentals remain important.
Google also says there are no additional technical requirements or special schema needed to appear in those AI features.
The official guidance is available in AI features and your website.
That matters because it separates two different ideas: AI may change how users consume information, but websites still provide the underlying content, evidence, products and services those systems need to reference.
What control do website owners have?
Control depends on the platform and the intended use.
For Google Search, Google says normal Googlebot crawling controls and Search preview controls such as nosnippet, max-snippet and noindex remain relevant to how content appears in Search and its AI features.
Google-Extended is a separate control for certain other Google AI uses and does not remove a site from ordinary Google Search.
Other AI companies publish their own crawler names and policies. Website owners should review those directly before changing robots.txt.
Blocking every AI crawler is not automatically the right decision
A blanket block may reduce unwanted reuse, but it can also reduce visibility in systems that increasingly influence discovery.
The right decision depends on the business model.
- A subscription publisher may value content control more heavily.
- An ecommerce retailer may want product information discoverable wherever customers are researching purchases.
- A local service business may benefit from being understood by AI search and recommendation systems.
- A specialist research publisher may want stronger restrictions around original data.
There is no universal policy that suits every website.
Why attribution matters
If an AI system uses information from a publisher, clear attribution and useful links can help preserve the connection between the answer and the original source.
For users, attribution also makes it easier to verify claims, inspect the underlying evidence and read the full context.
For publishers, meaningful referral traffic can help maintain the incentive to create original material in the first place.
Original content becomes more valuable, not less
Generic summaries are easy for AI systems to reproduce. Original reporting, first-hand experience, proprietary data and distinctive analysis are much harder to replace.
Google’s 2026 guidance for generative AI search specifically emphasises valuable, non-commodity content and a unique point of view rather than content that simply repeats what already exists online.
See Google’s generative AI optimisation guide.
What publishers should review now
- robots.txt: know which crawlers you allow or block and why.
- Search preview controls: understand how snippet controls affect ordinary Search and Google’s AI features.
- Analytics: watch referral patterns from AI and search platforms rather than relying on assumptions.
- Content value: invest in original material that cannot be replicated by summarising commodity information.
- Attribution: monitor whether platforms provide clear source links.
- Commercial model: decide whether discovery, licensing, subscriptions or direct traffic matter most to your business.
What small-business websites should do
Most small businesses are in a different position from large publishers.
Their website is usually designed to generate enquiries or sales rather than advertising impressions. For them, being understood by search and AI systems can be commercially useful.
The priority should therefore be clear, accurate business information, strong service pages, crawlable internal links and visible evidence that helps potential customers make a decision.
Our guide to getting a business found in AI search explains those fundamentals in more detail.
What happens to SEO if AI reduces clicks?
SEO becomes more focused on valuable visibility rather than raw click volume alone.
A search impression that answers a simple informational question without a click may have less commercial value than a smaller number of visits from people ready to compare providers, enquire or buy.
Businesses should therefore track outcomes such as qualified enquiries, calls, sales and branded demand alongside rankings and traffic.
Will publishers start licensing more content?
Licensing is likely to remain part of the discussion, especially for publishers with valuable archives, specialist datasets or original reporting.
However, licensing will not solve the issue for every independent website. Many smaller publishers will still need to decide how much access to allow through public web crawling and whether AI-driven discovery produces measurable value.
The key distinction: search visibility vs model training
Website owners should avoid treating all AI access as one thing.
Being crawled for a search feature that can surface links is different from allowing content to be used for other AI purposes. The controls and commercial implications may differ.
That distinction is why platform-specific documentation matters.
Bottom line
The tension between AI crawlers and publishers is fundamentally about control and value. Publishers want their work discovered, but they also need sustainable reasons to create it. AI platforms want access to high-quality information, but the long-term ecosystem depends on attribution, useful referral paths and clear controls.
For most businesses, the sensible approach is not panic or blanket blocking. Understand which crawlers are accessing the site, decide what each type of access is worth, and continue investing in original content that gives people a reason to choose the source itself.