The relationship between technology platforms and publishers is entering a new phase.
For more than two decades, search engines have sent billions of visits to news organisations, blogs, retailers and independent websites in exchange for access to their content. That arrangement helped build much of today’s internet economy.
Now, the rapid rise of artificial intelligence is changing the balance.
AI-powered search tools are increasingly providing direct answers to users instead of sending them to the websites where the information originated. At the same time, publishers are questioning how their content is being collected, summarised and reused by AI systems.
The debate has become one of the biggest issues facing the digital publishing industry.
A new era of AI-powered search
The launch of AI-generated search experiences has transformed how people find information online.
Instead of presenting users with a list of links, search engines and AI assistants are increasingly offering summaries, recommendations and direct answers within the search experience itself.
For users, the experience can be faster and more convenient.
For publishers, however, it raises difficult questions.
If a reader receives an answer without clicking through to a website, the publisher loses the opportunity to generate advertising revenue, subscriptions or brand engagement.
This shift has intensified concerns across the publishing industry, particularly among businesses that rely heavily on search traffic.
The rise of zero-click searches
Zero-click searches are not new.
Users have long been able to find information such as weather forecasts, sports scores, business opening hours and currency conversions directly on search pages.
Artificial intelligence has accelerated that trend.
AI systems can now summarise articles, compare products, answer complex questions and provide recommendations without requiring users to visit the original source.
Industry analysts increasingly believe that direct answers will become a larger part of the search experience in the years ahead.
As a result, publishers are preparing for a future in which search referrals may no longer deliver the same levels of traffic they once did.
Publisher traffic is under pressure
Many publishers have already reported declining search referrals over the past several years.
The reasons vary.
Changes to search algorithms, the growth of social media platforms and shifting consumer habits have all affected website traffic.
The arrival of generative AI has added another layer of uncertainty.
Media companies, independent publishers and specialist websites are increasingly examining how much of their audience remains dependent on traditional search engines and whether new business models are needed.
Some organisations are investing more heavily in subscriptions, newsletters, events and direct reader relationships in an effort to reduce reliance on search traffic.
Others are exploring licensing agreements with AI companies.
Licensing deals offer a new revenue stream
As artificial intelligence companies seek high-quality content to train and improve their models, some publishers have chosen to negotiate licensing agreements.
These deals can provide an additional source of income at a time when digital advertising markets remain highly competitive.
Supporters argue that licensing arrangements create a framework in which publishers are compensated for the use of their content.
Critics, however, say such agreements may benefit only a small number of larger media organisations while leaving smaller publishers with limited bargaining power.
The issue has highlighted growing differences in how publishers believe AI companies should access and use online content.
The crawler debate is becoming more important
At the centre of the discussion is the role of web crawlers.
Search engines have long used automated systems to scan and index websites so that pages can appear in search results.
The emergence of generative AI has complicated the situation.
Some publishers argue that there should be clearer distinctions between content being used for traditional search indexing and content being collected to support AI products and services.
They believe website owners should have more control over how their material is used and whether it can be accessed by AI systems.
Technology companies, meanwhile, argue that access to publicly available information has played a crucial role in the development of the modern web and in the improvement of search technologies.
The debate is likely to continue as AI products become more sophisticated and more widely used.
Regulators are paying closer attention
Competition authorities and policymakers are also taking a growing interest in the relationship between large technology companies and publishers.
Regulators in several markets are examining issues including platform power, transparency and the economic impact of AI on content creators.
Questions around data access, market dominance and fair compensation are increasingly becoming part of wider discussions about the future of digital competition.
Any regulatory intervention could have significant implications for both publishers and technology companies.
Why this matters for the future of the web
The dispute over AI crawlers is about more than technology.
It touches on the economics of publishing, the sustainability of independent journalism and the future structure of the open internet.
Publishers want to ensure that high-quality content continues to be financially viable.
Technology companies want to develop AI products that can provide users with faster and more useful information.
Finding a balance between those two objectives may become one of the defining challenges of the next phase of the internet.
What happens next?
The relationship between AI companies and publishers is likely to continue evolving.
More licensing agreements are expected. Regulatory scrutiny will probably increase. Publishers will continue to experiment with new business models, while technology companies are expected to refine how AI systems access and present information.
The outcome could reshape how information is created, discovered and monetised online.
For publishers, the question is no longer whether artificial intelligence will change the web.
The question is how they will adapt to it.