Cloudflare Radar has launched a new metric that shows how often artificial intelligence (AI) crawlers send traffic to websites compared to human traffic. The analysis reveals a concerning trend of AI bots consuming more content and sending the same or less traffic back to the source.
The metric is based on data from search engines, such as Google, Bing, and DuckDuckGo, which allow users to search for specific content online. It measures the ratio of aggregate HTML page requests from crawlers associated with a given platform to the number of HTML page requests from clients referred by a hostname associated with that same platform.
According to Cloudflare Radar’s data, some AI platforms make nearly 70,000 times more requests than referrals. For example, Anthropic’s AI platform Claude made nearly 71,000 HTML page requests for every referral. This trend is not limited to one platform but is observed across various search engines and AI models.
The analysis also reveals changes in crawling traffic over time, with some platforms experiencing an increase in crawling activity while others see a decrease. OpenAI’s GPTBot, for instance, had little-to-no crawling activity throughout the month.
This trend has significant implications for content providers, who are now facing more crawls and fewer referrals than ever before. Legacy search index crawlers would scan a site’s content less frequently, making revenue models more viable. However, AI models continue to consume more content despite sending the same or less traffic back to the source.
To address this issue, Cloudflare Radar has introduced new tools that allow site owners to block certain types of AI crawlers and make the exchange of value fairer for both sides of the equation. The company recommends that content creators audit and enforce their preferred policies for AI crawlers.
In addition to the new metric, Cloudflare Radar has launched an expanded Verified Bots directory, which provides a list of verified bots, including metadata about each bot, traffic graphs, and API access.
Source: https://blog.cloudflare.com