Anthropic’s ClaudeBot web crawler scraped iFixit’s website over 900,000 times within a 24-hour period, potentially violating the repair company’s Terms of Use. This excessive scraping triggered alarms and required iFixit’s devops team to intervene. iFixit’s CEO, Kyle Wiens, stated that this activity is strictly prohibited without prior written permission.
Anthropic initially linked back to an FAQ page stating that their crawler can only be blocked via a robots.txt file extension. However, after iFixit added the crawl-delay extension to its robots.txt, ClaudeBot respected the signal and stopped scraping.
Other website owners, such as Read the Docs and Freelancer.com, have also reported aggressive scraping by Anthropic’s crawler. This behavior is not new, as several months-old Reddit threads report a significant increase in web scraping activity by ClaudeBot.
Source: https://www.theverge.com/2024/7/25/24205943/anthropic-ai-web-crawler-claudebot-ifixit-scraping-training-data