news

Ignoring the website's anti-AI crawling policy, Anthropic crawler caused dissatisfaction among multiple website owners

2024-07-27

한어Русский языкEnglishFrançaisIndonesianSanskrit日本語DeutschPortuguêsΕλληνικάespañolItalianoSuomalainenLatina

IT Home reported on July 27 that according to The Verge local time on the 26th, ClaudeBot used by Anthropic visited the iFixit website in just 24 hours.Nearly one million times, which violates the site's terms of use.

This seemingly crazy move caused iFixit CEO Kyle Wiens to directly criticize it on X (Twitter). IT Home attached the following summary:

  • "If any of these requests had accessed our Terms of Use, it would have told you that our content is expressly prohibited. Don't ask me, ask Claude!"

  • "I know you're hungry for data, and Claude is really smart! But, you really need to 1 million hits to our servers in 24 hoursNot only are you taking our content without paying for it, you're also taking up our devops resources.Not cool at all!”

  • "If you want to talk to usContent Licensing and Commercial UseIf you do, we’re right here.”


iFixit's Terms of Use policy states that "copying, reproduction or distribution of any content on the site is strictly prohibited without the company's prior express written permission," specifically including "Used to train machine learning or AI models”。

Wiens has since said iFixit has added a "crawl delay" extension to its robots.txt file.

However, iFixit is not the only victim. Read the Docs co-founder Eric Holscher and Freelancer.com CEO Matt Barrie said in Wiens' post that their websites were also frequently crawled by Anthropic's crawlers.

These actions are not ClaudeBot's first offense. Back in April, the Linux Mint website forum attributed a site outage to the stress caused by ClaudeBot's crawling activities.