How do you stop AI crawlers from scraping your website?

The blocks that actually exist today

Three vendors publish training-specific robots.txt tokens, and those are the ones worth knowing by heart. OpenAI's GPTBot covers content that may be used in training its generative AI foundation models. Anthropic's ClaudeBot covers content that could potentially contribute to training. Google's Google-Extended covers whether crawled content may be used for training future Gemini models and for grounding.

A full opt-out from training across those three is short. Add a group per token with User-agent set to the token name and Disallow set to a single forward slash, at the root of every domain and subdomain you want covered. Anthropic's help page is explicit that you must do this for every subdomain you wish to opt out from — a robots.txt on the apex does not cover a docs or blog subdomain.

One quirk to know about Google-Extended: it has no separate HTTP request user agent string. Google's docs say the crawling is done with existing Google user agent strings and the robots.txt token is used purely in a control capacity. You will not find Google-Extended in your access logs, so do not conclude from an empty log that the rule failed.

Blocking training does not remove you from AI search

This is the part most people get wrong, and all three vendors are unusually clear about it. OpenAI states each setting is independent and gives the exact scenario: a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training. Google states plainly that Google-Extended does not impact a site's inclusion in Google Search and is not used as a ranking signal. Anthropic separates ClaudeBot from Claude-SearchBot for the same reason.

So the choice is not all-or-nothing. If your position is that you want to be citable but not trainable, that is a supported configuration: block GPTBot, ClaudeBot, and Google-Extended, and leave OAI-SearchBot and Claude-SearchBot allowed.

The reverse has a cost worth naming. OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. Anthropic says disabling Claude-SearchBot prevents indexing for search optimisation and may reduce visibility and accuracy in user search results. Blocking the search bots is a real visibility decision, not a free precaution.

User-triggered fetches are the gap in the fence

There is a third category besides training crawlers and search crawlers: fetches a person asked for. OpenAI's ChatGPT-User visits a page when a user asks ChatGPT a question about it. Anthropic's Claude-User does the same. Google's documentation classifies these separately as user-triggered fetchers, distinct from common crawlers.

OpenAI states the consequence directly: because these actions are initiated by a user, robots.txt rules may not apply. That is the honest description of the boundary. robots.txt is a convention for automatic crawling, and a person pasting your URL into a chat window is not automatic crawling.

Anthropic frames Claude-User as something site owners can control, and lists it alongside the other two bots with the same opt-out mechanism. The practical takeaway is that vendor behaviour differs here and you should read the specific page rather than assume one policy generalises. If your requirement is that no AI system ever fetches a given page on anyone's behalf, robots.txt is not the right tool and you need authentication.

Why robots.txt is a request, not a lock

The Robots Exclusion Protocol was standardised as RFC 9309 in September 2022, a Standards Track document that formalised the convention Martijn Koster defined in 1994 and added definition language, error handling, and caching rules. Standardisation made the syntax unambiguous. It did not make compliance mandatory, because nothing in the protocol can.

That means a robots.txt block works exactly as well as the crawler's willingness to honour it. The major vendors state that they do. Anthropic goes further and says its bots respect anti-circumvention technologies and will not attempt to bypass CAPTCHAs. Google says its common crawlers always respect robots.txt rules for automatic crawls, while noting the special case that AdsBot ignores the global user agent group with the ad publisher's permission — a useful reminder to read what each token actually promises.

Also budget for propagation. OpenAI notes it can take roughly 24 hours from a robots.txt update for its search systems to adjust. A change you made this morning is not evidence about behaviour this afternoon.

The long tail nobody maintains by hand

Three vendor tokens are the easy part. The Dark Visitors directory, a third-party listing rather than a primary source, currently paginates its known-agent catalogue across dozens of pages, spanning browser-driving agents, assistant fetchers, and third-party data providers that crawl on behalf of other companies. Names in that last category — commercial extraction and search-index services — are the ones most site owners have never added to a robots.txt file.

Keeping a hand-written allowlist current against that surface is not realistic for most sites. This is where a managed list or an edge control is a reasonable answer rather than a lazy one.

Cloudflare's AI Crawl Control, described in its docs as formerly AI Audit and available on all plans, is one such option: it reports which AI services are accessing content, allows per-crawler allow or block rules, tracks which crawlers actually follow your robots.txt directives, and lists a pay-per-crawl model as private beta. We have not tested it. It is cited here as a concrete example of the category — enforcement at the edge — not as a recommendation over any competitor.

Limits and things this cannot do

Blocking a crawler today does nothing about content collected before today. No vendor mechanism described here is retroactive, and none of them can remove content from a model that has already been trained. Anthropic frames its opt-out as signalling that the site's future materials should be excluded. Treat any block as forward-looking only.

IP blocking is worse than it looks. Anthropic's page explicitly warns that blocking its IP addresses may not work correctly or persistently guarantee an opt-out, because doing so impedes their ability to read your robots.txt file — meaning you can accidentally break the mechanism you were trying to enforce. Both OpenAI and Anthropic publish IP lists, which are better used for verifying that a request claiming to be their bot really is, than as a blocklist.

Verification is the one place you have solid ground. Google documents that its crawlers identify themselves by user-agent header, source IP, and reverse DNS hostname, and that all three can be used to confirm a crawler is genuine. Since a user-agent string is trivially forged, verify before you act on log evidence.

A useful next action

Decide the policy question first, because the syntax is ten minutes and the policy is the part that has consequences: do you want to be cited by AI search, trained on, both, or neither? Those are four different robots.txt files.

Then write the groups for GPTBot, ClaudeBot, and Google-Extended, deploy the file to every subdomain rather than just the apex, and wait a day before checking anything. After that, pull your access logs and look at what is actually hitting the site, verifying any bot that claims a known identity against the vendor's published IP list before you conclude anything from the user-agent string alone.