AI Crawlers and robots.txt: What to Allow If You Want to Be Cited

When AI companies started crawling the web at scale, plenty of sites responded with a blanket rule: block every bot with “AI” or “GPT” in its name. That was a reasonable reaction at the time. The trouble is that the same robots.txt file now also decides whether ChatGPT search, Claude and Perplexity can fetch your pages when someone asks a question you could answer. A rule written to keep content out of training data can quietly keep a brand out of AI answers as well.

The vendors have since split their crawlers by purpose, so you no longer have to choose between all or nothing. This guide lists the tokens that matter, what blocking each one actually does according to the vendors’ own documentation, and the mistakes we see most often in audits.

Three kinds of AI bot

Most vendors now run up to three separate agents:

• Search crawlers build the index the assistant searches when it answers. Block these and you are unlikely to be cited.
• Training crawlers collect content that may be used to train future models. Blocking them is a signal about training only.
• User-initiated fetchers visit a page because a person asked the assistant to look at it. Several vendors say robots.txt may not apply to these, since a human triggered the request.

The tokens, vendor by vendor

Two details catch people out. First, Google-Extended has no user agent of its own: Google crawls with its normal agents and reads the Google-Extended group as an instruction about use, not access. Second, blocking Google-Extended does not keep you out of AI Overviews. If you want to limit what Google shows from a page in Search, Google points to nosnippet, data-nosnippet, max-snippet and noindex instead.

A sensible default

For most commercial sites that want to be recommended by assistants but would rather not contribute to model training, a setup like this works:

# Search and answer engines: allowed
User-agent: Googlebot
User-agent: Bingbot
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

# Model training: opted out
User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /

# Everyone else
User-agent: *
Disallow: /account/
Disallow: /api/

Sitemap: https://www.example.com/sitemap.xml

Under RFC 9309 a crawler obeys the group that names it and falls back to the User-agent: * group only when no group does. So the named bots above ignore the catch-all rules entirely. If there are paths every bot should stay out of, repeat those lines inside each named group. Whether you block the training crawlers is a business decision; it does not affect the search crawlers.

Mistakes that block citations without anyone noticing

The firewall says no even when robots.txt says yes

Bot-management features at the CDN or WAF often challenge or block unfamiliar crawlers by default. Google’s own checklist for AI features asks you to make sure crawling is allowed “in robots.txt, and by any CDN or hosting infrastructure”, and Perplexity publishes WAF rules and IP lists for exactly this reason. Check your server logs for the search bots above and look at the status codes they receive. A wall of 403s means the robots.txt file is not the problem.

The content only exists after JavaScript runs

In a study of traffic across Vercel’s network published in December 2024, Vercel and MERJ found that none of the major AI crawlers they measured rendered JavaScript, including OpenAI’s and Anthropic’s. Crawler behavior can change, but if your product descriptions, prices or answers are injected client-side, assume an AI crawler sees an empty shell. Server-render or pre-render the main content, then check by fetching the raw HTML withcurl and searching it for a sentence you care about.

Subdomains with their own rules

Each host has its own robots.txt. A permissive file on www does nothing for docs. or help., which are often the pages with the most quotable answers. Anthropic’s documentation explicitly asks site owners to set rules for every subdomain they want covered.

Expecting changes to apply instantly

OpenAI says it can take about 24 hours for a robots.txt change to reach its search systems, and Perplexity gives a similar figure. Allow a few days before you conclude a change did nothing, and longer before you expect to see it in answers.

robots.txt is a request, not a lock

RFC 9309, the standard that defines robots.txt, says plainly that its rules “are not a form of access authorization”, and that the protocol “is not a substitute for valid content security measures”. Listing a private path in robots.txt even advertises that it exists. Reputable crawlers honor the file; nothing forces the rest to, and user-initiated fetchers are treated differently by design.

So if a document must never reach a third-party model, robots.txt is the wrong tool. Put it behind authentication, or do not publish it. Many organizations hit the same question from the other side: staff want an assistant that can work with internal contracts, records or source code, and that data cannot go to a public model at all. That is an infrastructure decision rather than an SEO one. Our parent company, AsymiLink Meta, covers the options on its page about offline and on-premise AI deployment, where the model runs inside your own network.

How to check your own site

1. Open /robots.txt on every host you care about and read the groups for the search tokens above.
2. Test a few important URLs against each token with a robots.txt parser that follows RFC 9309 matching rules.
3. Search your access logs for those user agents and confirm they get 200 responses, not 403s or challenges.
4. Fetch your key pages without JavaScript and confirm the main text is in the HTML.
5. Look for noindex or nosnippet on pages you expected to be quoted.

The Site Audit in AsymiLink SEO covers the first, second and fourth checks for you (your server logs are still yours to read). It parses robots.txt with RFC 9309 rules, reports whether Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot and PerplexityBot can crawl the site, flags a Google-Extended block separately (because it does not affect Search), and warns when the main content is missing from the server HTML. Once access is sorted, measuring how often AI answers mention and cite you tells you whether it made a difference.