robots.txt and AI crawlers: which bots to allow and how to check

robots.txt is a short text file at the root of your site that tells crawlers what they may fetch. For years most teams wrote it once and forgot it. AI assistants changed that: several of the crawlers that feed ChatGPT, Gemini and Claude read robots.txt, and a line written for a different purpose can quietly keep your pages out of AI answers.

This guide covers the four AI crawlers we check in the free AI SEO audit, what each one is for, which to allow, copyable snippets for common policies, and how to confirm what your site is actually doing.

The four AI crawlers to know

A few things in that table are worth reading twice.

OpenAI uses two separate crawlers. OAI-SearchBot is about search: when ChatGPT looks something up on the web to answer a question, this is the crawler connected to those results. GPTBot is about training future models. You can allow one and block the other.

ClaudeBot does both jobs. For Anthropic, the same user agent covers model training and Claude web results. There's no way, using this user agent alone, to allow one and block the other.

Google-Extended is about Gemini, not Google Search. It's the robots.txt user agent that tells Google whether your content may be used for Gemini training and grounding. It does not affect how your site ranks in Google Search. Blocking it won't hurt your search rankings; it will limit how Gemini can use your pages.

Which ones should you allow?

For most businesses that want to be recommended by AI assistants, the short answer is: allow all four. Being found and quoted is the point, and blocking crawlers removes your pages from the pool answers are built from.

There are good reasons to choose differently, and it's your call:

• You sell content. Publishers, course creators and data businesses may not want their work used for training. Blocking GPTBot while allowing OAI-SearchBot keeps you eligible for ChatGPT search results without contributing to training.
• You have private or sensitive sections. Block those paths for all crawlers, not just AI ones. Better still, put them behind a login, because robots.txt isn't access control.
• Server load. If a crawler is hitting your site too hard, limit the paths it can fetch rather than blocking it from everything.

The thing to avoid is blocking by accident. In our experience that's far more common than a deliberate choice: a staging file copied to production, a template from a forum, or a security plugin that added rules nobody reviewed.

How robots.txt rules work

A robots.txt file is made of groups. Each group starts with one or more User-agent lines and is followed by Allow and Disallow rules. A crawler looks for the group that names it most specifically and follows only that group.

That last part catches people out. If your file has a group for User-agent: * with careful rules, and a separate group for ClaudeBot, then ClaudeBot follows only its own group and ignores everything under the * group. If its own group says "Allow: /", it can fetch paths you thought you'd blocked for everyone.

A few other rules of thumb:

• Paths are prefixes. "Disallow: /admin" blocks /admin, /admin/settings and /administrator too.
• The longest matching rule wins when Allow and Disallow both match a path.
• An empty Disallow ("Disallow:" with nothing after it) means nothing is blocked.
• No robots.txt at all means every crawler is allowed by default.

Copyable snippets

Each snippet below is the full content of a robots.txt file. Each list item is one line. Replace example.com with your domain.

Allow everything, including AI crawlers

This is the simplest policy and the right one for most sites that want to be recommended.

User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml

Every crawler, AI or not, follows the * group because none has its own.

Allow AI search, block training

Stay eligible for ChatGPT search results while opting out of OpenAI training and Gemini training and grounding.

User-agent: *
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Sitemap: https://example.com/sitemap.xml

Note what this does to Gemini: Google-Extended covers grounding as well as training, so blocking it also limits how Gemini can use your pages in answers. Whether that's what you want depends on why you're blocking. ClaudeBot isn't listed here, so it follows the * group and is allowed; because it covers both training and Claude web results, you can't split those two for Anthropic.

Allow AI crawlers but keep private paths out

User-agent: *
Disallow: /app/
Disallow: /admin/
Allow: /
Sitemap: https://example.com/sitemap.xml

Because none of the AI crawlers has its own group, they all follow these rules. If you later add a group for one of them, copy the Disallow lines into it, or they won't apply.

The mistake to look for

User-agent: *
Disallow: /

Two lines that block every well-behaved crawler from your whole site. It's often left over from a staging environment. If you see this on your live site and didn't mean it, fix it today.

How to check your site

There are three places your crawler policy lives, and they have to agree.

1. Read your robots.txt

Open yoursite.com/robots.txt in a browser. Check that:

1. It loads as plain text, not an error page, a login page or a redirect to your homepage.
2. You can find each of the four user agents above, or you know they fall through to the * group.
3. The group each AI crawler follows allows the pages you want quoted: homepage, pricing, product pages, comparison pages, docs and blog.

The free AI SEO audit does this for you. It fetches your robots.txt, works out which group each of the four crawlers follows, and tells you whether each can reach your homepage and whether that looks deliberate.

2. Check your CDN, firewall and plugins

Some CDNs, firewalls and website plugins can block bots before robots.txt is even read, sometimes with a single switch labelled something like "block AI bots". If that switch is on, your robots.txt can say "Allow" and the crawler still gets an error. Look through your hosting, CDN and security settings for bot management and make sure they match your intended policy.

3. Look at your server logs

If you have access to logs, search for the user agent names. Requests from OAI-SearchBot, GPTBot or ClaudeBot that return 200 mean they're getting through. Requests that return 403 or 429 point to a firewall or rate limit. No requests at all for weeks can mean you're blocked upstream, or simply that you haven't been crawled yet.

After you change it

Changes to robots.txt take effect when crawlers next fetch the file, but the effect on AI answers isn't immediate. Pages need to be fetched again, and answers built from them change gradually. Don't expect a difference the next day.

The useful thing to do is record the date of the change and watch your AI visibility over the following weeks. If your pages start appearing among the cited sources, and your score moves on the questions those pages answer, the change worked. If nothing moves after a month or two, robots.txt probably wasn't what was holding you back, and it's time to look at the sources engines cite instead. Which websites AI engines cite is the next step.

Quick checklist

• robots.txt loads as plain text at the root of your domain.
• OAI-SearchBot can reach your homepage and key pages.
• GPTBot, ClaudeBot and Google-Extended are allowed or blocked on purpose.
• Any bot with its own group has the Disallow lines it needs.
• Your CDN, firewall and plugins don't override robots.txt.
• Your sitemap is listed.
• You've written down the date of any change.

Run the AI SEO audit to check the first four in seconds, then the free AI visibility check to see whether ChatGPT and Claude name you today.