# Robots.txt for AI Crawlers

Canonical: https://llmcitely.com/guides/robots-txt-for-ai-crawlers
Publisher: LLM Citely
Reviewed: 2026-10-30

A robots rule has a purpose, a user agent and a path. Work through those three pieces before deciding that a change allows search, limits training or protects information.

![A robots example blocks GPTBot while allowing OAI-SearchBot on the guides path.](https://llmcitely.com/images/editorial/ai-robots-purpose-rules.svg)

Illustrative supplied rules. A rule decision does not test a firewall, response header, authentication or an actual provider visit.

## Choose the control before the rule

Suppose a site owner wants public guide pages available to search while expressing a preference against training use. A broad “block AI” rule does not describe that goal well enough. Start by writing the purpose in ordinary language, then look up the agent that performs it.

OpenAI's [crawler documentation](https://developers.openai.com/api/docs/bots) distinguishes OAI-SearchBot for search from GPTBot, whose crawling may be used for foundation-model training. It describes ChatGPT-User as certain user-initiated actions, with different behavior from automatic crawling. The choice of one control should not be presented as the choice of all three.

Use a short mapping on the audit sheet: desired purpose, documented agent, affected paths and intended change. This makes a future review possible even if the provider changes its agent descriptions.

## Read a small rule set before copying it

This is the exact illustrative file loaded by the crawler checker’s “Load illustrative rules” button and used in the screenshot. The named search group permits the /guides/ path and blocks other paths; GPTBot is blocked separately. It is a teaching fixture, not a configuration to deploy unchanged.

```text
User-agent: *
Disallow: /private/

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /guides/
Disallow: /
```

Read each group separately. The OAI-SearchBot group does not inherit the wildcard group’s /private/ rule; its own Disallow: / is what blocks paths outside /guides/. This file says nothing about actual server responses or private-data authorization.

Before adapting it, decide which public paths should be searchable and test them individually. A production site that also wants public product pages discovered would need different search-group rules.

## Test a path, not the word “allowed”

Open the [AI Crawler Checker](https://llmcitely.com/tools/ai-crawler-checker), choose “Load illustrative rules”, and run the check for https://example.com/guides/analytics. With the file above, OAI-SearchBot is permitted by line 8, Allow: /guides/. GPTBot is blocked by line 5, Disallow: /. These are the inputs and decisions shown in the screenshot.

Change only the page URL to https://example.com/product and run it again: OAI-SearchBot is blocked by line 9, Disallow: /. A path under /private/ is blocked as well. Keep blank lines as shown if you want the same line numbers.

The expected rule decision and the observed server response belong in different columns. A parser explains a supplied directive; it does not establish that a request reached the host or passed access controls.

Google documents [its own group and path matching](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec). Use the relevant provider’s documentation for another agent’s behavior, and leave unimplemented provider-specific behavior for documented review.

![The actual rule checker permits OAI-SearchBot for the guides path and blocks GPTBot with the selected rules.](https://llmcitely.com/images/editorial/ai-crawler-rule-results.png)

The working rule checker applied to the supplied example. The winning line is shown for each token; this is not a live provider access test.

## Use the right mechanism for indexing and private content

A robots file is public. It describes crawl preferences; it is not a lock on a client report. A private report needs authentication and authorization even when its path appears in a Disallow rule.

Indexing has a different interaction. Google can discover a URL through links even when crawling is disallowed. If you rely on a noindex directive, Google needs to be able to obtain that directive. [Google's robots introduction](https://developers.google.com/search/docs/crawling-indexing/robots/intro) and [noindex guide](https://developers.google.com/search/docs/crawling-indexing/block-indexing) explain the distinction.

For a public page that should be discoverable, investigate an unexpected noindex or access restriction. For private data, keep access controls intact. The desired outcome comes from the page's purpose, not from making every checker row green.

## Check what the host actually returns

Once the rule decision makes sense, inspect the real host separately. Get the robots file from the correct scheme and hostname, then test the intended public URL using an appropriate response tool. Save a redirect, challenge or error as evidence rather than summarizing it as “AI blocked.”

If logs show requests under a crawler name, use the provider's published verification method where available. A name in a user-agent string alone is not enough to establish who made the request.

This step is often where an apparently correct file meets a different deployment setting. A CDN rule, authentication boundary or unavailable resource can prevent the intended access even when the supplied robots rule permits the path. The [HTML readiness check](https://llmcitely.com/tools/ai-readiness-checker) covers a different part of the page review.

## Leave a change someone else can explain

Save the previous file, the purpose of the change, the affected group and representative test paths. Then write down two results: the rule decision from the file, and the actual response if one was obtained.

For example, “The supplied OAI-SearchBot group permits the public path” is a narrower statement than “ChatGPT can crawl and cite the page.” The second would need additional access and answer evidence.

Include this record in a [GEO audit](https://llmcitely.com/guides/run-a-geo-audit). Use the [measurement method](https://llmcitely.com/methodology) when turning an access observation into a broader client report.

## Questions, answered

### Will blocking GPTBot also block ChatGPT search?

OpenAI documents separate settings for GPTBot and OAI-SearchBot. Review both groups and any host-level rules rather than assuming one setting controls every surface.

### Does allowing a crawler guarantee it will cite the page?

No. A crawl preference is only one part of access and eligibility; it does not guarantee retrieval, selection or citation.

### Can robots.txt protect a private client report?

No. Use authentication and authorization for private content. Do not rely on a public crawl preference to prevent access.

## Sources and further reading

- [Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots)
- [Introduction to robots.txt](https://developers.google.com/search/docs/crawling-indexing/robots/intro)
- [How Google interprets the robots.txt specification](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec)
- [Block Search indexing with noindex](https://developers.google.com/search/docs/crawling-indexing/block-indexing)

Examples marked illustrative are not customer measurements.

## Continue with the next step

- [Check a specific crawler rule](https://llmcitely.com/tools/ai-crawler-checker)
- [Inspect other page checks](https://llmcitely.com/tools/ai-readiness-checker)
- [Review measurement boundaries](https://llmcitely.com/methodology)
- [Include rules in an audit](https://llmcitely.com/guides/run-a-geo-audit)
- [Review a separate llms.txt navigation file](https://llmcitely.com/tools/llms-txt-validator)

## Your next step

[Check a specific crawler rule](https://llmcitely.com/tools/ai-crawler-checker)