SOURCE-LINKED FIELD GUIDE

AI crawler robots.txt:
search is not training.

A single “allow AI” rule hides several different decisions. Use the provider-specific controls for search discovery, model development, and user-triggered retrieval—then measure actual fetching separately.

Sources reviewed August 29, 20269 minute read

01

Map each provider

  1. 01

    OpenAI

    OAI-SearchBot supports ChatGPT search discovery. GPTBot is the separate crawler for content that may help improve foundation models. ChatGPT-User is user-triggered, and OpenAI says robots.txt rules may not apply to those requests.

  2. 02

    Anthropic

    Claude-SearchBot supports search-result quality. ClaudeBot supports model development. Claude-User retrieves pages when a user asks Claude to access them. Anthropic documents all three separately.

  3. 03

    Perplexity

    PerplexityBot supports search results. Perplexity-User is a user-triggered fetcher; Perplexity says it generally ignores robots.txt because the request was initiated by a user.

  4. 04

    Google

    Googlebot crawls for Google Search. Google-Extended is a robots.txt control token for Gemini training and grounding; it is not a separate HTTP user agent and does not affect Google Search inclusion or ranking.

02 · STARTING POLICY

Allow search.
Block training.

This example expresses one common policy, not a universal recommendation. Merge it with your existing rules, preserve any deliberate path restrictions, and re-check current provider documentation before publishing.

# Allow documented automatic search crawlers
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Googlebot
Allow: /

# Block documented model-development controls
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

03

Implement without guessing

  1. 01

    Audit the effective groups

    The most specific matching group and longest matching path determine access. Do not append a broad block without checking how it interacts with existing agent-specific rules.

  2. 02

    Test the public response

    Confirm that `/robots.txt` returns HTTP 200 as plain text, then evaluate the exact homepage path against each automatic search crawler. AnswerReady's free checker performs that bounded test.

  3. 03

    Verify requests in server logs

    Robots.txt expresses permission; it does not prove that a crawler visited. Validate user agents and published IP ranges where available before attributing requests.

  4. 04

    Keep user-triggered fetches separate

    Some user-initiated agents may not follow the same robots behavior as automatic crawlers. Treat them as a separate product and security decision.

PRIMARY SOURCES

Check the current rules

The companion public reference repository also provides CSV, JSON, and two complete robots.txt examples.

OpenAI crawlers

Current purpose, robots.txt behavior, and published IP information for OAI-SearchBot, GPTBot, and ChatGPT-User.

OpenAI documentation →Anthropic crawlers

Separate documentation for ClaudeBot, Claude-SearchBot, and Claude-User.

Anthropic documentation →Perplexity crawlers

Purpose and robots behavior for PerplexityBot and Perplexity-User.

Perplexity documentation →Google crawlers

Googlebot details and the Google-Extended product-token distinction.

Google documentation →

CHECK YOUR CURRENT POLICY

See which search crawlers
can fetch your homepage.

The free checker reads the public robots.txt response and explains each result. If your five revenue pages need clearer answers and evidence after access is fixed, the AnswerReady Fix Pack is $69 once.

Run the free crawler check →Start the $69 Fix Pack