Official-source reference
AI crawler access controls for answer visibility
For answer visibility, allow the provider's search or retrieval agent, not automatically its training crawler. OpenAI, Anthropic, Google, and Perplexity publish distinct controls. As of this review, xAI and X do not publish an official Grok crawler token in the documentation we found.
Robots.txt creates or removes crawl permission. It does not guarantee indexing, ranking, a mention, or a citation. A CDN, WAF, login wall, noindex rule, canonical conflict, or server error can still make an allowed page unusable.
Answer-first matrix
Which agent controls which use
| Provider | Search or answer access | User-requested fetch | Training control | Verification |
|---|---|---|---|---|
| OpenAI | Allow OAI-SearchBot for ChatGPT search eligibility. | ChatGPT-User handles some user-requested visits. Robots.txt may not apply. | GPTBot is the separate foundation-model training control. | Check the official OAI-SearchBot IP list or a verified-bot signal. |
| Anthropic | Allow Claude-SearchBot for search indexing and Claude-User for user-requested retrieval. | Claude-User is the named user-initiated agent and Anthropic says it honors robots.txt. | ClaudeBot is the separate potential model-training control. | Check the official Anthropic crawler IP-prefix JSON. |
| Allow Googlebot and keep the page indexed and snippet-eligible for Google AI Overviews and AI Mode. | Google-Extended controls grounding in Gemini Apps and Grounding with Google Search on Vertex AI. | Google-Extended also controls use for future Gemini model training. | Verify Googlebot traffic with Google's published methods. Google-Extended has no separate HTTP user agent. | |
| Perplexity | Allow PerplexityBot and traffic from its published IP ranges for Perplexity search visibility. | Perplexity-User handles user-requested visits and generally ignores robots.txt. | Perplexity says neither listed agent is used to collect content for foundation-model training. | Match the user agent and the corresponding official IP list at the WAF. |
| Grok | No official xAI crawler or robots.txt token was found in the official documentation reviewed. | Official X documentation confirms real-time web search, but does not identify its fetcher. | No provider-specific website training token was found in the reviewed xAI or X product documentation. | Do not trust a guessed token. Keep normal search access healthy and inspect logs until xAI publishes guidance. |
Starter policy
A merge-safe robots.txt pattern
If your existing User-agent: * group permits public pages and you have no more-specific group for those agents, the documented answer-access agents are already permitted and need no separate allow group. The pattern shows only optional choices to decline OpenAI and Anthropic foundation-model training. Keep your existing wildcard rules and test the final result on each host.
Do not paste a bot-specific Allow: /group into a policy with wildcard exclusions: a specific matching group does not inherit the wildcard group's disallows. If you must override a blanket block, copy every applicable exclusion into the specific group. Perplexity-User is not included because Perplexity says user-requested fetches generally ignore robots.txt. Grok is not included because no official Grok token was found.
# Keep your existing User-agent: * rules unchanged.
# Search and answer agents need no separate Allow group when that
# wildcard group permits the public pages and no more-specific group matches.
# Optional training choices. These do not control the search bots above.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /Provider details
What the official documentation actually says
OpenAI and ChatGPT
Recommended visibility setting: For ChatGPT search visibility, allow OAI-SearchBot and make sure your host or CDN accepts traffic from OpenAI's published searchbot IP ranges.
- OAI-SearchBot is the automatic search crawler. OpenAI says sites opted out of it will not be shown in ChatGPT search answers, though navigational links may still appear.
- GPTBot is separate. Disallowing GPTBot signals that content should not be used for training OpenAI's generative AI foundation models.
- ChatGPT-User supports certain user actions and is not the control for ChatGPT search inclusion. OpenAI says robots.txt rules may not apply to these user-initiated visits.
- OpenAI publishes separate IP-range JSON files for OAI-SearchBot, GPTBot, and ChatGPT-User. User-agent strings can be spoofed, so use the official ranges or a provider-verified bot signal when configuring a WAF.
- OpenAI says its systems may take about 24 hours to adjust after a robots.txt change. Eligibility still does not guarantee placement.
Anthropic and Claude
Recommended visibility setting: For Claude visibility, allow Claude-SearchBot for search quality and Claude-User for user-directed retrieval. Decide on ClaudeBot separately if you have a model-training preference.
- Claude-SearchBot navigates the web to improve the relevance and accuracy of Claude search responses. Anthropic says blocking it may reduce search visibility and accuracy.
- Claude-User is used when a person asks Claude to access a website. Anthropic says blocking it prevents retrieval in response to a user query and may reduce visibility in user-directed web search.
- ClaudeBot collects public web content that could potentially contribute to model training. It is independent from the two answer-access agents.
- Anthropic says its bots honor robots.txt and supports the non-standard Crawl-delay directive. It now publishes a JSON list of crawler IP prefixes for verification.
- Apply the policy on every subdomain you want covered. A rule on the apex host does not automatically govern another host's robots.txt file.
Google Search and Gemini
Recommended visibility setting: For Google AI Overviews and AI Mode, allow Googlebot and keep pages indexed and eligible to show a snippet. For grounding in Gemini Apps, also allow the Google-Extended control token.
- Google says there are no extra technical requirements for AI Overviews or AI Mode. A supporting page must be indexed, eligible for Google Search, and eligible to show a snippet.
- Googlebot robots.txt directives control crawling for Google Search, including its AI features. Noindex, nosnippet, data-nosnippet, and max-snippet control whether or how content can appear.
- Google-Extended is a separate robots.txt control for future Gemini model training and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI.
- Google-Extended does not affect inclusion or ranking in Google Search. It does not send a separate HTTP user-agent string, because Google crawls with existing user agents and applies the token as a control.
- Structured data should match visible text. Google says no special AI schema or AI text file is required for AI Overviews or AI Mode.
Perplexity
Recommended visibility setting: Allow PerplexityBot and its official IP ranges for search visibility. Do not assume that an Allow rule alone bypasses a CDN or WAF challenge.
- PerplexityBot gathers and indexes information for Perplexity search results. Perplexity recommends allowing the token and traffic from its published IP ranges.
- Perplexity says PerplexityBot is not used to crawl content for AI foundation models.
- Perplexity-User supports user actions and may fetch a page to answer a question and include a link. Perplexity says it is not a web crawler or training collector and generally ignores robots.txt because the request is user initiated.
- Perplexity publishes a separate JSON IP list for each agent and recommends combining user-agent matching with IP verification in WAF rules.
- Perplexity says crawler-setting changes may take up to 24 hours to reflect. Visibility and citation are still not guaranteed.
Documentation gap
Grok: no official crawler token found
X Help documentation reviewed on August 23, 2026 says Grok can conduct real-time web search. Neither that page nor the reviewed xAI Grok overview identifies an official crawler user agent, robots.txt token, published crawler IP range, or provider-specific webmaster control for that web search.
This is a report about the documentation we found, not proof that Grok never fetches pages directly. Until xAI publishes webmaster guidance, do not invent or rely on a guessed GrokBot rule. Keep pages accessible to established search crawlers, monitor verified server logs, and review new xAI documentation before adding a provider-specific allowlist.
Verification checklist
Test the whole access path
A correct robots.txt rule can coexist with an inaccessible page. Check the public path from discovery to rendered content before concluding that a provider ignored your preference.
- 01Fetch robots.txt without cookies and confirm it returns HTTP 200 on the exact host being tested.
- 02Evaluate the most-specific user-agent group and path rule for the target URL.
- 03Fetch the canonical page without login, cookies, or JavaScript challenges and confirm a successful status.
- 04Check page-level robots directives, X-Robots-Tag headers, canonical tags, and snippet restrictions.
- 05Confirm the page is linked from a crawlable hub and included in the XML sitemap with an honest lastmod.
- 06Review CDN and WAF logs. Verify claimed bots with official IP data or a provider-verified signal before allowlisting.
- 07Inspect rendered text. Important facts should appear in the HTML and structured data should match what visitors see.
- 08Record the test date and recheck official provider docs before making a permanent policy change.
Official sources
All sources below are first-party provider documentation. They were reviewed on . The Grok note is explicitly limited to what these official xAI and X sources published and what our official-domain search found.
- 1. OpenAIOverview of OpenAI Crawlers ↗
- 2. OpenAIPublishers and Developers FAQ ↗
- 3. AnthropicDoes Anthropic crawl data from the web? ↗
- 4. AnthropicPublished Anthropic crawler IP prefixes ↗
- 5. Google Search CentralAI features and your website ↗
- 6. GoogleGoogle's common crawlers ↗
- 7. PerplexityPerplexity Crawlers ↗
- 8. xAIWelcome to Grok ↗
- 9. X Help CenterAbout Grok ↗
After access, measure the answer
Crawl permission is one layer of local AI visibility. Use a fixed query set and keep the exact response, sources, date, and limitations together.