A page loading in your browser does not prove a search crawler can read it. Equally, a bot-looking user-agent in a log does not prove the request came from the company it names.
Choose the purpose before the bot
| Provider | Search discovery | Training | User-triggered retrieval |
|---|---|---|---|
| OpenAI | OAI-SearchBot | GPTBot | ChatGPT-User |
| Anthropic | Claude-SearchBot | ClaudeBot | Claude-User |
| Perplexity | PerplexityBot | Its published search bot is not described as foundation-model training | Perplexity-User |
These are documented roles, not interchangeable access switches. User-triggered fetchers may behave differently from automated crawlers; do not treat robots.txt as authentication or a secrecy control.
Check the page in layers
- Review the effective robots.txt rules for the specific agent and path. Inspect existing rules before adding a new group.
- Request the page and review status, redirects and headers.
- Check whether the useful content is present in the returned HTML, rather than hidden behind a login or interaction.
- Inspect CDN and WAF events for challenges, blocks and rate limits.
- Validate actual bot identity using the provider’s current verification guidance and published network information where available.
Diagnostic command
curl -sS -L -D headers.txt -o page.html \
-A 'OAI-SearchBot' https://example.com/service/This simulates a user-agent string from your network. It does not prove that a genuine OpenAI request will pass IP-based controls or receive identical content.
Avoid the common false fix
Allowing GPTBot does not substitute for reviewing OAI-SearchBot access. Similarly, allowing ClaudeBot is a separate training decision from allowing Claude-SearchBot.
Protect private customer content with actual access controls. For public discovery, review only the paths and agents consistent with your publishing policy. Do not broadly disable security based on a user-agent alone.
Frequently Asked Questions
No. It is one access observation. Discovery, processing and answer selection are additional steps.
No. It expresses crawler preferences; it is not authorization.
Next step
Run a quick public readiness check, then investigate anything it flags with your technical team.
Run Buzzlect’s public readiness check →
Related guides
- How ChatGPT, Perplexity and Google AI Choose Which Sources to Cite (and How to Explain It to Clients)
- Why Your B2B Brand Isn’t Appearing in ChatGPT Recommendations
- Google’s AI Content Guidelines: What B2B SEO Teams Must Know
- Entity Optimization for AI Search: A Packageable Service for Agencies
- How B2B Companies Can Measure Visibility in ChatGPT and Google AI
Sources and verification notes
The steps, examples and templates in this article are practical recommendations, not platform rules, survey data or measured client results. Hypothetical figures and examples are labelled as such. The sources below support the platform and technical facts only.
- OpenAI: Overview of OpenAI crawlers
- Perplexity: Perplexity crawlers
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
Want to see how your brand ranks in ChatGPT right now?
Run a free AI Visibility Audit to see where your business appears across ChatGPT, Gemini and Perplexity—and where competitors are being recommended instead.
Get Your Free AI Visibility Audit