AI Crawler Checker
See whether GPTBot, ClaudeBot, Google-Extended, PerplexityBot, and 16 other AI and search crawlers may read your site, which robots.txt line decides it, and whether you have an llms.txt.
Enter a page address to check that page; a bare domain checks the home page.
Build the robots.txt rules
Tick the crawlers to block, then add the rules to your robots.txt. Crawlers you leave unticked keep following your existing rules.
Nothing is blocked. Tick a crawler or pick a preset.
Blocking a search engine crawler removes your pages from that search engine, not just from its AI answers.
robots.txt is a request, not a lock: well-behaved crawlers follow it, and some user-request fetchers don't. Rules take up to a day to be picked up.
AI crawlers and what they do
From each company's own documentation, checked October 15, 2026.
AI training
Collect pages to train AI models. Blocking them does not remove you from AI answers.
- GPTBot OpenAI
- Collects pages that may be used to train OpenAI's models.
- ClaudeBot Anthropic
- Collects pages that may be used to train Anthropic's Claude models.
- Google-Extended Google
- Controls whether Gemini models may train on and ground answers in your pages. It is a robots.txt token, not a separate crawler, so blocking it does not affect Google Search.
- Applebot-Extended Apple
- Controls whether Apple may train its AI models on pages Applebot crawled. A robots.txt token only; Applebot still crawls for search.
- Meta-ExternalAgent Meta
- Collects pages for Meta's AI, including training its models.
- Amazonbot Amazon
- Crawls for Amazon products such as Alexa; Amazon says the data may be used to train its AI models.
- CCBot Common Crawl
- Builds the free Common Crawl web archive, one of the most widely used sources of AI training data.
- Bytespider ByteDance
- Collects pages for ByteDance, the owner of TikTok, including its AI models.
AI search
Index pages so assistants such as ChatGPT and Claude can find, quote, and link to you.
- OAI-SearchBot OpenAI
- Finds pages to show and link in ChatGPT search answers. OpenAI recommends allowing it if you want to appear there.
- Claude-SearchBot Anthropic
- Indexes pages to improve the search results Claude gives its users.
- PerplexityBot Perplexity
- Indexes pages so Perplexity can show and link them in its answers. It is not used to train models.
- DuckAssistBot DuckDuckGo
- Fetches pages for DuckDuckGo's AI-assisted answers. DuckDuckGo says it is not used to train models.
User requests
Fetch a single page when someone asks an assistant to read it.
- ChatGPT-User OpenAI
- Visits a page when someone asks ChatGPT or a custom GPT to. OpenAI says robots.txt rules may not apply to these visits.
- Claude-User Anthropic
- Visits a page when someone asks Claude a question. Blocking it can make you less visible in answers people ask for.
- Perplexity-User Perplexity
- Visits a page when someone asks Perplexity to. Perplexity says it generally ignores robots.txt for these visits.
- Meta-ExternalFetcher Meta
- Fetches a link when someone asks Meta AI to. Meta says it may bypass robots.txt.
- MistralAI-User Mistral AI
- Visits a page when someone asks Mistral's Le Chat to.
Search engines
Classic search crawlers. Their AI answers, such as Google AI Overviews, use the same crawl.
- Googlebot Google
- Crawls for Google Search, including AI Overviews and AI Mode, which cannot be blocked separately.
- Bingbot Microsoft
- Crawls for Bing, which also supplies the web results in Microsoft Copilot.
- Applebot Apple
- Crawls for Siri, Spotlight, and Safari suggestions.
About the AI Crawler Checker
AI companies send different crawlers for different jobs: some collect pages to train models, some index pages so assistants can cite you, and some fetch a page only when a person asks. Enter your site to see which of them your robots.txt allows, the exact line that decides each one, and whether you have an llms.txt. Then build the rules to block only what you want.
How to use it
- 1 Enter your domain, or a full page address to check one page.
- 2 See each crawler marked Allowed or Blocked, with the robots.txt line that decided it.
- 3 Tick the crawlers to block, or pick a preset, and copy the rules into your robots.txt.
Frequently asked questions
Should I block AI crawlers?
It depends on what you want. Blocking training crawlers such as GPTBot, ClaudeBot, and Google-Extended keeps your pages out of future training data but does not stop AI assistants from finding and citing you. Blocking AI search crawlers such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot can remove you from AI answers, which may cost you visits.
Does blocking Google-Extended remove my site from Google?
No. Google-Extended only controls whether Gemini may train on and ground answers in your pages. Google Search, including AI Overviews, uses Googlebot, so blocking Google-Extended does not change your search rankings.
Do AI crawlers obey robots.txt?
The major training and search crawlers say they do. Fetchers that act on a person's request work differently: OpenAI says robots.txt may not apply to ChatGPT-User, Perplexity says Perplexity-User generally ignores it, and Meta says Meta-ExternalFetcher may bypass it. To truly stop a crawler, block it at your firewall or CDN.
The checker says allowed, but AI tools still cannot read my site. Why?
robots.txt is only one layer. A firewall or CDN, such as Cloudflare's AI bot blocking, can refuse crawlers before they ever see robots.txt, and pages that need JavaScript to show their text can look empty to crawlers. This tool reads robots.txt only, so check your CDN settings too.
What happens if my robots.txt is missing or broken?
If /robots.txt returns 404, crawlers may fetch every page. If it returns a server error or times out, well-behaved crawlers treat the whole site as off limits until it loads again, which can quietly drop you from search and AI answers.
What are content signals?
Content-Signal is a newer robots.txt line, used by Cloudflare's managed robots.txt, that says how content may be used after it is fetched: search, ai-input for AI answers, and ai-train for training. The checker shows any it finds. Crawlers are not required to follow them.
Related tools
AI Output Cleaner
Clean ChatGPT and Claude text for Word, email, and WhatsApp
AI Token & Cost Calculator
Count tokens and compare GPT, Claude, and Gemini API costs
JSON Repair & LLM Output Validator
Fix broken JSON from ChatGPT, Claude, or Gemini and check it against a schema
llms.txt Generator
Create an llms.txt file that guides AI tools to your best pages
Speech to Text & Subtitle Maker
Transcribe audio and video to text and SRT subtitles in your browser
ChatGPT Table to Excel & CSV
Turn tables from ChatGPT, Claude, or Gemini into Excel and CSV files