which AI bots can crawl your site?
robots.txt, your firewall and llms.txt, checked against 34 AI crawlers from openai, anthropic, google, perplexity, meta, apple and more. free, no signup.
we fetch your homepage (as a normal browser, then as GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot), /robots.txt and /llms.txt. that's it.
robots.txt says who's allowed.
dashi shows who actually shows up.
dashi spots 60+ AI crawlers in your real traffic, by vendor — GPTBot, ClaudeBot, PerplexityBot and the rest — shows which pages each one read, and keeps them out of your visitor numbers. right next to your revenue and live visitors, on a 3D globe you can leave open all day.
free for now.
AI crawlers, by vendor.
the 34 crawlers this checker tests, from 17 companies — each token, purpose and robots.txt behavior taken from the vendor's own documentation (checked october 2026).
- ai searchindexes pages so ai answers can find, cite and link you
- trainingcollects pages to train ai models
- user-triggeredopens one page when someone asks an ai about it
| robots.txt token | vendor | what it's for | robots.txt |
|---|---|---|---|
| OAI-SearchBot | ai searchchatgpt search | follows robots.txtvendor docs | |
| GPTBot | trainingopenai model training | follows robots.txtvendor docs | |
| ChatGPT-User | user-triggeredchatgpt reading a page someone asked about | may ignore robots.txt (user-triggered)vendor docs | |
| Claude-SearchBot | ai searchclaude's search results | follows robots.txtvendor docs | |
| ClaudeBot | traininganthropic model training | follows robots.txtvendor docs | |
| Claude-User | user-triggeredclaude reading a page someone asked about | follows robots.txtvendor docs | |
| Googlebot | ai searchgoogle search, incl. ai overviews and ai mode | follows robots.txtvendor docs | |
| Google-Extended | traininggemini training and grounding | robots.txt-only token — doesn't crawlvendor docs | |
| Google-Agent | user-triggeredgoogle's agents acting for a user | may ignore robots.txt (user-triggered)vendor docs | |
| Google-GeminiNotebook | user-triggeredgemini notebook (notebooklm) sources | may ignore robots.txt (user-triggered)vendor docs | |
| PerplexityBot | ai searchperplexity search results | follows robots.txtvendor docs | |
| Perplexity-User | user-triggeredperplexity reading a page someone asked about | may ignore robots.txt (user-triggered)vendor docs | |
| Bingbot | ai searchbing search, which copilot answers draw on | follows robots.txtvendor docs | |
| Applebot | ai searchspotlight, siri and safari search | follows robots.txtfollows your Googlebot rules when robots.txt doesn't name itvendor docs | |
| Applebot-Extended | trainingapple intelligence training | robots.txt-only token — doesn't crawlvendor docs | |
| meta-webindexer | ai searchmeta ai answers and citations | follows robots.txtvendor docs | |
| meta-externalagent | trainingmeta ai training and indexing | follows robots.txtvendor docs | |
| meta-externalfetcher | user-triggeredmeta ai fetching a link for a user | may ignore robots.txt (user-triggered)vendor docs | |
| Amzn-SearchBot |  Amazon | ai searchalexa and amazon search | follows robots.txtwhen robots.txt doesn't name it, follows the rules you give other search botsvendor docs |
| Amazonbot | Amazon | trainingamazon products, may train amazon ai | follows robots.txtvendor docs |
| Amzn-User | Amazon | user-triggeredalexa fetching a page for a user | may ignore robots.txt (user-triggered)vendor docs |
| CCBot | trainingan open web archive anyone can download | follows robots.txtvendor docs | |
| MistralAI-Index | ai searchle chat search results | follows robots.txtvendor docs | |
| MistralAI-Training | trainingmistral model training | follows robots.txtvendor docs | |
| MistralAI-User | user-triggeredle chat reading a page someone asked about | follows robots.txtvendor docs | |
| DuckAssistBot | ai searchduckduckgo ai-assisted answers | follows robots.txtvendor docs | |
| YouBot | ai searchyou.com search | follows robots.txtvendor docs | |
| Kimi-SearchBot | ai searchkimi search results | follows robots.txtvendor docs | |
| KimiBot | trainingkimi model training | follows robots.txtvendor docs | |
| Kimi-User | user-triggeredkimi reading a page someone asked about | may ignore robots.txt (user-triggered)vendor docs | |
| AI2Bot | trainingopen language model training | robots.txt support not documentedvendor docs | |
| PetalBot | ai searchpetal search and huawei ai search | follows robots.txtvendor docs | |
| YandexAdditional | ai searchyandex ai answers | robots.txt-only token — doesn't crawlalso written YandexAdditionalBot. User-agent: * doesn't cover it — name it to opt outvendor docs | |
| Bytespider | trainingbytedance (no public docs) | robots.txt support not documentedno vendor docs |
how to allow or block AI crawlers.
robots.txt lives at the root of your domain (yoursite.com/robots.txt). a bot obeys only its most specific group, so a group naming a bot replaces the User-agent: * rules for it — repeat any paths you still want hidden.
keep AI search, opt out of training
the training-only tokens. ai search and answers keep working, because the vendors use separate crawlers for those.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: CCBot
User-agent: MistralAI-Training
User-agent: KimiBot
User-agent: AI2Bot
User-agent: Bytespider
Disallow: /block AI crawlers entirely
training, ai search and user-triggered bots. Googlebot and Bingbot aren't in it — blocking those removes you from google and bing completely. to keep a page out of google's ai answers, use nosnippet; for copilot, bing's noarchive or nocache meta tags.
User-agent: OAI-SearchBot
User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: ClaudeBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Applebot-Extended
User-agent: meta-webindexer
User-agent: meta-externalagent
User-agent: meta-externalfetcher
User-agent: Amzn-SearchBot
User-agent: Amazonbot
User-agent: Amzn-User
User-agent: CCBot
User-agent: MistralAI-Index
User-agent: MistralAI-Training
User-agent: MistralAI-User
User-agent: DuckAssistBot
User-agent: YouBot
User-agent: Kimi-SearchBot
User-agent: KimiBot
User-agent: Kimi-User
User-agent: AI2Bot
User-agent: PetalBot
User-agent: YandexAdditional
User-agent: Bytespider
User-agent: Google-Extended
Disallow: /let every crawler in
no robots.txt at all means the same thing. an empty disallow is the explicit way to say it.
User-agent: *
Disallow:robots.txt vs your firewall.
robots.txt is a request, not a lock. well-behaved crawlers read it and stay out of what you disallow — but a firewall, cdn or host setting can refuse a crawler before robots.txt ever matters.
cloudflare is the common one. it can serve a managed robots.txt that disallows known AI crawlers, its AI Crawl Control enforces blocks with a WAF rule that answers 403 (or 402 on paid plans), and since september 15, 2026 new domains block training and agent bots on pages that show ads by default.
that's why this checker requests your homepage as a normal browser and as five AI bots, and compares. we send each bot's published user agent from our own server, not the vendor's network, so a refusal means your edge rejects that user agent — firewalls that also check the bot's IP can still treat the real crawler differently.
what about llms.txt?
llms.txt is a proposal (llmstxt.org) for a short markdown file — a title, a one-line summary and lists of links — that helps AI agents find their way around a site. its author says it's used most for software docs, by coding agents.
none of the crawler docs in the table above say their crawler reads it, and google says you don't need special AI files or markup to appear in ai overviews or ai mode. so we check it, but as a nice-to-have, not a score.
questions
if I block GPTBot, do I disappear from ChatGPT?
no. openai runs separate crawlers: GPTBot collects training data, OAI-SearchBot powers chatgpt search, and openai says each setting is independent — you can allow OAI-SearchBot and disallow GPTBot. ChatGPT-User, which opens a page when someone asks about it, is different again: openai says robots.txt rules may not apply to it.
does blocking Google-Extended hide me from Google or AI Overviews?
no. google says Google-Extended doesn't affect inclusion in google search and isn't a ranking signal. ai overviews and ai mode come from the regular search index, so Googlebot's rules are what count there. Google-Extended controls whether google may use your pages to train future gemini models and to ground answers in the gemini apps and vertex ai.
robots.txt allows a bot — so why was it blocked?
because something in front of your site said no first: a firewall rule, bot protection, or a cdn setting like cloudflare's AI Crawl Control. robots.txt can't override that. our live test sends each bot's user agent from our server, so a refusal there means your edge rejects that user agent.
what are user-triggered fetchers?
bots that open one page because a person asked an ai about it, like ChatGPT-User, Perplexity-User and Claude-User. they don't crawl your site on their own. openai, perplexity, google, meta, amazon and moonshot say robots.txt may not apply to theirs, since a person asked; anthropic and mistral say theirs follow it.
how long until a robots.txt change takes effect?
it depends on the crawler. openai says about 24 hours for chatgpt search, meta says its crawlers may cache robots.txt for up to 24 hours, and duckduckgo says 72 hours for DuckAssistBot. the robots.txt standard (RFC 9309) asks crawlers not to cache it for longer than 24 hours.
what about ai.txt or noai meta tags?
we don't score them. none of the vendors in our table document reading ai.txt, and meta says it uses robots.txt "rather than non-standard formats like NoAI tags". the page-level controls vendors do document are things like nosnippet (google, apple) and noarchive (bing) — we flag those when your homepage has them.
what does this checker fetch, and do you keep it?
your homepage (once as a normal browser, then once per AI bot), /robots.txt and /llms.txt — about 8 requests, never more than 20. we keep the finished report for 10 minutes so shared links open instantly. we don't store your pages.
can I see which AI bots actually visit my site?
yes — that's what dashi does. it spots 60+ AI crawlers in your real traffic, by vendor, shows which pages each one read, and keeps them out of your visitor numbers, next to your revenue and live visitors on a 3D globe. it's free for now.