AI Crawler Access Checker
Audits your robots.txt from the AI crawlers' point of view: can GPTBot, ClaudeBot, PerplexityBot, Google-Extended and the rest reach you, and which ones are blocked?
- Sign-in requiredMembership is free
- Open on the Pro plan100 runs a day
- Saved to your historyReachable from your account
What does this tool look at?
Can ChatGPT, Perplexity and Google AI Overviews read you and cite you? Readiness checks for the new generation of search.
- Thirteen AI crawlers checked one by one
- Training bots separated from search bots
- What a block actually costs you in AI answers
- Checks whether you have an llms.txt file
What this tool measures
When ChatGPT, Perplexity or Claude cites a source, it read that source with its own crawler. If a single line in your robots.txt keeps those crawlers outside, no amount of content quality will put you in those answers.
The tool fetches robots.txt from your domain root and checks the AI crawlers one by one, asking whether each is allowed to reach the “/” path. It looks for your llms.txt file at the same time.
Bots are grouped by purpose, because the decision differs per group. Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) read you so they can cite you. User-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) open a link someone pasted into a chat. Training crawlers feed content into model training.
How to use it
- 1
Enter the domain
Give the domain rather than a single page; robots.txt lives at the root and governs the whole site. Subdomains have their own file and need a separate check.
- 2
Read the search crawler line first
One blocked crawler there fails the audit. Blocking training crawlers is a choice; blocking search crawlers is usually an accident.
- 3
Find where the block comes from
The rule is rarely hand-written: a security plugin, a hosting default or a robots.txt template copied years ago. Open the file and look for the directive.
- 4
Put your decision in writing
Separate training from search deliberately: you may want to stay out of training data while still appearing in AI answers. The Robots.txt Generator offers those as two distinct settings.
- 5
Re-check after editing
robots.txt is refetched within hours. Run the tool again after updating and confirm the rule resolves the way you intended.
When to use it
- Competitors keep appearing as sources in AI answers while you never do.
- After changing hosting or installing a security plugin, to see what the defaults brought with them.
- When answering the technical half of the question “where do we stand in AI search”.
- After deciding to keep your content out of training data, to verify the rule actually works.
Reading the result
No robots.txt found
No file means no rules, so every crawler counts as allowed. That is not an error, but it is not a decision either: passing the audit this way does not mean you considered the question.
Search crawlers blocked
A failure on this line means being cited in AI answers is technically impossible. Producing more content changes nothing while that holds; lift the block first.
Training crawlers blocked
This line reports as information rather than failure and counts how many are closed. If you do not want your content in training data it is the right setting, and it does not affect search visibility.
The Google-Extended case
That crawler serves Gemini training and AI Overviews and is separate from Googlebot. Blocking it does not lower classic rankings, but it reduces your chance of appearing in the AI Overviews box.
The llms.txt line
A missing file raises a warning. That is not a block, it is a missed opportunity: models could read a summary you wrote instead of inferring one from HTML. The llms.txt Generator handles it in minutes.
Questions about the AI Crawler Access Checker
Not your classic Google traffic, since Googlebot is a separate crawler. What you lose is your name and link appearing in ChatGPT and Perplexity answers. That channel is still small in many sectors but growing steadily.
The major providers state that theirs do. Still, robots.txt is a request, not a technical barrier. Content that genuinely must stay private needs server-side authorisation.
Access returns immediately, but re-crawling and reuse in answers takes time. A site closed off for a long stretch does not regain visibility overnight.
Usually not. Crawlers like ChatGPT-User only open a page when someone pastes your link into a conversation. Blocking that means blocking the person who shared your content.
Use it alongside these
The other tools in the same category. Run them together and you see the whole picture.
The tool found the problem. Now for the fix
We run technical SEO, content and AI visibility work end to end. Let us review your site and come back with a clear list of what to do.
Monday – Saturday 09:00 – 18:00