Robots.txt Checker
Downloads the domain's robots.txt, parses the rule groups, and flags the critical faults: a Disallow that closes the whole site, a missing sitemap line, unreadable syntax.
- Sign-in requiredMembership is free
- Open on the Ücretsiz plan5 runs a day
- Saved to your historyReachable from your account
What does this tool look at?
Everything that decides whether Google can crawl and index your page: tags, redirects, robots, sitemaps and server responses.
- Warning when a rule closes the entire site
- Whether a sitemap line exists and the address responds
- User-agent groups and their specific rules
- Syntax errors and orphaned lines
What this tool measures
robots.txt is a plain text file at the root of your site that tells crawlers where they may and may not go. Do not let its size fool you: one wrong line can wipe your whole site out of search. A file that says “Disallow: /” closes the entire site to Googlebot.
The tool downloads the file and parses the rule groups. Then it runs the rule Google actually applies: the longest matching path wins, and on a tie Allow takes precedence. So the question “is the homepage open to Googlebot” gets answered by calculation, not by guesswork.
It does not leave the Sitemap lines alone either; it requests the first three addresses and checks whether they really answer. Dead sitemap URLs discovered months after launch are common.
How to use it
- 1
Type your domain
You do not need the file path; the tool looks for /robots.txt at the root itself.
- 2
Look at the Googlebot access line
If it is red, drop everything and fix that rule. When a site is entirely absent from search, this is usually why.
- 3
Verify the Sitemap line
Add it if it is missing; if it is there, confirm in the audit that the address answers 200.
- 4
Clean up the unreadable lines
A misspelled directive is silently ignored. You can spend years believing a rule works when it never did.
When to use it
- After launching a new site, to confirm the staging “block everything” rule was removed.
- As the first place to look when organic traffic drops overnight.
- When admin or filter URLs show up in Google and you want to confirm the block is written correctly.
- Before submitting your sitemap in Search Console, to make sure it is declared in the file.
Reading the result
File not found (404)
Not a fault on its own: with no file, crawlers assume everything is allowed. But you cannot declare a sitemap or close off any area.
Googlebot blocked
Urgent. Usually a leftover “Disallow: /” from staging. After removing it, indexing takes days to return.
No Sitemap line
You lose the cheapest way for Google to find your sitemap. Add one line: Sitemap: https://yoursite.com/sitemap.xml
Content type is not text/plain
If the server serves the file as HTML, some crawlers never read it. Fix it in the server configuration.
Unrecognised directive
A typo or an unsupported field. The line does nothing; remove it or write it correctly.
Questions about the Robots.txt Checker
It does not. Disallow only blocks CRAWLING. A blocked address that other sites link to can still appear in the index by URL alone, with its content unread. To deindex it, put noindex on the page and allow that page to be crawled.
The longer, more specific path wins. With “Disallow: /admin/” and “Allow: /admin/help/” together, the help page stays open. If two rules are the same length, Allow takes precedence.
No. Google renders the page like a real browser to judge it. With style and script files blocked, it sees a broken page, and assessments such as mobile friendliness come out wrong.
Google reads up to 500 KB and ignores the rest. In practice, a robots.txt approaching that size means rules are being written address by address; use wildcards instead.
Use it alongside these
The other tools in the same category. Run them together and you see the whole picture.
The tool found the problem. Now for the fix
We run technical SEO, content and AI visibility work end to end. Let us review your site and come back with a clear list of what to do.
Monday – Saturday 09:00 – 18:00