Robots.txt Generator
Builds the robots.txt file that tells search engine and AI crawlers what to do, adds the sitemap line and the common blocking patterns.
- Sign-in requiredMembership is free
- Open on the Ücretsiz plan5 runs a day
- Saved to your historyReachable from your account
What does this tool look at?
Anything that is easy to get wrong by hand, generated for you: meta tags, schema, robots.txt, sitemaps and redirect rules.
- Search engine and AI bots handled separately
- Sitemap line added automatically
- Ready-made patterns for admin and filter URLs
What this tool measures
robots.txt is a handful of one-line rules telling crawlers which paths they may read. It is also the file where a single wrong line is most expensive, because one stray directive can drop an entire site out of search.
The tool builds the file from two decisions: is the site open to crawling, and are AI bots welcome. In open mode it writes the general rule, your blocked directories and the sitemap line. In closed mode the file contains only “Disallow: /” behind a loud warning, because that setting belongs on a staging server.
On the AI side there are two extremes. Allow, and explicit permission lines are written for GPTBot, PerplexityBot and ClaudeBot. Block, and eleven bots are shut out one by one: your content stays out of model training, and out of AI citations too.
How to use it
- 1
Enter the site address
The address is used only to build the sitemap line. Subdomains count separately: blog.example.com needs its own robots.txt, the root domain's file does not apply there.
- 2
Choose the general rule
Use open for a live site. Closed belongs on staging only, and a closed file reaching production means the site disappears from search within days.
- 3
List the directories to block
One path per line; a missing leading slash is added. Up to 100 lines are processed. Admin, account and filter URLs such as /admin, /account and /*?filter= are the usual candidates.
- 4
Decide on AI bots
Allow them if you want to appear in AI answers. Block them if you do not want your content in training data. This is a preference, not a technical right answer.
- 5
Publish at the root and verify
Placement details are below. Once uploaded, run the Robots.txt Checker to confirm the file is readable and the rules behave as expected.
When to use it
- Launching a site, when the closed staging file has to be replaced with the production version.
- After spotting cart, filter or account URLs in search results.
- When writing down your policy toward AI crawlers for the first time.
- When server logs show unfamiliar bots crawling heavily.
Reading the result
Where the file goes
At the exact root of the domain, reachable at example.com/robots.txt. A file in a subdirectory is read by nobody. The server must send it as text/plain.
In a Next.js project
Two options: drop the file at public/robots.txt, or write app/robots.ts returning a MetadataRoute.Robots object. Never both. The file in public wins the route and app/robots.ts is silently ignored.
In WordPress
WordPress serves a virtual robots.txt by default. A real file at the root replaces it. With Yoast installed you can write the same content through the file editor under Tools.
Blocking is not deindexing
robots.txt blocks crawling, not indexing. A blocked URL with inbound links can still appear as a result with no description. To remove a page from results use a noindex tag, and do not block that page in robots.txt or Google can never read the tag.
The closed-mode warning
In closed mode the sitemap line is deliberately omitted and a two-line warning sits at the top. Keep that warning: it is the one signal that tells you a staging file made it to production.
Questions about the Robots.txt Generator
No. The file is public, so every path you list advertises the directory you wanted hidden. Real protection means a password or server-side authorisation.
Classic search traffic is unaffected, since Googlebot is a separate crawler. What changes is whether tools like ChatGPT and Perplexity cite you. Blocking Google-Extended reduces your visibility in AI Overviews.
For any URL the longest matching rule wins, regardless of order. That is why an explicit Allow for /admin/public still works while /admin stays blocked.
Google usually fetches robots.txt several times a day, so changes apply within hours. The robots.txt report in Search Console shows when it was last read.
Use it alongside these
The other tools in the same category. Run them together and you see the whole picture.
The tool found the problem. Now for the fix
We run technical SEO, content and AI visibility work end to end. Let us review your site and come back with a clear list of what to do.
Monday – Saturday 09:00 – 18:00