Guides ·
How to use Robots.txt Generator
Robots.txt Generator. Build and test robots.txt rules. No. It stops crawling. A blocked page can still be indexed from links.
The short answer
To create a robots.txt file, choose a default rule, add Disallow or Allow paths for all bots or specific ones such as Googlebot or GPTBot, add your sitemap URL and download. A built-in tester shows whether a given URL would be blocked.
It runs in your browser. What you enter is not uploaded.
How to use it
Type or paste your input into the Robots.txt Generator.
Choose any options you need.
See the result update instantly.
Copy the result or download it.
Limits worth knowing
Does robots.txt stop a page from being indexed? No. It stops crawling. A blocked page can still be indexed from links. Use a noindex meta tag (and allow crawling) to keep a page out of search results.
How do I block AI crawlers? Add groups for user agents such as GPTBot, ClaudeBot, Google-Extended and CCBot with Disallow: /. The generator has a one-click preset.