WHAT IT DOES
Write crawler rules you can inspect before deploying.
Choose a user-agent, allow and disallow paths, crawl delay, and sitemap URLs to produce a deterministic robots.txt file. The generator keeps rule order stable, flags contradictory paths and incomplete sitemap URLs, and puts its caveats in the generated file.
Why use it?
Use it to draft crawler instructions with visible path and sitemap rules before publishing them on a site.
How to use this tool
- Choose the user-agent and enter rooted Allow or Disallow paths plus any absolute sitemap URLs.
- Generate and review the comments and conflicts. For example, Disallow: /drafts/ is a crawler instruction, not protection for private drafts.
- Publish the reviewed file at the site's intended robots.txt location and check it with the crawlers you care about. The generator neither fetches your sitemap nor guarantees a crawler will honor every directive.
Details and limitations
- The default starter group is for all crawlers and begins with editable Allow, Disallow, and Sitemap rows so the form is usable on first paint.
- Empty rows are ignored; non-empty paths must be rooted and sitemap values must be absolute HTTP or HTTPS URLs.
- Allow and Disallow are advisory crawler directives, not access control. The generated comments call this out beside the rules.
- Output is deterministic for the same selections and does not fetch or verify the sitemap hosts.