Robots.txt is the first thing a well behaved crawler reads before it touches anything else on your domain, and it is also one of the easiest files to get quietly wrong. A rule with a missing slash blocks more than intended, a rule placed under the wrong user agent group is ignored entirely, and a typo in the sitemap path means the file that is supposed to help discovery goes unread.
The file this tool produces follows the syntax every major crawler agrees on, and the preview updates as you edit, so you can see the exact plain text file before you upload it to your site's root.
Disallow is not noindex, and that distinction matters
This is the single most common misunderstanding about robots.txt. A disallow rule tells a crawler not to request a URL. It does not tell a search engine to keep that URL out of its index. If other pages link to a disallowed URL, a search engine can still list it in results, typically showing the URL alone with no title or description because it was never allowed to fetch the page to generate one.
Keeping a page out of search results reliably requires a noindex meta tag or HTTP header on the page itself, which in turn requires the page to be crawlable so the crawler can see that instruction. Disallowing a page and adding noindex to it at the same time works against you: the noindex tag never gets read because the crawler was told not to fetch the page in the first place.
- Use disallow to save crawl budget on pages that provide no value to index, such as internal search results or filtered duplicates.
- Use noindex, not disallow, when the goal is specifically to keep a page out of search results.
- Never disallow a page you have also marked noindex, since the crawler needs to reach the page to read that tag.
- Remember that disallow is a request, not enforcement. It only works because well behaved crawlers choose to respect it.
User agent groups and how crawlers pick one
Each group in a robots.txt file starts with one or more User-agent lines followed by the rules that apply to them. A crawler looks for a group naming it specifically, for example Googlebot, and if none exists it falls back to the group for User-agent: *, the wildcard that matches any crawler without its own dedicated group.
A common mistake is adding a specific group intending to add extra rules on top of the general one, not realising that a crawler matching a named group ignores the wildcard group entirely rather than combining the two. If you write a dedicated Googlebot group, it needs to repeat every rule Googlebot should follow, not just the ones that differ from the default.
Allow, disallow and how the most specific rule wins
Within a group, the rule that applies to a given URL is the one with the longest matching path, regardless of whether it appears as an allow or a disallow and regardless of the order the rules are written in. This lets you disallow a broad directory and then allow a single file or subfolder inside it, for example disallowing /private/ while allowing /private/public-notice.html, and the more specific allow rule wins for that one file.
Wildcards extend this further. An asterisk matches any sequence of characters, and a dollar sign anchors a rule to the end of a URL, which is useful for blocking a file type everywhere on the site, such as Disallow: /*.pdf$ to keep PDF files out of crawling without touching every other path that happens to contain the letters pdf. A capable disallow rules generator needs to support both wildcards correctly, since a rule missing the anchor can block far more of the site than intended.
Crawl delay and when you actually need it
Crawl-delay asks a crawler to wait a number of seconds between requests, which matters for a modest server being crawled aggressively by a bot that is not Google, since Google ignores crawl delay entirely and manages its own crawl rate through Search Console instead. Bing and several other crawlers do respect it.
Set it only if your server logs show a specific crawler causing real load, and start with a small number like 5 or 10 rather than guessing high, since an unnecessarily long delay slows down how quickly new or updated pages get discovered by the crawlers that do honour the directive.
Presets for common platforms and situations
This robots txt generator includes presets for the situations that come up most often: a fully open file that allows everything, a WordPress preset that blocks the admin area while allowing the admin-ajax endpoint it depends on, a staging preset that disallows the entire site for every crawler, and an ecommerce preset that blocks cart, checkout and account paths while leaving product and category pages open.
A preset is a starting point, not a finished file. Always check the disallowed paths against your own site's actual URL structure before publishing, since a preset written for a typical WordPress install will not match a heavily customised permalink structure or a completely different platform.
How this robots txt generator assembles the file
Every group you add becomes a block of plain text lines in the order User-agent, then Allow and Disallow rules, matching the format search engines expect. Sitemap directives are written on their own lines, conventionally at the end of the file, and can point at more than one sitemap if your site splits them by section.
The output is exactly what gets uploaded, nothing more. Save it as robots.txt and place it at the root of your domain, for example https://example.com/robots.txt, since a crawler only checks that one specific location and ignores a correctly formatted file placed anywhere else.
| Directive | Effect |
|---|---|
| User-agent | Names the crawler the following rules apply to |
| Disallow | Blocks crawling of a path, does not guarantee deindexing |
| Allow | Overrides a broader disallow for a more specific path |
| Crawl-delay | Requests seconds between requests, ignored by Google |
| Sitemap | Points crawlers at an XML sitemap, works outside any group |
How to use the robots txt generator
- 1
Start from a preset or a blank file
Choose a preset close to your situation, such as WordPress or ecommerce, or start blank and add your own groups.
- 2
Add a user agent group
Enter the user agent, for example * for all crawlers or Googlebot for one crawler specifically.
- 3
Add allow and disallow rules
List the paths to block or explicitly allow. More specific paths override broader ones automatically.
- 4
Add your sitemap and download
Paste your sitemap URL, review the preview, then download robots.txt and upload it to your site's root.
Related tools worth bookmarking
Sources and further reading
- robotstxt.org: About robots.txtThe original description of the Robots Exclusion Protocol that robots.txt implements, maintained as the reference for the format.robotstxt.org
- Google Search Central: Robots.txt introductionGoogle's own documentation of how its crawlers interpret robots.txt, including group matching and crawl delay behaviour.developers.google.com
- RFC 9309: Robots Exclusion ProtocolThe formal internet standard defining the syntax and semantics of robots.txt that this generator's output conforms to.rfc-editor.org
Frequently asked questions
Common questions about the robots txt generator.
Not reliably. A disallow rule stops a crawler from fetching the page, but if other pages link to it, it can still appear in search results without a description. To reliably keep a page out of the index, use a noindex meta tag on a page that is still crawlable, rather than disallowing it in robots.txt.
At the root of your domain, so it is reachable at https://yourdomain.com/robots.txt exactly. Crawlers only check that one location. Placing the file in a subfolder or under a different filename means it is simply never found, and the site is crawled as if no robots.txt existed at all.
No. Google ignores Crawl-delay entirely and instead lets you adjust crawl rate through Search Console if needed. Bing, Yandex and several other crawlers do respect it, so it is still worth setting if you have evidence a specific non Google crawler is putting load on your server.
Yes. Large sites commonly split their sitemap into several files by section, such as one for products and one for blog posts, and list each one on its own Sitemap line. Every sitemap listed should point at a valid, reachable XML file for it to be useful.
Crawlers treat a missing robots.txt as permission to crawl everything, which is often fine for a small site with nothing sensitive to keep out of crawling. A robots.txt becomes useful once you want to exclude specific sections, such as an admin area, a staging environment or filtered duplicate pages, or once you want to point crawlers at your sitemap explicitly.
The most common cause is that the rule sits under the wrong user agent group, so a specific crawler skips the wildcard group entirely and never sees it. The other common cause is a path typo, since matching is an exact prefix match rather than a fuzzy one, so a missing leading slash or a trailing character difference means the rule simply does not match the URLs you intended.