Robots.txt Generator

What it does

The Robots.txt Generator is part of the SEO Suite. It builds a standards-compliant robots.txt file from per-user-agent rule groups. Set the user-agent (or leave it as * for all), add allow and disallow path patterns, and the generator emits the directives in the correct order: User-agent, then Disallow/Allow, then Crawl-delay if specified, followed by any Sitemap declarations at the file end. Multiple groups stack so you can give Googlebot, Bingbot, and the catch-all * different rules in one file.

Common situations

You are launching a new site and want to block the staging environment, the WordPress admin, the cart and checkout pages, and a tag-archive area you have decided is low-value. The default group blocks the obvious patterns; the generator’s structure makes it easy to add the site-specific ones without remembering the exact directive syntax.

You have noticed an aggressive bot from a low-quality SEO crawler is hammering your server. Add a per-user-agent group blocking that specific bot from your whole site (Disallow: /) and leave the others untouched. Robots.txt is voluntary, but every reputable bot honours it. Many of the abusive ones do too, because they want to stay out of legal trouble.

A faceted-navigation system on a product catalog is creating thousands of low-value URLs that are getting crawled and indexed when they shouldn’t be. Block the URL parameters with a wildcard pattern (Disallow: /*?filter=) and the crawl budget reallocates to the canonical product pages. This is one of the highest-impact uses of robots.txt on large e-commerce sites.

You have a Search Console report showing Google is crawling private subpaths (admin, login, internal API) and you want to declare them off-limits. Add an explicit disallow for each, plus the sitemap declaration so Google knows where to find what should be crawled.

What you need to know

Robots.txt is a voluntary protocol. Bots that respect it will; bots that don’t will ignore it. Every major search engine (Google, Bing, DuckDuckGo, Yandex, Baidu) honours it, as do most academic and reputable third-party crawlers. Aggressive scraper bots and bad-actor crawlers do not. Robots.txt is the right tool for politely shaping crawl budget on legitimate bots; it is not a security measure for hiding URLs from the public.

The most common misconception is that robots.txt prevents indexing. It does not. It prevents crawling. Google can still index a URL it has not crawled (typically when other pages link to it), and the URL will appear in search results with a generic snippet (“A description for this result is not available because of this site’s robots.txt”). To prevent indexing, use the noindex meta tag or X-Robots-Tag HTTP header. Robots.txt blocks the bot before it can read the noindex, which is why noindex via robots-disallowed pages does not work.

The matching rules are precise: longest matching path wins, and on equal-length matches, Allow beats Disallow. Wildcards: * matches any sequence of characters, $ anchors to the end of the URL. So Disallow: /private/ blocks /private/ and everything under it, Disallow: /*.pdf$ blocks all PDF URLs, and Allow: /private/public-thing.html overrides the previous Disallow because it is a longer match.

User-agent grouping is sequential: every directive after a User-agent: X line applies to that UA until the next User-agent: line. The conventional pattern is to set User-agent: * first with broad rules, then add specific overrides for named bots. There is no way to apply rules to multiple specific UAs except by repeating them; consolidating into the * group is usually cleaner.

The Crawl-delay directive is supported by Bing, Yandex, and a few smaller bots. Google ignores it: the crawl rate for Google is configured in Search Console, not in robots.txt. So Crawl-delay: 5 shapes Bing’s crawl rate but does nothing for Google. Most sites do not need Crawl-delay; if your server is being overwhelmed by bots, the right fix is at the firewall, not in robots.txt.

Frequently asked questions

Does robots.txt prevent a page from appearing in search results?

No. It prevents crawling but not indexing. A page blocked in robots.txt can still be indexed if other pages link to it: Google will list the URL in results without a description. To keep a page out of search entirely, use a noindex meta tag or X-Robots-Tag header, and don’t disallow it in robots.txt (otherwise the bot can’t read the noindex).

What user-agents should I name?

In practice, only the bots you want to give different rules to. Most sites have one User-agent: * group covering everything. Add a named group for Googlebot only if you want Google to have different rules from *. Adding empty groups for every imaginable bot is noise.

Are wildcards supported?

Yes. * matches any sequence and $ anchors to end. Both are supported by Google, Bing, and most major bots, though they are technically extensions to the original protocol. Use them confidently for major search engines; for niche crawlers, the unwritten convention is to fall back to literal prefix matches.

What’s the difference between robots.txt and a noindex meta tag?

Robots.txt blocks crawling: the bot doesn’t fetch the page. The noindex meta tag blocks indexing: the bot fetches the page, reads the directive, and excludes it from the index. Use robots.txt to manage crawl budget and protect sensitive paths; use noindex to control which crawled pages appear in search.

Can I block ChatGPT and other AI crawlers?

Yes, by user-agent. The well-known ones are GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google AI training), CCBot (Common Crawl), and PerplexityBot. Add a group for each with Disallow: / to opt out of training-data collection. Whether they honour the directive is up to them; the major ones (OpenAI, Anthropic, Google) have publicly committed to respecting robots.txt.

What does User-agent: * cover?

Every bot that has not matched a more specific group earlier in the file. If a bot is named in a separate group, the * group does not apply to that bot. This is the most common mistake: adding restrictive * rules and assuming Googlebot will follow them when Googlebot has its own group earlier in the file.

How big can robots.txt be?

Google reads up to 500 KiB. Beyond that, content is ignored. Most robots.txt files are under 5 KB; if yours is approaching the limit, the file is doing more work than it should. Usually that’s because someone has been listing every URL to block individually instead of using path patterns.

Where should the file live?

At the site root, exactly at /robots.txt. Each subdomain has its own robots.txt: app.example.com/robots.txt is separate from example.com/robots.txt. Subdirectory rules don’t exist; you cannot have /blog/robots.txt.

Common problems

Problem: A page blocked in robots.txt is still appearing in search results.

Robots.txt blocks crawling, not indexing. If the page is linked from elsewhere, Google can index it as a URL-only entry without a description. Remove the disallow, add a noindex meta tag, wait for Google to recrawl, then the URL drops from the index entirely. Re-adding the disallow afterwards is fine.

Problem: Search Console reports “Indexed, though blocked by robots.txt”.

This is the noindex/disallow conflict in action. The page was indexed (probably from links elsewhere) and is now blocked from re-crawl, so Google cannot read any noindex you have added. Allow the URL temporarily, let Google read the noindex, then re-disallow.

Problem: Allow rule is not overriding the parent Disallow.

Check rule length. Allow only beats Disallow when it is a longer or equal-length path match. Disallow: /private/ and Allow: /private/ are equal length, so Allow wins. Disallow: /private/foo/ and Allow: /private/: Disallow wins because it is a longer match. The longest match always wins; equal-length ties go to Allow.

Problem: Robots.txt file shows in browser but Google reports “couldn’t fetch”.

Check the response’s Content-Type header: it should be text/plain (or anything non-HTML). If your CMS is serving robots.txt with Content-Type: text/html, Google rejects it. The other common cause is a 200 OK response with a “this is the file” HTML page rather than the robots content, usually a misconfigured CMS routing.

Problem: Different rules apply to different subdomains and managing them is messy.

Each subdomain has its own robots.txt. There is no inheritance. If you have many subdomains, generate a robots.txt for each one (often via the same templating logic) and serve from each subdomain root. There is no single-file solution.

Tips

• Always include the Sitemap: declaration. It is the only directive that helps Google rather than restricting it, and it costs nothing.
• Block staging and dev subdomains with a separate robots.txt per subdomain that disallows everything. Otherwise indexed staging URLs are inevitable.
• Check the file at https://yourdomain.com/robots.txt periodically. CMS updates and security plugins occasionally rewrite or remove it.
• Use the Robots.txt Tester to verify a path against the rules before deploying. It saves a lot of “did this actually block what I thought it would” debugging.
• The file is public. Anyone can read it, including competitors. Do not list “secret” URLs in robots.txt: listing them makes them discoverable. Use authentication or noindex for genuinely private pages.

Related tools in this suite

The natural pairing is the Robots.txt Tester. Once you have generated the file, paste it back in (or fetch a live one) and test specific paths against the rules per user-agent. The Sitemap.xml Generator builds the sitemap that the Sitemap: declaration in your robots.txt should reference.

What this looks like at scale

For a single site, a hand-edited robots.txt is fine. For multi-site organisations or templated sites where the rules need to vary by environment (production vs staging vs preview), the file should be generated dynamically: different content per environment, version-controlled, deployed alongside the rest of the site config. The WP Beacon Plugin audits robots.txt presence and validity as part of broader site monitoring, which catches accidental overwrites between deploys. That kind of per-environment configuration work is also what lives in the systems we build for clients.

Take it further

If your robots.txt has become complex enough that errors are frequent, the underlying problem is usually that crawl-budget management is being done in robots.txt instead of via canonicalisation or noindex meta tags. Talk it through and there is usually a cleaner path.