Build a valid robots.txt with the right group structure and directive order.
One absolute URL per line
Validation of this output
robots.txt controls crawling, not indexing. A disallowed URL can still be listed in search results without a description if other pages link to it — to keep a page out of an index entirely, allow crawling and serve a noindex meta tag. The file must sit at the root of each domain and subdomain to have any effect.
Block an admin area
The most common rule
Disallow: /admin/Allow everything
An explicit empty Disallow
Disallow:Block one crawler only
A named group overrides *
User-agent: GPTBotName the crawlers it applies to, or use * for all of them.
Allow and Disallow paths, one per line, starting from the site root.
Save it as robots.txt in your site root — it only works there.
Consecutive User-agent lines are grouped properly, and a rule-free group still gets the empty Disallow it needs to be meaningful.
The output is parsed back through the same RFC 9309 parser that powers the validator, so structural mistakes surface immediately.
Crawl-delay is offered with a note that Google ignores it, rather than implying every crawler obeys everything.
Generate a robots.txt file that crawlers will actually parse the way you intend. The structure matters more than people expect: consecutive User-agent lines share the rules beneath them, a rule line closes the header, and a group with no rules needs an explicit empty Disallow to mean anything. This builder handles all of that, adds your sitemap references in the right place, and parses its own output back through the validator so what you copy is known-good rather than assumed-good.
Keep going
Related pages in SEO Tools, plus what others are using right now.
More seo tools that pair well with this one.
Check whether a specific URL is allowed for a specific crawler, and see which rule decided it.
Check description length, truncation and keyword use before it reaches a results page.
Check whether your page title fits the search snippet, and where the keyword sits.
Produce a canonical link tag and check the URL for the mistakes that break it.
Build a valid XML sitemap from a list of URLs, with correct escaping and dates.
Check a sitemap against the protocol — structure, namespace, dates and limits.
Create strong random passwords with full control over length and characters.
Format, validate and minify JSON, with errors that point to the exact line.
Work out any percentage — of a number, as a share, or as a change.
Count words, characters, sentences and paragraphs as you type.
Convert between length, weight, temperature, volume, speed, data and more.
Count characters with and without spaces, against the limits that matter.
Find the mean, median, mode and range of a list of numbers, with the working shown.
Sample and population standard deviation and variance, both shown, with the working.
Turn a logo into favicon.ico, the PNG sizes a site needs, and the HTML to declare them.
Check a downloaded file against its published checksum, without uploading anything.
Stamp text like DRAFT or CONFIDENTIAL across a PDF, in your browser.
Turn JSON into readable YAML, with quoting and multiline strings handled properly.