Check whether a specific URL is allowed for a specific crawler, and see which rule decided it.
A full URL or a path beginning with /
The crawler token, or * for the default group
Paste a robots.txt and enter a path to see whether it may be crawled, and which rule decides.
Being allowed to crawl is not the same as being indexed. A permitted URL may still be left out of an index for other reasons, and a blocked URL can still appear in results — without a snippet — if other sites link to it. To keep a page out of an index, allow crawling and use a noindex tag.
A blocked path
See the rule responsible
/admin/settingsAn exception inside a blocked folder
Longest match wins
/folder/public/pageA named crawler
Ignores the * group entirely
GooglebotThe whole file, exactly as served.
A path beginning with /, and a user-agent token like Googlebot.
Allowed or blocked, with the exact line and pattern that decided it.
Knowing a URL is blocked is half an answer. The tool names the line and pattern responsible, which is what you need to fix it.
Longest-match-wins, Allow-on-tie and prefix-based agent matching are all per specification, so the result matches what a crawler would do.
A named group silently overriding the wildcard group is a common cause of confusion, and it is made visible.
Paste a robots.txt file and a path, and find out whether that path may be crawled — and crucially, which line made the decision. The answer often surprises people, because precedence is not what intuition suggests: the longest matching pattern wins regardless of where it appears in the file, an Allow only beats a Disallow on an exact length tie, and a crawler matched by name ignores the wildcard group entirely. All of that is implemented per RFC 9309 and the deciding rule is shown with its line number.
Keep going
Related pages in SEO Tools, plus what others are using right now.
More seo tools that pair well with this one.
Build a valid robots.txt with the right group structure and directive order.
Check description length, truncation and keyword use before it reaches a results page.
Check whether your page title fits the search snippet, and where the keyword sits.
Produce a canonical link tag and check the URL for the mistakes that break it.
Build a valid XML sitemap from a list of URLs, with correct escaping and dates.
Check a sitemap against the protocol — structure, namespace, dates and limits.
Create strong random passwords with full control over length and characters.
Format, validate and minify JSON, with errors that point to the exact line.
Work out any percentage — of a number, as a share, or as a change.
Count words, characters, sentences and paragraphs as you type.
Convert between length, weight, temperature, volume, speed, data and more.
Count characters with and without spaces, against the limits that matter.
Find the mean, median, mode and range of a list of numbers, with the working shown.
Sample and population standard deviation and variance, both shown, with the working.
Turn a logo into favicon.ico, the PNG sizes a site needs, and the HTML to declare them.
Check a downloaded file against its published checksum, without uploading anything.
Stamp text like DRAFT or CONFIDENTIAL across a PDF, in your browser.
Turn JSON into readable YAML, with quoting and multiline strings handled properly.