Home › SEO Tools › Robots.txt Checker
Free Robots.txt Checker
Fetch any website's robots.txt in one click and instantly test whether a specific URL is allowed or blocked for search engine crawlers. No sign-up needed.
We'll fetch that site's robots.txt and test this exact URL against it.
How to use the robots.txt checker
- Paste the full URL you want to test into the box above — for example a blog post or product page.
- Click Fetch & test. We'll fetch that site's robots.txt file and test your URL against its rules.
- Read the verdict: ALLOWED means crawlers can visit the page, BLOCKED means a rule is keeping them out. The matched rule shows exactly which line caused it.
About robots.txt
A robots.txt file lives at the root of a website (like example.com/robots.txt) and tells search engine crawlers which parts of the site they may or may not crawl. It works through simple Allow and Disallow rules grouped under User-agent lines that name the crawler they apply to, such as Googlebot or * for everyone.
Rules are matched by path prefix: Disallow: /private/ blocks everything under /private/, while Allow creates exceptions inside disallowed paths. When several rules match, the longest (most specific) match wins. Remember that robots.txt controls crawling, not indexing — a blocked URL can still appear in search results if other pages link to it.
Frequently asked questions
What is a robots.txt file?
A robots.txt file is a plain-text file placed at the root of a website (for example example.com/robots.txt). It gives search engine crawlers instructions about which parts of the site they are allowed or not allowed to crawl. It is a crawl instruction, not a security mechanism — it does not hide content from visitors.
What's the difference between Allow and Disallow?
Disallow blocks crawlers from crawling matching paths — for example Disallow: /admin/ keeps crawlers out of everything under /admin/. Allow creates an exception inside a disallowed path, for example Allow: /blog/ inside Disallow: / lets crawlers access only /blog/. When rules conflict, the most specific (longest matching) rule wins.
The checker says my page is blocked — what should I do?
If you want the page in Google, edit your robots.txt and remove or adjust the Disallow rule that matches it — for example, change Disallow: /blog/ to a narrower rule, or add Allow: /blog/important-post/. Save the file, then ask Google to recheck it in Search Console and request indexing for the page. Crawlers can take hours or days to pick up the change.
Does robots.txt prevent Google from indexing a page?
Not exactly. A Disallow rule stops Google from crawling the page, but Google can still index the URL without crawling it — for example if other sites link to it — showing it without a description. If you want a page removed from Google's index, use a noindex meta tag or HTTP header instead of (or as well as) robots.txt.