robots.txt Tester
Validate directives and test paths against your robots.txt in the browser.
Paste a robots.txt and a URL path; get back which user-agents can crawl it, which directive matched, and any syntax warnings.
allow: /admin/public/robots.txt is a small file with an outsized ability to break a site's visibility overnight. A single misplaced `Disallow: /` pushed to production during a staging-to-live migration has quietly de-indexed entire sites; we've fixed that exact mistake more than once. Because the file is plain text with no build-time validation, errors ship silently — there's no compiler warning when a rule accidentally blocks your entire CSS directory or shadows a more specific Allow rule you intended to take precedence.
This tester parses your robots.txt the way a compliant crawler does: grouping directives by user-agent block, then finding the most specific matching rule for a given path (longest-match wins, per the de facto standard most major crawlers, including Googlebot, follow). Paste your file, test a path, and see exactly which rule fires — before you find out the hard way in Search Console's Coverage report.
How crawlers actually resolve conflicting rules
robots.txt directives aren't read top-to-bottom like a firewall config; they're resolved by specificity. When both an Allow and a Disallow rule match a path, the rule with the longer matching pattern wins, regardless of which line appears first in the file. This trips up a lot of manually-written robots.txt files, where someone assumes a later Allow overrides an earlier Disallow purely by file order — it doesn't, unless the Allow pattern is also the more specific match.
User-agent blocks matter too: a `User-agent: *` block applies to any crawler with no explicit block of its own, but a crawler-specific block (`User-agent: Googlebot`) fully overrides the wildcard block for that crawler — the two blocks aren't merged, so directives you only wrote in the wildcard section won't apply to Googlebot if a separate Googlebot block exists.
What robots.txt can and can't do
robots.txt controls crawling, not indexing. A URL disallowed from crawling can still appear in Google's index — typically with no snippet, sourced from anchor text and external signals alone — if other pages link to it. If your goal is to keep a URL out of the index entirely, use a `noindex` meta tag or header instead, which requires the page to be crawlable so Google can see the directive. Blocking a page via robots.txt while also trying to noindex it is a common contradictory setup that leaves the page indexed with a blank snippet indefinitely.
- Disallow ≠ noindex — a blocked URL can still be indexed via external links.
- Longest matching pattern wins, not file order.
- Crawler-specific blocks fully replace the wildcard block for that crawler.
- Sitemap directives belong in robots.txt but are advisory, not access-control.
- Test every deploy — a single wrong slash can block an entire site section.
A pre-deploy checklist
Before any robots.txt change goes to production, test the paths that matter most: your homepage, your primary category templates, your sitemap URL, and any path you intentionally want blocked (admin panels, internal search results, faceted-navigation parameter combinations). Confirm both the wildcard and the Googlebot-specific behaviour if you maintain separate blocks, since they can diverge in ways that are easy to miss on a quick read-through.
Step-by-step workflow
- 1Pull your live robots.txt
Fetch it from yourdomain.com/robots.txt — always test the actual deployed file, not a local draft.
- 2Paste it into the tester
Keep comments and blank lines intact; they don't affect parsing but help you track sections.
- 3Enter a path and user-agent
Test your homepage, key templates, and any path you expect to be blocked.
- 4Read the matched rule
Confirm the directive that actually fired matches your intent, not just the overall verdict.
- 5Repeat for Googlebot and wildcard
Crawler-specific blocks override the wildcard block entirely — test both if you maintain separate sections.
- 6Re-test after every deploy
Treat robots.txt changes with the same caution as a production database migration.
Frequently asked questions
- Can robots.txt keep a page out of Google's index?
- Not reliably. Robots.txt blocks crawling, but a disallowed URL can still be indexed with a blank snippet if other sites link to it. Use a noindex directive on a crawlable page to reliably keep it out of the index.
- Why does Googlebot ignore my Disallow rule sometimes?
- Check whether a more specific Allow rule elsewhere in the file matches the same path with a longer pattern — that rule wins regardless of line order. Also confirm you're not testing against a stale cached copy of the file.
- Do I need a robots.txt file at all if I have nothing to block?
- It's not strictly required, but an empty or minimal robots.txt with just a Sitemap directive is good practice and avoids ambiguity for crawlers checking for one.
- Does robots.txt affect AI crawlers like GPTBot or PerplexityBot?
- Many AI companies' crawlers respect robots.txt directives for their named user-agent, but compliance varies by vendor and isn't guaranteed the way it broadly is with Googlebot and Bingbot. Check each vendor's published crawler documentation if you want to explicitly allow or block them.
- What's the difference between Disallow: / and no robots.txt file at all?
- No file at all is treated as no crawl restrictions — everything is allowed. Disallow: / under a matching user-agent block explicitly blocks that crawler from the entire site, which is the single most common accidental-deindexing mistake.
- Should I block staging or admin URLs via robots.txt?
- For genuinely sensitive URLs, robots.txt isn't real access control — it's a public file crawlers may still index the URL string of, and it doesn't stop direct browser access. Use authentication or a noindex header plus IP restriction for anything truly sensitive.
- ·Debugging crawler access issues
- ·Validating a robots.txt before deploy
- ·Confirming Googlebot behaviour
SERP Preview Simulator
See how your title and meta description render on Google — with pixel-accurate truncation.
Schema Explorer
Browse the JSON-LD blocks we ship — and copy the ones you need.
Meta Length Checker
Bulk-check title and description length across dozens of pages at once.
Programmes that use this exact analysis.
Technical SEO agency in Austin fixing Core Web Vitals, schema, and crawl issues. We ship the fixes that get you ranked and cited by AI.
Austin web design agency building high-performance, conversion-optimized marketing sites. We bridge the gap between aesthetic excellence and technical SEO requirements for Austin brands.
Austin SEO analytics agency wiring GA4, Google Search Console, Semrush, and Looker Studio into executive-grade dashboards. Monthly reports Austin CFOs actually read. Quarterly business reviews tied to revenue.