robots.txt
A plain-text file at the site root that tells crawlers which paths they may or may not crawl.
robots.txt is a plain-text file placed at a site's root (e.g. example.com/robots.txt) that gives crawlers directives about which paths they may or may not crawl, using a simple syntax of user-agent blocks, `Disallow` and `Allow` rules, and an optional `Sitemap` declaration.
Its directives are advisory, not enforced: a well-behaved crawler like Googlebot will respect them, but robots.txt provides no actual security or access control, and disallowed URLs can still be indexed (without their content being crawled) if they're linked to from elsewhere — for genuine index control, `meta robots` tags or the `X-Robots-Tag` HTTP header are the correct tools, not robots.txt.
The single most damaging and surprisingly common mistake is blocking CSS or JavaScript resources in robots.txt, which prevents Googlebot from rendering the page as a real browser would, directly harming how the page is understood and ranked — this was a bigger problem in the JavaScript-framework era than many site owners realise, since a blocked CSS file can cause Google's renderer to see a broken or empty-looking page.
A correctly-configured robots.txt is a narrow, deliberate tool: block genuinely low-value crawl paths (internal search results, admin areas, staging environments, certain parameterised URL patterns) to manage crawl budget, and leave everything else — especially rendering-critical assets — untouched.
How robots.txt directives are structured and interpreted
The file is organised into groups, each beginning with a `User-agent` line specifying which crawler the following rules apply to, followed by `Disallow` and `Allow` rules using path prefixes (not full regex, though `*` wildcards are supported by major crawlers). A `Disallow: /` blocks an entire site for the specified user-agent; an empty `Disallow:` allows everything.
Rule specificity, not order, determines precedence when rules conflict: the most specific matching path wins, which means a broad `Disallow: /search` combined with a more specific `Allow: /search/help` will correctly allow the help subpath even though the disallow rule appears first or is broader.
Crawlers cache robots.txt for up to 24 hours typically, meaning a change doesn't take effect instantly — a mistaken block can persist in a crawler's behaviour for up to a day even after being corrected.
Why robots.txt hygiene matters more in 2026
As AI-crawler user-agents (GPTBot, Google-Extended, PerplexityBot, and others) have proliferated to feed training data and retrieval-augmented generation systems, robots.txt has become the primary lever site owners have to decide, deliberately, whether their content should be eligible to inform AI Overviews, ChatGPT, or Perplexity's answers at all — a decision with real strategic weight that didn't exist in the classical-SEO-only era.
How to audit a robots.txt file
Fetch the live file directly at `/robots.txt` and review every rule manually — this is a small, human-readable file, and manual review catches ambiguous or overly-broad rules that automated tools sometimes miss.
Use Google Search Console's robots.txt report (under Settings) to confirm Google's last-fetched version matches production, and use the URL Inspection tool to confirm specific important pages, and specifically their CSS and JS assets, aren't being blocked.
Check the site's CDN or hosting-platform-generated robots.txt separately from any manually-edited version — some platforms auto-generate rules that can silently override or conflict with intentional site-owner rules.
Common robots.txt mistakes
Blocking CSS or JavaScript directories (`Disallow: /wp-content/themes/` or similar) is the most damaging common mistake, since it prevents Google from rendering the page correctly and can cause otherwise-good pages to be assessed as thin or broken.
A leftover `Disallow: /` from a staging environment accidentally carried into production is a frequent and severe error that blocks the entire site from being crawled at all, sometimes going unnoticed for weeks because it produces no visible symptom on the live site itself.
Using robots.txt to try to keep pages out of Google's index, rather than using `noindex`, is a subtle but common misunderstanding — a disallowed page can still be indexed (typically shown with no description) if other sites link to it, since Google never crawled it to discover a noindex instruction in the first place.
- Blocking CSS/JS paths, breaking Google's ability to render the page.
- A leftover staging Disallow: / rule carried into production.
- Using Disallow instead of noindex to try to prevent indexing.
- Auto-generated platform rules conflicting with intentional site rules.
A step-by-step robots.txt correction process
Fetch and manually review the live robots.txt file line by line, flagging any rule that blocks a CSS, JS, or image path, or that uses an unexpectedly broad wildcard.
Cross-check every disallowed path against the actual list of URLs Google has indexed for the site, using a `site:` search or Search Console coverage report, to catch cases where Disallow was mistakenly used in place of noindex.
Correct the file to allow all rendering-critical assets, remove any leftover staging-environment blocks, and add a `Sitemap:` line pointing to the canonical XML sitemap.
Resubmit or request a recrawl of the robots.txt file via Search Console, and re-check the URL Inspection tool's rendered-page preview for key pages to confirm assets are loading correctly.
robots.txt for local Austin-area service business sites
Local business sites built on common CMS or page-builder platforms often inherit a generic, platform-default robots.txt that may block admin or theme paths unnecessarily broadly, or fail to reference the sitemap at all — reviewing the platform's default file rather than assuming it's already correct is a quick, worthwhile check during any technical audit.
How robots.txt relates to canonical tags and sitemaps
robots.txt, canonical tags, and sitemaps together form the core technical-crawl-control toolkit, but each does a distinct job: robots.txt controls whether a path is crawled at all, canonical tags consolidate duplicate content that is crawled, and the sitemap proactively lists the URLs a site wants prioritised — using the wrong tool for a given problem (blocking instead of canonicalising, or vice versa) is a common source of confusion and error.
robots.txt correctness checklist
| CSS/JS blocked | 0 (target) | Never block rendering-critical resources. |
|---|---|---|
| Staging Disallow: / in production | 0 (target) | Check explicitly after every environment promotion. |
| Sitemap declared | Yes | Include a Sitemap: line pointing to the canonical XML sitemap. |
| AI crawler user-agents reviewed | Deliberate decision | GPTBot, Google-Extended, PerplexityBot etc. — allow or block intentionally. |
Frequently asked questions
- Is robots.txt enforced by search engines?
- It's advisory, not enforced. Well-behaved crawlers like Googlebot respect it, but it provides no actual access control — a disallowed URL can still appear in search results if other sites link to it, since it was never crawled to discover a noindex instruction.
- Should I block CSS or JavaScript in robots.txt?
- No, never. Blocking rendering-critical resources prevents Google from rendering the page as a browser would, which directly harms how the page is understood and ranked — this is one of the most damaging common robots.txt mistakes.
- What's the difference between robots.txt and noindex?
- robots.txt controls whether a page is crawled at all; noindex (via meta tag or HTTP header) controls whether a crawled page is included in the index. Disallowing a page in robots.txt does not reliably keep it out of the index if it's linked to elsewhere.
- Should I block AI crawlers like GPTBot?
- It's a deliberate strategic decision, not a default. Blocking them removes your content from being used in that platform's training or retrieval, which may reduce your visibility on that surface — weigh this against your GEO/AEO goals before blocking.
- How quickly does a robots.txt change take effect?
- Crawlers typically cache the file for up to 24 hours, so a change — including a fix to a mistaken block — may not be reflected in crawler behaviour instantly. Request a recrawl via Search Console to speed up detection of the update.
A site migration accidentally carried a staging environment's `Disallow: /` rule into the production robots.txt. Organic traffic dropped sharply within two weeks as pages fell out of the index; correcting the file and requesting recrawl via Search Console restored indexing within roughly three weeks.
Need this applied to your site?
We turn concepts like these into shipped work every week.