Crawl budget

Crawl rate limit is set by the search engine to avoid overwhelming a server, and adapts based on how quickly a site responds and how many errors it returns — a server that responds slowly or throws frequent 5xx errors gets crawled more cautiously, compounding any existing performance problem.

Crawl demand reflects the search engine's assessment of how valuable and how likely-to-change a site's content is. A site publishing genuinely new, valuable content frequently earns higher crawl demand than a static site of comparable size, and a page that's historically ranked well or been cited frequently tends to be recrawled more often than an obscure, rarely-visited one.

As AEO and GEO strategies depend on content staying current and being recrawled promptly to reflect updated pricing, availability, or facts that generative citation relies on, wasted crawl budget has a second-order cost beyond classical indexing: a stale, infrequently-recrawled page is more likely to be cited with outdated information, or simply passed over in favour of a competitor whose content the crawler has verified more recently.

Pull the Crawl Stats report in Google Search Console to see total crawl requests, response time trends, and a breakdown by response code and file type over the last 90 days — a rising trend in crawl requests to non-200 or low-value URLs is the clearest early warning sign.

Cross-reference server log files directly (log-file analysis) to see exactly which URLs Googlebot is requesting and how often — this is the most authoritative source, since Search Console's Crawl Stats aggregates and samples rather than showing every request.

Compare crawl frequency against a list of genuinely high-value URLs (top commercial and content pages) to check whether they're being recrawled proportionally to their importance, or whether low-value parameterised or faceted URLs are consuming a disproportionate share of requests.

Faceted navigation on ecommerce sites — colour, size, and sort-order combinations each generating a distinct crawlable URL — is the single most common large-scale cause, sometimes multiplying a catalogue of a few thousand products into millions of technically distinct, mostly duplicate URLs.

Session IDs or tracking parameters appended to URLs that aren't properly canonicalised or excluded create an effectively infinite space of near-duplicate pages a crawler can get lost in.

Soft 404s and thin, auto-generated pages (empty search-result pages, expired listing pages left live) consume crawl attention without offering any value in return.

Identify the specific URL patterns consuming disproportionate crawl activity via log-file analysis or Search Console's Crawl Stats, focused on parameterised, faceted, or otherwise low-value paths.

Use robots.txt to block crawling of genuinely low-value, high-volume parameter patterns where the pages have no independent SEO value at all, and use self-referencing canonical tags on near-duplicate variants that should be crawled but consolidated for indexing purposes.

Fix or remove soft 404s and thin auto-generated pages, either by adding genuine content, a proper 404/410 status, or a noindex directive as appropriate to the specific case.

Improve server response time and reliability where slow response is itself constraining the crawl rate limit, since a faster server directly increases the ceiling on how much a search engine is willing to crawl.

Monitor Crawl Stats and indexing coverage over the following months to confirm crawl attention is shifting toward the intended high-value pages.

Crawl budget is very rarely a genuine constraint for a local service business site of typical size (tens to low hundreds of pages) — the far more common local-site problem is the opposite: too few pages and too little internal linking, not too many low-value URLs competing for crawl attention. Crawl budget work is usually only relevant here if a site has accumulated a large number of thin, auto-generated location or service-combination pages without real content behind them.

Canonical tags and robots.txt are the two primary technical mechanisms for managing crawl budget once a problem is identified; the XML sitemap, by contrast, should only ever list the canonical, valuable URLs a site wants prioritised, since including low-value or duplicate URLs in a sitemap actively works against a crawl-budget remediation effort rather than supporting it.