XML sitemap
A structured list of URLs a site wants search engines to crawl.
An XML sitemap is a structured file, typically at `/sitemap.xml`, listing the URLs a site wants search engines to crawl, along with optional metadata such as `lastmod` (last modification date) that can influence recrawl prioritisation.
A sitemap is a hint, not a guarantee of crawling or indexing — Google is explicit that inclusion in a sitemap does not ensure a URL will be crawled or indexed, but it does make discovery easier, particularly for large sites, sites with weak internal linking, or newly-published pages that haven't yet accumulated internal links.
The sitemap's core discipline is restraint: it should list only canonical, indexable, genuinely valuable URLs — including non-canonical variants, redirected URLs, noindexed pages, or thin low-value pages actively works against a site, since it signals a mismatch between what the sitemap claims is important and what the actual page-level directives say, which can reduce Google's trust in the sitemap's usefulness overall.
For most local service business sites, a single, auto-generated, always-current sitemap covering pages, posts, and key media is sufficient; for large sites the sitemap is typically split into multiple files (by content type or by size, since a single sitemap file is capped at 50,000 URLs and 50MB uncompressed) referenced from a sitemap index file.
How sitemaps work mechanically
A search engine periodically fetches a site's declared sitemap (referenced via the `Sitemap:` line in robots.txt, or submitted directly in Search Console) and uses the listed URLs as a discovery and prioritisation input alongside normal link-following crawling.
The `lastmod` field, when accurate, is used as a freshness signal that can influence recrawl priority — a page whose `lastmod` genuinely reflects a meaningful content update is more likely to be recrawled promptly than one where `lastmod` is either missing or updated on every automated build regardless of whether the content actually changed, which search engines have learned to discount over time.
Why sitemap accuracy matters more in 2026
As AEO and GEO strategies depend on freshly-updated content being recrawled and re-verified promptly — updated pricing, new service-area detail, corrected facts that generative citation systems rely on — an honest, accurate sitemap with genuine `lastmod` values is a meaningful lever for getting that freshness recognised faster, whereas a sitemap that cries wolf with constant, meaningless `lastmod` updates loses exactly that leverage.
How to audit a sitemap
Fetch the live sitemap and cross-reference every listed URL against its actual HTTP status, canonical tag, and indexability directives — flagging any 404, redirect, non-canonical, or noindexed URL that shouldn't be there.
Use Search Console's Sitemaps report to check the ratio of 'discovered' to 'indexed' URLs from the sitemap; a large, persistent gap between the two is a signal worth investigating, either in content quality or in a mismatch between sitemap inclusion and actual indexability.
Common sitemap mistakes
Including noindexed, redirected, or non-canonical URLs is the most common mistake, usually the result of an auto-generated sitemap that isn't filtered against the site's actual indexability rules and simply lists every URL the CMS knows about.
A `lastmod` value that updates on every deploy regardless of actual content change — common with poorly-configured static-site generators — teaches search engines to discount the field entirely, forfeiting a genuinely useful freshness signal.
Forgetting to update the sitemap when new pages are published, or leaving deleted pages in the sitemap indefinitely, both waste crawl attention on stale or non-existent URLs.
- Non-canonical, redirected, or noindexed URLs included in the sitemap.
- lastmod values that update on every deploy regardless of real content change.
- New pages not added to the sitemap promptly.
- Deleted or redirected pages left in the sitemap indefinitely.
A step-by-step sitemap correction process
Generate or audit the sitemap to include only canonical, 200-status, indexable URLs, explicitly excluding anything noindexed, redirected, or duplicate.
Fix `lastmod` generation so it reflects genuine content changes rather than build timestamps, ideally driven by the CMS's actual content-update event rather than the deployment pipeline.
Split into multiple sitemap files referenced by a sitemap index if the site exceeds roughly 10,000–20,000 URLs, to keep individual files manageable and to allow more granular monitoring by content type.
Submit the sitemap in Search Console and reference it in robots.txt, then monitor the discovered-vs-indexed ratio monthly and investigate any large or growing gap.
Sitemaps for local Austin-area service business sites
For a typical local service site, the highest-value sitemap fix is usually simple: confirming the CMS's auto-generated sitemap actually excludes noindexed utility pages (thank-you pages, internal search results, tag archives) and updates promptly whenever a new service or location page goes live, rather than requiring a manual resubmission each time.
How sitemaps relate to robots.txt and canonical tags
The sitemap, robots.txt, and canonical tags should always tell the same story: a URL blocked in robots.txt or carrying a non-self-referencing canonical has no business appearing in the sitemap, and any mismatch between the three is a direct, checkable signal of a technical SEO bug worth investigating immediately.
Sitemap health checklist
| Non-canonical URLs in sitemap | 0 (target) | Only canonical, indexable URLs should ever be listed. |
|---|---|---|
| Indexed-to-discovered ratio | 80%+ target | From Search Console's Sitemaps report; lower ratios warrant investigation. |
| Max URLs per sitemap file | 50,000 | Hard technical limit; split into multiple files via a sitemap index above this. |
| lastmod accuracy | Reflects real content changes | Not tied to deployment timestamps regardless of content change. |
Frequently asked questions
- Does having a sitemap guarantee indexing?
- No. Google is explicit that sitemap inclusion is a discovery hint, not a guarantee of crawling or indexing. It makes discovery easier, especially for large sites or newly-published pages, but page quality and indexability still determine the outcome.
- Should noindexed pages be in the sitemap?
- No. Including noindexed, redirected, or non-canonical URLs in a sitemap sends a contradictory signal and can reduce Google's overall trust in the sitemap's usefulness. Only canonical, indexable URLs belong there.
- How many URLs can one sitemap file contain?
- Up to 50,000 URLs or 50MB uncompressed, whichever comes first. Larger sites should split content across multiple sitemap files referenced by a single sitemap index file.
- Does the lastmod date actually matter?
- Yes, if accurate — it can influence recrawl prioritisation as a freshness signal. But a lastmod that updates on every deploy regardless of real content change teaches search engines to discount the field, forfeiting its usefulness.
- Where should I submit my sitemap?
- Reference it in robots.txt via a Sitemap: line, and additionally submit it directly through Google Search Console and Bing Webmaster Tools, which also gives you a discovered-vs-indexed report to monitor over time.
An ecommerce site's auto-generated sitemap included 40,000 filtered/parameterised product-variant URLs alongside 3,000 genuine product pages. Rebuilding the sitemap to include only canonical product and category URLs improved the indexed-to-discovered ratio from 54% to 91% within two months.
Need this applied to your site?
We turn concepts like these into shipped work every week.