sitemap.xml
Definition
A sitemap.xml file is an XML document, normally served at /sitemap.xml, holding one entry per page. Each entry has a loc element containing the full absolute address, and may add a lastmod date, plus the older changefreq and priority hints that Google now ignores outright. One file may hold up to 50,000 URLs or 50 MB uncompressed; beyond that you split it and list the parts in a sitemap index. Crawlers find it either from a Sitemap line in robots.txt or because you submitted it once in Google Search Console. Listing a URL does not make it rank and leaving one out does not hide it — the file only answers the question of what exists.
Why It Matters
Crawlers reach pages by following links, so anything poorly linked is at risk of never being found. On a 40-page site where twelve pages are only reachable through a filter rendered in JavaScript, those twelve can sit unindexed for months while the eight pages in the header nav rank fine. A sitemap removes the guesswork for a few hundred bytes of file. The reverse is also true, and worth saying plainly: for a six-page site with every page in the navigation, a sitemap changes almost nothing. Where it earns its place is scale, weak internal linking, and telling a crawler that something changed.
How It Works
The crawler fetches /sitemap.xml, reads the loc values and queues each address it has not seen recently. The lastmod date is the part that does real work — when it moves, a recrawl is more likely to happen soon, and when it never moves on a page that has obviously changed, the signal gets discounted. Every entry should be a canonical URL on the same host as the sitemap, returning a 200; redirects, 404s and addresses on other domains spend crawl budget for nothing and show up as errors in Search Console. Compression is allowed, so a large file can be served as sitemap.xml.gz. In practice you rarely write the file by hand: a static site generator produces it at build time from the pages it rendered, and a publishing tool writes it from the pages it knows about.
Real-World Example
A recruitment agency zips a 40-page careers microsite and drops it on 99helpers, where it goes live at northstar-careers.99helpers.site. A month in, only the eight pages linked from the header appear in Google, because the other 32 role pages are reachable only through a JavaScript filter. Adding a sitemap.xml at the root, with a loc line per page and a lastmod taken from each file's date, gets the rest crawled inside a fortnight. Nothing about the pages themselves changed — the crawler simply learned they were there.
Common Mistakes
- ✕Listing URLs that redirect or 404 — each dead entry spends crawl budget and lowers trust in the whole file
- ✕Using relative paths in loc — the specification requires a full address including scheme and host, and a relative value is discarded
- ✕Stamping every page with today's lastmod at each build — when 300 dates change nightly the signal becomes noise and is ignored
- ✕Treating the file as an SEO lever — it helps pages get found, and that is the whole of it
Related Terms
robots.txt
A plain text file at the root of a site that tells search engine crawlers which paths they may fetch. It is a request that well-behaved crawlers honour, not a lock on the door.
Meta Description
A one-line summary of a page, written into a meta tag in the head of the HTML, that search engines often show as the grey text under a result. It affects clicks, not rankings.
Static Site Generator
A static site generator is a tool that turns content and templates into a folder of finished HTML files. Hugo, Eleventy, Astro, Jekyll and Next.js in export mode are the common ones.
index.html
index.html is the file a web server returns when a visitor asks for a folder rather than a named file. It is the default document for the site root and for every directory inside it.
Canonical URL
A canonical URL is the one address you nominate as the real home of a piece of content when several addresses serve it. You declare it with a rel=canonical link in the page head, and search engines use it to decide which version to index.
Put a file online in seconds
Drop in a document, an image, a page or a whole static website and share the link — free, with no build step and no server to set up.
Host a file free →