Crawl Budget
Definition
Google describes crawl budget as the product of two things. The first is the crawl capacity limit: how fast it can fetch from your server without degrading it, adjusted up or down according to response times and errors. The second is crawl demand: how much it actually wants your URLs, based on how popular they are and how stale its copies have become. The lower of the two governs. Google's own documentation is unusually direct about scale — a site with a few thousand URLs is generally crawled efficiently and needs no attention here, and the guidance targets sites above roughly a million unique pages that change weekly, or above ten thousand pages that change daily. Crawling is not indexing: a fetched URL still has to earn a place in the index.
Why It Matters
For a forty-page static site the honest answer is that crawl budget does not matter and time spent on it is time wasted. It starts to bite when the number of reachable URLs runs far ahead of the number of useful pages — faceted filters that multiply combinations, calendars that generate a page per date forever, session identifiers appended to every link, or a soft 404 pattern that keeps a thousand empty URLs returning 200. On sites like that a crawler can spend most of its fetches on rubbish and take weeks to notice a genuinely new page. The fix is almost never 'get more budget'; it is to stop generating URLs that were never worth fetching.
How It Works
A scheduler holds a queue of known URLs for your host, ordered by demand and refreshed from links, sitemaps and past crawl history. Fetch rate adapts to what the server does: fast 200 responses let the rate climb, while timeouts, 429s and 5xx errors make it back off, sometimes sharply. A robots.txt disallow stops a URL being fetched at all, which genuinely saves fetches, though it does not remove the URL from consideration for indexing. Server log files, or the Crawl Stats report in Search Console, show what was actually fetched and how long each response took — the only honest measurement of any of this.
Real-World Example
A studio publishes a 30-page documentation site on 99helpers at helio-docs.99helpers.site, served as flat static files from the edge. Every page returns in well under a hundred milliseconds, there are no query parameters and nothing generates URLs on the fly, so Googlebot works through the whole site in a couple of passes. Crawl budget never becomes a topic — it would only do so if the team later added a search page producing a unique crawlable URL per query.
Common Mistakes
- ✕Optimising crawl budget on a small site — below a few thousand URLs the queue is never the bottleneck, and the real problem is usually content or links
- ✕Blocking URLs in robots.txt to save budget and expecting them to drop out of results — blocking prevents fetching, not indexing, and the URL can still be listed
- ✕Reading a fall in crawl rate as a penalty — it usually tracks slow responses or a drop in how often the site changes
- ✕Letting tracking parameters create endless variants of the same page — each variant is a separate URL in the queue until it is consolidated
Related Terms
Search Engine Indexing
Indexing is the step where a search engine stores a fetched page so it can be returned for queries. It happens after crawling, and it does not follow automatically from it.
Soft 404
A soft 404 is a URL that returns a success status while the page itself says the content is missing. The words say not found; the HTTP response says everything is fine.
robots.txt
A plain text file at the root of a site that tells search engine crawlers which paths they may fetch. It is a request that well-behaved crawlers honour, not a lock on the door.
sitemap.xml
An XML file listing the pages of a site so search engines do not have to find them all by following links. It aids discovery; it does not affect ranking.
HTTP Status Code
An HTTP status code is the three-digit number a server returns at the head of every response, saying what happened to the request. It is the first thing on the first line — a 200 means here is the file, a 404 means there is nothing at that address.
Put a file online in seconds
Drop in a document, an image, a page or a whole static website and share the link — free, with no build step and no server to set up.
Host a file free →