Search Engine Indexing
Definition
Crawling and indexing are two different operations and almost every misunderstanding in this area comes from treating them as one. Crawling is the fetch: a robot requests a URL over HTTP and receives a response. Indexing is what happens next — the HTML is parsed, JavaScript may be executed, the page is rendered, the text is broken into terms, and an entry is written into the search engine's store mapping those terms back to the URL. Only indexed pages can appear in results. A page can be crawled and never indexed, which Search Console reports in plain words as 'Crawled - currently not indexed', and a URL can even be indexed without being crawled at all when it is blocked by robots.txt but linked from elsewhere, in which case it appears as a bare address with no description.
Why It Matters
The distinction decides which lever you reach for. If a page is not in the index because the crawler cannot reach it, the fix is a route in — a link, a sitemap entry, an unblocked path. If it was fetched and left out, no amount of linking helps, because the engine made a judgement about the page itself: too thin, near-identical to a page already held, or filtered out as a non-canonical duplicate. People routinely spend weeks on the wrong one of these two problems. The other consequence is timing: a new URL typically waits days rather than minutes, so 'it is not showing up yet' is often just the queue.
How It Works
The pipeline runs discovery, crawl, render, index, serve. Discovery comes from links, from sitemap.xml, or from a manual submission. The crawler fetches, honouring robots.txt, and stores the response. Rendering uses a headless browser, so content that only appears after a script runs is picked up in a second pass rather than the first. At indexing time the engine checks robots meta directives, resolves the page against near-duplicates and picks a canonical for the cluster, then writes the entry. The URL Inspection tool in Search Console reports each stage for a single address, including the last crawl date and the canonical Google actually chose, which is usually the fastest way to find out where a page stopped.
Real-World Example
A one-page product brief goes live at lumen-brief.99helpers.site with no links pointing at it from anywhere. Nothing happens for a fortnight, because no crawler has a route to it. Adding a link from the author's existing site gets it fetched within days, and 99helpers serves the file unchanged at a stable address, so the version Google indexes is exactly the one a visitor sees. Two weeks later a search for the exact product name returns it.
Common Mistakes
- ✕Saying a page is not crawled when Search Console says it was crawled and not indexed — those are opposite problems with opposite fixes
- ✕Submitting a URL repeatedly for indexing — resubmission does not improve the page, and the reason for exclusion is usually stated in the report
- ✕Expecting a blocked URL to vanish from results — blocking stops the fetch, so the engine never sees the noindex it would have obeyed
- ✕Judging coverage by a site: query — it is an estimate, not an index count, and the Pages report in Search Console is the actual answer
Related Terms
Indexability
Indexability is whether a page is technically allowed into a search engine's index. It is eligibility, not a promise — an indexable page can still be left out.
noindex
noindex is a directive telling a search engine not to keep a page in its index. It only works if the crawler is allowed to fetch the page and read it.
Crawl Budget
Crawl budget is the number of URLs a search engine is willing and able to fetch from one site in a given period. For most sites it is not a real constraint.
Google Search Console
Google Search Console is Google's free tool for site owners. It reports which queries brought people to your pages, which pages Google has indexed, and what stopped it indexing the rest.
robots.txt
A plain text file at the root of a site that tells search engine crawlers which paths they may fetch. It is a request that well-behaved crawlers honour, not a lock on the door.
Put a file online in seconds
Drop in a document, an image, a page or a whole static website and share the link — free, with no build step and no server to set up.
Host a file free →