Indexability
Definition
Three states get muddled and they sit in a line. Crawlable means a robot is permitted to fetch the URL and can reach it. Indexable means that, once fetched, nothing about the response forbids storing it. Indexed means it is actually in the index and can be returned for a query. A page is indexable when all of the following hold: robots.txt does not disallow it, it returns a 200 status, it carries no noindex directive in a meta tag or an X-Robots-Tag header, it is not excluded from its duplicate cluster by pointing its canonical elsewhere, it is not behind authentication, and it has some content once rendered. Fail any one of those and it is not eligible, however good the page is.
Why It Matters
The value of the term is diagnostic. When a page is missing from results, it is either not eligible or eligible and not chosen, and those have entirely different remedies. Not eligible is a checklist you can work through in ten minutes and fix with certainty. Eligible but not chosen is a judgement about the page — too thin, too similar to something already held, too far from anything anyone searches for — and no technical change will move it. Rewriting a page that is blocked by a stray disallow line wastes a week. The checklist is the cheap step, so it goes first.
How It Works
Work down the response. Fetch the URL as a crawler would and check the HTTP status code first: a 404, a 410 or a 500 ends the question, and a 301 moves it to another address. Read the response headers for an X-Robots-Tag, then the head of the document for a meta robots tag, remembering that a page blocked in robots.txt is never fetched, so a noindex on it is never seen and the URL can still be listed bare. Check the canonical URL the page declares, because a page canonicalised to a different address is asking to be dropped from results in favour of that one. Finally, view the rendered HTML rather than the source, since a page whose entire body is built by JavaScript may be empty at first pass. Search Console's URL Inspection tool reports all of these for one address at a time.
Real-World Example
A team copies a robots.txt from an old project into a new static site on 99helpers at brightmoor-site.99helpers.site, and the file still contains a blanket disallow from the staging days. Every page is crawlable-looking, well written and entirely absent from search. Re-uploading the two-line file without the disallow makes the whole site indexable again, and the first pages reappear in results within days — no content change, no links, just one line removed.
Common Mistakes
- ✕Treating indexable and indexed as the same thing — eligibility is a prerequisite, and Google still chooses what is worth storing
- ✕Checking only the meta tag — an X-Robots-Tag header does the same job invisibly, and it is easy to miss without looking at the response
- ✕Auditing the source HTML when the page is built by scripts — the rendered output is what gets indexed and may differ completely
- ✕Leaving a canonical pointing at the home page site-wide — every page then asks to be dropped in favour of one address
Related Terms
Search Engine Indexing
Indexing is the step where a search engine stores a fetched page so it can be returned for queries. It happens after crawling, and it does not follow automatically from it.
noindex
noindex is a directive telling a search engine not to keep a page in its index. It only works if the crawler is allowed to fetch the page and read it.
Robots Meta Tag
The robots meta tag is a line in the head of an HTML page that tells search engines how to treat that one page. It controls indexing and display, not crawling.
Canonical URL
A canonical URL is the one address you nominate as the real home of a piece of content when several addresses serve it. You declare it with a rel=canonical link in the page head, and search engines use it to decide which version to index.
HTTP Status Code
An HTTP status code is the three-digit number a server returns at the head of every response, saying what happened to the request. It is the first thing on the first line — a 200 means here is the file, a 404 means there is nothing at that address.
Put a file online in seconds
Drop in a document, an image, a page or a whole static website and share the link — free, with no build step and no server to set up.
Host a file free →