noindex
Definition
There are exactly two supported ways to issue a noindex. One is a robots meta tag in the head of an HTML document, written as a meta element with name="robots" and content="noindex". The other is an X-Robots-Tag HTTP response header carrying the same value, which is the only option for files that have no head section — a PDF, an image, a plain text file. Google and Bing both honour it. What does not work is a Noindex line in robots.txt: Google supported that undocumented behaviour for years and stopped honouring it on 1 September 2019. A noindex tells the engine to drop the page from results; it says nothing about whether the page may be fetched, and the two are separate questions.
Why It Matters
The trap is worth stating carefully, because it defeats people constantly: if you block a URL in robots.txt and also put noindex on the page, the crawler never fetches the page, never sees the directive, and the URL can go on appearing in results as a bare address with no title or description. Blocking prevents crawling. Only a directive the crawler actually reads prevents indexing. The order to apply is therefore counter-intuitive — allow the crawl, serve the noindex, and only then consider blocking the URL if you still want to. For genuinely confidential material neither mechanism is the answer: both are requests that well-behaved crawlers honour and others ignore.
How It Works
On the next fetch of the URL, the crawler parses the response, finds the meta tag or the header, and marks the page for removal from the index. Removal is not instant; it takes effect at the next crawl and processing cycle, which can be days or weeks depending on how often that URL is visited. The default pairing is noindex,follow, so links on the page are still traversed, though Google has said that over the long run links on a persistently noindexed page end up treated as nofollow anyway. To remove something urgently, the Removals tool in Search Console hides a URL for about six months, which buys time while the noindex takes effect.
Real-World Example
A consultancy uploads a client proposal to 99helpers at ardent-proposal.99helpers.site to share the link by email, and does not want it turning up in search. Because the page is a plain HTML file served as uploaded, adding one meta robots noindex line to the head is the whole fix. For the accompanying PDF, which has no head to put a tag in, the practical answer on a static host is 99helpers' password protection: the crawler only ever reaches the form, so there is no document to index.
Common Mistakes
- ✕Combining a robots.txt disallow with a noindex tag — the block stops the fetch, so the tag is never read and the URL can still be listed
- ✕Putting the tag in robots.txt instead of on the page — Google stopped honouring Noindex lines there on 1 September 2019
- ✕Expecting the page to disappear the same day — it drops out at the next crawl, which for a rarely visited URL can be weeks
- ✕Using noindex where a canonical belongs — for two versions of the same page you usually want consolidation, not deletion from the index
Related Terms
Robots Meta Tag
The robots meta tag is a line in the head of an HTML page that tells search engines how to treat that one page. It controls indexing and display, not crawling.
Indexability
Indexability is whether a page is technically allowed into a search engine's index. It is eligibility, not a promise — an indexable page can still be left out.
Search Engine Indexing
Indexing is the step where a search engine stores a fetched page so it can be returned for queries. It happens after crawling, and it does not follow automatically from it.
robots.txt
A plain text file at the root of a site that tells search engine crawlers which paths they may fetch. It is a request that well-behaved crawlers honour, not a lock on the door.
Password-Protected Link
A hosted address that asks for a shared secret before it will show the file or site behind it. One password covers everyone who has the link; there are no individual accounts.
Put a file online in seconds
Drop in a document, an image, a page or a whole static website and share the link — free, with no build step and no server to set up.
Host a file free →