Robots Meta Tag
Definition
The tag is a meta element with name="robots" and a content attribute listing one or more comma-separated directives. The common ones are noindex (keep this page out of results), nofollow (do not follow links on this page), noarchive (no cached copy), nosnippet (no description text), noimageindex (do not index images on the page), none (equivalent to noindex plus nofollow) and unavailable_after with a date. Display controls take values, such as max-snippet:-1 for no length limit, max-image-preview:large and max-video-preview:-1. Swapping the name for a specific crawler, as in name="googlebot", narrows a rule to that engine. The identical directives can be sent as an X-Robots-Tag HTTP header, which is how you apply them to a PDF or an image that has no head to hold a tag.
Why It Matters
This tag and robots.txt are constantly confused, and the difference is the whole point: robots.txt says which URLs may be requested, while the robots meta tag says what to do with a page once it has been fetched. That ordering has a hard consequence. Put noindex on a page and also disallow it in robots.txt, and the crawler never makes the request, never reads the tag, and the URL may go on appearing in results as a bare listing with no description. The correct sequence is to leave the URL crawlable, serve the directive, and only block it afterwards if there is still a reason to. Defaults matter too — a page with no tag is treated as index,follow, so silence is permission.
How It Works
The tag must sit inside the head element to be read reliably; placed in the body, it may be missed or ignored. A crawler parses the head, collects the directives, and applies the most restrictive interpretation when sources conflict — a noindex anywhere in the set wins over an index elsewhere. Directives injected by JavaScript are only seen after rendering, a later and less certain pass than the initial fetch, so a tag that exists only in the rendered DOM is a weaker instruction than one in the served HTML. Google's URL Inspection tool shows both the raw and the rendered version, which is the quickest way to prove which tag was actually served.
Real-World Example
A design team keeps a staging copy of a client site at oakline-staging.99helpers.site while the real site is built. Adding one meta robots line with content="noindex, nofollow" to each staging page keeps the whole copy out of results without touching the files that will eventually ship. Because 99helpers serves the uploaded HTML unchanged, re-uploading the pages with that line removed is all it takes to make the production version indexable later.
Common Mistakes
- ✕Disallowing the URL in robots.txt as well — the crawler cannot read a tag on a page it is not allowed to fetch, so the directive has no effect
- ✕Placing the tag in the body — only a tag in the head is reliably honoured
- ✕Adding the tag with a script after page load — it may be picked up at render time, but the served HTML is what gets read first and trusted most
- ✕Leaving a site-wide noindex in place after a launch — it is the single most common reason a brand new site never appears in search at all
Related Terms
noindex
noindex is a directive telling a search engine not to keep a page in its index. It only works if the crawler is allowed to fetch the page and read it.
nofollow
nofollow is a value in a link's rel attribute saying the publisher does not vouch for the destination. Since March 2020 Google treats it as a hint rather than an instruction.
robots.txt
A plain text file at the root of a site that tells search engine crawlers which paths they may fetch. It is a request that well-behaved crawlers honour, not a lock on the door.
Indexability
Indexability is whether a page is technically allowed into a search engine's index. It is eligibility, not a promise — an indexable page can still be left out.
Meta Description
A one-line summary of a page, written into a meta tag in the head of the HTML, that search engines often show as the grey text under a result. It affects clicks, not rankings.
Put a file online in seconds
Drop in a document, an image, a page or a whole static website and share the link — free, with no build step and no server to set up.
Host a file free →