robots.txt
Definition
robots.txt is a small text file served from exactly one place: the root of a host, at /robots.txt. Crawlers read it before they read anything else on that host. The syntax is a list of groups, each beginning with a User-agent line naming the crawler, followed by Disallow and Allow lines giving path prefixes. A file may also carry one or more Sitemap lines pointing at a sitemap.xml, which is how most crawlers discover a full list of pages. Nothing in it is enforced by the server, so a path listed under Disallow is still publicly readable by anyone who types the address. Treat it as traffic direction for robots, not as access control.
Why It Matters
A stray Disallow: / costs a site every organic visit it would otherwise have had. Google drops the affected pages within days and recovery takes weeks after the fix, which is why the line most often shipped by accident — the one copied from a staging site — is also the most expensive. The opposite error is quieter: leaving a staging copy crawlable, so two near-identical sites compete for the same queries and neither does well. Blocking also has a limit people rarely expect. A disallowed URL can still appear in results, listed with no description, if other sites link to it, and the crawler cannot see the canonical URL or noindex tag on a page it is not allowed to fetch.
How It Works
A crawler that wants to visit example.com first requests https://example.com/robots.txt. If it gets a 200, it parses the groups and picks the one whose User-agent best matches its own name, falling back to the wildcard group User-agent: *. Rules match against the path portion of a URL by prefix, with * as a wildcard and $ anchoring the end, and the most specific matching rule wins rather than the first one written. A 404 is read as permission to crawl everything; a run of 5xx responses is read by Google as a temporary instruction to crawl nothing, so a flaky host can suppress crawling without anyone noticing. On a static site the file is simply another file you upload, which means it has to sit at the top level of the deployed folder rather than inside a subdirectory.
Real-World Example
A design studio publishes a portfolio at fold-studio.99helpers.site and keeps last year's version in a /v1/ folder inside the same upload. Both get crawled, and the result for the studio's name sometimes points at the old draft. A two-line robots.txt — User-agent: * followed by Disallow: /v1/ — stops the draft being fetched, and a Sitemap line added underneath gives the crawler an unambiguous list of the pages that do count. Because 99helpers serves whatever you upload at the path you uploaded it to, the fix is one more file at the root and no configuration screen at all.
Common Mistakes
- ✕Using Disallow to hide something private — the file is public, so it advertises the exact path you wanted nobody to find
- ✕Putting the file in a subfolder — crawlers only ever read /robots.txt at the host root, so a copy in /public/ or /assets/ is never seen
- ✕Shipping the staging rule to production — Disallow: / and Disallow: /tmp/ differ by a few characters and nothing on the live site warns you
- ✕Expecting a blocked page to vanish from search — blocking stops the fetch, not the listing
Related Terms
sitemap.xml
An XML file listing the pages of a site so search engines do not have to find them all by following links. It aids discovery; it does not affect ranking.
Meta Description
A one-line summary of a page, written into a meta tag in the head of the HTML, that search engines often show as the grey text under a result. It affects clicks, not rankings.
Static Site
A static site is a website made of finished files — HTML, CSS, JavaScript, images — that are sent to the browser exactly as they are stored. Nothing is assembled on the server when a visitor arrives.
Canonical URL
A canonical URL is the one address you nominate as the real home of a piece of content when several addresses serve it. You declare it with a rel=canonical link in the page head, and search engines use it to decide which version to index.
Password-Protected Link
A hosted address that asks for a shared secret before it will show the file or site behind it. One password covers everyone who has the link; there are no individual accounts.
Put a file online in seconds
Drop in a document, an image, a page or a whole static website and share the link — free, with no build step and no server to set up.
Host a file free →