Bot Traffic
Definition
Bot traffic is every request that no one sat and made. It divides into the useful and the unwanted. The useful side includes search engine crawlers such as Googlebot and Bingbot, the preview fetchers that chat apps and social networks send when a link is pasted, uptime monitors, and archivers. The unwanted side includes content scrapers, price harvesters, vulnerability scanners walking a list of known paths, and credential stuffers. Industry measurements have put automated requests at roughly a third to a half of all web traffic for years, and for a small site with few real visitors the proportion skews higher still, because the bots arrive whether anyone is interested or not.
Why It Matters
The practical damage is usually to your understanding of the site rather than to the site itself. A view count that includes every crawler and preview fetch will tell a freelancer their proposal was opened nine times when the client opened it twice, and decisions get made on that number. Bandwidth is the second cost: a scraper pulling a 40 MB image folder on a loop can move more data in a night than a year of genuine readers. There is also a slower harm, which is that scrapers copy content wholesale and republish it, and the first sign is usually a duplicate page outranking the original. None of this is an argument for blocking everything — turning away search crawlers removes the site from search results entirely.
How It Works
Bots announce themselves, more or less honestly, in the User-Agent request header, and the well-behaved ones also read robots.txt before fetching anything and respect what it disallows. Because that header is just a string a client chooses, a scraper can claim to be Chrome, so identification also leans on behaviour: request rate, the order pages are fetched in, whether assets and fonts are loaded alongside the HTML, and whether the client runs JavaScript at all. Analytics that count at the edge filter known bots by matching the user agent against a maintained list, which catches the declared ones and misses the disguised. Firmer measures escalate from a challenge to a block returning HTTP 403, usually applied by a web application firewall. Whatever the method, the evidence lives in the access log, one line per request, which is where an unexplained traffic pattern is actually diagnosed.
Real-World Example
A photographer publishes a portfolio at studio-north.99helpers.site and sees 4,300 views in the first week, which seems wonderful until the detail is read. Most of it is one client fetching every image at four in the morning from the same network, plus a steady trickle of link previewers from a Slack channel where the URL was pasted. With bot filtering applied, the honest figure is about 260 visits from perhaps 90 people — smaller, but a number worth acting on. The 99helpers analytics show the countries and referrers behind those visits, which made the pattern obvious once someone looked.
Common Mistakes
- ✕Reporting raw hits as an audience — unfiltered totals flatter the site and mislead whoever is paying for it
- ✕Blocking all automated clients in a fit of tidiness — that includes Googlebot, and the site quietly leaves search results
- ✕Trusting robots.txt to keep scrapers out — it is a request, not a control, and the clients you most want gone ignore it
Related Terms
Web Application Firewall
A filter that reads each incoming HTTP request and blocks the ones matching known attack patterns. Unlike a network firewall, it looks at what a request says, not merely where it came from.
Access Log
A line-by-line record of the requests a server has answered — who asked, for what, when, and what came back. It records visitors, not account activity.
View Count
The number of times a page or file was successfully requested. It counts requests, not people — one person reloading four times produces four views.
robots.txt
A plain text file at the root of a site that tells search engine crawlers which paths they may fetch. It is a request that well-behaved crawlers honour, not a lock on the door.
Web Analytics
Measuring who visits a site and what they look at. The two methods — a script running in the visitor's browser, or counting each request as it is served — produce different numbers, and neither of them is wrong.
Put a file online in seconds
Drop in a document, an image, a page or a whole static website and share the link — free, with no build step and no server to set up.
Host a file free →