Character Encoding
Definition
Unicode assigns a number, called a code point, to every character in every writing system it covers, currently over 150,000 of them. An encoding decides how those numbers become bytes. UTF-8 is the one the web settled on: it uses a single byte for the 128 ASCII characters, two for most European accented letters, three for the bulk of Asian scripts and four for emoji and the rarer symbols. That design is why UTF-8 is backwards compatible with ASCII — a file of unaccented English is byte-for-byte identical in both. The older single-byte encodings, Windows-1252 and ISO-8859-1 chief among them, could only ever represent 256 characters and are the source of most of the damage still seen today. W3C surveys put UTF-8 above 98 per cent of web pages, and HTML5 has named it the required encoding for authoring since 2014.
Why It Matters
The failure mode has a name: mojibake, from the Japanese for character transformation. A UTF-8 file read as Windows-1252 shows the name Müller as Müller, a pound sign as £, and an em dash as a three-character jumble, and every one of those is a silent substitution rather than an error anyone gets told about. It is worst where a file crosses a boundary — a CSV exported from a web application and opened in Excel, a form submission stored and later printed, a database migrated between servers. Once mojibake is written back and saved, the original characters are usually gone for good. Fixing it costs nothing at the point of writing and can cost a day of scripted repair afterwards.
How It Works
Three places can declare an encoding, and they do not carry equal weight. The HTTP header wins: Content-Type: text/html; charset=utf-8 overrides anything inside the file. Next comes the in-document declaration — a meta charset tag, which HTML5 requires within the first 1024 bytes, or the encoding attribute of an XML declaration. If neither appears, the browser guesses from the bytes and the user's locale, which is where the trouble starts. A byte order mark, the three bytes EF BB BF at the front of a UTF-8 file, is a fourth signal: Excel on Windows relies on it to read a CSV correctly, while a JSON parser or a shell interpreter usually chokes on it. Note the charset parameter only applies to text types — application/json has no charset parameter at all, because RFC 8259 already mandates UTF-8.
Real-World Example
A membership list is exported as members.csv, uploaded to 99helpers, and published at directory.99helpers.site/members.csv. It is served as text/csv with charset=utf-8, so the Danish and Turkish names in it render correctly in the browser preview. One month the export tool is reconfigured and writes Windows-1252 instead; the charset in the header still says UTF-8, so the accented names appear as replacement characters within seconds of opening the link. Because the previous file is still in version history, the fix is a rollback while the export is corrected, rather than a scramble to rebuild the list.
Common Mistakes
- ✕Omitting the meta charset tag and relying on the browser's guess — the page looks correct on your machine and wrong on a reader's with a different locale
- ✕Declaring UTF-8 in the header while the editor saves Windows-1252 — the declaration does not convert anything, it only says what the bytes are supposed to be
- ✕Adding a byte order mark to JSON, a shell script or an HTML page — it is invisible in the editor and breaks parsers that expect the first character to be meaningful
- ✕Repairing mojibake by hand in a spreadsheet — re-export from the source with the right encoding instead, because the damaged file has already lost information
Related Terms
CSV File
A CSV is a plain text table: one record per line, fields separated by commas, with quotes around any field that contains a comma or a line break. It has no types, no formatting and no formulas — which is both its strength and the source of most of its trouble.
Plain Text File
A plain text file holds characters and line breaks and nothing else — no fonts, no sizes, no embedded images. It is the format that survives everything, and the one that carries no clue about how to read its own bytes.
MIME Type
A MIME type, also called a media type or content type, is the short label a server sends to say what kind of data a response contains — text/html, image/png, application/pdf. The browser decides what to do with the bytes based on that label, not on the file name.
URL Encoding
URL encoding replaces characters that are unsafe or reserved in a web address with a percent sign and two hex digits, so a space becomes %20. It is why a filename with spaces or accents can break a link.
HTML File
An HTML file is the source of a web page: nested elements that describe headings, paragraphs, links, images and forms. It is the one format a browser executes rather than merely displays — it fetches what the file references and runs the scripts it names.
Put a file online in seconds
Drop in a document, an image, a page or a whole static website and share the link — free, with no build step and no server to set up.
Host a file free →