Kodokon kodokon.com

Accented characters, UTF-8 and line endings

Understand why accents sometimes come out garbled and what those invisible line endings are hiding.

7 min · 3 questions

Open this lesson in Kodokon

A machine has no idea what the letter A is. All it handles is numbers. So to write text, everyone had to agree on a lookup table: the number 65 means A, 66 means B, and so on. That convention is called an encoding. The oldest one, ASCII, dates back to the 1960s and only covered the English alphabet: no é, no ç, no ü, and not a single Chinese character.

Every country then invented its own table, and chaos followed: a French file opened with the Russian table came out as gibberish. The modern solution is called Unicode, a single catalogue that assigns a number to every character in every language, emoji included. And UTF-8 is the standard way of writing those numbers into a file. Today the rule is simple and has no exceptions: always save your files as UTF-8.

TEXT
Written in UTF-8, read as UTF-8    ->  Creme brulee (accents correct)
Written in UTF-8, read as Latin-1  ->  Crème brûlée
Written in Latin-1, read as UTF-8  ->  Cr?me br?l?e
The same file read with the wrong table turns into gibberish.

Second invisible thing: the line ending. When you press Enter, no line gets drawn in the file. One or two invisible characters are added instead, and here again the systems never agreed with each other. macOS and Linux use a single character, LF (line feed), written \n. Windows uses two in a row, CRLF (carriage return then line feed), written \r\n. It comes straight from typewriters, where two separate moves were needed: push the carriage back, then roll down one line.

TEXT
What you see on screen:

Hello
Hi there

What the file really contains:

macOS / Linux (LF)     Hello\nHi there
Windows (CRLF)         Hello\r\nHi there
Two files that look identical on screen, two different contents in reality.

Knowledge check

Make sure you remember the key points of this lesson.

  1. What is going on when é shows up as é?
    • The file is being read with a different encoding than it was written with
    • The file is corrupted and has to be recreated
    • The accent was stripped out by the system
    • The font is missing
  2. Which line ending has Windows used historically?
    • CRLF
    • LF
    • CR on its own
  3. Which settings should you go for in your code editor?
    • UTF-8 for the encoding and LF for line endings
    • ASCII for the encoding and CRLF for line endings
    • Latin-1 for the encoding and CR for line endings