Kodokon kodokon.com

Internationalization: lang, dir, charset and Unicode traps

Use lang, dir and the encoding declaration wisely, and defeat the Unicode normalization and directionality traps.

9 min · 3 questions

Open this lesson in Kodokon

The lang attribute is much more than metadata: it drives screen reader voices, hyphenation (hyphens: auto does not work without it), the selection of CJK glyphs (the same ideograph is drawn differently in Japanese and Chinese), the quotation marks generated by <q> and the CSS :lang() selector. It is inherited: declare it on <html> with a BCP 47 tag (en, en-GB, ja) and override it locally at every language change.

HTML
<html lang="en">
  <p>She said: <q>hello</q>.</p>
  <p lang="fr">Elle a dit : <q>bonjour</q>.</p>
</html>
The quotes of q follow the language: “hello” then « bonjour ».

dir=rtl flips the default alignment and the visual order of characters with neutral directionality. For unpredictable user content, dir=auto picks the direction based on the first strongly directional character. The <bdi> element isolates a fragment from the bidirectional algorithm: without it, an Arabic username followed by ": 12 points" sees its punctuation reordered into gibberish.

HTML
<p dir="auto">مرحبا - inferred direction: RTL</p>
<ul>
  <li><bdi>مستخدم</bdi>: 12 points</li>
  <li><bdi>Alice</bdi>: 8 points</li>
</ul>
bdi protects the layout from bidirectional usernames.

The <meta charset=utf-8> declaration must appear within the first 1024 bytes of the document: the browser runs a prescan limited to that window before choosing an encoding. With no declaration and no HTTP header, the default encoding depends on the user's locale (often windows-1252 in Western Europe) - never UTF-8. A BOM at the start of the file, however, overrides everything else, including the HTTP header.

JAVASCRIPT
const composed = "\u00e9";
const decomposed = "e\u0301";
console.log(composed === decomposed); // false
const nfc = decomposed.normalize("NFC");
console.log(composed === nfc);        // true
Two visually identical “é”, yet different in memory.

macOS stores some file names in decomposed form (NFD): an "é" coming from an upload may therefore differ from the one typed on a keyboard. Systematically normalize all user input to NFC before storing or comparing it. Finally, .length counts UTF-16 code units, not perceived characters: a family emoji combines more than seven of them; use Intl.Segmenter to count real graphemes.

Knowledge check

Make sure you remember the key points of this lesson.

  1. Where must the <meta charset=utf-8> declaration imperatively be located?
    • Anywhere in the <head>
    • Within the first 1024 bytes of the document
    • Before the DOCTYPE declaration
    • Within the first 512 bytes of the <body>
  2. How does dir=auto determine an element's direction?
    • It examines the first strongly directional character of the content
    • It applies the direction declared on the <html> element
    • It counts RTL and LTR characters and keeps the majority
  3. Why can two strings that both display “é” be different for ===?
    • Because one is encoded in UTF-8 and the other in UTF-16
    • Because Unicode casing differs by locale
    • Because one uses the precomposed character U+00E9 and the other the decomposed sequence e + U+0301