A while ago, the following appeared in a database for which I am the admin:
Jubiläumsschrift.pdf
Jubiläumsschrift.pdf
Jubiläumsschrift.pdf
Jubiläumsschrift.pdfApparently, it should read Jubiläumsschrift.pdf, but the ä umlaut broke in a very peculiar way. Why?
Character Encodings #
For processing in computers, text must be converted into a sequence of bytes. One byte is 8 binary digits (bits) long and can thus take values from zero (inclusive) to 256 (exclusive). The process for this is called character encoding. In the early days of computing, a simple association between individual characters and numbers was used.
ASCII #
The first character encoding in widespread use was ASCII (American Standard Code for Information Interchange), which had only 7 bits, since the 8-bit byte was not universally adopted at that time. Therefore, ASCII encodes 128 characters. Among these are
- the 26 lowercase and uppercase letters of the Latin alphabet
- 10 digits
- some punctuation characters
- some special characters
- control characters. They encode e.g. the end of a file and the line break.
8 Bit Proliferation #
This works great as long as you only want to process English text and programming code (and do not need currency signs other than the dollar sign $ and do not use “typographically correct” quotation marks).
However, because computers were also in use in other countries by now, the character set had to be extended.
Meanwhile, the 8-bit byte was universally adopted. Since each character was stored in one byte anyway, ASCII left one bit free to use!
And thus the proliferation of character encodings began.
So it was possible to encode some more writing systems. The ISO 8859 family found widespread use and provides 15 encodings for Latin-based alphabets as well as Hebrew and Arabic.1 Since Microsoft does not value standards and interoperability particularly much, Windows has its own encodings, the so-called Windows code pages.
All of these encodings have in common that the first 128 characters are the same as in ASCII. Therefore, ASCII-encoded texts are correctly displayed in the 8-bit encodings.
This also works great if it is known which encoding was used for a given text. For your own files this is easy, but as the Internet gained more adoption, the exchange of data (in particular web sites and email) between computers increased. Thus the encoding had to be specified with each transferred text (and the software had to handle these encodings correctly).
But there were even further problems:
- Texts containing multiple languages whose characters are not in a single encoding cannot be processed.
- Scripts with more than 128 characters in addition to the Latin alphabet could not be supported (a completely different encoding had to be used instead).
- Since all 256 possible values in the 8-bit encodings were already assigned, new characters could only be supported by new encodings. For example, ISO 8859-1 (Western European) was superseded by ISO 8859-15, which also encodes the Euro sign
€. However, it still does not contain the uppercase sharp Sẞ, which was only introduced later.
Unicode #
To solve all of this, Unicode was developed, and it was supposed to be able to encode all characters used anywhere in the world. Initially it had capacity for 65536 (fitting in 16 bits), since that is sufficient to encode all characters of all living languages (haha, no).
However, Unicode is not really a character encoding but only a list of so-called code points.
Code points can be characters or components thereof or fulfill other purposes, like controlling line breaks or ligatures.
Referencing a code point is usually done in the form U+<number> where <number> is specified as a hexadecimal number and is filled to 4 places.
Anyway, a text is a sequence of code points. In order to represent this sequence of code points as a sequence of bytes (because only those can be processed by a computer)
a Unicode encoding is needed.
An important property of Unicode is forward compatibility: no future version of the Unicode standard will remove or repurpose a code point. And this promise was kept, even for characters whose meaning is unknown and which were likely included in Unicode by mistake.2
The first 128 code points are the ASCII characters. While it may seem strange that the dollar sign $ is part of the basic Latin alphabet, this ensures a limited compatibility with older encodings (more on that later).
Similarly, the code points 128 to 255 are the same as in ISO 8859-1.
But even the Unicode code points are not problem-free: many graphemes (known among non-linguists as “characters”) can be represented as code points in multiple ways.
For example, the uppercase A with diaereses Ä is encoded as code point U+00C4,
but can also be represented by U+0041 (Latin uppercase letter A) followed by U+0308 (combining diaeresis): Ä.
(It should be mentioned that U+0308 must be differentiated from U+00A8 diaeresis ¨.)
Similar redundancies can be found in the Hangul script, which is used for the Korean language:
Here, Unicode encodes both individual jamo (letters) as well as Hangul syllables, which are composed of jamo.
Furthermore, some ligatures like fi (U+FB01 Latin lowercase ligature FI) and ffl (U+FB04 Latin lowercase ligature FFL) are encoded in Unicode despite only representing the character sequences fi and ffl, respectively.
Even some characters used by multiple East Asian scripts, but with different meanings, are encoded as a single code point.
This controversial practice is called Han unification.
This is in part motivated by technical constraints: If all variants of all characters were encoded separately, the originally envisaged maximum number of code points of 65536 would not have been sufficient.
Additionally, the prevalent opinion was that the variants used in different languages are only different graphical representations and do not have a (sufficiently) different meaning to warrant encoding as separate code points.
It should be kept in mind, however, that the European scripts fared better in this regard: the Latin A, the Cyrillic А, and the Greek Α are all encoded separately.
Even some typographic variants of Greek letters have separate code points, such as θ and ϑ, ς and σ, or φ and ϕ.
With version 2.0 of the Unicode standard, the limit of 65536 code points was removed. This made enough space for less commonly used East Asian characters and numerous historical scripts that are no longer in use, such as Egyptian hieroglyphs and cuneiform.
Unicode Encodings #
The Unicode code points are only one part of the story. In order to store and transmit a text, represented by a sequence of code points, in a computer, it must be converted to a sequence of bytes. This conversion is defined by a Unicode encoding. They are categorized as fixed-width and variable-width encodings.
Today, UTF-8 is the most commonly used character encoding and has a variable width. One code point takes 1 to 4 bytes (8 to 32 bits) to encode. Because the first 128 code points (which are the ASCII characters) are encoded in a single byte, an ASCII-encoded text can also be interpreted as a UTF-8-encoded text. Furthermore, texts that mostly consist of these characters (i.e. those using the Latin alphabet) need less space than when encoded with other Unicode encodings and only a little more than with the legacy 8-bit encodings.
When using variable-width encodings, the number of code points can no longer be simply read from the number of bytes the text is stored in. Instead, the entire text must be processed, counting the code points. Fortunately, this is irrelevant in practice because the number of code points is meaningless anyway.
Back when Unicode was designed for a maximum of 65536 code points, there was the UCS-2 encoding, which had a fixed width of 16 bits (2 bytes) and thus encoded every Unicode code point in 16 bits. When it turned out that 65536 code points were not enough, UCS-2 was developed into UTF-16. This encoding is commonly used by software to represent texts internally and also has a variable width. In contrast to UTF-8, a code unit is 16 bits wide. Therefore, one code point is encoded in 16 or 32 bits (2 or 4 bytes). The fixed-width encoding still working today is UTF-32 (also known as UCS-4), which encodes every code point with 32 bits (4 bytes).
The Letter Salad #
So how did the letter salad from the start of the article come to be? In short, a UTF-8-encoded text was interpreted as if it were encoded in Windows code page CP1252 and the result was again encoded in UTF-8. This cycle was repeated multiple times.
In the word “Jubiläumsschrift”, the letter ä is not in the basic Latin alphabet and thus not part of ASCII.
Therefore, it is the only letter that is encoded in two code units (each of which is 1 byte long).
Overall, “Jubiläumsschrift” encoded in UTF-8 is thus:
J u b i l ä u m s s c h r i f t
4A 75 62 69 6C C3 A4 75 6D 73 73 63 68 72 69 66 74This byte sequence was then wrongly interpreted as CP1252-encoded text.
For all ASCII characters (the letters of the basic Latin alphabet in our example), they are the same characters that were originally encoded.
Only for the ä is there a difference: in UTF-8 it is encoded with the two bytes C3 A4.
In CP1252, however, each character is only one byte long. ä is encoded as E4.
The byte sequence C3 A4 instead corresponds to the two characters ä.
Thus the above byte sequence is interpreted as this text:
4A 75 62 69 6C C3 A4 75 6D 73 73 63 68 72 69 66 74
J u b i l à ¤ u m s s c h r i f tNow the cycle repeats: the text is encoded back in UTF-8. Now, the characters à and ¤ are the only non-ASCII characters and are encoded in two bytes each:
J u b i l à ¤ u m s s c h r i f t
4A 75 62 69 6C C3 83 C2 A4 75 6D 73 73 63 68 72 69 66 74Interpreted as CP1252, this text results:
4A 75 62 69 6C C3 83 C2 A4 75 6D 73 73 63 68 72 69 66 74
J u b i l à ƒ  ¤ u m s s c h r i f tIf this is repeated some more times, we eventually arrive at the monstrosity from the start.
Trying it out #
Here you can see for yourself which texts result after one or multiple wrong encodings. Hint: Choose an initial text that contains non-ASCII characters.