UTF-8 Was Sketched on a Diner Placemat. It Now Runs the World

Hinzugefügt: 2026-05-26

[00:00:00]According to the historical record, September 2nd, 1992, a diner somewhere in New Jersey. It's late, the food is getting cold, and one of the two men at the table has stopped eating. He's writing on the placemat, not on paper, not on a whiteboard back at the lab. On the paper placemat under his plate in a roadside diner, he's sketching out a system to represent every written language that has ever existed, Latin, Japanese, Arabic, Chinese. And though he doesn't know it yet, every script not yet digitized. The man writing is Ken Thompson. The man watching is Rob Pike.

[00:00:30]By Friday, three days later, the entire operating system they worked on would be running on what was scribbled on that placemat. Almost nobody expected it to survive a year. This is the story of UTF8, the temporary hack that quietly ate the world. To understand why two men were sketching on a placemat, you have to understand the disaster they were trying to escape. Here's the myth most people carry. That text on a computer just works. You type a letter, the computer stores a letter. Anyone in the world opens it and sees the same letter.

[00:01:00]Clean, obvious, solved decades ago. It was not solved. For most of computing history, text was a war zone. The original sin was a number, 256. Early computers stored each character in one bite. One bite holds 256 different values. For English, that was plenty.

[00:01:20]The American standard ASI used only the first 128 of those slots, the uppercase and lowercase letters, the digits, the punctuation, a handful of control codes.

[00:01:29]Half the bite sat empty. But the world does not write in English. So every country grabbed that empty upper half of the bite values 128 through 255 and stuffed in their own characters. Western Europe defined Latin 1 with the accented vowels and the N. The result was chaos because nobody agreed on what those upper values actually meant. You've probably seen the ghost of this war. You open an email or a web page and the apostrophes have turned into this. Those little smears of nonsense have a name.

[00:02:00]The Japanese coined it because they suffered the worst of it. Moji bake, character transformation, garbled text.

[00:02:07]And in much of the world, it was far uglier than a stray accent. Japanese, Chinese, and Korean have thousands of characters. They could never fit in a single bite. So they built their own multibbyte systems. Shiftjis, EU, big five, a dozen incompatible standards.

[00:02:23]Each one quietly assuming the rest of the world used the same one. They didn't. So opening a document from another country was a gamble. Same byes, different code page, completely different text. A Japanese email arrives on an American machine and becomes a column of question marks and empty boxes. This is the part people forget.

[00:02:41]In the early 1990s, the internet was about to go global, and it was on track to fragment permanently along language lines. A network where a message couldn't reliably survive crossing a border. The web was about to be born already broken. Something had to encode every language at once. In a way, every existing computer could already read.

[00:02:59]Right now, on the device you're watching this on, there are hundreds of millions of bytes of text rendering cleanly. a Japanese article, a Spanish headline, a Thai government document, all on the same screen without a single one of those boxes. That happened because of a decision made at a diner booth in New Jersey by two people who had no idea it would matter this much. Nobody hired them to do it. They just decided the thing in front of them was broken and fixed it. That's the wall Thompson and Pike walked up to, and it looked impossible. Here's the trap. The obvious fix is just use more bytes. give every character a fixed generous slot, 16 bits, enough for tens of thousands of characters, one number per character. No more 256 character ceiling. A standards body was already doing exactly that. It was called ISO 10646, the early universal character set. And plan 9, the operating system Thompson and Pike worked on, had already adopted a 16-bit version of it. They hated it because a fixed 16-bit encoding breaks the one thing the entire existing world was built on, ASI. Suddenly, the letter A isn't one bite anymore. It's two. Every program ever written, every file, every Unix tool that walked through text one bite at a time looking for a slash or a new line, all of it broke. You'd have to rewrite the world. So, the real problem wasn't how do we fit every language. It was something much harder. How do you fit every language on Earth into a system that only understands ASI without breaking the trillions of text and millions of programs that already exist?

[00:04:27]That's not an encoding problem. That's a constraint problem and the constraints were brutal. Constraint one is the foundation. If A stays exactly as it was, one bite, the value 65, then every English document ever written is already valid in the new system. Day one, no conversion. The world doesn't break. It just keeps running. Constraint two is subtler. In a multi-bite sequence, say a Japanese character, not a single one of those bytes is allowed to accidentally look like an ASKI letter. because if it did, an old program scanning for a slash or a space could find one inside a Japanese character and slice the text in half at the wrong spot. Constraints one and two are engineering. Constraint three, that's where the committee had already failed because there was already a proposal on the table. An engineer named Dave Proser at Unix System Laboratories had drafted the proposal for the Xopen committee. It was clever.

[00:05:15]It was almost there. It was also not self-synchronizing. In Proser's design, if you landed in the middle of a stream, a corrupted packet, a file you opened halfway through, you couldn't always tell where you were. To be sure, you'd have to scan from the very beginning. On a small file, fine. On a large one, that's a problem that compounds with every bite. Thompson's fix traded away a few bits of efficiency to buy one absolute guarantee. Every single bite announces what it is, a start bite or a continuation bite. No ambiguity ever.

[00:05:44]So, you can land anywhere in a file the size of a library. look at one bite and recynchronize in a single backward scan.

[00:05:51]Here's how he did it. This is the scheme Thompson sketched on that placemat, the one they probably threw out with the dishes. Read it slowly because the whole modern world is hiding in those bits.

[00:06:00]The high bits of each bite are a length code. They tell you instantly what kind of bite you're looking at. A bite that starts with zero is a plain old ASKY character. One bite. Done. That's the entire backward compatibility promise encoded in a single leading zero. A bite that starts with 1 1 is the start of a multibbyte character. And the number of ones tells you how long it is. 1 1 0 means two bytes total. 1 1 0 means three. 1 1 1 0 means four. And a bite that starts with 1 zero is never a start. It's always a continuation. The middle of a character. That last rule is the magic. Drop a needle anywhere into a stream of text. The middle of a file, a corrupted packet, anywhere, and look at one bite. If it starts with 1 zero, you've landed inside a character. So you step backward until you hit a bite that doesn't. You recynchronize in microsconds. You can never get permanently lost. That's the difference between a clever design and a design that runs the planet for 30 years. And the proof of how right it was, they didn't argue about it for months.

[00:07:02]Thompson sketched it Wednesday evening.

[00:07:04]They coded it that night. By Thursday, it worked. By Friday, the entire plan 9 operating system was running on UTF8 top to bottom, 2 and 1/2 days. Then they called the committee back and told them, "We already built it and it's better." 6 days later, September 8th, Thompson sent the final proposal to the committee.

[00:07:23]They agreed it was better than theirs and adopted it. Now, jump forward 30 years to the device in your hand.

[00:07:29]There's a distinction almost everyone blurs, and it's worth 30 seconds because the whole modern stack depends on it.

[00:07:35]Unicode is the list. It's the universal catalog that assigns every character on Earth a single permanent number called a code point. The letter A is code point 65. The euro sign is code point 8364. A flame emoji has its own number, too.

[00:07:51]UTF8 is how you store and send that number, how a code point turns into actual bytes on a wire or a disc.

[00:07:57]Unicode says what the character is. UTF8 decides how it travels, and UTF8 won that job almost completely. Let's run a few characters through the placemat scheme. Start with the simplest, one bite, starts with zero. It's identical to the asie it was in 1963. An English text file from 60 years ago is still a perfectly valid UTF8 file today. Nothing about it ever had to change. That's the backward compatibility promise kept for half a century. Now, the euro sign, a character that didn't even exist when the placemat was written. Three bytes.

[00:08:29]The first one starts with 11 1 0. The length code saying three bytes. Here we go. The next two start with one zero.

[00:08:37]Continuations. The system Thompson sketched in 1992 swallowed a currency symbol invented in 1999 without changing a single rule. And then there's this one. The character that proves the placemat is still in charge of your phone. The fire emoji code.1 F525 in UTF8. Four bytes. The first bite starts with 11110.

[00:09:00]the four byte length code, the deepest one in the scheme. The other three are continuation bytes, each starting with one zero. Every fire emoji you have ever sent, everyone in every group chat on Earth is that exact four byte sequence.

[00:09:14]Traveling through wires and towers according to a rule scribbled on a diner placemat before emoji existed, before the modern web existed, before most of the people sending them were born.

[00:09:24]That's the part that should stop you.

[00:09:25]The placemat sketch wasn't updated to handle emoji. It didn't need to be. The design was general enough from the first evening to absorb characters its inventors could never have imagined. And here's the quiet irony at the center of this whole story. UTF8 was the underdog.

[00:09:39]Two engineers and a placemat against an international committee with years of meetings and formal proposals. The committee's instinct was the natural one. More structure, fixed widths, careful formal design. The placemat's instinct was the opposite. Stay compatible. Stay simple. Let one leading bit do the heavy lifting. The simple, elegant thing didn't just win. It outlived almost everything that was built to replace it. We like to believe the systems running our lives were designed deliberately in clean rooms by people who knew exactly what they were building and how long it would last.

[00:10:11]It's comforting. It feels safe. The truth is messier. An enormous amount of the modern world rests on decisions that were improvised under pressure in a hurry by a couple of people who were just trying to make the thing in front of them stop being broken. UTF8 was supposed to be temporary. It was a hack.

[00:10:27]Nobody in that diner thought they were writing the foundation of all digital text for the next 30 years. But that's also the hopeful part because the reason it lasted isn't luck. It lasted because it was simple. Because it respected what already existed instead of demanding the world rebuild itself. Because one elegant idea, one leading bit did the work that a mountain of committee structure couldn't. The lesson the placemat keeps teaching. The thing that survives is rarely the most sophisticated design in the room. It's the one humble enough to stay compatible and clever enough to need almost nothing. UTF8 is one of those invisible old technologies quietly holding up the entire modern world. And it is very far from the only one because there is another one. A language that was declared dead in 1985 that no university wants to teach that should have been retired decades ago and that still runs the bank account behind your card right now. We break down exactly how and why nobody can turn it off in our video, why banks still run on a language declared dead in 1985. The placemat is gone. The hack is still running. Watch that one next.