Tokenization Just Got 1,000x Faster! (Gigatoken)

[00:00:00]Type a sentence into an AI model, and before it can do anything else, this happens. Your words get chopped into tokens. It runs on every prompt you send and every model that gets trained. It is the plumbing most people rarely think about, and that plumbing has been slower than it should be. Turning text into tokens is a real measurable bottleneck.

[00:00:21]This week, one developer shipped a tokenizer that runs 100 times faster than OpenAI's tiktoken. Not a benchmark trick, 24 GB of text every second on an ordinary server.

[00:00:34]And the catch you are bracing for is not there. The output is identical, token for token. It is called gigatoken, open source, MIT licensed, one command to install. Pip install gigatoken, and it slots straight into the code you already wrote. Why care how fast a tokenizer runs? Because it is the step every training job waits on first, and the tools it laps, tiktoken and Hugging Face, are already fast multi-threaded Rust. The person behind it teaches a Stanford course on building language models from scratch. Here is the number that set r/LocalLlama on fire this week. Take 11.9 GB of real web text, an actual training corpus, not a toy string, and tokenize it with GPT-2's tokenizer. On a big AMD server, gigatoken finished the entire file in under half a second. That works out to 24 and a half GB every second.

[00:01:29]It is hard to picture a number that big, so anchor it. A typical novel is about half a megabyte of text. At 24 GB a second, that is on the order of 50,000 novels tokenized every second on a single machine. To feel why that is wild, you need to know what it beat.

[00:01:47]Hugging Face's tokenizers library is the default that almost every open model ships with. Tiktoken is the tokenizer OpenAI wrote for GPT-4 and its newer models. These are not lazy baselines.

[00:02:00]They are the fast, trusted, industry standard tools. So, put them on the exact same GPT-2 job. Hugging Face managed about 25 MB a second. TikToken came in around 36. Look hard at the units. Those are megabytes, while GigaToken is doing gigabytes. It is not a tall a bar on the chart. It is a different axis of the chart. Run the division and it stops sounding real. On that workload, GigaToken is roughly 900 times faster than Hugging Face and 680 times faster than TikToken. Same input, same token IDs coming out. Three full orders of magnitude sitting in the middle. And you do not need a data center chip to see it. On an Apple M4 Max, a laptop, that same GPT-2 tokenizer ran at nearly 9 GB a second. On that machine, that is close to 1,300 times faster than Hugging Face. The single biggest multiple in the whole table came from a MacBook. It is also not one lucky tokenizer that happened to line up with the code. Llama, Qwen, DeepSeek, GLM, Phi, Nematron, GPT-OSS, the model families you are actually running, every one of them clears 20 GB a second on that server. Scroll the benchmark table and it is the same result row after row.

[00:03:17]Three very different chips, one story. A 144-core AMD server, an Apple laptop, and a consumer gaming CPU were all tested and all three land in the gigabytes per second range.

[00:03:30]The speed tracks the code he wrote, not the price of the hardware you happen to run it on. The result that matters most is the cheapest one. On a Ryzen 9800X3D, the chip sitting in a lot of gaming PCs, GPT-2 tokenizes at 6 GB a second.

[00:03:47]That is still 68 times faster than TikToken on a machine that might already be under your desk. Here is the stat that stuck with me. At the server's rate, you could tokenize all of Common Crawl, roughly the readable internet, about 130 trillion tokens in under 6 and 1/2 hours. The entire public web chewed through in a single afternoon.

[00:04:10]So, how does one PhD student beat two teams of specialists by 100 to 1? The surprising answer is what he did not do.

[00:04:17]There is no exotic new algorithm hiding in here, and there is no secret hardware. It is the same math the whole field already runs. The method is byte pairing coding, the same scheme everybody uses. It has been the standard since GPT-2, and GigaToken leaves it completely untouched. Same merge rules, same vocabulary, same token IDs. Diff the output against Hugging Face, and it is byte-for-byte identical.

[00:04:43]Byte pairing coding itself is almost too simple. You start with raw characters, find the pair that shows up most often, glue it into one new token, and repeat that a few thousand times. That is how a model learns that the letters t h e should collapse into a single token for the word "the".

[00:05:01]But before any of that merging happens, the raw text has to be pre-split into rough pieces, words, spaces, punctuation, numbers. And that pre-split step, not the clever merging that gets all the attention, is where almost all of the runtime actually goes.

[00:05:16]Almost every tokenizer runs that split with a regular expression engine because the original GPT-2 code did. Regex is flexible and easy to get right, so it rarely gets rewritten. It is also slow because it walks through the text one single character at a time. GigaToken threw the regex out and split the text by hand using SIMD. Those are the wide CPU instructions, AVX-512 on Intel and AMD, Neon on Apple and ARM, that process a whole batch of bytes in one shot instead of one at a time. Where the regex reads one character per step, this reads 32 or 64 bytes in a single instruction. Then it refuses to do the same work twice. The first time it sees a given word, it tokenizes it and files the answer away. Every time after that, it just looks the answer up. Real text is the same few thousand words on endless repeat, so that cache starts paying off almost immediately. That sounds obvious, but it is the genuinely hard part. Language has a long tail.

[00:06:18]There is always some new rare word arriving. So, the cache keeps growing and can spill right out of your CPU's small fast memory. Designing that cache hierarchy, so it stays fast, is most of the real engineering.

[00:06:31]The rest is a stack of small unglamorous wins. Cut the branch mispredictions.

[00:06:36]Stop the threads from chattering at each other and barely ever cross back into slow Python. No single one of these is dramatic on its own. Stacked on top of each other, they multiply into a 100x.

[00:06:48]And the benchmark is not even rigged in his favor. Gigatoken reads the whole 11 GB file and finds all the split points itself, while the other two libraries get handed pre-split chunks to start from.

[00:07:00]It is doing strictly more work than its rivals, and it still wins by three orders of magnitude. The part one keep coming back to is correctness. He ran it against Hugging Face across all 20,400 documents in that corpus, and every single token matched. You are not trading accuracy for speed here. It is the exact same output, just produced far faster. It is fair to ask where it falls down, and he tells you himself.

[00:07:27]The older sentence piece tokenizers, the ones Gemma and some Mistral models rely on, only get about 10 to 20 times faster, not a thousand. WordPiece is not supported yet, and Windows is best run through WSL for now. He flags all of it plainly in the read-me.

[00:07:43]There is also an honest note about AI right in the read-me. He says most of the code was written by hand and you can verify that in the commit history with AI brought in near the end to widen compatibility and to squeeze out the final 4x of performance.

[00:07:58]It is a refreshing level of transparency. Actually using it is almost anti-climactic in the best possible way. You wrap your existing tokenizer in a single function call and from that point on it behaves exactly the same, same methods, same results. It just runs far faster underneath. There is one line to make it act like a hugging face tokenizer, one line to make it act like ticktoken or a native giga token API that reads your files straight off the disk and skips Python entirely to hit those full gigabyte per second numbers.

[00:08:31]And it already understands nearly every common tokenizer you would reach for.

[00:08:35]Dozens of model families work from a single install across Python versions 3.10 through 3.14 on both x86 and arm chips. You are unlikely to hit one it does not handle.

[00:08:47]The developer is Marcel Rad, a third-year computer science PhD at Stanford. He helps teach CS 336, the well-known language modeling from scratch course alongside Percy Liang and Tatsunori Hashimoto.

[00:09:01]He studied physics and math at Oxford and NTNU in Norway, spent time at MIT and CERN, and won a GPU kernel hackathon last year. This is a serious systems person. He posted it a few days ago and the reaction was immediate. It has already blown past a thousand GitHub stars. It is trending on r/localllama and his own PhD advisor is publicly cheering it on. For a low-level tokenizer library, that is a genuinely loud launch. To see why speed here is worth so much, think about training.

[00:09:32]When you build a model from scratch, which is exactly what his Stanford course has students do, you push hundreds of billions of tokens through the tokenizer. At the old rates, that pre-passing alone could run for days, and it is not only about training runs.

[00:09:46]Every time a live model answers you, it has to tokenize your prompt before it can think. Multiply that by millions of requests a day, and even inference time tokenizing turns into a recurring cost worth cutting. So, here is why this reaches you, even if you never train a model.

[00:10:04]Tokenizing used to be a tax on every project. Hours of pre-passing before a single training step could even start.

[00:10:10]Take that from 20 megabytes a second to 20 gigabytes, and the tax basically disappears.

[00:10:16]Cheaper data prep means faster experiments and quicker retries, which matters most when you are training or fine-tuning on a budget instead of a warehouse of GPUs. The invisible first step of the whole pipeline just stopped being the slow one, and that is the lesson I would keep. The boring plumbing, the part most of us never bother to profile, is often exactly where the biggest speedups are still hiding. Go find one. The link to gigatoken is in the description.