Google Shrunk 31GB of AI Memory Down to 4GB (TurboQuant)

[00:00:00]Last month a headline detonated across tech. Google shrank 31 GB of AI memory down to four, and the number is real. 10 million documents that ate that much RAM folded onto a box a quarter the size and search faster than Meta's FAS. But look closer because the memory it shrinks is not the model at all. It is the vector index. Every rag app, every AI agent, every semantic search leans on one of these. It is how machines find things by meaning. And at scale that index becomes a monster in your cloud bill because holding all those numbers in memory is not cheap. The normal way to shrink it comes with a tax. You train a codebook on your own data first, which is slow and breaks when the data shifts. Turbo quant throws that whole step out. It skips training, never peaks at your vectors, and still lands the same compression. That property has a name, data oblivious. The compression is byte for byte identical whether it runs on your vectors or mine. Now the catch the headlines missed. Google wrote the math, but Google did not ship the tool everyone keeps screenshotting, which raises two questions. Who actually built it and how do you shrink something you never once looked at? To see the trick, you have to start with what a vector really is.

[00:01:10]When an AI reads a word, an image, or a whole document, it turns that thing into a long list of numbers, a single point floating in a space with hundreds of dimensions. Things that mean similar stuff land close together. So search stops being about matching letters and starts being about matching meaning. The problem is how heavy those points are.

[00:01:29]One embedding from a modern model can be over 1,500 numbers long and each number is usually a 32-bit float, 4 bytes a piece. So a single vector already weighs in around 6 KB before you have stored one word of the text it came from. Now line up 10 million of them, which is a pretty ordinary size for a company knowledge base. Multiply it out and you are holding roughly 31 GB of raw floating point in fast memory all day just so your app can look things up.

[00:01:57]That is the map every query walks across, and it never gets to rest. Older compression does shrink each number, but it plays a sneaky game. To rebuild the values later, it has to store a full precision constant for every little block of data. And that bookkeeping can add one or two extra bits to every single number. So, the budget you were trying to protect leaks right back out.

[00:02:17]The fancier method, product quantization, learns a codebook by running K-means clustering over a big sample of your vectors first. It works, but it means a whole training pass before you can index anything, and you have to run it again every time your data drifts or your corpus grows. That is friction, and friction is what kills adoption.

[00:02:36]Here's where TurboQuant does something that feels almost like cheating. Strip the length off each vector, and what is left is pure direction, a point sitting on the surface of a giant high-dimensional sphere. And directions, it turns out, have a hidden statistical habit you can exploit, as long as you set them up right first. So, it sets them up by spinning them. Every vector gets multiplied by the exact same random rotation matrix, one shared spin for the whole data set. And here's the payoff.

[00:03:01]After that rotation, each coordinate independently follows a known bell curve, no matter what the original data looked like. Random text, product photos, medical records, it does not matter. That single fact is the whole unlock. Because the shape of every coordinate is now predictable, you can work out the best possible way to round it ahead of time, straight from the math, without ever peeking at one real vector. The data does not teach the compressor anything, because the geometry already did. Concretely, you pick how many bits you can spend. Two bits gives you four buckets to sort each number into. Four bits gives you 16, and the exact bucket boundaries come from a classic technique called Lloyd-Max, tuned to that predicted curve. Every one of those boundaries is computed once, from theory, and reused for every data set. Then you bit pack the results as tight as they go. A 1,500 dimension vector that used to sit at 6 kilobytes now fits in 384 bytes.

[00:03:56]That is 16 to 1. And because the rounding was optimal, the search results barely move. Google stacks two of these ideas together. The first, the rotate and round part, spends most of the bits capturing the core of each vector.

[00:04:09]The second is a one-bit sign trick called QJL that spends a single leftover bit cleaning up the error the first stage left behind at basically zero cost.

[00:04:18]Add them together and the distortion lands within about 2.7 times of the Shannon limit, which is the hard mathematical floor for how little information you can throw away at a given size.

[00:04:28]In plain terms, there is almost no better compression physically possible at that bit budget. So, who actually built the thing in the headline? Google published TurboQuant as a research paper, accepted to one of the big machine learning conferences, and left it there as math. Then an independent developer named Ryan Kodry read the paper and did the unglamorous work of turning it into software people can install. He called it Turbovec and wrote it in Rust with clean Python bindings on top.

[00:04:55]The headline feature is online ingest.

[00:04:57]You add vectors and they are indexed on the spot with no separate training step and no rebuilding the index as your collection keeps growing. It even drops straight into tools like LangChain and LamaIndex. And his benchmark is the exact clip that went viral. That same 10 million vector index, the one bloated to 31 gigabytes as raw floats, now rides in roughly four and answers queries faster than FAISS while it does. That result is Turbovec, the community tool, not something Google handed out. That speed is not luck, either.

[00:05:28]Kodry hand-wrote the low-level search kernels in the chip's own SIMD instructions, one version for ARM and one for Intel. On ARM, they beat Meta's heavily tuned FAISS fast scan by 10 to 19% across every setting. And on Intel, they trade blows, winning the four-bit cases.

[00:05:45]It also filters right inside the search, not after it. Hand the query a list of allowed IDs from a permission check or a database filter, and the kernel only ever scores those candidates.

[00:05:56]So, hybrid search that mixes keywords and meaning stays exact, and selective filters actually run faster instead of slower, and the whole thing runs on hardware you already own. There is no managed service to sign up for, and no data ever leaving your walls, which is the difference between a demo and something a hospital or a bank will deploy for private air-gapped retrieval.

[00:06:16]Now, remember Google's own reason for building this.

[00:06:20]The same rotation math also squeezes the KB cache, which is the scratch memory a model fills up while it reads a long conversation.

[00:06:27]That cache is what makes 100,000 token chats so expensive, and Turbo Quant cuts it by around six times down to three bits with almost no drop in answer quality, and a smaller cache is a faster cache. On a top data center GPU, the four-bit version computes attention up to eight times quicker than uncompressed keys, and since that memory traffic is where a huge slice of the serving bill lives, the cost roughly halves on long context work. Put differently, cutting the cache six times over means fitting about six times as many long chats onto the very same GPU.

[00:06:59]For anyone renting those cards by the hour to serve big context windows, that is not a benchmark bragging point. That is the invoice getting shorter. Wall Street felt it immediately.

[00:07:10]When the paper made the rounds, memory chip makers took a hit. Micron and Western Digital both sliding a few percent in a single session, and traders started openly calling it Google's deep seek moment. The fear being that far cheaper AI memory means far fewer memory chips sold. That panic is probably backwards, and history says so. Every time compute or memory gets dramatically cheaper, we do not buy less of it, we find 10 new uses and buy more, an old idea called Jevons paradox. Cheaper indexes likely mean more apps shipping big memories, not a smaller pile of chips. And it is already leaving the lab. The rotation trick now ships inside Qdrant, one of the most popular vector databases, so a lot of developers get it as a checkbox.

[00:07:53]There is a separate port that squeezes the memory of local language models and a plain Python reference build for anyone who wants to read it. One fair caveat before you tear out your stack.

[00:08:03]What Google actually released is a paper and an algorithm, not a supported product with a support line, so the production grade tooling is still up to the open source to harden. The math is shipping. The polished download button mostly is not yet. And that is what makes this genuinely useful rather than just clever. The win is not really the compression ratio, it is the total absence of friction. Because with no training run and no codebook to babysit, you can bolt it onto a system you are already running this afternoon. Step back and the whole thing is almost funny in hindsight. One shared random spin turned the problem from compress my data into compress anybody's data. And that in the end is how 31 GB of AI memory politely folded down into four.

[00:08:45]If watching a giant model shrink into a tiny footprint is your thing, the next one is wilder. A 700 billion parameter model running on a laptop with no GPU at all. Same instinct, squeeze the memory, push to the absolute edge. That link is on screen.