Google Made AI Memory 8× Smaller (TurboQuant)

[00:00:00]Google just compressed a 31 GB AI vector index down to roughly 4 GB. The headline sounds like it shrank an entire AI model, but the real breakthrough is more specific and potentially [music] more useful. It compressed the vector index used by rag systems, AI agents, and semantic search. [music] Whenever an AI reads a document, it converts its meaning into an embedding, a long list of numbers called a vector.

[00:00:25]Documents with similar meanings produce vectors that [music] sit close together, which is why a search for reset my password can find a document titled recover account access. The system is matching meaning rather than exact words. That becomes expensive at scale because a single vector can contain hundreds or thousands [music] of 32-bit numbers.

[00:00:44]Multiply that by millions of documents, and the index can consume tens of gigabytes of RAM. A database containing 10 million vectors, for example, may require around 31 GB just for the vectors [music] before storing the documents, metadata, or application itself.

[00:01:00]Traditional compression methods already exist, but many require a training stage.

[00:01:05]Product quantization studies a sample of the data set and creates a codebook describing how those particular vectors should be compressed. [music] It works well, but the codebook is tied to the data. So, changing or expanding the data set >> [music] >> may require retraining it. Turbo Quant removes that step entirely. It is data oblivious, which means it can compress vectors without [music] first studying the data set. It starts by normalizing each vector, separating [music] its length from its direction. Then it applies the same random rotation to every vector. Imagine millions of arrows pointing through a space with hundreds of dimensions. Turbo Quant rotates that entire space, so [music] the arrows move together and their relationships remain intact. Similar vectors still remain similar, but their individual values now follow a predictable bell-shaped distribution.

[00:01:52]>> [music] >> That predictable pattern is what makes aggressive compression possible. Because Turbo Quant already knows how the rotated values are likely to be distributed, it can calculate efficient rounding rules in advance.

[00:02:04]Instead of storing every coordinate with 32 bits, it may use only two or four bits.

[00:02:10]Two bits provide four possible values, while four bits provide 16.

[00:02:15]Each original coordinate is placed into the nearest available bucket. Normally, reducing a precise number to only a few possible [music] values would destroy accuracy.

[00:02:24]Here, however, the buckets are optimized [music] for the distribution created by the rotation, so the vectors become much smaller while their important relationships are largely [music] preserved. That is how a vector index can fall from around 31 GB to roughly four. The viral demonstrations often use TurboVec, an independent open-source implementation of Google's TurboQuant research.

[00:02:47]TurboVec is written in Rust and includes Python bindings, which makes it easier to connect with existing AI applications.

[00:02:53]Because no codebook needs to be trained, new documents can also be compressed and inserted immediately.

[00:02:59]The system does not need to rebuild the entire index whenever the knowledge base changes.

[00:03:04]TurboVec also uses SIMD instructions, allowing the processor to compare several compressed values at once.

[00:03:11]As a result, the smaller index may not only use less memory, but also search as fast as or faster than some optimized FAISS configurations.

[00:03:21]The same idea could also help compress an AI model's KV cache.

[00:03:25]The KV cache stores information from previous tokens, so the model does not need to process the entire conversation again every time it generates a new word.

[00:03:34]As the conversation grows, this cache consumes more GPU memory, making long context windows expensive.

[00:03:41]Compressing it could allow the same hardware to process longer conversations or serve more users.

[00:03:47]There is still an important limitation.

[00:03:49]Google released a research technique, not a polished commercial database.

[00:03:53]Open source implementations need more production testing, integrations, and long-term maintenance.

[00:03:59]But the key breakthrough is already clear.

[00:04:02]Turboquant does not simply make vectors smaller. It transforms unpredictable vectors into a predictable shape, allowing the same compression rules to work across completely different data sets.

[00:04:13]And that is how 31 GB of AI memory can fit inside four.