Google's Plan To Boost Its AI Models

[00:00:00]Tell me about this new chip that Google is working on.

[00:00:03]>> Yeah, so Google is working on a new chip. It's informally dubbed Frozen V2 inside the company. And essentially the idea is that it could be way more efficient even than the TPUs at the time of launch. They're projecting it could be 6 to 10 times more efficient than the TPUs at the time of launch. And that's because it works slightly differently by etching the model itself into the chip.

[00:00:25]So fewer decisions have to be made at the time that the model is running.

[00:00:29]>> Etching it into the chip. So I mean, so why is it called frozen?

[00:00:34]>> So the name frozen has to do with the idea that like the model is kind of frozen into the chip to some degree. So essentially the way that a regular chip works, it's general generalizable, which means that it can do a lot of different things, but that means that there are a lot of different decision points that the chip has to run. And so like this chip makes things more efficient by combining some or sorry by like presetting some of that essentially. And so like it doesn't have to make all those decision points.

[00:01:01]>> And so how would it be different then from the TPU?

[00:01:05]>> Yeah. So the TPU is more of this generalizable kind of chip versus okay these frozen chips which would like specifically run like this model architecture.

[00:01:14]>> I see. So I mean if if this frozen chip I mean if it's really only um I guess it's only meant to run Gemini models sounds like so is it only meant to be used internally by Google then is that the idea?

[00:01:28]I mean, so a lot of like Google Cloud customers also run Gemini models and so this is something that could theoretically also be run by external customers, but this is still this is still something that's pretty early. So it's not expected to launch until 2028 at the earliest. So I think a lot of these strategy questions might still be be like in the process of being worked out, >> right? because and I mean the reason I'm asking this question is because we of course know that the TPU was mostly used internally but now they're at the point where they're really trying to >> outfit uh customer data centers if I recall correctly with these TPUs. So I mean your point is well taken that that it's not just people uh at Google running Gemini, it's it's all sorts of customers. So that could very much be strategy.

[00:02:15]>> How does this compare with other inference folks's chips? uh you know we've had the uh CEO of Samaova on the show uh OpenAI is working on their own ship. How would this compare to that?

[00:02:26]>> Yeah. So a lot of people are trying to do the same thing which is essentially um driving down the cost of inference because everyone's in this big compute crunch. So Google's chip works a little bit differently because of this tourism design. It's actually something somewhat similar to what this Canadian chip startup Talis is doing. said they're also trying to like like specifically like etch a model into the chip so that it is more efficient.

[00:02:54]>> Has Google being compute constrained? Is this a way to get at that problem?

[00:02:59]>> Yeah. So they talk about that constantly on earnings calls. The idea that like their cloud division is compute constrained. Like Google reports earnings this week. I would expect them to also talk about that this week. And uh tell me TSMC capacity that I imagine is also top of mind here for this new chip.

[00:03:18]>> Yeah, so Google is going to have to find fabrication space for this chip. It's not totally clear to me that it's going to be like a onetoone exchange in the sense that like in order to make these chips at like seals from TSMC space for the TPUs like that's definitely possible but it's also been suggested to me that this is enough of a different fabrication process that it might use different space so it might not be a zero sum game for them.

[00:03:44]>> Great well Aaron I want to thank you for coming on. That is Aaron Woo our open AI and Google reporter here at the information