Stop Renting Your Cognitive Infrastructure - Thiyagarajan Maruthavanan, Kalmantic Labs

[00:00:00]One of the largest retailers in the country spent close to $200 million on inference with Anthropic and decided that things got way out of hand and built their own infrastructure. I'm pretty sure most of you have read the news from Uber CTO on how they had planned a budget of their tokens for an entire year and it got over in month four. I'm also confident that half of you in this room have come to a very similar conclusion that as time goes by the cost of intelligence really built.

[00:00:27]Using inference feels like, you know, it's one of the most inexpensive thing.

[00:00:30]But then this is very different from using a phone where you get a bill once every month and then you have like a specific set of amount that you can actually anchor your mind to. But in case of using these rented intelligence platform, they are like prepaid. You load credits. It's almost as if you're loading credits inside a casino. You put some and then you pull it and then you are so addicted to it then you end up doing more and more of it and by some time you realize that you've blown past the threshold that you had mentally kept in mind. And I had this experience myself. I built an app called Ultrazone or I experienced the inference cost ballooning here. Suno.com has anybody heard about Suno.com? Yeah, Suno.com is is this application that allows a user to turn a text prompt into music. What I was interested in is is doing the reverse which is given a particular song, what prompt could have actually generated it. This is something that I wanted. So I built this. I was having a lot of fun using this application, shared it with a few friends and spread wide around. I had hundreds of thousands of users, but then the cost ballooned way more than what I had anticipated. Hundreds of thousands of dollars I had to spend on inference.

[00:01:41]Now, this happens for many reason. There are many talks that are there at AIE itself where people talk about how you need to manage your context better.

[00:01:50]And many people forget about doing compression of their input token. And when there are agent loops, then there are many of these calls that are happening which are very, very wasteful.

[00:02:00]The inference endpoint that is consuming this is is completely unaware of the shape of the workload and which is why this happens. And I have this other issue that had happened. 3 weeks ago, my key got stolen. Someone in China got hold of it and then was sucking my endpoint dry. I could see the cost rise up from 7,000 to 7,500 dollars to 8,000 and going and so forth.

[00:02:22]Thanks to my co-founder who heads the research and technology, we were able to arrest it at 10,000 dollars. Otherwise, it could have been 100,000 dollars. Now, many people suggest that the alternative to rented intelligence platform is is to use token factory. Token factory is is basically saying that why are you paying money to Anthropic and OpenAI? Instead, go open source, have these open source models that are already deployed somewhere in the cloud, and then they are provisioned as tokens per second.

[00:02:48]There are neo clouds and then there are inference endpoint providers who actually do this. In fact, there is also an argument saying that, you know, you can build this token factory locally.

[00:02:56]There are AI Twitter influencers who actually talk about building inference in your garage, in your basement. Buy GPU cards, rig them up together, and then you could actually run a local token factory. In fact, I was in inspired by that a little bit. I bought my own DGX box and then I first moved Ulta Sono from Anthropic to DGX box. It worked well. I ran into this one issue of memory being the bottleneck. And then, it was good enough that I started building my next applications. I started having agents. I have some agents that I need for running my research lab. So, these agents started shaping up inside the DGX box, you know, two, six, eight, 12. And it worked all right. The issue though is is that, you know, it may not be reliable for enterprise, which is what I exactly faced. Three enterprises reached out to me to replicate the same setup for them. But for enterprises, renting and leasing don't cut it. Bill is a problem, but then there are secondary set of problems that makes it extremely ineffective approach. The enterprises that I reached out to me, one was a fund, another a hospital, and the third a tax practice. And each of them had different wall that they had hit.

[00:04:06]The fund, it was an investment fund, they were running an investment analyst on an email client architecture, and they didn't want somebody else to dictate as to what the rate limit that they could consume. So, control become a big issue for them to actually go with token factories. Hospital had a different issue. They used the use case, it worked well, but later when they went through an audit, a third-party vendor dependency was redlined, and then they couldn't go forward. The tax practice was a completely different issue. In a tax practice, what is happening is this, when an intelligent generates a recommendation, you want to be able to recreate it. And when you don't have access to the in-depth of the model, you will not be able to do this, and that became the issue. So, that brings me to the most important point of this presentation. Where do you sit? When do you stop renting infrastructure? If you're a startup, if you're a founder who is doing pre-product market fit work, so you're still figuring out that the use case that you have, if there is demand for it, you can get by by renting. But if you're post-product market fit, you cannot afford to And if you're an enterprise who's already budgeted a project, which means you're telling that you are assuming that this particular use case has product market fit, then again, you cannot ignore to build your own infrastructure. Which is what I realized, and I said that this this situation is this like, if you're going to a new city, you may initially start with saying that I don't want to buy a house. Let me actually rent and see. Sometimes you might even Airbnb.

[00:05:34]You experience the environment, you experience the city, the neighborhood, but then eventually you have to buy the house. You cannot raise a family in an Airbnb. As I went through this experience, I decided I came to the conclusion that I need to build my own inference infrastructure for the apps, the agents, and the scaling of the apps that I'm building. And I call this as just infra. And while I went through this exercise, I realized that there is optimization to be done at multiple layers. Even at the renting and the lease layer, you can do optimization around input cost to token management, and then, you know, context management, and so on and so forth. Some of those experiences that I've had in the last couple of months combined it into an open-source project and published it as just token max. If you have used headroom from Netflix, then this is an alternative to it. We benchmarked against headroom and on many parameters, just token maxes is far superior. If this is a thing that is of interest to you, give it a try, maybe a GitHub star if you like it. I also wrote the book called peak inference infraeconomics of AI inference when you have to think about building your own inference infrastructure. The AI market is is very different compared to the rest of the technology market that used to exist because here the rules of the game change every three to six months, which means it becomes a very noisy marketplace. You talk to someone like Jensen, he would say token factory is the future. You hear someone like a Satya Nadella, he will say unmetered intelligence is the future, it is going to be local. And then when you hear new clouds and inference endpoint providers, they'll say, "Hey, inference endpoint providers are the ones that are going to capture the value in the marketplace."

[00:07:08]Now, my experience walking from application to agents to scaling them and then building my own inference infrastructure taught me that if you want to learn, you can rent, but if you want to earn, then you have to own. And if there was the one sentence that you had to take away from this entire presentation, it is that. Rent to learn, own to earn. But then, you have to come to your own answers. Thank you. And if any of these topics are of interest to you, then I'm happy to talk to you about renting, about just token max, about how to build your own inference infrastructure. I'm here at the AI Engineer's conference for the next 3 days. Hit me up on Twitter at mtraj or through my site mtraj.com.