Big Ideas 2026: AI Infrastructure

Added: 2026-06-02

[00:00:00]This is Frank Downing, research director at ARC, focused on AI cloud computing and software. And today we're going to go over our AI infrastructure section of our big ideas 2026 report, which you can find on the arc-invest.com website. So jumping right into what we see in the current state of the AI market is a continued explosion in demand. uh when we look at for example the tokens inferenced on open router over the last year since December of 2024 they've grown over 25fold and what's really driving this is a reduction in cost and an increase in performance of the leading models combined with models finally being built into products that we use in our everyday life both on our uh time spent as a consumer using AI in our day-to-day and AI getting rolled into the workplace both through uh kind of general subscriptions like chat GPT for enterprise or anthropics cloud product and through kind of vertical specific applications and we talk through all of that in detail in our AI productivity section. Um but you know focusing on cost declines for a second uh this is something that's really important and foundational in our research at ARC and it helps us understand when a technology is ready for prime time and will grow to uh meet more and more use cases over time. Uh we use this for example in studying electric vehicles in the past to know that the cost declines in electric batteries would allow electric vehicles not only to be possible but profitable at scale. And we're seeing these same dynamics play out in the AI space. Uh there's a lot of benchmarks out there.

[00:01:30]If you look at artificial analysis which does an index of many of the leading benchmarks uh the cost to achieve a certain level of intelligence on that benchmark has fallen by 99% over the last year. And we see this across many different benchmarks. Um, which again we we we speak to in in in other sections of this deck. Um, but this kind of evokes this concept of Jevans paradox where a decline in cost actually increases the market size and increases the demand because you're unlocking new use cases. Uh we see that in domains like software development for example where as coding models have become more intelligent relative to the price you're paying. Uh software developers are leaning in harder and harder and using and consuming more tokens uh than they were in prior years. Uh and what is all that doing? It's driving a huge investment in the underlying infrastructure required to run generative AI. We've seen the long-term trend of data center system spending which was growing at a 5% annual rate over the last uh 10 years prior to the launch of CHACHBT. It was you know growing slowly from 150 billion to about $200 billion per year. that has inflected upwards and since the launch of Chachib accelerated to grow at a 29% annual rate hitting 500 billion uh nearly $500 billion in 2025 and the current market estimates are for that to grow uh to nearly $600 billion uh in 2026. Uh if you're wondering what data center systems are, these are the compute servers, networking to connect them and storage that's attached to them that are put into data centers. Uh so really the the core IT equipment driving these workloads uh it doesn't include which is a whole another big expense the data center actual facility and power that gets connected and and enabled uh uh to uh to power all those IT systems.

[00:03:21]Um and I think you know this is where we spent a good amount of time in our research this year which is adding historical perspective on where we are in the cycle. there's a lot of fears of uh being in an AI bubble in the market today and we wanted to shed some light on that. Uh looking at the left side of this chart uh we looked at you know how how big is the scale of this investment relative to other times in history and you know the last time we saw a capex cycle like this was the tech and telecom bubble of the late 90s and early 2000s.

[00:03:50]uh and those numbers I mentioned on the past slide are part of the the reason why this ratio of the capex relative to GDP for uh technology companies has been growing over time and is now reaching levels that haven't been seen since that late '9s period. Uh and actually I think what's interesting about this is if you look since the previous times this chart bottomed uh both after the dotcom bubble burst in around 2002 and then after the great financial crisis uh in in 0809 we've actually seen a consistent rise of tech capex as a percentage of GDP over time. Uh, and this actually intuitively makes sense if you think of, you know, compared to the year the iPhone was launched versus now, how much more technology is represented in our day-to-day lives and how much bigger these companies just are uh large cap tech today um relative to their size uh a decade and a half ago. Uh on the right side chart, I think this is helpful on the market cycle context of of answering that question of of where are valuations and are we in a bubble? Is the market um multiple higher than it was postcoid for example when the market bottomed? Yes.

[00:04:56]The S&P 500 has gone from a PE multiple of around 20 to around 30. Um so it's definitely higher though off what I would consider an overly pessimistic base judging by how the market has performed since then. And if you look at uh large cap technology companies specifically uh we've highlighted where the current mag six are uh as of the beginning of 2026 which is around a market multiple of 40 and comparing that to what large cap tech you know the Cisco oracles the IBMs Microsoft is the only company that's in both of these lists uh where they were through the late 90s period they actually peaked at over a hundred times earnings. Uh so if you looked at where they hit 40, that was in about 1997. Um so multiples are elevated but partially because these companies are so profitable already and are funding um this uh buildout with a larger percentage of free cash flow than companies uh in the 90s and because the the actual returns on AI are real. There there is real revenue flowing into the cloud divisions of these businesses in particular. Uh we see multiples healthier. Um and you know this would imply there's there's a long way to go before we reach the same euphoria that was seen in 1999 and 2000 for example.

[00:06:12]Uh so so in the next slide here we we look at you know what's evolving in the heart of AI compute. You know these are the the chip designers that are that are making the um the hardware that makes it possible to run generative AI at scale.

[00:06:25]And of course what we've seen since um well since they started building data center chips between 2012 and 2014 um but really since launch of CHBT NVIDIA has been the star of the show. Uh now three years into the generative AI revolution we're seeing uh the beginnings of a broadening out of uh the market for compute providers. Uh and a really compelling story is AMD who's coming to market with uh more competitive chips having competed against Nvidia on in the gaming space on the consumer side and has competed against Intel very successfully in the data center side. AMD went from uh almost 0% market share in 2017 to 40% today in data center CPUs. And we think there's a similar share gain story possible for AMD in the data center space with regards to GPUs. um they've won customers like OpenAI and Meta for example uh and are looking to extend that list as they have new chips that come to market and are even more competitive relative to Nvidia. Um on the research side we looked at performance uh using a variety of semi-analysis benchmarks of AMD versus Nvidia on small models versus large models. And what you can see here on small models uh relative to the cost of the chips, AMD has already caught up and is actually more performant on a uh performance per dollar basis or the amount of tokens you could generate per dollar for the same model uh than Nvidia is on the large model side. However, you could see where Nvidia still has this huge advantage. uh their Rackcale solution Grace Blackwell uh can deliver uh 15.5 million tokens per dollar of total uh cost of ownership compared to uh the current stats for uh AMD's top-of-the-line which is about 2.9 million tokens. So, so this rack scale offering really is an advantage for Nvidia in the market today. But what we're going to see through the rest of 2026, which is important for forecasting where we're going to go, is that AMD AMD is coming to market with their own rack scale solution called Helios that they expect to go toe-to-toe with Nvidia's next generation Vera Rubin. So, we're excited to see uh when these products both come out in the second half of this year to see where these performance numbers go. Uh but what we can see from the customer orders is early indications are looking like it's going to be a compelling product. uh not in the chart because uh the the benchmark data hasn't come out for it yet but in the table is Google's TPU which is another uh important part of what's happening uh in the AI compute market which is uh you know the maturing and the scaling of custom silicon projects within the hyperscalers. Uh so all of Nvidia's biggest customers actually have different efforts to build their own AI chip. Google's TPU, Amazon's Tranium and Microsoft's Maya series. Uh Microsoft has uh really not gotten a good shot on goal yet. Uh Trrenium from Amazon would be the next most developed and Google having worked on the TPU for over 10 years is definitely the most mature um uh in the market. Uh they're running all of their internal uh Gemini workloads on TPU and trained their latest model Gemini 3 on the TPU and it is one of the most capable uh frontier models out there. Uh so we're seeing uh increased investment by Google on their TPU stack and that's benefiting companies like Broadcom that are the back-end design provider uh or basically the silicon partner for Google and really like the go-between between Google and TSMC to help them scale that product. Uh so you know if we take all that into context and we look out through the rest of this decade we are you know again at around 500 billion in annual data center system spending. Now, we think with the scaling of AI applications and the continued cost declines we're seeing, that total investment could nearly triple to 1.4 trillion by 2030. And again a large part of that being spent on compute and the composition of that compute market will change uh pretty significantly where we have already the change uh in market share from traditional CPUdriven computing to accelerated computing powered by GPUs and these uh custom silicon projects or AS6 application specific chips developed by the hyperscalers uh they're already taking a majority sca uh share of incremental compute spend and we think within that uh custom silicon and could grow to a third or higher of spend as some of these projects like the TPU and Tranium continue to scale. Um so all this is setting up for what we think is uh a sustainable infrastructure buildout.

[00:10:51]There's a chance that we go too crazy and the market overheats between now and then. Uh but all of our research on how AI is being used and you know when you think about it is really early in the deployment cycle relative to how um how we integrated we think it can be into our daily lives especially at work where enterprises have you know paused and then now are scrambling to try and figure out how can I best use AI. We think every business just like they became common users of the internet will become common users of AI and the businesses that become power users will be the ones that lead you know really become the next leadership in the market. Um so with that uh that's a a brief rundown of our AI infrastructure section uh of big ideas 2026. We're really excited about uh the research this year uh and the insight it's giving us into the market uh and we're excited to uh keep on learning and following innovation.