Use AI to Strike Back at AI Memory Prices

Added: 2026-05-13

[00:00:00]If you're buying any kind of computer right now, you're definitely feeling the pain that is memory pricing. Because no matter if you're buying a mini PC, a notebook, a workstation, a single server, a rack of servers, or an entire data center of servers, memory pricing has become a major limiting factor. The reason, of course, is because the AI buildout is consuming a ton of memory, creating a lot of demand, and a surge in pricing. And just like everybody else, we're getting hit hard by memory pricing. And so, I had this idea. I was like, "Wait, why don't we use AI to strike back at memory pricing?" And that's the topic of today's video. So, let's get to it. Hey guys, this is Patrick from Sth. Memory has gone from something that was annoying and part of buying and building servers and what have you to something that is completely limiting how much people can deploy right now. Just to give you some example, these little dims right here, we never actually even took out of their little ESD bags. I bought them about a year and a half ago, and I was haggling with someone over eBay on on eBay for the 32 gig ECC DDR5 RD dims and I think I spent like $106. The guy wanted like 108 and I was trying to get 104. We ended on 106 and like I just looked the exact same seller is selling the exact same memory for over $750.

[00:01:12]And if that doesn't tell you that people are going to be absolutely crushed by memory pricing. I don't know what does.

[00:01:20]But at the same time, we also have a fleet of servers that we need to refresh plus new applications that are coming online all the time. And frankly with the agentic AI stuff we have a lot more compute demand than we had previously.

[00:01:30]So using AI to strike back at these memory prices we are trying to answer three questions. The first one how much memory do we actually need to deploy?

[00:01:37]The second one if we were to go and degrade or trade some memory performance for that memory capacity would we still be okay in terms of performance? And the third one is well if we do lose that performance how many more cores do we need to make up that difference? and would it be worthwhile just to go buy more cores versus buying more memory?

[00:01:55]Now, as a quick disclosure, we're going to use an AMD Ryzen Thread Ripper workstation from Zidex just to go and use for some of our testing when we want to look at two channels and four channels of memory. When we talk about eight channels and 12 channels, we're using the AMD Epic. We have to say that AMD is sponsoring this video, but frankly, this should be applicable to just about everybody. Now, in front of me, you're going to see a bunch of memory and also a CXL memory expander behind me. and those we also didn't purchase. So, we have to say, of course, that this is sponsored, but we're using just a lot of gear here and this took a ton of testing. So, hopefully you guys understand uh how crazy this whole project was to put together. But to me, it was really important to scale all the way from a workstation up to a server where you could have 12 channels of memory, two dims per channel, 24 dims total, and really talk about the entire continuum of how memory performance or how memory population impacts performance. Super fun to get to show you guys these results. So, well, let's get to it. Now, the first way that we used AI to really strike back at memory pricing was really to get an idea of are we deploying too much memory in servers.

[00:02:55]Like a lot of folks, you know, frankly, the memory pricing hadn't really been an issue for a long time. Dims were relatively cheap and it really made sense a lot of times to go from a 32 gig dim to a 64 gig dim because, well, we might need that extra capacity at some point. And I think a lot of folks, not just on their servers, but all the way down to their desktops, their notebooks, what have you, were using more memory just because it was so cheap to go do so. But it's not that way anymore. But the greatest part about this is the fact that no matter what your operating system is or your operating environment really, you can use the tools that are provided to do monitoring or that your organization has to go and figure out are you deploying too much memory in your servers because well that's just what you had been doing for a long time.

[00:03:36]If you're using Nanix, Red Hat, Ubuntu, if you're using Proxbox, XCPNG, any of these kinds of things, you know, you're definitely also seeing things like swapping, ballooning, memory usage, all that kind of stuff. That's all normal metrics for these platforms to collect as well. Windows, both desktop and server supports this. You can even see it on your Windows desktop. If you just go bring up your task manager, you can go see your memory usage really easily.

[00:03:59]For your Kubernetes cluster, you may have like coupe state matrix. You may also have a cloud provider like Microsoft Azure, Amazon, Google Cloud Platform. Any of these guys also have platforms or metrics that will tell you, hey, you know, here's how much memory you're using. And depending on your setup, you could have a bunch of these.

[00:04:15]You could have some VMs and workloads that are running in the cloud. You could have other things that are running on VMware or Linux hosts and be using those tools. You could also have uh, you know, broad tools like data dog or New Relic or any of those types of things. Just a custom Prometheus Graphana dashboard or something like that. Most organizations have this memory usage data and really the key to what we did was we just literally said hey here are the data sources. We exported a bunch of data over time and we said here's what our our actual memory usage looks like across our fleet both at our host level and also a lot of times at our VM level and said hey what do we actually need like what are we using are these systems or all these VMs that have ballooning memory are they actually using that? Now the important thing here is not to just do it over a short period of time. If you're only doing like a 30 second, 5 minute interval, hour interval, you're not going to catch all of the weird things that you need in terms of memory spikes. If you're at a business, you have an endo quarter and you have a big spike in sales activity, maybe you have applications that are running that sales pipeline and your deal workflows and all that kind of stuff that all of a sudden are spiking like crazy at the end of the quarter, but for the rest of the year they're pretty slow. or maybe on weekends, easy peasy, but on weekdays, oh my gosh, there is a crazy amount of load on your servers while people are up and working. So, what we did was we actually took an entire year's worth of data, exported it, and then used AI to go build scripts to go analyze that data and find patterns, find the cases, if there were any, that were out of memory.

[00:05:39]And what we found was in a lot of hosts that we were deploying like 512 GB of memory, realistically, we could have gotten by no problem with 256 gigs. That may not seem like a lot, but deploying half the memory almost has your memory cost. So that's definitely an easy win to go out and do. And it's really one of those ones that we would not have cared about a year ago, but nowadays we definitely need to care about because that could be the difference between deploying one server and several servers. And guys, everybody has different setups. I'm talking about this in a very general term. But what I would definitely do is not just take every AI recommendation here with a grain of salt. I would actually go and drill down because the other thing you would probably want to do is look at what the memory pressure is. So if you can get the metrics on like you know how active the memory is in these systems that would also give you a lot of data because something that will be useful is to figure out how memory sensitive or memory bandwidth sensitive these applications are as well as you know how much the capacity varies in the applications. And of course, we operate a ton of different platforms for the size of organization we are because well, we we have a test lab. But that also meant that we had a whole bunch of like cobbled together just total hodgepodge of systems. And so we needed to go and get access to a whole bunch of different systems, export that data easily and then do that analytics. And it was just going to be a pain to go do.

[00:06:53]It wasn't that we couldn't do it. It was just it would take a lot longer if we did it manually. So it became a great application to use AI to really push back and figure out where those pockets of performance capacity and what those different needs were. Really the idea was to stop doing what we've always done and do something that's more informed by data because we have a new constraint which is the price of memory. But of course capacity is only one aspect that we could optimize on because the other idea we had was well could we optimize on memory bandwidth and the performance of our systems based on how we populate that memory. Now guys, there are too many different types of platforms out there for me to go into each one of them individually, but from a high level, each platform that you have has some number of memory channels as well as a maximum speed that those memory channels will run at a given number of dims installed. What we mean by that is that you may have a Epic 95 series platform with a 12 channel memory controller, but you may be able to run that at DDR56400, but you may also only be able to run it at like DDR5, 5200 or 4,800 depending on your environment. Now, of course, DDR4 is a different type of memory for really older generations of systems, but they're still relevant because a lot of people are running DDR4 systems these days and they ran for a long time. Also, DDR5 is a little bit different because unlike DDR4, you can't go and take like a consumer unbuffered DIM and put it into a uh, you know, platform that supports RDMs because they're actually keyed differently. The RDMs that you use in a server versus the you know unbuffered memory that you would use in a client desktop. We have entire guides on this on the ST main site. But while we talk about the maximum number of channels in a platform, for example, a consumer desktop might only have a two channel memory controller, whereas the Thread Ripper system that we're using has four memory channels. A platform like the AMD Epic 90005 Turin series will have 12 memory channels. There's also the capability in some of these platforms to run at two dims per channel mode, which means that you have 12 channels and you can run 24 dims, albeit at a reduced speed. And why that matters is simple. If you have 12 dims at 64 gigs, you can get the same capacity by running 24 DIMs at 32 gigs each, which may end up costing less. Now, the trade-off, of course, is that you are going to spend or you're going to have less memory performance because instead of DDR5 6400, you might be at DDR5 4,800. And so, there is a trade-off because you're losing a significant amount of memory uh performance because you're losing that clock speed. But on the other hand, you still have your 12 memory channels. Now, another option is exactly what this is here, and this is a eight channel AMD Epic Turin 90005 series platform from ASRock Rack. We actually did a review on this not too long ago. And this is a bit of a different system because even though the AMD Epic 95 series supports up to 12 memory channels, two DIMs per channel, so 24 DIMs, this particular motherboard only has eight channels of memory that are available. So, we essentially have an eight channel platform and we're going to run at that high speed because we can't do two DS per channel. We can't even get all the way out to 12. Now, in the industry, there is something called the stream benchmark which has been around for forever and you'll see stream triad numbers posted all online and those numbers are really interesting because they mostly focus on the performance of just the memory subsystem. How much bandwidth can you get out of a memory subsystem which of course we just covered is really dependent on how fast the memory controllers are on the CPU, how fast the dims are that you installed and also how many channels you have installed and uh you know what what the operating speed of those dims are in those channels, right? That's essentially what that is covering. And something you're going to see in the industry is that oftent times the stream benchmark is used as a highlight feature for how much faster a new generation of chips is versus an older generation. Oh, this new one's twice as fast. Well, it's twice as fast because you went from DDR4 3200 and 8 channels to DDR5 4,800 at 12 channels.

[00:10:46]And that's that's why you got so much more performance, right? But stream is really looking at like what is purely memory bandwidth bound. And let me be clear in the AI workloads for example like if you look at GPUs, there are a lot of inference workloads for example that are very memory bandwidth bound.

[00:11:00]And that's exactly what we mean. But on the other hand, traditional workloads that run on CPUs oftentimes are more computebound. So you may have a HPC thing that is mostly memory bandwidth bound, but there's some compute elements. So it's not quite as bad or sensitive to memory bandwidth as a stream, but the vast majority of workloads like well over half or more over probably more than you know 2/3. I I have no idea what the actual number is, but like most of the workloads that you have, they're not going to be super super sensitive to memory bandwidth. But we've always said that. And so I thought, well, how do we actually test it? And what we're going to use now is our agent STH, which is our new benchmark suite that covers really what we did when we went and looked at the profiling behind how we were using agents. We generated billions of tokens and saw, hey, here's what the Agentic CPU workloads looked like against those, you know, actual real world, like what are the CPUs even doing at this point?

[00:11:54]We've been using it internally to go and profile different systems. Figure out is one system faster than another system and why are pores faster than e- cores?

[00:12:01]How much faster in aentic workloads are they? How sensitive is everything to the memory bandwidth of the system. So once we figured out what the AI agents were doing, the next step was to figure out well what kind of pressure are they putting on system? Are they CPU heavy?

[00:12:12]Are they memory bandwidth heavy? Or are they a mix of both? Are they communication heavy? Are they all hitting the exact same cache line at the exact same time? That's why we're getting weird performance, right? We went and did a ton, I mean months and months of profiling to figure out exactly how these things are operating and be able to come up with a good benchmark because of it. And this whole test setup provided an excellent opportunity to go look at the sensitivity of the different workloads to memory bandwidth. And we decided to come up with an extreme case because you know honestly if you're 15 10 15% memory bandwidth difference, most applications are going to see low single digits in terms of impact. So really we wanted to do something that was a little bit more stark and that's why we started out with using this Thread Ripper system. Now, this is a Zidex Thread Ripper system, and we put both the Thread Ripper 9970X as well as the 9980X in there just because we want to see 32 and 64 core results. That really is less of an exciting thing here. What's more exciting is really the way that the memory came. This system came to us from Zidax with two 64 gig DDR5 6400 ECC RD dims installed. But of course, it's a quad channel memory platform. So, we could just have four DIMs in there and we would in theory get twice the memory bandwidth. The other thing we could do though is we could put, you know, lower speed or lower capacity dims in there.

[00:13:25]So, we could get 128 gigs with two 64 gig dims or four 32 gig dims. We could also put DDR5 4800 dims in there. we can have our DDR 56400 on the other end of the spectrum. And so this was a really awesome platform because it gave us, you know, more than a doubling of the memory bandwidth from like a low config up to a high config. And also because it's N5, just like our AMD Epic 90005 Torren series, it gave us a great way to go look at two and four memory channels and then scale up to 8 and 12. And also because we had the AMD Epic 9575F, which is a highfrequency 64 core part along with the Thread Ripper 64 core part and all Zen 5, it made it very easy to do this comparison or at least get close to being able to do this comparison. As a quick note, we're using the highest certified DDR5 6400 in these platforms, but depending on the actual platform that you're using, the actual memory speeds may vary. So always check the systems that you're planning on using to figure out exactly what memory speeds they support. First things first, by far the biggest determinant of your memory performance is really based on how many memory channels you have installed. You just frankly get more from that in most cases than going from like a lower speed memory to a higher speed memory. So just keep that in mind that you always want to fill as many memory channels as you possibly can because that will probably get you the most performance you can. So what agent STH does, which is really cool, is that not only can it look at an entire workload on a single CPU, but we can also do things like we can run a entire workload on a single core or we can give agents more cores. So they can run on two cores, four cores, eight cores, 16, 32 cores, whatever it ends up being. And we can get different splits based on how many agents are running at a given time. And to kick this off, we really did a 2x2 matrix where we did two channel and 4 channel DDR5 4800 and 6400 to see what the sensitivity to adding more channels or to changing the speed of those channels are. And in our 128 gig configuration where we're using in four channels 32 gig dims and in two channels 64 gig dims, you'll see that by far the number one thing that moved the needle was really having those four channels of memory. So, if this Zidex system came with four channels of DDR5 6400, that would have been better in terms of performance than if we had, you know, two channels of DDR5 6400 like the system came at, which is fine because you can always upgrade it later. And that's exactly what we did. And what we found relatively consistently is exactly the behavior that we would expect, which is slower memory was lower performance, faster memory was faster performance.

[00:15:48]But our impact even though we were varying this massively guys was only maybe I don't know 6 to 12% on average across our entire suite. Something we found during our profiling was that our compression benchmark actually was way more impacted by the memory bandwidth than it was by the actual CPU performance because the memory bandwidth in our scenario ended up becoming the main determinant of performance. And so it was no surprise that in our compression test but also our rag pipeline test we saw that the memory bandwidth was a huge impact. But on other tests, it really wasn't. Something else that we found which was really neat is that because we have such large caches, being able to keep data close to the CPU and avoid going back to memory as often meant that we could actually get way more performance. So what we saw was when we had, you know, highly single threaded workloads, two two cores, four cores, all those kind of things. You you tended to see that a lot of the memory pressure was actually relieved by the large caches. But once we got up to having 16 or 32 cores for an agent, we started seeing the impacts of having to go to memory a lot more often because we didn't have as much cache just dedicated to the core, right? So, of course, as you would expect, the more, you know, more work that's being done, the less of that overall work that you can fit in your local caches means that you're going out to memory more often, but it is something that's important when you're doing your profiling. And of course, with our new tool, we can actually run multiple tiles of agents on same CPU at the same time. And when we did that, we saw a pretty similar pattern. Now, once we had that baseline of doing DDR5, 4,800, and 6400, and 2 and 4 channel, and we kind of had an idea of what the answers were going to be, we knew what to look for when we scaled this up to Turin because, of course, more cores takes a lot longer to run the benchmarks. We actually had to do a little bit of our own profiling to be able to push it onto the larger platforms. But when we did that, we wanted to test what happens if you are using DDR54800 or if you're using 6400 in 8 channel 12 channel or what happens if you go and you do two dims per channel on that 12 channel memory controller. Because if you do 24 dims on a single CPU like we've seen in some of our server reviews, that means that the memory often only runs at DDR5 5200 speeds. And that also means that running DDR5 4800 instead of DDR5 5200 might make a little bit more sense because you're actually losing less than 10% of your memory bandwidth anyway. And this memory downclocking from going one dim per channel to two dims per channel was one of the reasons that we're doing this whole DDR5 4800 to 6400 in the first place cuz we kind of knew where this was going to go. What you'll see on this chart is that as we would expect, you do get more performance when you have more memory channels and at faster speeds.

[00:18:20]But on the other hand, downclocking memory also means that you get less performance. Now I would really treat the two and four channel memory results as a little bit different because they are running on Thread Ripper versus Epic. But at the same time, you can definitely see the impact of going from 8 channel memory to 12 channel memory or even just kind of get an idea in terms of the curve between going from two channel to 12 channel. But even doing 2ch on Zen 5 cores depending on the power doesn't really mean that you know you're going to get six times the overall system performance. Instead, it's a lot more muted than that. Now, especially on the mostly memory bandwidth band ones, those are the ones where you're going to see a huge difference in terms of maximum versus minimum performance here, but realistically, of course, 12 channels DDR5 6400, that's performing the best, but there are other configurations that are not too far behind in the vast majority of workloads. And we saw the same general shape when we went from having one agent across all 64 cores all the way down to having many different agents running on the same system at once. Okay, guys, I know that was a lot of numbers, but here's the practical reason. We actually are re-recording this video because when we originally recorded the video, a 64 gig DDR5 6400 dim was something like $1,500. That was only about a month, month and a half ago. But now on eBay, if you were to go look at current pricing, it would be almost $2,000. But that gave me an idea. You know, what would be the difference between a DDR5 6400 and a DDR5 4800 DIM, a little bit older generation, and we're going to use eBay. Now, of course, it's a little bit weird because it's like used pricing and all that kind of stuff, but most vendors actually do have lower pricing for DDR5 4800 versus DDR56400. And you see it all the time on things like desktop memory kits where lower speed memory tends to cost less because it makes sense. You don't have to hit those higher timings, those higher frequencies, and therefore you get a higher yield and that means that you can have a lower cost memory module. And so my thought was, well, let me go look up what the pricing is and just for fun say, hey, let's go look at let's go get a dollar per dim pricing and come up with what it means in terms of performance lost for dollar saved.

[00:20:23]Now, a weird thing with this methodology was that the DDR5 5200 dims, those things, guys, cost so much that they often cost as much as a DDR5 6400. So, to me, I would never go buy a DDR5 5200 dim uh just to go put into like a two dim per channel turn platform because, you know, like why the heck would you do that, right? There's I I think that from the performance data uh and from the cost aspect, I just I just don't know why you would. But I I at least wanted to just call that out since you're not going to see it here. And for our price data, we also used the lowest price listing of things that were not for parts and working, so actually working modules from a seller that had enough modules that you could actually fill an entire server. Now, of course, that means that we're kind of getting the lowest eBay pricing and your actual pricing may be a little bit higher, but overall the shape, I think, is what we really care about and also some of the magnitude information because when we looked at this, what we found was that yeah, you can save thousands of dollars by using lower cost memory or lower speed memory versus higher speed memory.

[00:21:21]And specifically the interesting one is when you have something like 12 channels of DDR56400 and you're trying to decide should you go and put say 12 channels two dims per channel of 32 gig dims and use DDR5 6400 versus 4800 here. What we've proven with our data is that you actually don't really lose that much performance because you're only going from DDR5 5200 to 4,800 which is less than 10%. And as a result, your performance difference, even on the memory bandwidthbound ones, is sub 10%. And on most workloads, it's within a percent or two. Yet that simple change can save you thousands of dollars in a system. Now, of course, you are losing performance. So then the next question is, well, what if I could buy more cores with that performance? And that's where the analysis got a little bit weird because depending on what you use for your system or your core cost, you could say like, you know, if you're using lower spec cores or lower frequency cores, maybe it's like $80 a core, $100 a core, $120 cuz my, you know, my OEM is a little bit more expensive or whatever it ends up being.

[00:22:20]You can save so much money from using lower speed memory that you can get way too many cores. And what I mean by you can get too many cores is that if you were to use that, you know, 80 to $120 per core, you could end up adding or saying that, you know, you have the money to go buy higher core count CPUs by a factor that is more than the actual amount of cores that you can put in a socket, right? The full Zen 5 cores, you're limited to 128 in their current generation Turin. Also, there's the Turin dense of course, which goes up to 192, but then you have Zen 5C, and that's not what we are testing. So I just want to be very clear that that would be a little bit different because the caches are different of course on that and clock speeds. So we just capped it at 32 cores because I think like after you go to 32 cores you're especially talking about like you would need more memory capacity. But the bottom line on this chart is simple.

[00:23:07]Unless you have something that is super memory bandwidth sensitive an HPC application and you know and you should be doing the profiling behind figuring out what that those applications are.

[00:23:15]Unless you absolutely need that performance, then the answer is save money on your memory and get lower speed memory, which is weird. Nobody talks about that. Go to two dims per channel and use DDR54800 when you do it, and you're really not losing that much for the majority of workloads. And it also allows you to deploy more systems. And from a strategy perspective, especially if you have a fleet of servers and you have a mix of applications that some are very memory bandwidth sensitive and some are not, because we've done that AI profiling, well, now we have a really good way to go look at which workloads need that memory bandwidth and which workloads don't need that memory bandwidth because we can see the applications and VMs that are really putting that memory pressure on. And so what you could do is let's say that your fleet said, "Oh, well, we need to have, you know, x number of 64 cores per socket, two sockets, and we need to have 1.5 tabytes of memory because that's the memory capacity we found out that we need." Well, you could actually have some systems in that 1.5 TBTE memory footprint that are 64 GB DIMs. And, you know, therefore, you get full memory performance, full memory bandwidth, and uh and you're you're running at DDR5 6400. Hey, that's great. But you could also run other systems where you run two dims per channel and have just less memory bandwidth and way lower cost per system. Okay guys, now I'm not going to lie, this was a pretty long one, but I still do have some key lessons learned.

[00:24:29]Like first off, you know, had we not had the AI agents be able to do this, we wouldn't have had the bandwidth internally as a team to go do all of this profiling and of course look at it over such a long period of time, go through it in such detail, and just find some of the weird things that we found.

[00:24:43]Every environment is different, so you're just going to have to go and bite the bullet and go and do that profile.

[00:24:47]Nobody wants to do it because nobody has time to do it. But now with AI agents, well, now you can strike back at that AI memory pricing because you all of a sudden have a tool that isn't necessarily just dependent on your time to be able to go and do all this data analytics. That's an awesome use of AI, but of course double check all of your results. Another key lesson learned, let's talk about CXL for a moment. Now, most modern PCIe Gen 5 servers also have the ability to run CXL, which essentially allows you to repurpose some PCI lanes in a system and use them for extra memory expansion devices like this one that we have next to me. These devices literally, you can see that there are four DDR5 DIMs in here and you could run DDR5, say 4,800 or whatever you want in here. But the key thing is that these allow you to add more memory capacity to a system using lower capacity per dim, but also maintain the full performance because you can still do one dim per channel and you can keep that that same high performance on your main memory channels, but also add additional memory bandwidth because these are not using your primary DDR5 channels. They're using CXL instead. So, this is actually a way to use lower capacity memory. also take advantage of some applications aren't necessarily as memory sensitive, but at the same time get the benefits of using even slower or uh just lower capacity dims in a system and really get those cost savings. We have tons on CXL and the ST main site and also on YouTube. So, go check those out. And guys, look, I get it. Whether you're buying a mini PC, a laptop, an edge server, a single just data center server, rack of servers, or entire data center worth of servers, guys, it is absolutely crazy how much the price of memory is impacting our day-to-day operations. And really, we all know that the impact of that is because AI is creating such a demand for compute that's also creating a huge demand for memory. And there just not enough fabs in the world to go and make all the memory that we need. And it's really causing prices to skyrocket. But now we can use AI to go do the profiling to figure out how our workloads could work in terms of how much capacity we need to deploy per core, but also how much memory performance or how much memory bandwidth we need in a system and how sensitive our workloads are to those. If at work you work in it, go do a project, figure this out, show it to management and say, "I deserve a raise at a bonus because I figured out how to save you a ton of money in a world where memory prices are going up every single month."

[00:27:09]Other people are profiting off AI, so why can't you? Now guys, with all these videos, I love to hear what you guys think. So, let me know down in the comments. But also, if you did like this video, we'll give it a like, click subscribe, and turn on those notifications so you can see whenever we come out with great new videos. As always, thanks for watching. Have an awesome day.