Distributed Home Data Centers: Arch Rao

[00:00:00]Hi, I'm David Sandalode. This is the AI, Energy, and Climate Podcast. Several months ago, I saw a news story that caught my eye. It was about a partnership between three companies.

[00:00:10]Nvidia, the PY Group, which is one of the nation's largest homebuilders, and a technology company called SPAN. The partnership is begun installing many data centers at homes around the US. The idea is to use existing electric interconnections and excess power capacity in homes to build distributed networks of computing capacity and then pay homeowners for this service. Now there's there's a lot of innovative activity happening at the grid edge these days including with virtual power plants, vehicle-to- grid charging, and much more.

[00:00:44]This partnership struck me as one of the most innovative. One can imagine it helping address three problems at once.

[00:00:50]First, the long lead times data center developers are facing to secure electric interconnections around the US today.

[00:00:56]Second, the enormous community opposition to large centralized data centers. And third, rising electric bills for homeowners. It could be a triple win. At the same time, one can imagine a number of barriers to launching a partnership such as this, including challenges with fiber connections and more.

[00:01:17]So I was thrilled when Arch Ralph the CEO and founder of SPAN agreed to join me on the AI energy and climate podcast for a conversation about his innovative partnership with Nvidia and the PY Group. Arch founded SPAN in 2018 after spending time at Tesla and other companies and graduating from Stanford.

[00:01:34]He's a visionary in a number of respects building a company that's transforming grid edge architecture and he may be onto a big idea when it comes to data centers. I hope you enjoy our conversation. Arch Ralph, welcome to the AI, Energy, and Climate Podcast.

[00:01:49]>> Thanks for having me, David.

[00:01:52]>> Well, thanks for joining us. I'm really looking forward to this conversation because I read about your very interesting announcement with Nvidia and the PY Group, a major home builder a month month or two ago, and I've been looking forward to learning more about it ever since. And as I understand it, the idea is to create a network of small computing nodes, kind of mini data centers at homes around the United States. So, so tell us kind of what what's your product and how's this going to work?

[00:02:19]>> Yeah, absolutely. Um, for a long time now, SPAN has been uh solving the digitization of the electricity grid problem. So, at its core, we've been building uh a u router for electrons, if you will, that helps us increase the utilization of the existing grid. So a couple of years ago as I was um thinking about what is the biggest driver of load growth for us across the country it's it's undeniably compute and compute is morphing from being uh compute for training which is a cost center to build models to compute for inference which is a revenue center where these models are being applied to deliver inference value agentic inferences physical AI etc and that inference doesn't have to look the same in terms of how you structure the compute doesn't have to be large contiguous racks of GPUs running proprietary algorithms in a large shell.

[00:03:10]Most of us run tasks that can be served by small clusters of GPUs. So if you take the power side of the equation that we've been solving with with span and uh try to patch that with what I think is this burgeoning demand for inference compute. Uh the answer is extra distributed compute that is purpose-built for inference that can leverage existing power infrastructure using our digital power control systems.

[00:03:36]>> And so you're putting GPUs about a dozen of them as I understand it. Uh that's right. At homes uh and >> maybe just some of the basics for people so they can help visualize this. What is what does your unit look like? Does it make noise? Does it generate heat?

[00:03:52]>> Yeah. So really think about the device as being a um slightly larger than an air conditioning unit size appliance.

[00:04:00]It's uh taking all of the key elements that you would see in a large scale data center and miniaturaturizing it into a purpose-built device that has a condition shell, has power monitoring and power controls, has resiliency, has high bandwidth networking with fiber connected directly to it, and a uh a system with a uh rack of GPUs and CPUs that can do the inference work. Uh and we've also done quite a bit of systems integration to directly pair that with uh a liquid cooling system, a plate based liquid cooling system that is also self-contained with um a quiet heat pump. Um so in a traditional data center you would have uh let's say a 42U rack which is like a roughly 6ft tall rack with air cooled servers running CPUs and GPUs in them with a an area of fans high-speed fans that are trying to move air through it. those tend to be very very loud and you didn't care about that noise when you were in a large server room if you will or in a large data center. Uh obviously this is a problem that needed to be solved in our solution because we're going to put this in the side of your home or the side of a commercial building and we've done that by essentially taking the thermal mass away with convective heat transfer with a liquid and then having one large fan that spins rather quietly uh with a heat pump. So it's less than 60 dB at a meter if you will. And not surprisingly, computers also move into a model where having direct liquid cooling enhances both the life longevity of the GPUs and the performance of the GPUs. And uh that it's a natural sort of intersection of I think technology development in both those areas that we've implemented in Extra.

[00:05:36]>> Yeah.

[00:05:37]>> So it's fascinating. So talk about the power system aspects of your earlier unit. How much power does it draw? Um and more.

[00:05:47]>> Yeah. So we we have publicly talked about um you know a couple of tiers of our product that we're building. So you can think about um each unit as having a defined power envelope for it load which is the GPUs doing the compute work and then cooling load that's and the balance of system load if you will that's managing the system. In a nominal residential setting especially in a new home construction setting uh the incoming power is around 200 amps of uh power. That's the conductor sizing if you will. From our fleet of span homes, even fully electric homes, we've seen empirically that there's about 80 amps, roughly 19 kilow of unused capacity in the 200 amp conductor capacity, if you will, that's almost always available.

[00:06:31]Right? So, the the data that we have from our fleet is uh virtually all of our homes uh 200 amp homes roughly 98% or greater have 80 amps of underutilized capacity 100% of the time across the fleet. Not surprisingly, this also cascades down to 100 amp homes that have around 40 amps of ND plus capacity. But 40 amps is not large enough to build a sufficient density of GPUs for inference applications. But 80 amps 19 kow is actually pretty well sized. So within that, let's say 19 to 20 kow power envelope, we have around 5 to 6 kW of cooling to take the rejected heat away from the GPUs and we have around 12 to 13 kow of compute. that corresponds to either eight B300 uh GPUs or 16 RTX Pros or looking ahead there are uh GPUs from AMD and Cerebras and other companies that also fit within that power envelope pretty really well. That's the modular system size if you will that we're building initially >> and and so almost all homes are rated for this much power you're saying.

[00:07:32]>> All new homes by design are rated to 200 amps if not greater.

[00:07:36]>> Yeah. So before extra one of the big problems we were solving for new home construction partners like PI for example is how do you avoid overbuilding even beyond 200 amps like for the last 10 or 15 years single family homes across the US which is about 800,000 to a million homes being built every year where we were being built at 200 amps of copper coming in that was the the design standard if you will from the utility coming into a single family home with the advent of electrification with the desire to have homes ready to add EV charging in the future or add electric water heating in the future or induction cooktop in the future. The load calculation was pushing the size of the conductor up to 320 amps or 400 amps even. And for a long time now, the last three or four years, we've been helping home builders avoid that cost which they have to pass through to the home buyers by using span. That'll ensure that you never exceed 200 amps because quite frankly 200 amps is 48 kW at 240 volts of contiguous power, concurrent power draw. That almost never happens. Right now, we're applying that same logic to say, what if I were to use that, you know, 20 kow out of the 50 kilowatt available to your home to be able to deploy an extra node. I'll preemptively answer a question that often comes up.

[00:08:48]Needless to say, if you go upstream from a single home, the transformer level, it's not a linear sum in terms of how they size the transformers or the substation upstream. So, we're not proposing to deploy an extra node in every new home. In a community, we've done a tremendous amount of power flow math to say a roughly 25 to 30% penetration of extra nodes into a new home community does not in any way impact the upstream infrastructure. We can very very comfortably operate that system within that power envelope.

[00:09:17]Walk me through that because presumably you're increasing the base load draw on the system and so transformers are going to be running hotter and that's going to have some wear and tear on the on transformers maybe other parts of the system. Right.

[00:09:31]>> Yep. That's right. So let's talk about it from a a nodal perspective. What we do from a compute and load control perspective and a battery perspective and let's talk about it at a zonal perspective. Let's say a collection of homes if you will, right? Um a typical new home construction um uh you know power system modeling would have a single pad and a transformer often a 100 KVA transformer sometimes 167 KVA transformer powering a collection of 8 to 10 single family homes right um and then you you cascade that upwards you get to sort of megawatt scale substations that power a community of dozens of homes if you will right or hundreds of homes what we are designing is uh every extra node will be paired with a span panel and a battery and every sort of simplified way every extra home will have a neighbor to the left and the right that also gets a free span panel and a battery. So at a at a the home that has the extra node we have the ability to throttle compute or move compute workloads around when needed to other extra nodes that might have capacity. We have the ability to throttle your home loads with span which is what we've been doing for years now.

[00:10:40]And we have a onsite on premise battery that can inject power into our bus bar as well. So our bus bar is rated at 225 amp. So think of it as like an ongrid micro grid that's doing some clever realtime optimization. Then beyond that at a fleet level or let's say a zonal level we have two additional homes that also have load controls and batteries that can inject power into the AC grid that they share downstream of that 100 KVA transformer. So we're able to ensure that in in some at at a at a fleet level, even if you take hundreds of homes in a new home community, a third of them have extra nodes and all of them have span panels and batteries, we're able to ensure that the the network is is very resilient to this increased demand.

[00:11:25]I'm sorry to interrupt, but I really have to tell you about our Substack. The DSR network Substack is the absolute best way to follow deep state radio, the DSR daily, words matter, need to know, silicons consciousness, AI energy and climate, and the daily blast. Sign up to get notified every time a new episode drops, [music] and stay uptodate on the latest news and expert analysis. You can also support us by becoming a paid member. Paid members [music] get an ad-free listing experience, select episodes 2 days early, access to live streamed episodes, and 50% off of David's Need to Know Substack. Please consider joining us at dsrnetwork.substack.com.

[00:12:10]That's dsrnetwork.substack.com.

[00:12:15]Thank you. And back to the show.

[00:12:19]>> Talk some about the commercial aspects of this. I'm not sure how much you can say about pricing, uh, how much is public right now, but what what as much as you can say what are the costs of these items and what does this look like to a homeowner? If I wanted to if I want to do this, what would I need to pay in order to get your service and what would I earn from it?

[00:12:38]>> Absolutely. Yeah. So um without giving away too much I think at at a high level the way to think about the cost of our system is um there's a very clear speed to power advantage that we can deploy compute capacity today as opposed to waiting four or five years to build a 100 megawatt data center but there's also a very measurable capex advantage so if you think about the cost of the compute that is uh you know par passive like it's the same to us as it is to a large scale data center if you buy computer the same volume that's on the order of $30 million per megawatt is the the going rate if you will for inference compute or training compute the in a traditional data center uh the rest of the system so the the land the the shell the cooling the gen set the transformers the batteries all of that combined sighting permitting is on the order of 15 to $20 million a megawatt today that's for 100 megawatt scale data center because we've taken away all of that complexity of having to land design development interconnection uh you know gen sets sub transformers etc by the way, all have exceedingly worse lead times and instead transformed it into a single node that has all of those components pre-fabricated at a factory.

[00:13:48]Our cost is 3x to 5x cheaper on that $15 million number. So we're at like sub $5 million a megawatt today and we're not quite at the scale that we plan to be at in the next 6 to 12 months. Right? So there's that capex advantage that then translates into economic value for the compute offtaker which is very very clear along with the speed to power advantage. We do have an increased opex compared to a large scale data center because we're buying energy at effectively retail energy prices from the utility. The average utility rate across the US is around 12 cents a kilowatt hour as opposed to let's say a bespoke rate you can get 4 to 5 cents a kilowatt hour at a large scale data center. But even at that increased opex at the site level energy opex at the site level our capex is so low that our effective payback is much sooner than a traditional data center. So that's at the let's call it like asset economic uh level, right? And there's the consumer economics. We have to be mindful of the fact that essentially you are the landlord. Let's say you're an extra node host. I'm borrowing a small amount of physical space and I'm also utilizing some of your utilities, your your power and ISP, your fiber connectivity, right?

[00:14:52]And we've tried to flip the script to say let's make it even simpler because the computer is so valuable. What if if I were to just be able to uh pay for your energy and internet use as you would use normally? So each extra node has the potential to essentially offer the individual consumer between $200 and $400 of economic value each month, which in most parts of the country fully compensates you for the cost of your energy consumption and your let's say home internet connection with much higher bandwidth fiber that we can deliver to your home. So your estimate is $200 to $400 a month of value being generated per unit um per month basically and then you deliver that to the customer through reduction in bills.

[00:15:35]>> Yeah, we're saying that's the value we will give to you. That's not that we the value that the compute generates is far greater than that because compute demand is still uh significant and let's say fairly volatile. Uh each node generates you know close to over $10,000 of economic value per month in at the current uh compute pricing, right? And keep in mind much of that goes towards paying the the cost of the capex, right?

[00:15:58]You have to pay for the the server, you have to pay for the GPUs, you got to pay for the cooling system, you got to pay for the installation cost, the operating cost, etc. Net of all of that, we have the ability to compensate quote unquote you as the the host customer, host partner for us uh on the things that I think matter most to most Americans, right? Or or anybody really. It's like, can you bring the cost of my, you know, utilities down? And by doing that and and maybe you're going to this later in the conversation, we're able to shift the narrative from not in my backyard to yes, please in my backyard, right?

[00:16:29]>> That's a great motto. I like that. Yeah.

[00:16:32]Um just on the um another part of the infrastructure issue here just to go back for the minute. Um talk talk about the fiber that you need here. Is it >> are are homes connected with right fabric? Because I know in general the uh downloads a lot faster than the upload in most home connections and it sounds like if I understand what you're doing, you're going to be needing a lot of upload speed to make this work. What does that challenge?

[00:16:58]>> Yeah. Uh home home internet as we know it is very different than what you think about enterprise uh you know what I would symmetric connectivity on fiber.

[00:17:05]So on on the one hand um there is there has been quietly a massive amount of investment going into building out fiber infrastructure. In fact all of the new home partners that we're working with typically are working with uh retail providers like AT&T or Verizon or Spectrum or Comcast who are already landing fiber to the new home construction sites and often have fiber coming all the way to the customer u lot if you will right. Um we have deployed nodes right now that are able where we're able to very easily subscribe to uh high quality 1 gig per second symmetric uplink and down link fiber per site and uh that is what the node extra node receives and you might get from the switch something that is comparable to what you might be able to buy from AT&T retail for example like 300 400 megs per second type of service concurrently to having an extra node sitting on your premise. Now a lot of the IP that we've developed also goes into what we call the secure or stationation layer. Uh if you think about it your network traffic is not at its peak capacity all of the time back and forth just like you would think about car traffic or energy traffic if you will. So by designing the systems where we have enough on extra node compute and memory where the model which is the heavy you know weights uh and and uh you know several gigabytes of of data if you will being stored locally the traffic becomes the tokens going in and out and the tokens going in and out emphatically what we're seeing with nodes that we have deployed is not at the gig gigabit scale it's usually in the megabit scale of traffic flowing back and forth. So once installed, once deployed, uh what we're building in the residential sites, one gig of symmetric uh fiber connectivity seems plenty for most applications. In some of the commercial sites we're going into, we're able to source 5 gigs or 10 gigs of symmetric uh fiber connectivity as well.

[00:18:56]That obviously for certain applications becomes meaningful.

[00:19:00]>> Let me just keep probing on different aspects of this. It's such an interesting idea and it's it's so innovative and creative. Another question that occurred to me involves it's right now >> GPUs are improving in quality. We're you know going from Blackwell to Ruben to Fineman and Nvidia GPUs near there other innovations in GPUs that are happening.

[00:19:18]>> Do you envision physically swapping out GPUs from these units or some other >> much like you would see anything else what's your plan?

[00:19:27]>> Yep. planned up solicits, right? Like you plan for a threeyear halflife and like a five to six year uh full life of these operating systems of these GPUs and and CPUs and not just because of their um uh expected operating life but also because of the as you rightly mentioned technology evolution that's happening very very rapidly. So the servers are designed to be field replaceable. So the servers have u purpose-built connections for power networking and cooling. So the direct liquid cooling that goes into it that are all quick fit connectors and the servers sit on trace that we can pull out. So obviously with authorized access to the site be it a um a service request where let's say a server or GPU is not performing we can do a very quick field swap uh and we will not do any IT maintenance on site and if it's a planned uh swap as in we've reached 5year life of the asset and now we want to upgrade it to something else the core infrastructure is the real value we've essentially now built a large network of distributed power and compute infrastructure capabilities where we can upscale the computers needed that that power networking and cooling will always remain >> and talk about how you see this scaling.

[00:20:39]What's the vision here? Will you be entering into deployment agreements with with with with home builders with utilities with what are your channel partners in trying to make this work?

[00:20:49]>> Yeah, you know the we broadly think about the partnerships we're framing in in three categories. There's host partners, there are technology partners and then there are offtake partners.

[00:20:58]Right? [snorts] In that order, we had to solve for host partners, the homebuilders and eventually the homeowners. Uh we have a number of commercial uh real estate partners that uh have sites that are not suited for large scale data center operations at all. Like they might be, you know, stranded power. They might be less than a megawatt, less than 5 megawatt. Uh but they often have uh physical space, you know, either uh on the rooftop of a building or on the sideyard of a building, if you will. So we've uh we've now acrewed um partnerships in the host side where we can deploy just over the next year over a gigawatt of inference compute without breaking a sweat and that's existing sites. If you look at new homes that are built every year, we can deploy an additional gawatt of compute every year if you attach our product at like 20 25% attached to new homes being built. Right? So there's the existing site model where we can deploy several gigawatts and just with the existing pipeline of host partners we can do a gigawward and then we have this evergreen model we can deploy into. Then we have technology partnerships. We have partnerships with the likes of invi Nvidia who are who've been incredible in helping us think through the roadmap just as you talked about server evolutions, liquid cooling evolution, uh what is the evolution of models and what type of models can be run on what type of compute and ultimately what we're saying is once we've built the infrastructure you can choose as a offtaker what combination of comput and models you want to run across a distribution of sites. So if by end of next year let's say we have tens of thousands of sites that are up and running totaling a gigawatt of compute you have a tremendous amount of flexibility in determining what combination of physical compute and or GPUs and what models you want to deploy. Now the host partnership side or the offtake partnership side we are we are working with the um uh the large uh hyperscalers and the frontier labs that are desperately wanting to find more compute >> [snorts] >> uh or more power to deploy compute because many of them actually have access to uh the servers but not really a place to put them in and we're able to solve that problem for them. Uh and that opportunity is in the hundreds of megawatts to gawatt scale compute per year. You've got another category of inference aggregators. So the folks that are building enterprise solutions um that are um that are not the companies that are building the data centers, right? And they themselves are second in stack uh from a capex perspective compared to the large hyperscalers are the ones that are primarily building data centers and we're able to give them access to lower cost compute across across the country. And then the third tier is just merchant comput which is where we are active today where we have compute nodes that you can go rent today if you go to like a vast AI or lightning you can go find an extra node you won't know it's an extra node because to you it's just cloud compute and you can choose how many servers you want how many GPUs you want for how many weeks or months and it gives you a price and you can just rent it right >> so [clears throat] you have a retail window right now you could people can just go to your website and design their own x-free unit >> not not through our website so we're not trying to become the marketplace We put our compute node today in third party marketplaces which are uh which is where an academic institution or a small startup could just go and rent GPUs right um ostensibly like you today when you go uh say write a query on cloud or chat JPT uh as a consumer or even enterprise you don't really know where that query is going to which compute in which part of the world is serving that that particular request and it's kind of the same idea our orchestration layer aggregates across all of our sites to give you essentially a seamless cloud of a large network or clusters of GPUs that if they meet the same quality of service that you would find from a traditional data center is virtually the same value to you, right?

[00:24:43]>> So if one of our listeners either at their home or their workplace wants to do this, wants to buy or rent a extra unit, where do they go to?

[00:24:52]>> Yeah. So you we don't sell extra units.

[00:24:55]We will build, own, and operate them. If you want to be a host partner, that's that's a great conversation for us to have. we will rent or we will uh be happy to partner with folks who want to off take you know meaningful tranches of compute capacity directly from us. So, so, to illustrate the point, if you have an extra node sitting on the side of your home, you're not necessarily using, you're not the direct user necessarily of that node. You might be when personal AI becomes more relevant or physical AI becomes more relevant and you have robots that want to share context and there's a compute right there that's super low latency. That's great. But today, the vast majority of comput is just going to be going back into this pool of aggregated compute that we have available. And then we're be building this optimization layer that helps you determine based on the type of inference that you're trying to do. What is the right uh physical uh low latency node to send the query to? What type of model is embedded in those uh devices? How much uh contextual memory do you need? And how much bandwidth do you need? And we're able to optimize that. It's like a traffic routine if you will. Right.

[00:25:58]Well, it's a fascinating product and a really fascinating idea that meets the moment. uh you've been very generous with your time. Is there anything we haven't touched that you would like to say any message you'd like to get out?

[00:26:10]>> Yeah, I think fundamentally what I've been really passionate about for the last couple of decades and what I think SPAN is building is uh infrastructure, right? I think the the piece that um you can you can choose to be a betting person on is you know how big is AI going to be? Which comput company is going to win? Which model company is going to win? And I'm saying we're not placing bets on that. What we're placing bet on is building infrastructure for maintaining our our dominance or lead in AI. And that comes down to investing much like we did in railroads and much like we did in power systems, we're doing that with now taking what is otherwise a very very analog electrical grid and making it digital much like we did with the telecom infrastructure 20 years ago. Right? So and power and comput are two sides of the same coin as I mentioned earlier. So a lot of what we're doing is uh you know we think of ourselves as an energy transition company and for me the energy transition is less about the the the effect which is the gas to electric transition. It's the cause which is the analog to digital transition and that at its core is our mission.

[00:27:17]>> Yeah.

[00:27:18]>> Well I always close this show with two questions for our guests and the first question is how are you using AI tools in your day-to-day life? Uh we use enterprise cloud here at uh at span. Um I do not use AI to write my email responses for me or do calendaring for me. I often use AI uh either cloud or perplexity and I often go back and forth with that. uh in terms of doing research in in the last um in the last I would say year and a half to two years as I've been obsessed with uh building extra um I I've had to go learn a lot about compute which is not my you know my academic work was all in energy and power conversion so I've been using AI to become smarter on all of these things and it's a pretty good tool >> it's an amazing tool I'm using in the same way it's incredible for climbing the learning curve on different topics Just unbelievable. Uh, final question.

[00:28:14]Could you please recommend three books, reports or articles to our listeners?

[00:28:19]Could be something old or new.

[00:28:23]>> Um, that's a good question. Uh, there are a couple of books that I often go back to. Um uh there there's a there's a book um from an anthropology book called Guns, Germs, and Steel that was published I want to say almost 20 years ago that talks a lot about kind of the industrial revolution and sort of the evolution of medicine and how that uh that essentially >> translated to how we think about uh economic growth across the globe, right?

[00:28:55]And I think there are some very interesting parallels there or you know as people often say history repeats itself or has at least a a rhythm or a rhyme to it, right? Uh that that's one that often comes to mind for me. Um I'd say another book that I've enjoyed in the past that I tend to go back to is The Diamond Age by Neil Stevenson.

[00:29:14]That's a really good fiction book. uh you know if you're trying to escape some of the um let's call it uh dystopian views of uh AI that are presisted out there I think it's good to go read some of these things from 2030 years ago again some of these ideas are becoming more real now than they were before but I think that's a good one to go back to and my bedside book that I'd like to go back to just just for pleasure is Saki Hmundo collection of short stories it's it's a fun read you know >> well Artrao CEO, founder of SPAN. Uh, you have a fascinating idea, fascinating product. Thank you very much for joining us on the AI, Energy and Climate Podcast. Thank you for having me, David.