Unlocking Earth’s Intelligence | Lukas Geiger & Sean Wohltman | Google

[00:00:08]All right, good morning everybody. How are you? This is so exciting for me.

[00:00:12]This is literally one of my favorite events of the entire year. Um, I've been coming since it's existed. Uh, I did miss one year two years ago. I had a good excuse. It was a really nice day to fish. Um, so I I missed that one, but other than that, I love being here. Uh, and I'm very pleased to have this time up on stage. Um, so I'm going to talk a little bit about um, embeddings. I know this is a spatial data science conference. I don't probably need to spend too much time talking about embeddings, but they're really foundational to my entire presentation.

[00:00:40]And I mentioned fish. So you can think about uh, embeddings as the geometric similarity uh, that we can encode to make it easier for machines to understand the semantic similarity of of different things and distinguish them.

[00:00:53]Uh which is why if you know you train a model uh on different types of tuna, you can show um Gemini uh a picture of somebody holding a tuna. Um and then that's the elephant tuna uh that I caught while we were preparing this presentation uh with the Cardo team a couple uh two weeks ago uh last week rather. And uh so you can then say okay based on your understanding what kind of tuna is this? Uh, and your Gemini will be able to obviously detect, you know, the the pectoral fins of the elephant tuna go past uh uh the dorsal fins. Um, and I didn't edit this. Gemini can be extremely cruel sometimes for for me at least if you read that last line. My cardiologist says I'm fine and I can fish. But anyway, um, so anyway, so that's a little bit about about embeddings. Uh, now we'll talk about um some of our exciting foundation models.

[00:01:41]My colleague Jeremy Malik uh was here last year and started talking about this work that we were doing um which we published this year and released fully as alpha earth foundations. How many folks in the audience have have heard of this and looked at it? Okay, cool. Um there is a cardo connector already. Uh we'll talk a little bit about that. Uh what we did basically was build a global 10-meter virtual satellite and this this satellite is trained on various different inputs. Sentinel 2 at 10 meters is a big component, but we also have nonop EO imagery. We also have radar imagery. We have climate imagery.

[00:02:14]We even have Wikipedia in there. And so this data set combines all these different uh uh uh inputs into uh a video. And for every 10 me 10 m pixel on Earth for every single year, this is an annual product that goes back to 2017, it has a spatial window of about 1 and a half kilometers around that point. Um and what we do is we create a a video to understand trajectory and we we we then create this satellite. So it allows us to take very very sparse uh measurements in some cases and create a more continuous um uh data set. Um so you can you can access uh Google uh alpha earth foundations through Google Earth Engine.

[00:02:54]Um, and what that allows you to do is either open it directly in Earth Engine, access it via the Python API, access it through the UDFs that Cardo has built um in into their platform, or you can access it through BigQuery directly as well um using the ST region stats function uh in in BigQuery. And so um to go a little bit deeper, what this actually is is the embeddings as a raster data set. It is a 64dimension embedding space. And since it's a raster data set, that means we have 64 bands of data. Now, these bands do not correspond directly to the inputs. It's not like we're looking at the the red band of Sentinel one and and a SAR band uh VV or something like that. That's not what it is. So, you can put them into RGB space and you can visualize them and you can see that like water looks like water.

[00:03:46]That just happens to be that that there's some some similarity there. Uh and there's some some difference between the other things that are around it. So uh the data is is available. All the embeddings are there. But really those bands are just embeddings. They are not they are not actual inputs. But what can you do with it? Well, why is this important? Uh there's a number of things that you can do with it. I'm going to talk about uh similarity search and then also about change detection. That's what we'll have time for today. Um so similarity search uh things that are close together in that embedding space are more similar to things that are not uh close to each other. So, if I go to Vietnam and I click into this uh uh mangrove area here and then I apply a similarity search, a dot product search across all of the other uh points in the image, I'll see these really bright areas and those are um uh more similar to mangoes. And that works if you take that same point and apply it over to Bise. It works if you take it to the Florida Everglades and you can find that that similarity. Uh you can try this out for yourself. There actually is a a link uh that you can you can take a look at here to look at this uh embedding similarity demo. It's a lot of fun. Um one thing I love trying go click on a golf course. You'll find a ton of them across the United States. And go look at China or Saudi Arabia. You won't see them. Uh it's really interesting. So the way that this actually works under the hood, uh we have all of this embedding in here. And the powerful thing about it is we spent all the time and all the power to actually build this embedding data set. And now you can just do simple nearest neighbor searches or just a dotproduct um uh equation to find that similarity without having to retrain and do all that inference um in the data set. So it's extraordinarily powerful and it's really really cheap for us to run at that point. Um so then the other thing that you can do with change detection so I mentioned it was an annual product currently today. We are looking at subanual products in it as well. And the really interesting thing there is that through time uh stability uh it really shines. So if an area isn't changing uh the embeddings will be similar year to year and the areas that are going through change uh will will actually stand out quite differently. So for instance if we look at uh the central valley of California here um we take a look at uh various different uh regions. We've got some forest and timber regions. We've got uh a water uh we got a a dam here. And then we've got these uh agricultural monitoring fields and then suburban development. Um so if we look at each one of these and we go back and forth between 24 2024 and 2020, you can see there's quite a bit of change. A lot of the agriculture is probably crop rotation. The phenology is is being picked up there. Uh 2020 there was a drought year in California. You can see the difference in in the water levels there. Uh and then in uh the forest region uh you can actually see the evidence of burn scarring uh from a forest fire. If you chart this out, um this is all in a a uh uh negative one to one space. You can actually see uh places that have changed more uh between between the areas popping out here in in the bright white. So things again that are similar will be clumped together.

[00:06:47]Things that change their class uh give a good indication of change. Uh Diego, I don't know if you're here. Uh Diego did an awesome uh uh workshop yesterday uh at the at the workshop day where he actually used um uh Cardo connector to grab these uh embeddings and then trained uh because it's like a it's a satellite data set essentially a virtual satellite used uh existing wildfire parameters uh and this has been published. It's great it's a great workshop to go work through um to actually come up with a statewide California fire risk uh score. So really practical application there. Um and it was really really great application of using this. That's what we want to see.

[00:07:27]Um okay so I'm gonna switch gears a little bit now. This is been the domain of of Google Earth Engine uh for many many years kind of at that 10 meter level global scope. Um and and that was you know all that work was being uh pioneered by Google DeepMind. uh the past year I spent a lot of my time working with Google research and a new class of models um that we're talking about here today for the first time publicly uh as remote sensing foundation models. So we've been working with uh three uh core satellite imagery providers Vantor which was Maxar Planet Labs and Airbus. Um they've been interested tester of these new remote sensing highresolution foundation models. These are models that Google research built for our Google mapping team internally to help us do common tasks to be able to take in aerial imagery, do segment uh semantic segmentation on it uh for mapmaking. Uh and the models perform extremely well and this is a very important task. Um they also you know we're using this to delineate solar panels. We're using it to uh classify buildings when we're helping out with crisis response. Um and then we also do some super resolution on Sentinel 2. So very very valuable foundation model set here. But the ones that are actually the really really interesting ones have turned out to be these multimodal remote sension visual language models. Um and the idea is to try to understand what's in the image not just the values of it but understand the objects that are in it and what that represents. So for instance you could say okay here's the old part of Barcelona. Um and then you know that's picture of roundabouts. Um and so these models uh we needed to train them uh to be able to do this and understand what's there. So to do that um we wanted to be able to build as broad as possible foundation models and then you could build heads on these models for specific tasks. If you want to have a semantic segmentation, if you want to do object detection, um etc. But in order to build these models, we needed to have a data set to train it on. Uh if you look at some of the things that Gemini has trained on, you know, we're talking about the order of hundreds of billions of images, like the entire web index that Google has for imagery and YouTube videos and everything like that. Out of that, there was really only about a million pictures from you can imagine like a you know a Times News article or something like that that you might have a um an image of the Lahina fires and you've got uh you know satellite imagery with a caption. So really it was a pretty small amount of images that Google had access to. So what we had to do is actually go out and build uh a new data set in the order of a couple hundred million images. So we took all of our mapfax data, the data that is our places API, the data that makes our road network. Uh and then we also took um uh remote sensing those images that we did find from the web. And then we wanted to make sure that we did this not just where Google is heavily indexed on population but globally. And for each of these, we started to load in the actual data sets onto onto the images, overlay them, and tell tell Gemini um what's actually in that image. And so Gemini went through and generated hundreds and hundreds of millions of of both long form and short form captions to build this these foundational model data sets.

[00:10:33]And you can see some of the examples of what that looks like. Um you know, it could be talking about very simple garden design or aerero photography of boats in the sea. But you're starting to understand that wow, we're starting to get a description of what's in this image, not just the radiometric values to it. Um, okay. And so what comes out to be really, really important if you want to try to train imagery on these models and then do some similarity search and finding is a capability in BigQuery called vector search. Has anybody used BigQuery vector search before for all two people? Great. Um, really really interesting uh product uh extraordinarily fast index to allow you to do this type of work. And so the basic workflow if you're a satellite imagery provider and you've got your daily take is you can take an image run it through the uh image encoder part of this running in Vertex AI and then you can take those embeddings and store them as columns in in BigQuery. And then when you have another image if somebody says oh I'm looking for something that looks like this. Well, then you run that in Cloud Run uh at andvertex and you can then uh do the image transformer, find the similar embeddings and get the result of things that look like the thing that you were looking for. It also works because it's multimodal to be able to go in and say, "Okay, I want to do a a query for ships in Athens Harbor." And because the text and and um image embeddings are in the same model and in the same space, you're going to get an embedding that is very close to what you actually have from the from the input image. So, one of the partners that I mentioned uh is Planet Labs. Uh if you're not familiar with Planet, they've got a constellation of satellites that capture the Earth at 3 mters. They just announced uh Owl is going to be at 1 meter uh daily. So they plan they image the planet every single day and they've gone wild with this and I'm super excited about what they've done. What they realized is that they have a a georreerence plot for all these areas that they're capturing. And so if you looked over time just like we showed with the uh alpha earth foundations you could actually detect change but you could actually do it off of the text input as well. So this is a superpower.

[00:12:38]you can now say I am specifically looking for things that was empty undeveloped land and is now data center this was not possible before this model um and so that's that's what you they're doing right here they're actually allowing you to find that specific thing uh anywhere on the earth and if you say yeah yeah that's what I mean by data center then it can go out and branch out and find all the things that look just like that um some other examples here of dense forest and and clearing after development um if I'm looking for factory facilities etc And one of the really interesting things, one of the hard problems that that's out there in the geospatial space that I think this is going to help solve is uh detecting specific types of construction, right?

[00:13:18]Detecting um when a nuclear plant is being built uh or a coal power plant in China is being built. And so if you take all that data and you stack it up like like planet Ken, something else really interesting uh unfolds and that is the similarity of phases of construction can actually be clustered themselves. You know, and this could be things like, oh, all of a sudden the land is now clear and then the next phase of the construction is, oh, some big yellow things showed up and they started doing some more more construction. Now there's materials that are on site and these things actually tease themselves out based solely on what the model is seeing. uh in each image and you can use Gemini to actually build a intelligence report uh to monitor these key events and have tipping and queuing to understand what is happening and that's what they've done here. They took the the model outputs they put it into Gemini. Gemini created an infographic of the different phases of construction um put together a summary and put this out as basically a finished Intel product.

[00:14:17]Now, one of the things that Jav mentioned is that, you know, our roles are changing, right? If I'm an intelligence an analyst, this doesn't scare me. This is fantastic. This gives me even more power to be able to do more analysis in a given day, leveraging that AI, factecking that AI, making sure that I can apply my expertise as an analyst.

[00:14:36]So, don't be scared of this. I think this is really revolutionary and I think it's amazing what what we can do together with the machines. And speaking of that, um, we obviously I think the theme of this conference over the last two days has been about, uh, the importance of of spatial agents and and what I say here is the path to planetary scale geospatial reasoning. It's all these pieces working together. Uh, and Google's going to have a lot of the answers here. We're not going to have everything. So, it really is going to rely on firstparty data from Google, third party data from other satellite providers or other data providers, uh, and platforms like Cardo to be able to help build agents and stitch them all together. We do have in the uh recently released Google Earth Pro and Pro Advanced um we have an early version of uh Earth AI uh that allows you to open up a Gemini window and ask questions about the data that you have loaded. So for instance, I can draw a polygon here if I'm trying to build utility sales uh utility scale solar and I can say, hey, I need to find uh available parcels that are greater than uh 40 acres for this project. It's a it's a simple selection, but this is where we're just getting started, right? you can start to picture a future where you could register agents. Um you could have your own capabilities, your own data set uh that you might want to analyze in BigQuery.

[00:15:47]And we can start to have this um this spatial future where these agents work together uh to to democratize access to these questions. And you know this is the 20th anniversary of Google Earth and and it's a really really important year and I think it's I'm a little biased. I mean that's what I've been doing for the last 20 years but um it really did democratize how people access GIS information. you didn't have to have a RGIS license and you didn't have to have um a background in GIS uh to be able to go out and explore the world and understand what's happening in it. And so I think we are on the the precipice of this whole new uh forefront of AI where we're going to be able to bring all this uh uh capability and reasoning and capabilities together to be able to analyze data uh far better than we ever could in the past. So thank you so much for uh the time here today. Next, I want to uh introduce my colleague uh Lucas Guyer from uh Google Cloud. He is an AI and ML uh cloud customer engineer who helps strategic partners uh get their agents published in the agent marketplace. Thank you all.

[00:16:47][Applause] >> Hello everyone and thank you so much Sean for the warm welcome on stage. I uh and it's really a privilege being here actually in my alma mater. Um I think uh last I I got my computer science degree from Columbia University and today I've been a Google cloud customer uh engineer and a ML specialist um for about the past four years and the biggest question that I get uh almost every single day is how do I build deploy and go to market with my AI agents and this you know has so many follow-up questions for example how do my AI agents work together. How do I distribute this across my entire enterprise workforce? How do I distribute this against uh across my entire customers enterprise workforce?

[00:17:40]Uh how do we control the costs? And they keep on going, right? And Google Cloud's answer to this is the AI agent marketplace.

[00:17:49]This is the proven path for our partners and our customers to build and procure their AI agents. Google Cloud Marketplace connects you with over 1,500 listings with a sub subset of them actually already A2A enabled. A2A is our agentto agent protocol which allows different agents to communicate with each other. This is also the foundation of our integration with Gemini Enterprise and is what facilitates that communication between them. And for customers, what this means is that you can purchase there. There are three main value ads for our customers. The first one is you can find and procure AI agents more efficiently. We have this tool called AI agent finder which allows you to search uh based off of your use case, industry, tool calls, whatever it may be uh and find the agent that's right for you. And then we make it very easy for you to procure that agent. We give you the pricing based off what the partner decides. and through a singleclick deploy you can put that into Gemini enterprise. The second is prevetted agents. All of our partners go through a prevetting process. Basically we uh make sure that they are following the A2A protocol. Uh we make sure that um you know it's a valid use case and we make sure that we do some like pen testing uh for security purposes. And the third is because you're deploying to Gemini Enterprise, you have this governance control. So a big problem that I hear from a lot of customers is, you know, these IT admins have requests for millions of different agents across their entire organization. I mean, imagine, you know, that you're an IT admin for uh thousands of of workers.

[00:19:36]Every single one will have their own preferences and ways to use them. Gemini Enterprise centralizes that and allows the IT admin to then assign specific agents to specific people or specific groups so that they can manage that governance.

[00:19:51]And for our partners that want to build agents, we make it super easy for them as well, right? So because Google Cloud has uh Google Cloud AI platform, which you can see in the big screen above me, um we give you everything that you need in order to build that AI agent.

[00:20:07]everything from the state-of-the-art enterprise uh uh large language models, Gemini, Anthropic, and over a hundred different others in our model garden to um an SDK so you can actually build the agent. We call this agent development kit. We've since open sourced this and um the third is a a fully managed agent runtime built on top of cloudr run. We call this agent engine. And so throughout all of this, we give you the flexibility and choice to kind of choose and pick what you want to uh build your agent with. And all of it is uh fully A2A enabled and and supports MTP tool calls as well. And so for our market, for our partners, there are four key value ads. The first is a simplified onboarding process because we centralized on the A2A protocol. You now can upload your agent card and we read uh you know the agent description, the what skills the agent has and the security schema and we just read the metadata and we make the listing for you. So that process is somewhat more automated. The second is a clear agent validation framework. I've already kind of talked about this but it's very we go through a validation step to make sure that these uh agents are vetted. And number three, uh there's a flexible monetization strategy. So based off of how you want to monetize these AI agents, whether it be a credit based system, a usagebased system, or u a custom pricing model, the it it really doesn't matter to us. Um we can support you through that through the agent marketplace. And the fourth thing is we have automated entitlement and billing so that every time a customer goes to procure an agent, you get a notification or your sales team gets a notification and you can follow up or do some upselling strategies depending on your business processes. And then there's also automated billing so that you can charge uh your customer automatically.

[00:22:09]And an excellent example of this is the Cardo site selector agent. I think we've shown a video of this already at the conference. Um, but this is an AI agent that Cardo built uh which helps you analyze and select different commercial sites through the use of advanced spatial analytics. This agent is fully built on the GCP AI platform also using Gemini and some of the cool techn geospatial technologies that we just covered.

[00:22:34]And uh what you see behind me is the image of what this looks like within Gemini Enterprise.

[00:22:41]And again this is uh this allows you to govern not just high code agents which is what we call cardo because it was built by developers but also allows you to uh build no code agents and then centralize and share access to those across your entire enterprise.

[00:22:58]So you I know this was a very quick uh quick presentation but I uh want to just say again uh from the agent marketplace uh we give you everything whether you are a customer looking to find and use different AI agents or you're a partner looking to build and distribute your AI agents. Um and I look forward to what you either build or uh use in the future. I want to thank you and enjoy the rest of the conference.

[00:23:26][Applause]