Build for AI Glasses from Meta by Oscar Falmer and Abhijit Srikanth from Meta

[00:00:07]Thank you everyone. Hi everyone. Thanks a lot for joining. So happy to see so many of you here, so many familiar faces.

[00:00:14]It's really great to meet all together this week at AWE.

[00:00:17]Even more in the coming everyone is coming. So thank you. So yeah, for those who don't know me yet, my name is Oscar Fadner. I'm a wearables developer advocate in the wearables partnership team at Meta supporting the growing community of developers and partners building on wearables. And today I'm very happy to present building for AI glasses from Meta.

[00:00:39]Today today I'll talk to you about the AI glasses available, what developers can build for them.

[00:00:45]I'll kick off first with the non-display ones and then the display glasses and then my colleague Abby will join to do a great demo. Before we dive into the glasses specifics, I wanted to remind everyone Meta's vision.

[00:01:01]So at Meta, we're creating a new computing paradigm that enables people to be more present, connected and empowered into the world. We deeply believe that this new type of computer is people first, specially and contextually aware with AI at its core.

[00:01:18]To realize this vision, we introduced new types of socially acceptable wearables, devices with novel input, output and AI technology all working seamlessly together.

[00:01:28]Our journey has already begun with our line of Ray-Ban Meta you're very familiar with Oakley Meta and continues today with our Meta Ray-Ban display and Meta neural band.

[00:01:39]Let me first tell you about the non-display ones with Ray-Ban Meta and Oakley Meta.

[00:01:45]So yeah, we have introduced actually over 40 different Ray-Ban Meta and Oakley Meta styles and colors to meet all the needs from consumers adapting to you know to your you can to to your style. With the latest generation, we even brought 3K video recording and expanded battery life as well as Oakley metal with a centered POV camera.

[00:02:08]I'll let you all take future pictures and >> [laughter] >> let me tell you about the the features.

[00:02:14]So there are as you know there's so many use cases you're probably familiar with you can capture moments listen to music podcast do you know voice calling share moments with your friend livestream what you're living hands-free do video calling and even use the multi-model AI capabilities we have especially the latest model also meta new spark which you may not heard of and has rolled out in the last months.

[00:02:41]Um I also want to highlight that now this meta AI glasses are available in 24 countries globally supporting 31 languages. So we're super excited to be leading the space with already millions of AI glasses sold across the world.

[00:03:01]And we deeply believe and that's why we're here today we deeply believe developers will play a huge role in this new AI glasses category.

[00:03:10]Our goal is to build a general purpose computing platform with apps and experiences from meta and our developer community. And now starting with a device access toolkit developers can build experiences today with you know our AI glasses.

[00:03:25]Um So the devices toolkit is basically our SDK for iOS and Android which enable to gain access to the AI glasses sensors and basically provide a dedicated hands-free experiences to your mobile app users.

[00:03:40]You have access to the camera with the ultra-wide 12 megapixel camera to the speaker microphone directly into your app and to your existing user base allowing you to tailor experiences you know, to the user environment hands-free.

[00:03:55]Uh you can stream uh videos. Uh you can uh record uh you can capture uh photos.

[00:04:01]And I want to just give a quick example with this kind of chess coaching app.

[00:04:05]So, let's say you had a an existing This is a thing on iOS.

[00:04:08]Yeah, you you have an existing uh mobile uh chess coaching app, and you're basically uh let's say you intend to add a hands-free experience with glasses. Um I just want to go a bit to the flow to the user flow. So, basically the way it works for the end user is that they're in your app. Uh they get a suggestion that they could use uh their uh Meta glasses. Um and you can basically it basically trigger the Meta AI app uh to allow the user to um confirm the permission to use the sensors. Um and then once it's accepted, it brings them back into your app.

[00:04:41]Um and now the user can basically start the coaching experiences uh experience uh using the device as a toolkit. You can now start a session uh which will start a uh video live stream. Uh and you can capture anytime you want a frame, send it to your app, run the model you want, basically provide the analysis you want or the best suggestion for the next move, and basically send it back as a as an audio feedback to the user. Uh without having needed You don't even need a display for that, right? You can just broadcast it directly to the glasses.

[00:05:10]Um Next, and in order to optimize the the the experience for the user, and also to optimize kind of the battery usage of the of the glasses, you can also pause or resume uh the session at any time.

[00:05:26]And once uh this is all done, uh basically the user can just end the session. Um and yeah, and basically that's it. They've been, you know, coming from your app from your mobile app with your users, uh they've had the like basically an experience with their Meta um uh glasses. Uh let me show you an example from the community with Stijn from Belgium that did um Actually, before I show it, uh because it's going to be a making some sound. I'm sorry.

[00:05:50]Uh, so basically it's a city tour uh, location-based experience. Um, its integration uh, captures uh, what's what it's seeing around the city and thanks to a pre-config- configured LLM uh, provides additional information uh, to the user via via speakers and also via the mobile app because you can still display information obviously on your mobile app. Let's Let's watch it.

[00:06:17]>> What's this building?

[00:06:18]>> Hello. Welcome to Ghent. I'm your AI city guide. How can I help you explore this beautiful city?

[00:06:25]>> Can you tell me something about this building?

[00:06:28]>> That striking building ahead is the Stadshal or city pavilion. It's a really dramatic contemporary structure with that undulating wooden roof. It sparked quite the controversy when it was built, but it's now an accepted [music] part of Ghent's landscape. I'm opening more information on the app now. So, check it out.

[00:06:46]>> What can you tell me about this place?

[00:06:49]>> Based on the visuals, you might be inside the Winter Circus. It's a remarkable historic building that was originally built in 1899 as a permanent winter circus venue and has since been beautifully restored.

[00:07:03]>> Hey, where am I now?

[00:07:06]>> It looks like you found the famous Graffiti Straatje, also known as Werregarenstraat. This narrow alleyway is a living, constantly evolving outdoor gallery of urban art. I'm opening more information on the app, so check it out.

[00:07:18]The artwork here is constantly changing as new artists add their creations over older pieces. It's a truly unique spot in Ghent.

[00:07:27]>> Yeah, so Stend did a great work uh, earlier this year and actually we recorded um, um, a video session coming on YouTube actually. Should be tomorrow. Uh, actually this is the first time I I mention it. So, we recorded a video uh, where he goes into details on how he made that happen um, and the specific technical details. So, check that out tomorrow on the I think Meta for Developers YouTube channel.

[00:07:49]Uh there's another example I want to show.

[00:07:52]Uh this one is with our partners 18Birdies uh providing real-time location-based information when you play golf as well.

[00:08:04]Hey 18Birdies, what's my distance to the pin?

[00:08:07]>> The pin is 127 yd. I'd recommend a gap wedge.

[00:08:11]>> [music] >> Guys, guys, let's get a photo. Hey 18Birdies, take a photo.

[00:08:32]Check it out. Oh, that's sick.

[00:08:39]Awesome. So, uh I want to next talk about what you've been waiting for. I see many people wearing them already in the room. Thank you for coming with your Meta Ray-Ban display. Woo.

[00:08:48]>> [laughter] >> Thank you.

[00:08:50]Uh let's talk about the Meta Ray-Ban display.

[00:08:53]Uh so, you know, the Meta Ray-Ban display, our first display-based glasses available to consumers.

[00:09:01]Um to remind you, you know, the specs, uh it's integrating a full-color, high-resolution 600 by 600 uh megapixel display into one lens on the right side uh which others cannot see, which is actually quite hard to do. So, you know, it's a very private experience um alongside with all the existing features you're familiar with.

[00:09:19]12-megapixel camera, microphone, speaker. Actually, I didn't mention earlier, but there's also a touch pad also uh the right side. Uh all of that into a compact, lightweight form factor you can wear like all day long. Um it even actually includes uh transition lenses uh which turn the you know, them into sunglasses uh when it's bright around you, adapting, you know, really well to indoor and outdoor environments.

[00:09:41]Actually, fun fact, I even had a friend that was using using them while skiing.

[00:09:46]You know, it's an extremely bright environment. She was like, "Oh, actually, you know, I can use them also pretty well while skiing." So, anyway, it's a working This This is very convenient. Um in order also to interact with those, you know, we the user can basically use this Meta neural band that I'm wearing right now.

[00:10:03]This is a lightweight EMG wristband with haptic feedback, which allows the user to to control the display interface with natural gestures, you know, such as tap and swipe in a very battery-efficient way because, as you know, it's not using actually the, you know, the camera to track with computer vision. It's just like basically getting the sensors here and providing the gestures to your glasses.

[00:10:25]Um and obviously, for them to fit best to everyone, there's there's a black one, but there's also the sand sand one, which looks which is great.

[00:10:32]Um and so, what can user do with them?

[00:10:34]First, you know, that basically allows to add different kind of visual informations, such as, you know, navigation, having some visual feedback from Meta AI with multimodal AI experiences, having live caption translation, and even doing messaging or video calling directly on your on your display.

[00:10:55]Um and for developers, that's, you know, I think what you've been waiting for.

[00:10:59]Obviously, there is still also the device access toolkit SDK, which connects to your iOS and Android apps, and basically can you you can use use this. And you have also the other way, which is web apps, which basically allows to build standalone, you know, web apps on your on your glasses, and it doesn't need to rely on any mobile app.

[00:11:19]Let me dive into a little bit specifics of both of those. So, for device access toolkit, you can access the sensors, as we discussed.

[00:11:27]And now you can obviously, yeah, already access show information to display. And in addition to that you can get uh the different gestures uh such as tap and swipe uh from the neural band.

[00:11:38]Uh current Yeah.

[00:11:40]I was going to Yeah, for web apps, sorry. I was going to say for web apps uh we also have, you know, this new path. Um and as mentioned, these are standalone apps which don't connect to any iOS or Android apps. They use HTML, CSS, JavaScript, and I uh square-sized apps which you can use on your glasses.

[00:11:59]Uh you can also access the IMU sensor uh which is very convenient. Uh the GPS uh you can have also GPS when it This There's not a GPS here on the glasses.

[00:12:07]It come from the phone, but it send uh directly to your glasses.

[00:12:10]Um and this is, you know, IMU is very convenient also like for example for fitness experiences.

[00:12:15]Um also microphone and camera are currently uh not accessible on this. We just uh launched that a month ago and it's still uh like device access toolkit uh currently in developer preview. So So ple- please come and build with us. So before uh before we have the great demo from Abby, um I want to show you a video of the, you know, some examples actually we built internally. Uh this was for the announcement. Uh internally we were super excited about it. All the teams were building very cool stuff and this is what we uh showed for the launch.

[00:12:54]>> [music] [music] >> Yeah, so and I want to highlight also uh um another project that's been We saw so many uh great projects from the community globally. Uh it's been extremely inspiring to us and uh you know, we're we're so delighted from that. I want to show actually viral also apps from staying that I showed uh like same same person I showed you earlier.

[00:13:34]He made this uh running app that got uh viral also as well uh where you know, just running around. And by the way, I've you know, spent a quite a while chatting with him. It was an actual run.

[00:13:44]It's just not for the video. He actually ran and he was actually telling me how that was actually a compelling and useful to him. Uh so, this was just more than a demo. It actually is working is working. And what he did on this video specifically uh is actually at a shadow a kind of ghost previous run that he's showing. Uh that's what you see that is already there. And he's basically running on that. And that basically allows him to follow the path.

[00:14:08]Sounds good. So, now that we have a good overview of developer tools, uh I would like to bring Abby on stage uh wearables partner engineer who is going to do a great live demo. Thank you, Abby.

[00:14:18]>> [applause] >> Perfect. Hey all. Um thanks, Oscar. Uh he kind of covered most of the parts of uh device access toolkit and uh how to build for web apps.

[00:14:31]Uh but here I'm here to kind of dig into how exactly to build for web apps in less than 4 minutes. And that's kind of a live demo that I'm doing. So, hopefully I have some blessings from the demo gods.

[00:14:45]Um All right. Uh and just checking I have some internet access as well. All right. Hey all. Uh Abby, partner engineer at uh Meta uh working on the wearables side mostly supporting the web apps stuff. And I'm super excited to uh you know, you know, you know, build out the web app support for all the developers that we have been hearing for almost a year now. So, All right.

[00:15:08]Uh Let's see here.

[00:15:15]Okay, perfect. Uh I will um I actually start by just pasting out the app idea into Codex and uh while it builds, I will possibly, you know, talk through it because we exactly have 4 minutes and that's my goal to build out a 4-minute web app and show you guys or prove to you guys that we can build out a web app to your glasses under 3-4 minutes.

[00:15:41]Okay?

[00:15:42]Um So, while while this runs, uh I basically opened Codex, uh fully permissions granted, and I let the AI do its job. Um starting the session now.

[00:15:57]Uh what what we are doing here is basically we are taking an we're taking AI-first approach, right, to build out a web app. Web app, again, it's not a completely new Meta SDK that we are launching. It's completely using W3C APIs. So, any web developers can actually build one out of the box.

[00:16:16]Uh So, to start, uh again, about the AI-first approach, we have built out this AI plugin that basically I have loaded in uh supporting Codex, Cursor, Cloud Code, or any of your AI coding agents.

[00:16:33]Uh and then as part of uh this whole AI-first approach, we have also open-sourced our AI plugin, if you can see here on GitHub, uh which you can just possibly just run a couple of commands to install all your AI skills onto Cloud Code or Cursor, again, any of your favorite coding agents.

[00:16:57]It's basically a set of skills that talk about all the APIs that we support, all the capabilities that we support, um the kind of display guidelines that you have to build for, which is basically for like a 600 by 600 viewport, uh the D-pad input, which is basically top-down, left-right, and and just a few other uh um guidelines that you follow, which is all part of the skill. Again, you don't have to worry about that. You just tell your coding agent to build one for you.

[00:17:25]So, back at what we are building as part of this demo itself, right?

[00:17:30]Uh I'm actually building out a simple web app taking advantage of all the capabilities that we are showing uh or as part of the glasses itself to build out a simple step counter app, which is basically taking advantage of the accelerometer, gyroscope, uh all the IMU sensors to count the steps while you walk, kind of like a heads-up display.

[00:17:51]And um with that, also using the magnetometer to show kind of the direction in which you're walking, the north, south, east, west, and and the distance and the speed using the GPS, which is coming from your phone.

[00:18:04]So, we'll see uh if uh we're anywhere close to building one.

[00:18:13]Not yet, I guess.

[00:18:14]But yeah, anyway, uh we have our whole documentation for build, setup, test, and you know, trouble to troubleshooting. And we also just announced the web apps MCP, which is kind of talking or directing you exactly to the documentation. And we have also seen developers actually build out already web apps not just through the AI coding agents, but also using the AI wipe coding platforms, the online ones, uh by just directing the the prompt with this docs uh URL or the GitHub URL and telling telling AI to build one for you.

[00:18:51]Right, so uh all right, we're we're close.

[00:18:56]We're close. All right. Uh so, the one last check that's doing is uh I'll be running this uh whole web app on my Chrome browser for now using a Chrome extension that we just built out. By the way, this is like super early preview, which uh I'm going to show.

[00:19:12]Um well, the port is blocked, so I'll be using the 8 802, I guess.

[00:19:19]Um Perfect. It's already up and running.

[00:19:24]So, this is the Chrome extension.

[00:19:25]Without the Chrome extension, this is exactly how it's going to look, which is basically the 600 by 600 viewport. Um but with the extension, you're kind of simulating how exactly it looks on the display glasses. Right? And uh one of the limitations right now is I can't cast exactly what I'm seeing from my glasses to my laptop. So, I've actually built out this app.

[00:19:46]Uh I I built out this app a similar app 30 minutes back and recorded the whole thing using this new feature that we just announced for the display recording, right? So, I was just walking across the hall to record this whole thing just to prove to you that this app works. Um and I actually walked almost 0.1 mile or 0.2 miles, I guess, uh just to record this. So, we have the direction working, the speed, and the counts the steps count, all of that working.

[00:20:15]Uh yeah, that that's mostly it. I think I took almost 4 minutes uh 5 minutes to build out the web app. Uh um and yeah, that's mostly it. And uh >> [applause] >> that's Before I give it back to Oscar, yeah, before I give it back to Oscar, I just want to say that I have actually built out a web app for what I'm talking right now as a speaker note, which is on my display glasses, and I'm actually just using my neural band to go left and right. So, if you want to check it out, you can meet me after the call.

[00:20:49]>> Thank you.

[00:20:49]>> ALL RIGHT.

[00:20:51]>> [applause] >> HERE WE GO. THANK YOU SO MUCH, ABBY. AND YEAH, ABBY UH was actually working on that uh this morning. Uh we were wrapping it up, so I thank you. And yeah, it's just that easy. Uh I think these skills files make it really seamless. Uh you don't have to be a developer. Actually, you know, we have a lot of power users that are just building, you know, what you know, they use like an app for what they need. So, I recommend you trying it out. And if you if you do, please share online or reach out because we're super happy to to discover them.

[00:21:30]Uh before we wrap up, I wanted to share a couple um resources that we have. So, um as you mentioned, we have great documentation for the device assist tool kit as much as web apps on our GitHub with the SDKs, the sample codes. You also have these skills files for your AI coding agents.

[00:21:49]And we also about a month ago launched a Reddit community at Meta Glasses for Dev. So, please if you build anything great, please share it there. That's the best way to kind of surface it to everyone in the community and also everyone at at Meta.

[00:22:03]Um you can find all these resources and more at the official Meta Wearables developer center.

[00:22:09]And thank you for joining. Thank you.

[00:22:11]Thank you so much.

[00:22:14]>> [applause] >> I I I also want to highlight that in 5 minutes there is Dilemme with a VR developer advocate doing a great session in the room nearby. So, if you're interested, feel free to join.

[00:22:26]And if you have any feedback, also you can scan and said and say you had a great time.

[00:22:31]>> [laughter] >> Thank you. Thank you everyone.

[00:22:36]>> [music]