Reimagining a 50-year-old interface (the mouse pointer) with AI

Added: 2026-05-17

[00:00:00]Pointing is really at the core of a lot of the interactions we have when we collaborate with other people.

[00:00:06]For more than half a century, the mouse pointer has been the one constant across every website, digital documents, and workflow we use.

[00:00:15]What if behind the pointer there was an AI model like Gemini actually listening to us, paying attention to the screen, and trying [music] to interpret whatever we're saying like another person would?

[00:00:30]I'm Adrian. I am a researcher at Google DeepMind. My job involves doing a lot of prototyping, a lot of experiment with users, and really trying to understand people [music] and how to create systems that actually satisfy their needs.

[00:00:44]The focus of this research project is an experimental [music] AI-enabled pointer with the ability to understand not only what you're pointing at, but also why it matters to you and how [music] to act upon it.

[00:00:56]Our first focus was really how can we build a system that can really understand fluid user intent?

[00:01:02]Could you get these two ingredients and also this one and add them to my shopping list here?

[00:01:10]Done. The way we actually made this work in our initial prototype was by saying keywords like this, that, here, or there. If I hover on the note the AI-enabled pointer knows [music] the data that's behind the scene. Make this orange.

[00:01:28]By typing the word this it added this actual text note to the prompt. [music] We can really have the pointer dig through all of the layers of data. We can have voice, we can have text, we can have image understanding.

[00:01:41]Can you make this 8:00 p.m.? I've updated the draft to start at 8:00 p.m.

[00:01:46]And then have Gemini write code to satisfy the user intent whenever they're moving the pointer across different apps. Can you show me how to go from this place to that place? Here are the directions between the two locations.

[00:01:59]All of those windows are going to be communicating with the pointer, creating the prompt on the fly. I am using head tracking here. Hey Gemini. So, can you generate an image based on this whole menu here?

[00:02:12]I would like you to use the style in this image.

[00:02:15]Okay, I'm generating the image now.

[00:02:17]Beautiful. Gemini transferred the content of the menu here, as well as the style from the bird, >> [music] >> into the new image. It's really magical what you can do when we mix voice and pointing and visual understanding at the same time.

[00:02:32]I imagine a new type of operating system, AI showing me content I might find useful, me pointing back at the content, sharing attention, [music] and sharing the canvas like if I was working with another person.