A new experiment from Google DeepMind could change how users interact with AI assistants by turning the mouse pointer into a real-time input tool for Gemini. The system is designed to help AI understand what users are pointing at on their screens, making interactions faster and more natural.
DeepMind’s experimental “AI pointer” uses cursor position and hover activity as context signals instead of relying only on typed prompts. The idea is to reduce the need for users to explain tasks in long sentences or manually copy and paste information into chat windows.
For example, a user could hover over a data table and simply say “make a pie chart,” allowing the AI model to understand the request based on the screen area being highlighted. The system can also support tasks such as highlighting a recipe and asking to “double these ingredients,” pausing a video frame to generate a restaurant booking link, or turning a handwritten note into an interactive to-do list.
According to DeepMind, the technology is aimed at removing friction in AI-assisted work by reducing the need for highly detailed instructions. The company believes cursor position and screen region could become important inputs alongside text and voice in future AI workflows.
The AI pointer could also benefit developers and teams building AI-native software tools. Researchers say the technology helps bridge the gap between what users see on their screens and what the AI model understands, without forcing users to manually assemble context.
DeepMind noted that the demonstrations are still experimental and should not be viewed as confirmed product features. The company has not announced a release date, a commercial product, or the Gemini model version powering the system.
Researchers said the availability of AI Studio may become the first indicator for developers looking to understand how the feature could eventually be implemented in AI applications and workflows.
Also read: Viksit Workforce for a Viksit Bharat
Do Follow: The Mainstream LinkedIn | The Mainstream Facebook | The Mainstream Youtube | The Mainstream Twitter
About us:
The Mainstream is a premier platform delivering the latest updates and informed perspectives across the technology business and cyber landscape. Built on research-driven, thought leadership and original intellectual property, The Mainstream also curates summits & conferences that convene decision makers to explore how technology reshapes industries and leadership. With a growing presence in India and globally across the Middle East, Africa, ASEAN, the USA, the UK and Australia, The Mainstream carries a vision to bring the latest happenings and insights to 8.2 billion people and to place technology at the centre of conversation for leaders navigating the future.
