Bandung, Indonesia Sentinel — Google is rolling out a significant update to its visual search app, Google Lens now enabling users to ask near-real-time questions about something while capturing the object of interest through video. According to TechCrunch, the upgrade enhances the app’s functionality by allowing users to record video through Lens and query objects of interest in the footage. It is available to English-speaking Android and iOS users with the Google app installed.
Lou Wang, Director of Product Management for Google Lens, explained that the new feature leverages a specialized version of Google’s Gemini AI model, which is responsible for analyzing video content and generating answers to user questions. Gemini, Google’s advanced family of AI models, powers a variety of products across the company’s ecosystem.
“For example, if you’re curious about a group of fish swimming in circles, Lens can provide an explanation for the behavior along with additional resources for further learning,” Wang said during a press briefing.
Google Lens employs AI to identify key frames in the video, focusing on the most “interesting” and relevant moments in relation to the user’s question, Wang noted. These frames are then used to ground the AI-generated response from AI Overviews. AI Overviews is a Google Search’s AI-powered tool that summarizes information from the web.
To access this video analysis feature, users need to join Google’s Search Labs program and opt in to the “AI Overviews and more” experimental features. Once activated, users can hold down the shutter button in the Google app to enter Lens’ video-capturing mode, ask questions while recording, and receive answers provided by AI Overviews.
“This feature came from observing how people are currently interacting with Lens,” Wang said. “When you make it easier for users to ask questions and satisfy their curiosity, they tend to adopt these tools more naturally.”
Kajang Tribe, Guardians of Indonesia Rainforest For hundreds of years
The introduction of video capabilities for Google Lens follows Meta’s recent preview of a similar feature for its Ray-Ban Meta AR glasses. Meta bring real-time AI video capabilities to the glasses, letting wearers ask questions about their surrounding. OpenAI has also hinted at plans for its Advanced Voice Mode tool to analyze videos in real time and respond with context-aware answers.
However, Google appears to have taken the lead in this race, though with some limitations. Unlike real-time interaction, Lens operates asynchronously, meaning users can’t chat with it as they would in live conversations. It’s also important to note that during the press briefing, Google did not offer a live demonstration, and the company has a history of overpromising on the capabilities of its AI.
Google Lens Images Search Optimization
In addition to video analysis, Google Lens can now perform combined searches with images and text. Users can hold down the shutter button to take a photo, ask a question verbally, and receive AI-generated responses, a feature available to all English-speaking users, even those not enrolled in Labs.
Lens is also receiving new e-commerce capabilities. Starting today, when Lens recognizes a product in an image, it will display information such as price, deals, brand, reviews, and stock availability. This product identification feature works on both newly snapped and uploaded photos but is limited to select countries and categories, including electronics, toys, and beauty items.
“For instance, if you spot a backpack you like, Lens can identify it and provide details like pricing and reviews instantly,” Wang said.
Google is also integrating an advertising component into this feature. The product results page will include relevant shopping ads, offering users additional options and prices for the items they’re interested in.
(Raidi/Agung)
