Google's New AI: From Phones to the Real World

Voice-activated assistants that allow hands-free phone use have become commonplace. Google's foray into this technology culminated in the 2019 release of the new Google Assistant for Pixel 4. This technology promised users instant, hands-free access to their phones, including multitasking across apps. However, the reality fell short of the hype. Users had to use specific phrases instead of speaking naturally, and the assistant was only compatible with a limited number of apps.

While other tech companies are exploring how large language models (LLMs) might power voice assistants capable of using any app, Google's focus has shifted. Instead of primarily focusing on improving phone-based AI assistants, the company is prioritizing AI that can help users in the real world.

This new direction is rooted in the understanding that many questions and problems arise in physical contexts without digital equivalents. Google's answer is Project Astra, an AI assistant that allows users to point their phone (or, in the future, smart glasses) at objects in the real world to receive information or assistance.

Project Astra will be incorporated into Gemini Live, an interactive experience that enables natural, two-way conversations with AI. The voice aspect of Gemini Live is anticipated to launch this year, followed by camera capabilities.

This focus on real-world applications also extends to personal data. The Gemini-powered Ask Photos feature transforms users' photo libraries into a searchable database of real-world knowledge. For example, users can take a picture of information and have Google organize it for them, such as creating calendar entries from a child's syllabus or generating shopping lists from a recipe photo.

Google's shift from phone-based AI to real-world AI is ambitious. While a revamped phone assistant might appeal to some, Google's pursuit of an AI that can help users navigate the complexities of the physical world is a bold and potentially transformative endeavor. Project Astra's goal of being a "universal AI agent that can be truly helpful in everyday life" speaks to this ambition. By providing AI with a live view of the world through camera input, Google aims to tackle a fundamental challenge: bridging the gap between real-world contexts and digital queries.