When the Apple Vision Pro was released early last year, it was perceived as a groundbreaking leap into the future of extended reality (XR). For weeks, it dominated conversations and captured the imagination of tech enthusiasts and developers alike. But as quickly as the buzz started, it faded. The world seemed to move on, shifting its attention back to the rapid advances in AI.

Yet, for those who saw the Vision Pro not as a polished consumer product but as a high-end developer kit, this quiet period felt inevitable. The hardware was indeed revolutionary, but the tools needed to harness its full potential were still evolving. One critical limitation was the lack of direct access to the device’s main camera—a necessary feature for unlocking AI-powered visual understanding in immersive apps.

Now, with Apple enabling camera access (at least for business apps) and Meta planning to activate it early this year for Quest, a new era of possibilities is unfolding. These updates promise to empower developers to craft cutting-edge mixed reality (MR) experiences that deeply integrate AI’s ability to perceive and interact with the real world. The question is: How will the combination of XR and AI launch a new wave of innovation?

Getting down to business

See it, fix it

Imagine a factory technician walking the shop floor and noticing that something seems off with a particular machine. In those cases where the machine lacks a built-in display for its status, the technician would typically have to open an app on a phone or tablet and manually search for the issue. But with an MR headset, this process becomes much more intuitive.

Using computer vision, the headset scans the machine in real-time, revealing wear, misalignments, or potential failures. Problematic components are immediately highlighted in the technician’s field of view, eliminating the need for guesswork or looking at a touchscreen.

Once an issue is detected, the system can provide step-by-step visual guidance to resolve it. For instance, it can overlay instructions for replacing a faulty part, ensuring precision and minimising downtime. Additionally, AI can suggest preventive measures, such as scheduling maintenance for components prone to frequent failure.

Virtual rehearsals, real-world impact

Training employees for complex or high-stakes tasks is often resource-intensive, but a combination of XR and AI can revolutionise this process. In healthcare, for instance, medical trainees can rehearse delicate surgical procedures in a virtual operating room, with AI providing real-time guidance to refine their techniques and ensure precision. This immersive approach allows for mastery of skills without the risks of practising on actual patients.

In the retail sector, XR and AI can help sales personnel prepare for challenging customer interactions. Trainees can engage in simulated scenarios where AI-powered virtual customers exhibit a range of emotions and behaviours, from confusion to frustration. AI can then also be used to evaluate the trainee’s responses, offering feedback on tone, body language, and problem-solving strategies. These simulations can help develop the empathy, communication skills, and composure needed to deal with high-pressure situations.

Creating better CX

While the above examples primarily benefit B2B applications, these advancements can also unlock exciting new possibilities for everyday activities, directly benefiting consumers:

Meal magic

Imagine stepping into your kitchen, scanning the contents of your fridge, and having an AI instantly suggest recipes based on the available ingredients. This AI-powered recipe app, leveraging the mixed-reality headset camera and AI algorithms, can analyse your ingredients and propose meals.

Once a recipe is chosen, the app guides you through the cooking process step-by-step. Ingredients can be highlighted in your field of view, ensuring you pick the right items and quantities. Virtual timers can hover above your pots and pans, alerting you when it's time to stir or check on your dish. Additionally, the AI can provide real-time tips, making cooking both educational and enjoyable.

While you can already use augmented reality (AR) on your phone today to achieve this, the whole experience is underwhelming and not very convenient. With a mixed-reality headset, the experience becomes hands-free and frictionless. You can also add virtual screens around you, maybe even watch a movie or follow along with a cooking show, making the whole process not just efficient, but also entertaining.

Try it, buy it

Shopping for clothes, too, is being revolutionised by XR and AI. Users can now snap a photo of their outfit and experience virtual try-ons with a 3D model of themselves. While Meta's Codec Avatars and Apple Vision Pro Personas currently lack full-body modelling, future updates are likely to change that. Alternatively, advanced AI can generate a 3D model of a person from just a single front-facing image, opening up even more possibilities.

This virtual try-on experience lets users see how different outfits look and fit, making it easier to make confident purchasing decisions. With AI-powered style suggestions, users can discover fashionable ensembles by combining pieces seamlessly. It can even recommend outfits tailored to their existing wardrobe, ensuring new purchases perfectly complement what they already own.

A “see” of possibilities

Apple has opened access to the main camera on the Vision Pro through its Enterprise APIs, though this feature is currently restricted to business apps. Meanwhile, Meta has announced that its Passthrough API, granting similar camera access to their Quest 3 line-up, will roll out early this year. Together, these updates will mean that developers will finally have the tools to harness computer vision for real-world applications on the industry’s leading platforms.

Though the potential is phenomenal, there are still some limitations too, like the bulky form factor of mixed-reality headsets. In their current form, these devices are not likely to become part of everyday outdoor life. Users aren’t going to walk down the street or commute to work wearing MR headsets, if only to avoid being called “glassholes” (remember those?). Instead, their adoption will likely be confined to homes and workplaces.

As access to XR hardware capabilities like main cameras becomes mainstream this year, the possibilities for AI-powered applications will be vast. Yet, developers must remain grounded in the realities of how and where these devices will be used, focusing on practical and impactful solutions within these constrained spaces.

Future reality

Head-mounted displays like the Meta Quest and Apple Vision Pro excel at high-precision, intensive tasks, particularly in B2B settings. Above all, the combined future of XR and AI lies in their seamless integration into everyday life, driven by more accessible and user-friendly devices.

In fact, companies like Meta and Snapchat are already showcasing easy-to-use interfaces such as the Meta Orion and Snapchat Spectacles. These devices resemble regular glasses and are designed for everyday tasks.

Meta's Orion glasses stand out as impressive for several reasons. Unlike bulkier headsets that require setup, the Orion glasses are designed for simplicity and ease of use. They can be easily put on and taken off, making them highly convenient for everyday wear. Additionally, the form factor is more stylish, encouraging users to discreetly wear them in public.

It’s important to note that the Orion glasses are currently not available yet to the general public due to high manufacturing costs, but their concept is already influencing the market. Developers have access to similar devices, such as the Snapchat Spectacles, which, although slightly larger and with a narrower field of view (FOV), provide a glimpse into the true potential of these types of glasses.

The combination of AI and XR will make them powerful tools for everyday activities. The popularity of Meta's Ray-Ban glasses, despite lacking XR elements, underscores the importance of form factor and shows that consumers value comfort, simplicity and discretion.

As XR capabilities are integrated into these sleek designs, they will become indispensable tools in daily life. Future wearables will transform our approach to app development, demanding a shift towards spatial thinking. By leveraging both AI and XR, we can gain deeper insights into our surroundings and create more context-aware experiences. For instance, applications can adapt to people's emotions and provide personalised interactions based on their mood. They can also adjust to different environments, enhancing the user experience whether indoors or outdoors. These are just a few examples, but they give a glimpse into the possibilities and how these adaptive experiences will enrich our everyday life.

Reimagining collaboration

Remote collaboration has become the new normal, yet current solutions often lack the sense of presence and connection that in-person meetings provide. Seeing colleagues as rectangles on a screen feels impersonal and can hinder engagement. XR and AI have the potential to redefine how we work together by enabling immersive, avatar-driven interactions that bring remote participants into shared spaces.

In this envisioned future, colleagues use MR headsets or AR glasses to join meetings as life-like 3D avatars. These avatars, animated with the help of AI, replicate subtle facial expressions, hand gestures, and even gaze direction, creating a sense of authenticity and presence.

While this technology isn’t fully realised yet, several companies are making significant strides. Meta has already launched Horizon Workrooms, allowing teams to collaborate in virtual spaces, albeit with relatively basic avatars. They’re also developing codec avatars, highly realistic digital representations that replicate subtle facial expressions for a more immersive telepresence experience. However, this technology remains in development and is not yet publicly available. But that does not mean that we ought to underestimate their potential, which promises to be huge.

Apple currently leads the field with the most advanced avatars, which have been introduced with the Vision Pro. Users scan themselves with the device to create realistic avatars that can be used in apps like FaceTime and Zoom. These avatars display emotions and feature lip-syncing that responds naturally to speech. While Apple hasn’t yet introduced a dedicated platform for immersive collaboration, their avatar technology provides a strong foundation for such tools, whether developed in-house or by third parties.

Access to passthrough cameras can take hybrid meetings to the next level by seamlessly blending physical and virtual spaces. For instance, a meeting host could share their real-world environment, captured in real-time by the headset’s cameras, creating a virtual version of the physical room. Avatars of remote participants could then join this virtual space, not just as passive observers but as active collaborators who can interact with the shared environment.

For example, if the host writes on a physical whiteboard, the passthrough feed ensures everyone sees the updates in real-time, regardless of their location. Similarly, remote participants could highlight or annotate objects within the virtual version of the room, enabling a level of interaction that bridges the gap between physical and virtual presence.

Risk & innovation

Enabling AI and access to camera images also raises significant privacy, ethical and environmental challenges and concerns. 

If apps can send images to AI systems over the internet, it is crucial to establish clear guidelines and permissions to ensure that users are informed about how their data will be used. Smart glasses, lacking the computational power to run AI models on the edge, could offload processing to a connected phone. This approach keeps data local, enhancing privacy and security.

On top of that, the constant monitoring of our environment through smart glasses could lead to a surveillance state, making it essential to protect user privacy with clear guidelines and robust encryption. Additionally, the energy consumption associated with processing large amounts of data, whether on the Cloud or locally, will increase a company’s carbon footprint, which is pretty controversial in these times of climate change.

Moreover, constantly monitoring and recording our surroundings raises ethical questions about consent, especially in public spaces. It is important to develop ethical frameworks that respect the privacy and rights of individuals.

While the integration of AI and camera technology offers exciting possibilities, it is also vital to think about these risks and provide clear guidelines and a good ethical framework. This, however, should not stop us from innovating.

A true power couple

XR and AI are an up-and-coming power couple in the tech world, each amplifying the other's strengths. XR creates immersive environments that bridge the gap between the physical and virtual worlds, while AI provides the intelligence to understand, adapt, and enhance these experiences. Together, they offer immense potential to revolutionise industries.

As these technologies evolve, the companies that invest in exploring their synergy will lead the next wave of innovation. By integrating XR's immersive potential with AI's cognitive capabilities, businesses can not only address current challenges but also shape a future where technology becomes more personal, connected, and transformative. In fact, just as we completed this article, Google announced its new AndroidXR and essentially positioned it as an interface for its Gemini AI. Their move signals a major shift in the tech landscape and hints at what lies ahead.

‍