When Apple launched the Vision Pro, many expected it to immediately revolutionise spatial computing. The reality has been more nuanced. The initial buzz has quieted, and mainstream adoption has been slower than some anticipated. Yet, beneath the surface, significant advancements continue to reshape what's possible with this technology.

We remain optimistic about Apple's vision (pun intended) for the future. The platform isn't just about what it can do today, but what it's evolving to become. The features we're exploring below represent not just gradual improvements, but foundational capabilities that unlock entirely new categories of applications. Let's examine these breakthroughs that have us excited about the Vision Pro's future.

Real-time AI Through Camera Access

As the camera access capability is now available on the Apple Vision Pro via enterprise APIs, a new realm of hands-free, real-time AI experiences becomes possible. In our proof-of-concept (PoC), we integrated Gemini 2.0 Flash to create an interactive AI assistant that responds instantly to your environment, guiding users with real-time insights through AR highlights, all while keeping their hands completely free.

Bridging Realities: AI-powered Augmented Assistance

Traditional AI assistance relies on text or voice input, creating a fundamental disconnect during hands-on tasks. By leveraging Apple Vision Pro's camera access and Gemini Flash's multi-modal processing, we've bridged this gap, enabling AI to deliver real-time, contextual guidance in an immersive AR environment where it matters most.

How it works

The system detects objects through AI-generated bounding boxes in 2D, then maps these to 3D space. By utilising the Vision Pro's environmental scanning capabilities, we precisely position highlights and visual instructions in the user's physical space, creating a seamless blend of digital assistance and real-world interaction.

Transforming workflows through spatial AI

This hands-free AI approach in AR dramatically boosts efficiency while reducing cognitive load. Users receive contextual insights exactly when needed without breaking their workflow to give input. As both the Vision Pro and AI models continue to evolve, this foundation will enable increasingly sophisticated assistance across industries—from medical procedures to complex manufacturing tasks.

Redefining Interaction Through Object Tracking

In this proof-of-concept, we integrated object tracking with an IoT device. Using a Philips Hue lamp as our test case, we've created an intuitive control system that combines precise eye tracking with gesture recognition. By leveraging eye tracking, users can simply look at the lamp and toggle it on or off with a pinch gesture. An AR overlay also allows users to adjust the lamp's colour effortlessly, keeping an eye on their smart home devices in an entirely new way.

Training and tracking objects

Using an iPhone's LiDAR and image capture, objects are scanned and trained via Apple's Create ML. This process generates a file that enables precise tracking on Apple Vision Pro. Built natively with Swift and ARKit, the system delivers robust tracking, particularly for stationary objects.

Revolutionising environmental control

This fusion of object tracking, spatial computing, and IoT connectivity transforms how we interact with our environment. Beyond smart home applications, this capability could revolutionise industrial settings where operators need to monitor and control multiple systems simultaneously. The ability to simply look at a device and control it through intuitive gestures eliminates friction in ways that were previously only imagined in science fiction.

Accelerating Intelligence Through Edge AI Processing

What if AI could run directly on the Apple Vision Pro without relying on the cloud? By processing everything on-device, latency is reduced, enabling real-time interactions with enhanced speed and privacy. This approach puts intelligence in sight rather than in the cloud.

‍

In this proof-of-concept, a simple heart gesture triggers the device to capture a picture. A machine learning model, MobileNet V2 in this case, then processes the image locally, detecting and identifying objects without any external processing.

Advancing on-device capabilities

On-device AI enables applications that simply weren't viable when dependent on network conditions. From instant language translation of physical documents to real-time accessibility features for those with disabilities, Edge AI processing opens doors to experiences that feel truly magical. Perhaps most importantly, keeping sensitive data on-device addresses growing privacy concerns that have surrounded cloud-based AI solutions.

Looking ahead, we envision integrating lightweight LLMs that could generate natural language descriptions, allowing the system to verbally communicate what it sees to the user in conversational speech. While current technical constraints limit multi-modal LLM implementation on-device, text-based LLMs could soon enhance these experiences with more natural interactions.

Shaping the Next Era of Spatial Computing

The conversation surrounding the Apple Vision Pro has matured from initial hype to a thoughtful exploration of practical applications. With camera access, object tracking, and edge AI processing now available, organisations face an imperative to leverage these capabilities for competitive advantage—whether through operational efficiencies, enhanced customer experiences, or novel service offerings.

What's particularly exciting is the vast uncharted territory ahead. Many sectors have yet to discover how XR and AI integration can transform their operations. This is precisely where collaboration creates exceptional value—when businesses with compelling challenges partner with specialists who can envision innovative solutions. The most transformative applications often emerge through this joint exploration, connecting emerging technologies with concrete business outcomes.

The journey of spatial computing is a marathon, not a sprint. With these foundational building blocks now in place, we remain confident that technologies like the Apple Vision Pro will redefine our relationship with computing. For organisations ready to explore these possibilities, our combination of XR expertise, AI capabilities, and strategic insight makes us an ideal partner to guide your journey—identifying the unexplored use cases that will deliver genuine value in your specific context.

‍