Have you ever wished you could bring your wildest ideas to life and blend them seamlessly with the real world? Well, Generative AI just brought us one step closer. Thanks to recent advancements in AI and a pinch of creativity, this dream is now becoming reality. In this article, our AR wonderboy Stijn Spanhove shares a few of his personal experiments that use multiple AI systems to augment the real world into a world where anything is possible. Even accordion-playing Shreks.

Paint your surroundings with Stable Diffusion

Stable Diffusion is a well-known text-to-image diffusion model that allows for the creation of photorealistic images from any text input. What sets it apart is its open-source nature, allowing others to build on its impressive capabilities. One such feature in Stable Diffusion is inpainting, which allows you to blend generated images with your surroundings in augmented reality.

By inputting a prompt, Stable Diffusion automatically blends in the generated image, and the SLAM engine provided by 8th Wall's WebAR framework anchors it to the real world, allowing for seamless integration as you move around. With Stable Diffusion, the possibilities of creating immersive, augmented reality experiences are endless. 

Create an AR scene using ChatGPT prompts

By now, we’re all quite familiar with ChatGPT. And as some of you will know, ChatGPT is also capable of generating code. And that opens up a whole new world of possibilities. In this experiment, we tried creating a completely new, out-of-the-blue, AR scenario by combining ChatGPT’s code with A-Frame, a framework used for building 3D/AR/VR experiences. 

With a little prompt training, the results are just staggering. Notice how it can understand context and references such as “put the item closer?” or “add a green sphere to the left?” This might look ‘basic’ at first sight. But imagine the possibilities of projecting a 3D model of about anything, anywhere you want, just by typing it in your phone. 

Transform the world around you

So far, we showed how to augment your surroundings and how you create an entirely new AR scenery. But what if we take it up a notch? In this example, we 3D scanned a statue to create a precise depth map. Next, we positioned the 3D scan in the real world using a VPS (visual positioning system). Finally, we wanted to alter our statue to our liking.

By prompting up some results in Stable Diffusion, we projected our generated results onto the 3D model, making it visible through our phone lens. A quick way to transform the world around you, easily generated by a few prompts.

And, honestly, who doesn’t like a good ol’ Shrek on your local plaza?

Fun experiments. True potential.

Too gimmicky for you? Yes, but actually no. If anything, you should try to look further than these fun adaptations we did on the verge of the real and augmented world. Generative AI entered our world only a few months ago and, already, people are coming up with creative and innovative applications. Applications that can support real business cases.

Just think about this: what’s the difference between that Shrek statue you just saw and a detailed, easily generated 3D model of your DIY idea, future house or even new business venue?