When our Augmented Reality developers are not busy creating applications for companies such as Telenet and Woody, they like to read interesting blog posts about their research field. These essays are often written by people like Robert Scoble, the inventor of the word "Virtual Reality," or Matt Miesnieks. The latter has been in AR for more than 15 years and worked at large tech companies like Samsung and Layar before he founded his own company named 6D.ai.

We like to play with his platform - which enables developers to create AR apps that interact with reality - as it really offers some powerful new capabilities for Augmented Reality, like realtime meshing, occlusion and visual localization - all on consumer hardware, which is very impressive.

We recently had the honor to interview him - an opportunity that we didn't just let pass. Someone with more than a decade of experience has obviously seen the AR landscape change dramatically in recent years. Or not?

"Well, not entirely," Matt says. A lot of the use cases of AR are pretty obvious - the same use cases have been around for ten years. Think about applications in gaming and entertainment or industrial automation and remote field assistance. It’s the user experience that always has been broken. There always has been an aspect that was a blocker, whether it’s the hardware, the content that was floating, impractical markers…’

The AR Cloud

Matt soon realized that it was important to make the content engaging. AR must feel like it is part of the real world, but that’s easier said than done. ‘What if you want to put something in AR in your living room, but you want it to go behind your couch? You’ll need a digital model of that couch to include the digital content. It sounds simple, but getting a digital model of everyone’s living room is really, really a challenge. Eventually we realized you need a 3D model of the entire world.’

That software is bigger than you can run on a smartphone. In addition, it is also important that the content persists after your device is turned off, that someone else can see it on his device or that you can share it. ‘When you realize that all of these capabilities need to exist outside and independent of anyone or one device, you need to host it in the cloud’, says Matt.

‘That’s how we came up with the term ‘the AR Cloud’. Not really in the sense that your content lives in the cloud the way your website is in the cloud, but more in the matter that these enabling technologies were fundamental to AR being able to work at all. And this 3D model of the world that all this content is going to depend on, is going to be an incredibly valuable asset that every AR app is going to tap into.’

‍

Software like this, that runs fast and real-time is exactly the tech breakthrough that 6D achieved, thanks to the crowdsourcing of the models. ‘Our short term objective is mapping the use cases we can solve today - and that’s why we’re working so much with enterprises. They mostly focus on use cases that work with handheld form factors. A lot of it is around capturing spaces in 3D, building digital twins, connecting them to live data feeds, overlaying that on a model... The AR view with the overlaying content is almost secondary to that really capturing the 3D data.’

AR & AI: match made in heaven

As the name of Matt’s company implies, AI is playing a big role as the enabling technology for what he’s building at 6D.ai. ‘Everything we do today depends on AI and neural networks, but it’s not visible. As the maps of the world we’re creating get into more and more apps, the layers above the geometry of the world, become more and more based on semantic understanding of the world.’

One of the things he mentions, is the segmentation of objects. How do you know if a blob of the mesh is a different thing to another blob of the mesh, for instance? ‘In order to do that, you have to identify what those things are. This is a chair, this is a couch or a car… Next step is tracking moving objects and separating them out from the mesh. Eventually you get stuck to predicting behaviour. You see a parked car, and already take into account that it is likely to move, so your app has to behave in a different way than if it’s a tree.’

In order for AR to take off in the short term, Matt sees three driving forces. ‘You need viable technology, you need a use case that someone will use and a big enough market of that use case to be commercially viable.’

Even when these three conditions are met, there is still work ahead. ‘There’s still a lack of understanding from these fuzzy concepts of which everyone think it’s the future. People will try the HoloLens or ARKit and they’re like “Well, that’s today, but how do these things connect?” How do you show that? Not by writing Medium posts but by saying ‘Try this’ and then they get it.

Although his company is currently focussing on enterprise applications, Matt thinks the biggest opportunity is in the consumer market - but he doesn’t think the consumer is ready today. ‘That’s why we’re focussing on specific use cases in enterprise. The first one is what we’re calling inspections. Think about big home sharing companies that want to have 3D models of all their properties in order to run their models through machine learning algorithms to get floor plans, checklists of facilities and that stuff. It’s cost-saving and makes them able to scale up and automate. Another opportunity are the digital twins. Big factories don’t want to spend 15.000 euros on a Lidar scanner and train people to use it and send it from site to site every couple of months. They want to be able to send people who wear a headset or wave their phone around to get a constantly refreshed model of that site, whether it’s under construction or actively in use.

This is an example of what 6D.ai does: making a digital twin of a space (in this case: our reception desk) by scanning it with your smartphone.

The BlackBerry era

6D.ai isn’t Matt’s first company. Way back in 2010, he was running a startup named Dekko. ‘Dekko was one of those companies that build the right stuff but just too early’, he says. ‘A couple of mayor things have changed since that time. It sounds almost trivial, but ARKit and ARCore were massive enablers. Most people don’t realize how difficult it was or it is to build a consumer grade 6DoF tracking system. There’s a lot of software and hardware needed. For a long time they were two separate worlds, but once you’ve got a good 6DoF tracker everything above that is software. That’s a massive change in terms of technology enablers. The next big revolution is that neural networks in 3D computer vision have become insanely capable.’

In contrary to the early days we now have technology that makes things possible, and all the noses are in the same direction, but now people are wondering what the next steps are between today and the promised land - and how long it’s going to take to get there. According to Matt, there’s one big player that needs to get involved: Apple.

‘I always say that the one thing that matters is Apple launching a pair of glasses, but even when they’ll do that it will take some years until we have tens of millions of these in the market. There’s still some work to get semantic algorithms down on the phone and figuring out what needs to run online and offline, but mostly the science is solved there. Next to that there are still some though problems around GUI and user interface that need to be solved on the device - whatever it will be - but from what I’m hearing there are strong capable teams that know what they’re doing in all of the mayor platform companies. When you compare it to the rise of smartphones, it feels like we’re at the Symbian and BlackBerry era. They exist, they functionally work, the user experience is still clunky, businesses are getting value out of it, but it’s still a tiny segment compared to the consumer mobile phone market. It’s kind of at the point now where all the pieces are there, but we just need someone to tie them all together. A couple of years ago all these pieces weren’t there. It definitely means we’re a couple of years, at worst, of someone coming out with a product that your mom or a completely naive consumer may have no interest in, but an early adopter enthusiast would probably want to buy and wear it.’

We totally agree - and we can’t wait to discover what’s next.