Why AR will give you "super powers" in the future

the HoloLens 2 restarts augmented reality for Microsoft. It offers a new fit, a wider field of view and adds detailed hand and eye tracking. But one thing is still not is a consumer device.

Alex Kipman is a technical collaborator of Microsoft and the inventor of Kinect and HoloLens. During a visit to the Microsoft campus of Redmond to try HoloLens 2 before its debut, we were able to talk to Kipman about his vision of the direction of computing, what becomes of HoloLens and the distance that separates us from a future where ordinary people will actually wear advanced RA helmets.

For Kipman and Microsoft, a helmet like HoloLens is one of the many devices in which sensors will digest the world with AI. As Kipman says, on a helmet, it's the HoloLens. In a house, it's a smart camera. On a car or drone, it's a standalone vehicle. This time, HoloLens 2 aims to connect its magnetic resonance experiences in the cloud to other devices, including iOS and Android, and to feel more than ever as a working tool.

But when will this world of augmented magic become something for the rest of us?

Below is a revised version of our conversation and a video interview with Kipman, from the moment CNET spoke to him on January 31, 2019.

How much of these technologies will the general consumer see in the short term?

I have no interest in overprinting and making a group of people believe that these products are consumer products. And then … go to the height of disillusionment when people say, "My God, I do not use this instead of a PC, instead of a phone, rather than a phone." 39, a television ". These devices need to be more comfortable, they have to become more immersive and, ultimately, they must have better value for money. There is a threshold in the trip where there is enough immersion, comfort, value, where I will be pleased to advertise a consumer product. This is not it.

Where are we?

It's impossible that I guess in itself. … I think humans are terrible predictors of time. In business, in front-line worker scenarios, we find great value when this kind of thing is transformative. Let's be clear: if I can take the best watermark, the best product existing in this space today and that remains "not ready for consumers", you can be the product judges of everyone in this space.

Kinect started with consumers, on the Xbox. Have you ever thought about coming back to this route for the HoloLens?

Look, like everything in life, you learn. I'm incredibly proud, of course, of the work we did with Kinect, and I think Kinect has transformed the world. But, look, people in the living room do not want to get up and play games … they want to lean back and enjoy and want the precision of a controller in their hands. We had not found so much signal in the living room for entertainment with devices like Kinect at the time.

But you know, look at Alexa today. What are these things doing? They recognize people, they recognize the word. They do a lot of things that Kinect did in 2010. So, obviously, there is room in people's homes for devices like Kinect, who recognize people, who recognize the things they contain and who understand the context of who you are. But we find much more signal with Kinect in business workloads. They do not tend to go through a proxy computer. That's why we then sold Kinect at the end of its life and just launched Azure Kinect, which is of course still connectable to a PC but also connects directly as an IoT device to our cloud.

Where does the mixed reality go in the next five years and what role does Microsoft play?

I will not guess five years, to be honest with you. Let me say for the duration of this product, say more in the category one to two … I think all success will be business scenarios, mainly front-line workers, and will increase with time in knowledge worker scenarios. So I will give you the forecast for the next two years. These next two years, they are still related to the company.

What do you think killer apps will be?

I think communication … basically defines a secular trend in computer science. It turns out that more often than not, it is the innovation in the communication stack that changes things. Mail to real mail, real mail to mail, SMS, Snapchating. It's like going from a still to a video to a teleportation. For example, my daughter can teleport to play with her cousins ​​in Brazil, as a parent. I do not have to travel around the world to visit all my partners. How much will I like it? If you could have this level of current experience and you're in New York, and you're in San Francisco, is it still immersive with me in Redmond? It is not difficult to imagine the presence as a lethal experience for mixed reality devices.

Regarding the replacement of the monitor … think of sitting in front of your PC for "n" hours a day. It can be immersive and comfortable. Would you like to spend so much money for [the HoloLens] on your head with a keyboard and mouse compared to buying a 30-inch monitor at $ 500? Probably not, so we try to focus a lot on things that you simply can not do otherwise. If all goes well, we will all be living very soon in a world where we will interface, interact and instinctively manipulate technology in our day, and you will not connect via monitors.

In terms of eye tracking, what challenges or opportunities do you see?

I think that in the end, our quest is to make people, places, and things understand artificial intelligence. You want to get as much signal as possible in this conversation. If you want to teleport somewhere, I want to be able to know what you are doing and have that level of understanding so that I can really teleport you. I still want your facial expressions to go through. It's something that fascinates me a lot, the eyes and the emotion of your eyes – we have so many signals to convey to mine to create more immersive and more comfortable experiences.

I think a lot about the safety of these devices. We are going to have advanced recognition of iris, the most secure biometric system with iris recognition via HoloLens, which allows me to get all this data safely, all my information from comfort, all my personalizations. And finally, the idea is that we can close the loop. I know at any time where the device is in relation to your eyes when we start to form the hologram. Without this signal, the image can be correctly obtained or not. Which means that your eyes and your brain are doing all the math for everything that bothers me. This results in fatigue at the end of the day. We can load most of these calculations on the device and adjust the hologram as you move, creating sharper and more immersive holograms.

One thing that sets your technology apart from others in the field is that you do not have physical control at all, using your hands. Will there be a controller or a haptic?

100%, we like haptics. We started this trip 11 years ago with a contribution. Kinect was about having sensors on the edge that looked at the environment to understand people, places, and things. We went from Kinect input innovation to HoloLens, more output entrance. The last is to have these things in my world exchange energy. Having zeros and ones, which turn into photons, turns into energy so that I can push a hologram, and that pushes me away with equal force. So, I can hold the hologram and feel the temperature of a hologram. We can call it a haptic feedback. Much more sophisticated than you traditionally thought [it]but another level of immersion. The minute I throw you a hologram and you can catch it and it pushes you away … ooh, immersion just took a crank forward. As soon as I hold a hologram and the temperature is there, it changes the level of immersion and the credibility of the experience.

Now, although this is absolutely in our dreams, we also believe that humans are tool builders. I would not want my doctor to operate without tools, only with bare hands, no more than I would like to eat my food tonight without fork or knife. We have no dogma: "You can not have anything in your hands." In fact, in our handsets of virtual reality, you have in your hands things: tools, controllers. This device could work here, but I do not know if you've seen it, it's lighted. All you see is the lights above the hologram. This is not such a great experience. It is very easy for us to create a version of it in IR so that you do not see the light. It is also absolutely in our roadmap to think about keeping things in control. Not just the things we create. What if I am a person with a real physical hammer? We hold a cup of coffee and I still want to touch my hologram?

How long do you use HoloLens every day?

Several hours a day are the short answer. We actually designed HoloLens on HoloLens. Wearing HoloLens and watching the model in 3D is a much more visceral way of understanding the space and the creation found there. But look, when I'm in a meeting, I do not wear HoloLens. I am often in my office and I use my keyboard, my mouse and my computer monitor to perform a large number of tasks. But I wear the camera several hours a day, as do most members of the team.

What's the only thing that bothered you while you made the HoloLens 2?

That's all. I dream that one day there will be more than just one problem that prevents us from sleeping at night. We count HoloLens in miracles. You know, we can not have double-digit miracles in a given product cycle. This is how we size the number of innovations or problems that we will integrate into a publication. This carbon fiber enclosure is a huge problem. This is where we can essentially make the device much more comfortable and much more stable. But shipping carbon fiber that does not look like carbon fiber … was incredibly difficult, remains incredibly difficult and poses many problems. Inventing a new display engine: it was a huge miracle. The innovation in the lenses, in the steam room at the back, in the system of adaptation, and make the tailoring system expandable for business, so that you can put it under a protective helmet, for a lot of things.

It's just the material. Manufacturing, construction on scale, performance? A completely different set of problems. Articulate your hand to feel instinctive. Follow your eyes to work over glasses. How do you create this platform from the edge to the cloud? To stay awake at night and say, "Oh my God, how can we take all this and make it open?" If you do it badly, we will have to live with some of these decisions for the next decade and more.

If you solve all the problems, what is the final state of the dream you want?

My dream state is that I walk on a plane, man, and that every person in this plane carries our product. By the way, this is not this product. This is probably not the next one either. But in the end, the goal is that these things transform humans, they give people and organizations the power to do things they just were not able to do before, they allow us to move space daily and time as if we were born instinctively with superpowers. It's a work of a lifetime, but I certainly can not think of anything else to do with my life.