HoloLens 2 at your fingertips: a practical magic

I am at Microsoft headquarters to attend a demonstration of HoloLens 2. I am driven into an underground corridor. Large sliding metal doors separate the rooms. Arriving at destination, I find the reconstruction of a car workshop. It smells like rubber tires. In front of me, there is a mountain bike on a pedestal. Tools and parts are distributed throughout the room.

I am handed a HoloLens 2 and I am told that I will learn to repair the bike.

Microsoft HoloLens are back and refitted. The new version of the company's 3-year mixed reality headset, now available available to order $ 3,500 and comes later this year, slips easily over my head and my glasses. It looks like an industrial tool, a welder's mask. I can see well. I go through the eye tracking setup and a grid of dots appears. I am the points, from one corner to the other, side by side. It works. The new Microsoft Dynamics 365 Guides application is now launched. This is my game of instructions.

The best way I can describe is like the detailed Google Maps instructions for real world instructions – or as a floating Lego manual for reality. I move my eyes on each map step by step that floats in the air in front of me. I am told to put the bike in neutral. Now a floating arrow bows through the 3D space to show me the gear shift on the bike and where I should move it. I do it. I move my eyes to the next step. Now, I'm adjusting a loose bar on the bottom. A long dotted arrow points to a toolbox against the wall, pointing to a ratchet. I take almost the wrong one, realizing that the arrow is pointing me to another tool. My eyes go to the next card. Another line of bow shows me the path that leads to a basket of nuts to grab the right piece to screw.

Sometimes the arrows do not align perfectly. Would I take the wrong tool, I wonder? Would the program correct me if I made a mistake? I should have tried to grab the wrong tool. Nevertheless, the positioning of the arrows pointing in space is strange, like a 3D browser. My mind, strangely, jumps on Nintendo and my experience with Lab, a game for Nintendo Switch that helps players make elaborate cardboard creations with on-screen instructions. Imagine that, but in 3D, with instructions scattered everywhere, indicating real things. The world in Lego kit.

There is no spider-zapping game here. I do not play Minecraft or Halo. Four years later Microsoft introduced the HoloLens helmet like a game door to the magic worlds In addition to being a business device, the tone has changed. HoloLens 2 is a handy device to help businesses help their employees. RA has become a tool to move things forward. And the improved comfort of the helmet, its improved field of vision, as well as better tracking of the eyes and hands are immediately apparent. It is also connected to a larger number of Microsoft cloud services that will blend into iOS and Android apps too.

After the wild, Willy Wonka dreams I've explored at Magic Leap Last year, Microsoft's HoloLens 2 resembled the selfishness of Magic Leap's identity. But both sides might be needed to determine where this technology should go next.

And that HoloLens is not meant for ordinary people – the mass consumer market – at all. This is for factory workers, in places that can spend thousands of dollars for a work tool. In fact, Microsoft may not be watching consumer headphones for years. For now, the company is simply trying to find a better way to mix reality in a practical way … and patiently waiting for applications, developers and the rest of the connected world to continue to evolve with it.

Practical magic

Augmented reality is no longer a chimera. This is an area explored by Apple and Google in phones, and many hopes for the manufacture of headsets, including Magic Leap, have tried to understand how to design a convenient and comfortable way to carry holograms all the time.

The first HoloLens was seen as a feat because it was autonomous, wireless and without connection. It just worked. The new HoloLens 2 from Microsoft is not really a whole new concept, but it emphasizes three essential elements for the latest helmet: eye tracking, wider field of view and better hands tracking. It is also much more comfortable and allows goal-holders, like me and my CNET colleagues, to use the helmet by simply sliding it over our glasses. There is a Qualcomm 850 mobile processor to drive everything, as well as Microsoft's own artificial intelligence engine, replacing the Intel processor of the old HoloLens.

The new design may look very similar to that of someone who has never worn HoloLens before, but it is significantly less bulky and gives the impression that it weighs less heavy too. It weighs less, of a fraction – 566 grams or 1.25 pounds, or compared to the original 579 grams of HoloLens, or 1.28 pounds – but that sounds more because the breakdown weight has changed, so a little thicker at the back now fits around the back strap, while the front visor is smaller. The center of balance is now slightly behind the ears and gives the impression of "wearing a baseball cap". I loosened the headband, put the headphones on my glasses and everything went well. The new change of balance makes everything a lot less painful to wear for more than 5 minutes. It's like putting on a backpack with better designed shoulder straps.

It even has a lift-up visor. The visor is inclined to allow anyone to make eye contact or perform regular work more easily, as with any computer software designed by a partner. Windows Mixed Reality VR Headsets. I loved being able to stop right in the middle of a demonstration to quickly clean my glasses or scratch my forehead.

The HoloLens 2 hardware is still autonomous, just like the first one, and does not have an extra bag worn on the belt, like the Magic Leap One, but it also means that the helmet is bigger than the Magic Leap One head. used glasses. (It's also over a thousand dollars more.) But it's also much more sympathetic to my vision. The Magic Leap One requires that I put contact lenses or that I expect prescription lenses for the headset: Magic Leap does not even support my prescription, yet. The HoloLens 2 only works on my glasses. I know which one I prefer.

Eye tracking: A no reading?

Eye tracking has not yet been an important factor in VR and RA. But it will be. The first HoloLens did not have eye tracking. That's what Magic Leap One does, and VR headsets for high-end businesses, such as the HTC Vive Pro Eye and the VR-1 Varjo, are starting to include it. Eye tracking can recognize your eyes with internal cameras, so you do not have to move your head. HoloLens 2 also added eye tracking.

Eye tracking HoloLens 2 has a dual purpose: it can measure the movement of the eyes and use it to interact with virtual objects. Microsoft is also using new eye tracking cameras for biometric security. The HoloLens 2 has an iris scanning function via Windows Hello. Users can instantly log in to Windows and launch their personal account or remember their personal headset preferences.

Even more impressive, eye tracking HoloLens 2 works with ordinary glasses, even thick like mine. Most of the eye tracking technologies I've used before have had some flaws when I used my glasses. First demos of the HTC Vive Pro Sometimes Eye did not work if I did not detach the virtual reality headset. Similar experiences have occurred with some experiments using Tobii's vision tracking technology. None of these problems occurred during the few HoloLens 2 demos that I had.

The only real use of the eye tracking that I have experienced was a brief demonstration showing how my quick eye movements could select a virtual object without even moving my head. I've exploded little virtual crystals by fixing them and ordering them to burst. But there are many practical uses: Business software companies such as Tobii already use eye tracking to create scans and heat maps of where you look, to enhance training.

The possibilities of eye tracking as a whole are much more strange. Alex Kipman, a Microsoft technical associate, explains that HoloLens 2's vision tracking cameras could also measure your emotions through minimal eye changes, as well as the position of your gaze.

How much will HoloLens begin to anticipate how you feel, maybe even think?

At Microsoft's Human Factors lab, where the comfort and accessibility of hardware is tested, we enter a room surrounded by prototypes of headset models and a table filled with different rubber ears. Carl Ledbetter, senior director of design at Microsoft, shows us how the fit of the new helmet has been carefully measured against a wide range of heads and ears, testing fatigue and eye comfort. But also, in a corner, a mannequin head studded with a sensor net rests on a table and looks like an accessory Minority Report. It's an EEG detection headset.

"We use it to measure brain wave activity and we can measure the burden on someone's mind," Ledbetter says. "We did not necessarily use as much on HoloLens, but we see it as an opportunity … we use it for other things."

"Are you reading on HoloLens 3?" Ian Sherr of CNET, who visited the facilities in Redmond, Washington with me, asks.

"Mmm-hmm," said Ledbetter, perhaps half-joking. Maybe not.

Finally, a wider field of view

Holographic magic is not so magical if your ghostly beings are cut off from view. The first HoloLens had a field of view of about 30 degrees, which gave the impression of seeing virtual objects through a window the size of a playing card game a few inches from your face.

HoloLens 2 extends its field of view to 52 degrees, which, according to Microsoft, is more than double the effective viewing surface. It's like you're looking at holograms through a window the size of a soft cover book. The vertical viewing area is also larger. Table holograms and virtual screens the size of a monitor make all the difference. It still means that some of the 3D effects are cut off, because my peripheral vision is not covered by anything and I can see in all directions. Nevertheless, it is significantly better than the Magic Leap One, which already offered a wider field of vision than the original HoloLens.

The effective resolution has increased to the equivalent of a 2 KB display per eye compared to HoloLens' original 720p, but the image density is still the same, at 47 pixels per degree. PPD is a means of measuring the density of pixels in optics, such as pixels per inch on a phone or tablet. Kipman calls this the resolution of "retina". I would always say that I can see the pixels if I really look closely, as opposed to my everyday vision. It is also sharper than the typical VR headset. (Varjo's new VR headset has even more dense PPD resolution, but not everywhere else.) But hologram-like effects still look ghostly, a bit like Magic Leap and the first HoloLens. They are bright and sufficiently present to interact in the interior spaces that I have tried. That it will be better outdoors, although? Microsoft says it.

The graphics performance is not radically different from the first HoloLens, except for the size of the viewing area. And, depending on the processors, Magic Leap could still be able to produce better graphics, at least if they do not rely on the cloud.

Everything works in a way that would make your eyes meet if you are not an optics specialist. Zulfi Alam, head of optical engineering at Microsoft, explains how the displays work with many graphics and a magnifying glass. Like the first HoloLens, the visor is equipped with waveguides that bend the light and direct it into the eye with a MEMS (Electronic Microelectronic System) system using LCOS (Liquid Crystal on Silicon).

This is new in HoloLens 2: the original HoloLens used a small screen that further limited the field of view. The display is now etched with a mirror-based laser system that creates an image of 120 fps with three lasers, drawn on the fly on the fly as an old-fashioned monitor, but at high speed. It allows an extra wide field of view and also means that the unlit areas are completely transparent. We watched a tiny mirror moving quickly, vibrating, on the table. Suffice to say that it is complicated. But the end results look good.

Reach out and almost touch objects (with your fingers and even your eyes)

HoloLens 2 has no physical controller: Microsoft relies entirely on hand tracking and voice commands. Manual tracking has also taken a big step forward. The sensors can now recognize up to 25 points of articulation per hand across the wrist and fingers, as well as the direction of the palms, which means flexing of the fingers, a movement of the hand and a capacity grip. Early HoloLens used finger-clicks and gesture-based gestures. This time, it's pinch, shoot, press. It feels far more advanced than the manual tracking of the Magic Leap One, but I am not given much chance of using it.

I test all the experience of what Microsoft calls the "Shell demo". In a living room resembling the Great Room on campus, I see a series of holographic objects on tables: a piece of motor, a windmill. I approach them and see the outline of a box surrounding them. I'm told to take a corner of the box and shoot. I do and the whole hologram gets bigger or smaller, like dragging the corner of a window.

It's literally 3D windows. To move something, I stick my hand to the center of the object, make a fist and move my fist. I can also put both fists in and spread them, and the object expands. It's strange. Not having a haptic feedback is a little exotic. But everything works.

I then see a shining crystal shape across the room, near the real sofa. There is another box, but this time, a triangular read button. I press with my finger. This starts an eye tracking demonstration.

I look at four crystals and when my eye passes to each of them, it sparkles. It feels effortless. I say "explode" and the one I watch in bursts. Microsoft then shows another example of eye tracking: a floating hologram for hummingbirds floats next to a Wikipedia-type text box containing an article about hummingbirds. I read and as my eyes go down, the text box starts to scroll. Sometimes the scrolling is too slow or too fast, but I learn to stall it with my eyes. In reality, the control of the eyes makes it possible to move things too, no hand is necessary.

I've already seen eye tracking demonstrations, but still, the whole thing is fascinating. I want to try something more artistic, even more advanced, but this demo was all I had, so I can not say how more detailed interactions could work.

What will it really look like in the field?

I think back to my virtual bike repair training session in the guides. I wonder if I will remember those instructions later. Would the training be maintained or would I become dependent on the instructions step by step? A few weeks later, as I write this, I do not really remember what I was doing in this room. I sometimes remember how I become "Google Maps Blind" when I drive, and how I forget where I'm going, by submitting to instructions. Some taxi drivers prefer to memorize the cards and internalize knowledge instead of. I ask if this kind of step by step education for navigation could decrease learning – or help? It's a good question, I'm told.

Clearly, there may still be no definitive answers. But HoloLens 2 shows possibilities. Step-by-step guides are designed to be as easy to create as PowerPoint slides. Perhaps, in the near future, it will be the way people will leave instructions in the real world to others so that they can find them later and navigate, like ghostly guides. Google plays in a similar territory on phones with AR in Google Maps.

Connect in the cloud

Microsoft is also focusing on multi-user mixed reality this time, focusing on cloud services via Microsoft Azure that will place mixed reality points that multiple people can experience simultaneously, in HoloLens or even on phones. Google immersed in multi-user AR last year, and Apple too, via ARKit on iOS.

Imagine sharing the same 3D object and working together on the same 3D model. The Dynamics 365 Layout application from Microsoft will memorize objects through the cloud in order to stay connected to a location that others will find. The "cloud anchors" will also be stored via the cloud, to ensure that everyone shares the same vision, even on the iPhone. iPad and Android devices using AR, via Microsoft applications sharing the same AR tools.

We only receive a brief demonstration of the group collaboration, in a room with a large circular table, a HoloLens 2 on the head. Next to me are my colleagues from CNET, Ian Sherr and Gabriel Sama. They also carry HoloLens 2. We meet as an intergalactic delegation. On the table is a vivid 3D virtual map of the Microsoft campus, created in the headset. A woman, standing up from the table, stands up and talks about future plans for Microsoft's expanded campus. Ian and Gabriel are watching too. We see the same thing from three different points of view. And above each of their heads is their floating name. Mine also floats over my head, if I could see it. This is a demonstration of Microsoft's position on mixed reality: collaborative, multi-user. To show how all of this can be multi-platform, some assistants with a hand-held phone stand by us, also showing how 3D holograms can also be seen on a phone screen. We share the same thing.

I notice immediately how the field of vision has improved. I see everything on the table without the effect being cut.

All is not perfect. Microsoft warns us in advance that we use earlier engineering prototypes. Indeed, the landscape of Microsoft campus that deploys in semi-transparent 3D on the table eventually tilted a little when I turn around and look at the room in which I find myself. He readjusts himself. But I'm wondering: if the client companies ended up using the computer science space to report the reality, how many problems would they tolerate?

Microsoft cloud services aim to seriously improve the quality of mixed reality. If the current HoloLens allow centimeter accuracy, the additional cloud computing enhancements will give it millimetric accuracy. Similarly, the quality of 3D rendering will be improved. Kipman shows us in just a few slides how the current mixed-reality graphics can be good enough to be seen, but not enough to create with. He demonstrated that a block engine could look much more detailed thanks to Azure cloud rendering added.

Microsoft is also planning to use Azure services to provide more of what HoloLens is doing to improve graphics on the device from 100,000 rendering polygons for a 100-million polygon 3D object. Ultimately, the goal is to reduce the size of the headset and push as much as possible to the cloud. At the moment, this means that HoloLens 2 should be able to tap much more into accounts and cloud documents. But it's hard to see how all this will come to fruition.

At the moment, the HoloLens 2 is designed to be largely autonomous. He is always made to be autonomous, able to work in offline mode, as was the original HoloLens. It connects via Wi-Fi, but not with the cell. Part of this, according to Microsoft, has to do with where the HoloLens 2 is intended to be used. But finally, when 5G networks are able to cover the world of high-speed data, HoloLens are likely to evolve into a much more powerful and cloud-dependent device.

Lily: HoloLens 2 from Microsoft: Why is the cloud really based on you? (excerpt from Mary Jo Foley from ZDNet)

Always missing: no haptics, no control

It should be noted that one of the physical controllers is missing from the HoloLens 2. Just like the original HoloLens, the headset is designed to work only with manual and voice commands. Although manual detection has been greatly improved, the lack of any tactile controller or force feedback has thrown me to the ground and made me feel a little pissed off. The Magic Leap One has a single one-handed physical controller that, while limited, adds a tangible sense of reality to the interaction with things and provides a haptic and vibrant feedback.

Microsoft thinks about controls and haptics – but it has not happened yet.

"We like haptics," Alex Kipman tells me. "The minute I can throw you a hologram and you catch it and it pushes you away … ooh, the immersion just took a crank in. The minute I hold a hologram and say, It is hot – cold, warm, warm – it changes the level of immersion and the credibility of the experience. "

While Kipman claims that haptics are "absolutely in our dreams," Microsoft is not yet using controllers similar to those that already exist. Microsoft Windows VR Headsets. "We have no dogma that you can not have something in your hands – in fact, in our virtual reality headsets, we have pretty decent things that work with the same set of sensors as our HoloLens. " But Kipman still does not see the current Windows VR controllers come into play, despite Microsoft's "mixed reality" on these accessories.

Maybe with the HoloLens 3? "It's also absolutely in our roadmap to think about keeping things in our hands, not just the things we create, and if I'm a person with a real physical hammer, and my hand is busy, or am I holding a cup of coffee, and I still want to touch my hologram? "

I am also curious about how the new controls will feel over time. Entering objects and pressing buttons is more realistic. But is realism what I want, or shortcuts and comfort? How will it be played on HoloLens 2?

"We think about that," says Ledbetter, about fatigue screening over time. "Do you try to do things and your hands are full and you talk to someone … what is the best interaction?" It introduces us to the world of software … but you're in the right place to do it. think . "

The future is still strange and wide open

It is clear that Microsoft is relying on its Azure cloud computing for HoloLens 2 to do more, and if that means more precise placement of objects in 3D space, more detailed graphics and mapping, both better. Kipman also points out that the future of products such as HoloLens is really part of a broad continuum. The HoloLens 2 computer vision-activated tracking helmets will also look like autonomous camera and UAV navigation camera sensors, and look like the world's scanning cameras in homes, factories and devices. (In fact, Microsoft is selling a new cloud version of Kinect which integrates HoloLens sensors.)

It looks like a physical world filled with advanced devices, relying on a cloud that will move at ever greater speeds. Perhaps the 5G, allowing all these future HoloLens devices and AR headsets to be more cloud-based than on the headsets. The HoloLens 2 has no cellular; it's Wi-Fi and Bluetooth only. The LTE world is not ready for this yet.

Même Kipman admet que le HoloLens 2 ne convient pas à tout le monde ni à toutes les situations. Alors que Kipman utilise les HoloLens plusieurs heures par jour, "il arrive souvent que je sois dans mon bureau et que j'utilise mon clavier, ma souris et mon écran d'ordinateur pour effectuer un grand nombre de tâches".

Mais quand vient la 5G et l'haptique, quoi alors? Microsoft joue clairement le jeu long, comme tout le monde dans le monde AR / VR / MR. Le prochain HoloLens n'est peut-être pas si loin après tout. Kipman hésite même à faire des prédictions dans cinq ans: "Je ne devinerai pas dans cinq ans, pour être honnête avec vous. Laissez-moi dire, pour la durée de ce produit, disons plus dans la catégorie un à deux. .. Je pense que tous les succès seront liés à l'entreprise. "

Et peut-être qu’à ce moment-là, ce sera le super appareil 5G auquel je m'attendais depuis le début.

(Initialement publié à 09h20, heure du Pacifique)