The Facebook prototype casts your eyes on a VR headset

Facebook Reality Labs wants to help people see your eyes while you’re in VR, even if the results fall somewhere between slightly disturbing and nightmarish. Earlier this week, FRL published an article on “VR reverse passthrough,” a recipe for making VR headsets less physically insulating. Researchers have come up with a method to translate your face to the front of a helmet, although they point out that it is still firmly experimental.

“Passthrough VR” refers to a feature that displays a live video feed from the cameras of a headset, allowing users to see the real world while wearing the device. Facebook’s Oculus Quest platform, for example, shows users a pass-through flow when they step outside of their VR space. This is useful for quickly ditching VR, and it can also enable some form of augmented reality by adding virtual objects to the camera feed. But as FRL notes, people around a headset user can’t make eye contact, even though the wearer can see them perfectly. It’s embarrassing if viewers are used to seeing their friend or colleague’s face uncovered.

FRL scientist Nathan Matsuda decided to change that. A blog post explains that Matsuda started in 2019, when he mounted a 3D display on an Oculus Rift S headset. The screen displayed a virtual rendering of the top of his face and custom eye-tracking cameras captured the location. where Matsuda was looking, so his avatar’s eyes could point in the same direction. The result was essentially Matsuda wearing a telepresence tablet showing a copy of his own face – which is arguably just as awkward but with a more intriguing postmodern touch.

FRL chief scientist Michael Abrash – understandably – didn’t find the idea very practical, according to the blog post. “My first reaction was that it was a bit of a wacky idea, a novelty at best,” he notes. “But I don’t tell researchers what to do, because you don’t get innovation without the freedom to try new things. “

Matsuda ran with the concept, and over the next two years he led a team to develop a svelter design. The team’s prototype headset – which he revealed ahead of next week’s SIGGRAPH conference – adds a stack of lenses and cameras to a standard VR headset display. Stereo cameras capture an image of the face and eyes inside the helmet, and their movement is mapped to a digital model of the face. Then, the image is projected onto a bright field screen facing outward. This display creates the illusion of looking through the lenses of thick glasses and seeing a pair of eyes, although in reality you are still seeing an animated copy in real time. If the wearer returns to full VR, the display may go blank to signal that they are no longer engaging with the outside world.

The result is a pair of octagonal glasses that would look comfortable in a Terry Gilliam movie. FRL used a simple rendering of a virtual face, but it also showed the system with its more realistic codec avatars, as seen below.

FRL recognizes that not all individual system components are revolutionary. HTC already has a face tracking add-on for its Vive Pro headsets; it maps movement to an avatar inside the VR, not an outward facing screen, but the principle is similar. This week’s article focuses on the potential of light-field displays and the system’s opportunities for better in-person social interactions.

HoloLens-style projection glasses theoretically leave your face much clearer than passthrough screens – although many of these glasses have dark lenses, and like On the road to virtual reality Please note, the light projected on transparent glasses may also block your gaze. But as companies like Apple experiment with passthrough designs, new research from Facebook shows that a solid screen isn’t necessarily a barrier to eye contact … sort of.