Meta’s display chief names 10 features for the perfect VR headset

Image: Meta

Der Artikel kann nur mit aktiviertem JavaScript dargestellt werden. Bitte aktiviere JavaScript in deinem Browser and lade die Seite neu.

There are still many hurdles to overcome before virtual reality reaches its technical potential. The head of Meta’s display division lists the challenges.

Douglas Lanman has been with Meta for eight years and heads the department that develops display systems. In June, his team unveiled new prototype VR glasses. The prototypes meet different technical challenges such as resolution, brightness and form factor. The researchers’ goal is a VR screen capable of visually reproducing physical reality while fitting into a compact helmet.

At this year’s Siggraph conference, Lanman gave a presentation on the ten main obstacles yet to overcome to develop an almost perfect VR headset. The following is a list of these ten challenges, including a brief explanation. You can find the video of Lanman’s speech at the end of the article.

The ten challenges

Highest resolution

Current VR headsets don’t come close to human vision in terms of resolution. For virtual worlds to look as real and crisp as if they were physical and for texts to read well even at medium distances, the resolution of VR screens must increase significantly.

Meta-states 8K per eye and one 60 PPD pixel density as a preliminary target. For comparison: the Meta Quest 2 (test) only reaches 2K per eye and 20 PPD.

Sehtafelvergleich zwischen der Auflösung dreier VR-Brillen: Rift, Quest 2 and Butterscotch.

The higher the resolution, the clearer the image. Oculus Rift, Meta Quest 2 and the Butterscotch prototype in comparison. | Image: Meta

With the Butterscotch prototype, Meta has developed an experimental headset that allows researchers to experience “retinal resolution” and evaluate its immersive effect. The development and production of high-resolution displays is not the biggest problem. The question is rather to house the computing power to power these high-resolution screens. Foveal rendering and cloud streaming could help, but they themselves present major technical challenges.

Wider field of vision

A lot also needs to be done in terms of field of view, Lanman finds. Jthe human horizontal field of view is about 200 degrees wide. commercially available VR headsets typically achieve a horizontal field of view of 100 degrees. But, there is still room for improvement in the vertical field of view as well.

A wider field of vision poses great challenges for lens technology, which is manifested by image distortions at the edges of the field of view. Here too, the question of computing power arises. The wider the field of view, the more pixels the VR headset should display. This results in higher energy requirements and more waste heat.

Ergonomics

VR headsets are still heavy and bulky. Meta Quest 2, for example, weighs over a pound and protrudes nearly 3 inches from the face. VR helmets should ideally be comfortable wear for longer periods of time and be much narrower and lighter.

Ein Mann mit Holocake-1-Protoyp und leicht transparent darüber gelegt eine VR-Brille mit herkömmlichem Formfaktor.

The image shows the Holocake-1 prototype, which comes in the form of sunglasses. Shown in transparency is an Oculus Rift from 2016. | Image: Meta

Pancake lenses and holographic lenses could help. Meta’s fully functional Holocake 2 prototype shows the direction the form factor could take. The problem: Holocake 2 uses custom lasers as its light source, which are not yet developed to the point of being mass-produced.

Display with vision correction

The perfect VR headset must be able to detect and compensate for users’ visual impairments so that conventional glasses or contact lenses are not required to see well in virtual reality. Will the prescription frames fit under the VR headset without scratching the lenses or pressing down on the wearer’s face? Consumers should no longer have to deal with such questions in the future.

The problem could be solved with special accessories or, even better, with a lens that can be adapted to the visual acuity of the user.

Innenseite der Quest 2 mit über den Linsen schwebenden Sehkorrekturaufsätzen.

Lens attachments with vision correction already exist. But they are not an ideal solution. | Image: Meta

variable focus

The human eye can’t naturally focus in VR environments, which is particularly troublesome at close range and can lead to eyestrain and headaches after a longer period of time. This phenomenon is called in technical jargon vergence-accommodation conflict.

To solve this problem, researchers at Meta have developed a screen that supports “progressive vision.” The display simulates different focal planes as well as blurring and helps the eye see the virtual world as if from nature. Meta’s varifocal prototypes are called “Half-Dome”.

Eye tracking for everyone

Eye tracking is a key virtual reality technology. It is the foundation of many other important VR technologies such as progressive vision (see point 5), foveal rendering and distortion correction (see point 7). It also enables eye contact in social experiences and new forms of interaction.

Ein Augenpaar, so wie es von einem Eye-Tracking-System erfasst wird.

The shape of the pupil differs from person to person. This is a challenge for eye tracking systems. | Image: Meta

The problem with eye tracking is that it doesn’t work equally well for everyone and sometimes has dropouts. A reliable solution with broad demographic coverage is needed. Otherwise, the technology frustrates rather than helps and is rejected by consumers.

Distortion correction

Lenses inherently introduce image distortion that must be corrected by software. The slightest movement of the pupil causes fine but noticeable distortions. They impair visual realism, especially in combination with other technologies such as progressive screens.

To speed up the development of correction algorithms, Meta researchers have developed a distortion simulator. It can be used to test different lenses, resolutions and fields of view without having to build special test headsets and lenses.

High Dynamic Range (HDR)

Physical objects and environments are much brighter than VR screens, even indoors with artificial lighting. Meta has built Starburst, a VR prototype capable of displaying up to 20,000nits. For comparison, a good HDR TV offers several thousand nits while Meta Quest 2 only offers 100 nits.

Zuckerberg hält den wuchtigen Starburst-Prototyp an zwei Griffen.

Mark Zuckerberg holds the Starburst prototype. | Image: Meta

Starburst can realistically simulate lighting conditions in closed rooms and nighttime environments. The current prototype is so heavy that it hangs from the ceiling and is extremely energy-intensive.

According to Meta, HDR contributes more to visual realism than, say, resolution and variable focus, but it is furthest from practical implementation.

Visual realism

The perfect VR headset should be translucent in both directions. VR users should be able to see the environment as much as the environment can see VR users. This is both for reasons of user comfort and social acceptance.

The helmet’s sensors record the environment and display it as a video image in virtual reality. This technology, called passthrough, already exists in commercially available VR headsets, albeit of rather poor quality. Meta Quest 2, for example, features a grainy black and white passthrough mode. Meta Quest Pro is supposed to significantly improve this display mode with higher resolution and colors.

Ein Mann mit einer herkömmlichen VR-Brille und ein Mann mit dem Reverse-Passthrough-Prototyp.

On the left is an Oculus Rift that blocks the view of the VR user’s eyes. On the right is an early prototype reverse passage. | Image: Meta

However, this does not yet achieve a perfect reconstruction of the physical environment. One issue yet to be addressed is that passthrough technology captures a perspective of the world that is spatially displaced from the eyes, which can be irritating during prolonged use. To this end, Meta is investigating AI-assisted gaze synthesis that generates perspective-correct viewpoints in real time and with high visual fidelity.

Otherwise, which Meta calls “Reverse pass-through“, strangers see the eyes and faces of virtual reality users and can thus make eye contact or read facial expressions. This could be made possible by inward-facing sensors and an external display (light field). This technology is still far from ready for marketas Metas research shows.

Facial reconstruction

Co-presence and metaverse telephony are the ultimate goals of Meta. The company wants people to meet one day in virtual spaces and feel like they’re in the same room. For this purpose, Meta is looking for photorealistic codec avatars, but they are still very expensive to produce and render.

A first step in this direction are VR headsets that can read the facial expressions of VR users in real time and transfer them to virtual reality. Quest Pro will be Meta’s first headset to feature face tracking.

Quest Pro can recognize facial expressions and transfer them to VR. | Image: Meta

Siggraph Lecture by Douglas Lanman


About Marion Alexander

Check Also

Warby Parker Inc. (NYSE: WRBY) Receives Medium “Holding” Rating from Analysts

Shares of Warby Parker Inc. (NYSE: WRBY – Get Rating) received a consensus rating of …