Mixed reality in the car: BMW turns the M2 into a racing game controller

What happens when you equip the new BMW M2 with a mixed reality headset and a gaming PC? The sports car itself is transformed into a controller that can be used to race around a virtual circuit, collect points and set best times. But the dangers are in the real world.

Table of contents

A very special M2
Varjo XR-3 offers high-resolution Focus Area
1. Cameras project real world in VR
Virtual circuit on real open space
1. Discs become a projection surface
Merging real and virtual worlds
1. BMW picks up numerous sensor data
2. Pipeline does not show any latency
Project could enable driver training

There is a lot of technology in this very special M2. What is meant is not the 460 hp straight six-cylinder, with which BMW itself outperforms the previous M2 CS, or the new cockpit with curved screen and BMW OS 8, but what makes the sports car a controller in a mixed reality environment. For the Web Summit in Lisbon, BMW equipped the new model (G87) with a gaming PC in the trunk, among other things, and the manufacturer revealed at least enough about the technology that a GeForce RTX 3090 (test) can handle the rendering.

A very special M2

In addition to the PC in the trunk, the eye-catching foiling, the ART Smarttrack 3 used for head tracking in the windshield and, last but not least, the Schroth four-point belts, indicate that not everything is going right in this M2 and some things differ greatly from the series are not offered for the production model and are installed in the prototype for two reasons. On the one hand, the driver is kept in the bucket seat on the fast lap and on the other hand, BMW has deactivated the airbags for legitimate reasons.

Varjo XR-3 offers high-resolution focus area

The background is that drivers do not shoo the M2 around a circuit in the classic way with an unobstructed view, but wear a mixed reality headset, which – should it happen despite all the safeguards – does not go well with an airbag. BMW connects the gaming PC in the trunk to the Varjo XR-3, a high-resolution HMD that is clearly aimed at the B2B segment with a base price of 6,500 euros and annual costs of 1,500 euros. Among other things, the XR-3 features a “Focus Area”, i.e. a particularly high-resolution area of 27 × 27 degrees in the driver's central field of vision, in which a very high 70 PPD (pixels per degree) can be achieved with 1,920 × 1,920 pixels. Varjo relies on a micro-OLED panel surrounded by an LCD with 2,880 × 2,720 pixels (30 PPD) per eye in the peripheral field of vision. Horizontally, the XR-3 has a field of view of 115 degrees. The MR glasses cover 99% of sRGB and 93% of DCI-P3 and work at 90 Hz.

Cameras project real world in VR

Two 12 MP cameras on the front, which record a video signal at 90 Hz with low latency, make the XR-3 a mixed reality instead of a normal VR headset in order to not only present the wearer with a virtual world, but also one into which snippets of the real world can be projected, resulting in the mixed reality experience that BMW uses for the “M Mixed Reality” in the M2. The approach is therefore different than with AR, which projects virtual elements onto a transparent pane in front of the user, which often has a semi-transparent holographic representation that entails a corresponding loss of quality. The cameras deliver a significantly higher quality, but do not reach the dynamic range of the human eye.

Virtual circuit on real open space

The structure presented at the Web Summit is relatively easy to explain. On an open space in the south of Lisbon, BMW has created an area that is large enough to safely move the M2 at higher speeds, but for the driver with the headset it is not in Lisbon, but on a virtual circuit , whose dimensions correspond to the free space. However, the driver is not isolated from the real world in that the entire cockpit is recorded by the cameras in the HMD and connected to the virtual world.

Discs become the projection surface

BMW achieves the fusion of the real and virtual worlds by defining the vehicle's windows as a projection surface for the virtual environment, while the vehicle visible in the HMD is an image of the cameras in the HMD. What the driver sees directly in front of him, i.e. the steering wheel and the entire cockpit of the car, is not rendered on the PC, but actually exists and can be operated like driving without an HMD.

Merge real and virtual world

The supposed blind flight is therefore not at all, because BMW has managed the connection between driver and car as well as real and virtual environment so convincingly and free of latencies that the two worlds merge directly with each other. Getting used to the environment like raw eggs or exploring the limits of the system? none. Even those inexperienced with VR were practically on the hunt for best times on site and wanted to drive additional laps, after only three were actually planned per person. None of those present experienced nausea, although the prototype was driven at high speed in relation to the narrow course.

On the winding virtual circuit, which can be completed in less than 20 seconds, coins can be collected and gates that change position must be driven through so that there are no penalty points. 16.891 seconds was the personal best, but project manager Alexander Kuttner's clock stopped after 16.2 seconds despite his relaxed driving style. No wonder, since Kuttner was responsible for the production management of BMW M motorsport vehicles before his time as “Project Lead Digital Driving Experience XR”. BMW picks up numerous sensor data

How BMW achieves such a realistic experience anchored in a virtual world can be explained in part technically, but the company does not yet want to lift the veil of all the details. First of all, the manufacturer collects and processes all sensor data related to the movement of the vehicle. According to the editorial team, the steering angle, accelerator and brake pedal position as well as roll-pitch-yaw angle and GPS position are determined. The driver's head or HMD is tracked via the Smarttrack 3 in the windshield. In addition, the Varjo XR-3 supports eye tracking, which BMW uses for foveated rendering to reduce the graphics quality in the peripheral field of vision and thus the load, or to obtain more resources for higher quality in the direct field of vision.

Pipeline does not detect latency

BMW also manages the immersion very well with the perceived latency-free implementation. In view of the complex pipeline and the challenge of moving a real sports car at high speed in a closed area in a virtual environment, this is not a matter of course. BMW renders the virtual race track with the Unreal Engine 4 and merges it with the camera images in Varjo software. The assembled image is then output on the HMD.

Project could enable driver training

And why all this? A product for end customers is not forthcoming, which is not surprising given the prerequisites. However, driver training courses or trips for potential customers are conceivable, for example. Even on a limited open space, you could practice certain combinations of curves on a real racetrack. A complete course would require a very large area, but sections could be realized as long as there are no differences in height. Potential customers could experience a dynamic driving style of a sports car in the dealer's parking lot without being reduced to a boring course with cones.

In addition, the mixed reality solution also allows for a multiplayer experience. Although two real cars did not compete on the same open space in Lisbon, according to Alexander Kuttner, a second driver could log into the session purely virtually via another PC and compete against the driver in the real car. In Lisbon, however, the only opponent was the stopwatch – and your own driving skills.

ComputerBase received information about this article from BMW at an event held by the manufacturer in Lisbon . The costs for arrival, departure and hotel accommodation were borne by BMW. There was no influence on the part of the manufacturer or an obligation to report.

This article was interesting, helpful or both? The editors are happy about any support by ComputerBase Pro and disabled ad blockers. More about ads on ComputerBase.