How Apple’s upcoming mixed reality headset will work

While companies like Meta and Microsoft are forced to reduce their investments in virtual and mixed reality technologies due to the economic downturn, Apple continues to actively work in this direction. Famous insider Mark Gurman of Bloomberg, citing his sources, detailed how Apple’s mixed reality headset will work.

The device is an ambitious attempt to create a 3D version of the iPhone operating system, with eye and hand tracking systems that could set the technology apart from competing products.

The roughly $3,000 headset, which is expected to launch later this year under the likely name Reality Pro, will take a new approach to virtual meetings and immersive video, aiming to shake up the VR industry currently dominated by Meta. This is a very risky move for Apple, which has been entering a new product category since the release of the Apple Watch in 2015, and the company needs to impress consumers.

The company’s technology development group of more than 1,000 people has been working on the project for more than seven years, and Apple expects it to become a new source of revenue — especially as sales growth for the company’s other products may stall this year.

But virtual reality has proven to be a challenge for the biggest tech companies. While some predict the market will top $100 billion by the end of the decade, VR headsets are still considered a niche product — and Meta has already lost billions trying to expand the business.

Apple’s goal is to offer something new, Gurman writes. According to people familiar with the product, eye and hand tracking capabilities will be the main selling point of the device, which is expected to cost about twice as much as rival devices. Its core features will include advanced FaceTime-based video conferencing and meeting rooms.

The headset will also be able to show video content with the effect of presence, serve as an external display for a connected Mac and reproduce many functions of the iPhone and iPad.

Here’s how it will work: the headset will have several external cameras that can analyze the user’s hands, as well as sensors in the body of the gadget to read the gaze. This will allow the user to control the device by looking at an on-screen element – a button, app icon or list entry – to select it.

Users then have to pinch their thumb and index finger together to activate the task – without having to use any controller. This approach differs from other headsets, which typically rely on physical manipulators.

Like the latest Meta headset, Apple’s device will use both virtual and augmented reality. With VR, users see images and content in glasses. AR, on the other hand, overlays digital content onto images of the real world.

The headset will have two ultra-high-resolution displays developed in conjunction with Sony to work with VR, as well as a set of external cameras to provide “end-to-end” augmented reality. This means that users will see the real world through the cameras located on the headset. Apple will offer users with prescription glasses special lenses that will be placed within the enclosure itself.

The device will have a so-called “Digital Crown” – like the Apple Watch – which will allow you to switch between VR and AR. Apple expects that to be the highlight of the product, according to people familiar with the product’s development. Given that the headset is still several months away, some features could still be canceled or changed, they added. An Apple spokesperson declined to confirm or deny the information when asked by Gurman.

The FaceTime software on the headset will realistically reproduce the user’s face and entire body in virtual reality. These avatars will allow two people – each with an Apple headset – to communicate and feel as if they are in the same room. The technology differs from the virtual meeting rooms on the Meta headset, which creates a more cartoonish avatar of the user.

Due to the massive processing power required for this feature, the headset will only support realistic avatars during one-on-one video chats. It will still allow FaceTime sessions with several people, but additional users will be displayed in the form of an icon or Memoji – personalized emojis from Apple.

Apple plans to unveil the device as early as this spring, although the timeline could still shift, according to people familiar with the development process. That would allow the company to discuss the product at its annual software developer conference in June and then release it later this year.

As with some of Apple’s previous big bets, the company plans to start slowly. It aims to start early production of the device as early as February in China and is considering a US-only launch of the product to begin with.

Price is also expected to limit the product’s appeal, and Apple is already working on a cheaper version — due for release in late 2024 or early 2025 — that could cost closer to $1,500. That’s how much a mixed reality headset from Meta costs.

Apple plans to sell about 1 million units of its new headset within the first year. That compares with more than 200 million units of the iPhone, which is the company’s biggest source of revenue. In a rare move, the company also doesn’t plan to make a profit from the initial version – even at the high price – suggesting it’s looking at the platform for the long term.

Immersive video viewing will be the main function of the new device. Apple has discussed developing VR content for the platform with half a dozen media partners, including Walt Disney and Dolby Laboratories. The company is also working on updating its own Apple TV+ content to work with the headset. In 2020, Apple acquired streaming company NextVR to create sports content in virtual reality.

Apple plans that the headset will have a special video viewing function that will give viewers the feeling that they are watching a movie on a giant screen in a different environment, such as a desert or outer space. But although the video on the headset will be exciting, its speakers will be less powerful. Therefore, users will need to wear AirPods headphones to get full spatial sound – the effect of surround sound.

The device will also have productivity-enhancing features, including the ability to serve as an external monitor for a Mac computer. With this feature, users will be able to see their Mac’s display in virtual reality, but still control the computer with a trackpad or mouse and a physical keyboard.

The headset’s operating system, internally called xrOS, will have many of the same features as the iPhone and iPad, but in a 3D environment. It includes the Safari web browser, photos, mail, messages and a calendar application. As well as applications for the company’s services, such as the App Store for installing third-party software, Apple TV+, music and podcasts. The company is also working on health tracking features.

Apple users should experience a familiar environment. When they put on the headset, the main interface will be almost identical to that of the iPhone and iPad, with a home screen with a grid of icons that can be rearranged. Users will be able to pin widgets such as weather, calendar appointments, email and stock indicators to their app icons.

When users need to enter text, they can use the Siri voice assistant or the iPhone, Mac or iPad keyboard. However, unlike the Apple Watch, an iPhone will not be required for the headset to work. The company is developing technology that will allow users to type text in the air with their hands, but such a feature is unlikely to be ready for the initial launch.

Gaming is expected to be a popular offering from third-party developers, and Apple has built its own VR game engine. In 2017, the company released ARKit and other tools to help developers create augmented reality games on the iPhone. This helped pave the way for the creation of apps, games and services for the headset.

Apple’s device will contain a variation of the M2 chip used in the latest Macs, as well as a dedicated processor for graphics and mixed reality. This second chip will be called the Reality Processor, according to trademark filings filed by the tech giant.

But creating powerful enough processors raised another concern: overheating of the device when it’s on the user’s face. To solve this problem, Apple decided to move the battery from inside the headset to an external unit. It lies in the user’s pocket and is connected using a cable. Another feature is the presence of a cooling fan, as in high-end Mac computers.

The headset can work for about two hours from one battery, which corresponds to the level of competing products. The battery, however, is large: about the size of two iPhone 14 Pro Max stacked on top of each other, or about 6 inches tall and over half an inch thick. However, some internal prototypes for software development have a built-in battery and charge via USB Type-C.

Meta, on the other hand, placed the batteries on the back of the headset in a way that helps balance the device on a person’s head. Apple’s approach may prove less user-friendly, especially if they’re watching an entire movie on a headset. Some testers complained that the product can be bulky.

Relatively short battery life – about 20 hours less than the latest MacBook Pro from Apple – can create its own problems. If users want to watch multiple movies or play games for hours, they may need to buy multiple batteries and change them frequently.

Apple has acknowledged these issues and is trying to set realistic expectations for the product. One of the advantages of the device, according to the company, is that it can encourage customers to visit Apple retail stores – not necessarily to buy the product, but to try it out. After that, they can purchase another device, such as an iPad or AirPods.

To showcase the new headset, Apple is creating a “store-within-a-store” concept – an area in its retail outlets dedicated to product demonstrations. The company already did something similar when it released the Apple Watch, which turned into one of its biggest businesses with a turnover of $41 billion.

The initial headset will be made of aluminum, glass and soft ear cushions – and will resemble Apple’s $550 AirPods Max headphones. The product will have a curved screen on the front, speakers on the sides and a headband that helps fix the device on the head.

This will differ from the mostly plastic design of competing products, which tend to attach the device to the user with multiple straps.

The function of tracking the movement of eyes and hands may become the most memorable element of the headset. As with previous big bets, Apple likes to include a revolutionary interface that sets its products apart from the competition. In the case of the iPod, it was the control wheel. iPhone and iPad have a multi-touch approach. And in the Apple Watch, it was the Digital Crown digital scroll wheel.

Now Apple hopes the headset’s sci-fi interface will make its latest product a winner.