Tracking brain activity and heartbeat: one of the developers of Apple Vision Pro talked about the creation of the headset
Renowned developer Sterling Crispin shared his experience working on the Apple Vision Pro augmented and virtual reality (AR/VR) headset. He spent 10% of his life on this project. This is what the developer wrote on Twitter.
Much of his work at Apple is protected by a non-disclosure agreement, he said. But there are also things that have become public property thanks to patents. For example, much of his work at Apple involved detecting the mental state of users based on data from their bodies and brains when they were in an immersive mode.
“So, a user is in a mixed reality or virtual reality experience, and AI models are trying to predict if you are feeling curious, mind wandering, scared, paying attention, remembering a past experience, or some other cognitive state. And these may be inferred through measurements like eye tracking, electrical activity in the brain, heart beats and rhythms, muscle activity, blood density in the brain, blood pressure, skin conductance etc.” he explained.
According to him, a lot of tricks were used to make specific predictions possible. They are described in several patents that mention the developer. For example, one of the coolest results was predicting when a user is going to click on something before they do.
“Your pupil reacts before you click in part because you expect something will happen after you click. So you can create biofeedback with a user’s brain by monitoring their eye behavior, and redesigning the UI in real time to create more of this anticipatory pupil response,” he shared.
Also, to determine the cognitive state, the users are quickly shown the visual or sound signals that they could not perceive.
“Another patent goes into details about using machine learning and signals from the body and brain to predict how focused, or relaxed you are, or how well you are learning. And then updating virtual environments to enhance those states. So, imagine an adaptive immersive environment that helps you learn, or work, or relax by changing what you’re seeing and hearing in the background,” said Sterling Crispin.
We remind you that while working on the headset, Apple engineers created 5000 new patents. The main focus of Apple Vision Pro is not so much to immerse the user in virtual reality, but to expand its possibilities by combining the real world with the digital one.
Sterling Crispin is best known as the creator of Cyber Paint, an app where users can create drawings in virtual reality. In 2018, he started working at Apple.