NVIDIA has announced the open-source development of its Audio2Face tool, which generates realistic facial expressions and lip movements for 3D avatars based on audio. Developers can now use the model, SDK, and training framework for free to create characters for games, apps, and even live streams.
Audio2Face analyzes the acoustic characteristics of the voice — intonation, phonemes, timbre — and converts them into animation data that is superimposed on a 3D model. This allows for precise synchronization of lip movement and facial expressions both in pre-recorded content and in real time. NVIDIA emphasizes that the technology can work with emotional nuances of speech, making digital characters more "alive".
Developers have already integrated Audio2Face into their projects: in particular, Farm51 used it in the game Chernobylite 2: Exclusion Zone, and the technology will also appear in Alien: Rogue Incursion Evolved Edition.
Along with the models, NVIDIA has also opened up a training framework that allows you to adapt the system to specific scenarios. Plugins for Unreal Engine 5 and Autodesk Maya are also available, simplifying integration.
Thus, the discovery of Audio2Face can significantly speed up the creation of facial animations and reduce the costs of manual animation, opening up new possibilities for games, media, and virtual services.
Read also: Zoom will introduce photorealistic AI avatars for video meetings