All it takes is a short video, and BakedAvatar can replicate the 3D head model of the character from this video. At the same time, accurately capture its facial features, simulate expressions and head movements.
And it can also render the 3D head in real time, edit and adjust the expression and posture of the avatar.
It can be achieved even on ordinary mobile devices.
Here is an example of a specific application scenario to help understand the functionality of this project:
Suppose you are a virtual reality (VR) game developer working on a new game in which characters need to be customized based on the player’s actual appearance. With BakedAvatar, you can:
Avatar creation: Players are provided with a short video from which BakedAvatar creates a highly realistic 3D head model. This model not only accurately captures the player’s facial features, but also simulates their expressions and head movements.
Real-time rendering: In the game, this head model can be rendered in real time, meaning that when players move or change their expressions in the game, their virtual characters will react accordingly in real time.
New perspective compositing: You can view the head from different perspectives, even if the original video was only shot from one angle. This means the software can create a complete 3D head for you to view from any angle.
Multi-device compatibility: Whether players are using a high-performance gaming PC or a mobile device, BakedAvatar ensures that avatars run smoothly across different platforms without latency.
Interactive editing: You can also perform face reenactments, expression editing, and pose editing on 3D characters, enhancing the interactivity and realism of the game. For example, you can modify the expression of the avatar, such as changing the smile to a surprised expression, and these changes will be displayed instantly on the avatar.
All of these features are able to be performed in real time without sacrificing rendering quality, which is important for applications that require rapid response and real-time updates, such as video games, virtual reality, and teleconferencing. In short, BakedAvatar provides a highly flexible and interactive tool for creating and manipulating realistic virtual avatars.
Projects and Demonstrations:https://buaavrcg.github.io/BakedAvatar/
Paper:https://arxiv.org/abs/2311.05521
GitHub:https://github.com/buaavrcg/BakedAvatar