Two new technologies for delivering realistic and real-time facial capture, retargeting and rendering are at once game-changing and a little frightening.
It may currently feels as though every month brings some new breakthrough in performance capture, facial rigging and character rendering, but two of the latest technologies to surface are especially stunning, and also practical enough to have a near-immediate impact on the field of photoreal human character animation.
The first comes from Cubic Motion, a UK-based facial animation company. Claimed to be the world’s first real-time high fidelity live performance system, the tech is expected to be available for full rollout in two year’s time. Right now, the company is working with games developer Ninja Theory for the creation of the lead character in upcoming title Hellblade: Senua’s Sacrifice and this month’s Games Developer’s Conference in San Francisco saw the two collaborate on a technology showcase for the real-time system.
Utilising Cubic Motion’s proprietary system in conjunction with a facial rig built by 3lateral and rendering via Unreal Engine, the live presentation saw Cubic Motion team member Melina Juergens playing out scenes within a capture station while wearing a custom rig to track facial movements (though noticeably fewer were used for this live demonstration it’s capable of tracking some 200 markers), with the resultant real-time animation immediately mapped onto a real-time rendering of the game’s titular female lead. Lag on the system was shown to be so minimal that the audio performance from Juergens synchronised perfectly with the CG character’s delivery.
"Today we witnessed the reality of a computer generated character driven by the facial expressions of a human being, and to levels of quality, fidelity and efficiency never previously seen before in a live setting, all delivered in real-time," said Andy Wood, Chairman of Cubic Motion. “This might just go down as a little piece of history as it is a paradigm shift in CG production methods and content generation for video games, films and live television.”
Tim Sweeney, CEO of Unreal Engine creator Epic Games was even more forthcoming, proclaiming that “There is no question that what the camera was to the 20th Century, the game engine is today.” At the very least, the implications for game animation creation, previz and CG-animated filmmaking of the kind pioneered by director Bob Zemeckis are enormous.
Equally profound, albeit in a rather more unnerving way, is Face2Face, a software tool developed by researchers at the Max Planck Institute for Informatics, the University of Erlangen-Nuremberg and Stanford University. The intent is simple enough: the real-time capture of one person’s speech and facial movements, for remapping directly onto another person’s face. The results - as demonstrated by a series of performance reenactments utilising existing footage of familiar faces including Donald Trump, Vladimir Putin and Barack Obama - are astonishing.
In fact the sample sequences were created by the software using standard, pre-canned RGB target footage (obtained from Youtube) of the leading political figures, along with an off-the-shelf webcam. Even with such basic video inputs, the software is able to track the original facial performance and deform the deform the original target actor using a “dense photometric consistency measure”. Even the mouth interior is accounted for, by matching the newly retargeted expression with the best fit from the original footage and warping this to fit. The re-rendered face is then blended with the target footage for a seamless blend with the original real-world illumination.
While currently only intended as a research project, there is talk of making it commercially available at some point. At which point, either political satire will get infinitely more entertaining or the potential for criminal and government misuse will be so great that we’re all doomed. Watch this space.