Not just for VR: Volumetric video holds huge advantages for traditional video too

At the end of September, Facebook unveiled a volumetric 360-degree camera built in a collaboration with RED Digital Cinema. Known as the Manifold, the new system may be first and foremost a VR camera, but volumetric cameras also hold great promise to traditional video industry.

What is Volumetric Video?

Volumetric video is the process of capturing moving images of the real world, people and objects that can be later viewed from any angle at any moment in time (known as six degrees of freedom).

The volumetric video market is expected to grow from $578.3 million in 2018 to $2.78 billion by 2023, according to a recent market research report.

Most of the media attention surrounding volumetric cameras such as the Manifold has focused on its potential to create more immersive and compelling virtual reality content. When paired with the positional tracking systems within most VR headsets, such as Facebook’s own Oculus Rift and Quest, volumetric video enables viewers to navigate new worlds on their own.

In essence, the viewer gets to be the director, choosing each and every ‘shot’. But that same potential will also give traditional filmmakers new capabilities previously unavailable to them.

“It would be stupid of me to say that (the Manifold) isn’t about VR,” said Director of Photography Andrew Shulkind, who moderated a panel about the new camera at Oculus Connect 5. “It absolutely is, but it’s about more than that.”

What Does that Mean?

The “more than that” mentality is what most appealed to Red Digital Cinema, Facebook’s collaborator on the new camera system.

Rather than volumetric video’s applications for 360 video and VR, Red CTO Uday Mathur said they were most excited by the system’s potential for VFX capture, plates capture, and virtual green screen. In fact, they see that as the camera’s main applications until the VR ecosystem develops further – not the other way around.

“We currently see this fitting alongside a traditional camera to take a whole different type of view of the scene that can be massaged and adjusted in post,” Mathur said. “It takes the obligation of capture and sort of lessens the load of what has to happen on the set. It gives creative flexibility to the post production team.”

The possibility of using the camera to capture immersive content is secondary to its potential for traditional film projects, according to Mathur. “For the traditional cinematographer, it’s a whole new type of tool they can use, while also enabling a new type of experience.”

How It Works

Volumetric video captures the scene from multiple viewpoints and then that information is used to create a depth map of the space. For example, each Manifold can capture every direction from the system with at least four sensors, and seven sensors in most places.

The Manifold features 16 RED Helium 8K sensors, each paired with a custom Schneider 180-degree, 8mm fisheye lens with an aperture of F/4.0.

Like most other 360-degree cameras, the Manifold can stitch these images together to create a 3D, 360-degree video. But these multiple camera angles can be used to generate a depth map – for that, Facebook has partnered with Otoy – and infinite perspectives on the scene.

Once a volumetric video is captured, it could potentially be used far beyond its original purpose: to create additional assets for the intended project but also as a virtual set for a limitless number of other projects.

“The ultimate dream is to plop down two of these cameras somewhere in the jungle, take a few frames, re-light the whole thing later and put some actors in,” said Kate Wurzbacher, Head of Camera at Here Be Dragons. “I think that’s the ultimate dream. We’re at the very beginning of that.”

New Solution, New Problems

For every problem a volumetric video camera like the Manifold solves, it creates new ones. As you can imagine, a setup like this would mean capturing and storing an insane amount of data.

Although the Manifold’s single SMPTE 304M cable (which powers and controls the camera as well as transfers data to a storage rack located up to 100 meters away) nakes data wrangling simpler than multiple cords, multiple SD cards, etc, this setup is by no means simple and easily accessible.

Facebook/Red’s grand plan to solve this problem relies only on time. “That storage will get cheaper over time,” Mathur said. And smaller. But for now, the storage constraints limit the camera’s potential to be used outside of the studio – in my opinion, the exact types of places a camera with these capabilities would shine.

Volumetric video will also present new workflow challenges. Facebook has partnered with post production software companies, including Adobe, CaraVR, Nuke, Foundry, Otoy and others, to build out the workflow for video of this kind. But it’s not only about the tools, but also the people and the process.

For example, it will require more collaboration between post, pre and production teams earlier than usual. And production teams will likely be more deeply involved later in the game to ensure post production teams – equipped with more control than ever before – stay true to the original vision.

“Some positions on the film crew might even be eliminated,” said Facebook’s Engineering Director Brian Cabral. But new positions would be created. For example, since perfect continuity of each and every object is key, Cabral estimates there will one day be a job in which a crew member ensures no one moves anything in the scene.

Cabral realises those applications are ways away. But, as he said, the Manifold is the culmination of the last five years of Facebook’s immersive video efforts, as well as the launching point for the next five years.

He compares the Manifold to his work at Nvidia, developing the first GP GPU (general-purpose computing on graphics processing units). “Now (the GP GPU) is de rigueur,“ he said. “We’re going to start seeing people doing interesting capture – not just for VR… And it will just be de rigueur.”

“Of course, you’ll bring one or two of these to a set,” Cabral continued. “We’ll just capture everything and we’ll make VR assets, we’ll make AR assets, we’ll make 2D assets from it, but let’s just capture it because everything on set is expensive and we’ll figure it out later. That will release a ton of creativity.”

Other Players, Other Approaches

Of course, Facebook isn’t the only company investing in volumetric video. And a system like the Manifold isn’t the only way to do it.

For example, in January of this year, Intel opened a volumetric video studio in Los Angeles.

Its approach to volumetric video comes from the outside. Rather than a single system capturing the scene from the inside out, like Manifold, Intel Studios surrounds the 10,000-square-foot capture area with dozens and dozens of cameras.

The technology is already in use at a handful of football stadiums around the country to capture plays from all angles and recreate 360-degree highlights and player perspective video, but Intel has also partnered with Paramount to investigate the studio’s use for cinema.

However, implementing 38 specifically placed 5K cameras connected to servers processing up to 1TB of data every 15 seconds (as in the case of those stadiums) would still be more prohibitive than Facebook’s inside out volumetric camera (or any inside out volumetric camera), regardless of what Manifold may ultimately cost.

Additionally, outside in volumetric may be ideal for high-quality captures of people and things in a limited space in a highly controlled environment, but inside out volumetric may be optimal – and flexible – for capturing spaces and places outside the studio.

Google also has a hand in the volumetric video game. When light field camera maker Lytro shut its doors earlier this year, Google scooped up plenty of the company’s employees.

Light field cameras record light from every direction through every point and space and use this information to map out a space in 3D. For example, Lytro’s Immerge camera system offered about one meter of free movement for viewers, as well as more realistic texture and lighting.

Although Lytro’s camera technology could also allow creators to change depth of field, focal point, shutter speed and aperture in post, and eliminate the need to use a green screen for visual effects, the company honed in on VR in 2015.

Facebook and Red have made it clear from the very beginning that Manifold is not just a VR camera.

“The beauty of (Manifold) is that it’s the ultimate versatile tool,” Shulkind said at OC5. He envisions using the camera to create deliverables not just for VR headsets but also content specific to viewing experiences, whether the audience is watching on a smartphone or a TV, from an aeroplane or in a theatre.

“(The Manifold) isn’t about VR or AR or even the future of immersive entertainment, but it’s writing the code for the future of content and how we’ll engage in it,” Shulkind said. “I can see a waterfall of deliverables that are all relevant, that make me feel like I’m not missing the shot, each shot in a different way.” All captured by one camera.

Whether or not you believe it, it is a captivating vision.

However, the launch date and price of Manifold have yet to be announced. The camera in its current form comes with plenty of limitations. And such a shake-up in the basic principles of cinematography will take ages to figure out creatively.

Believe me, I realise there is plenty of uncertainty surrounding volumetric video. What I am certain about is that there will be plenty of applications for volumetric video – both within and beyond virtual reality.