RED and Facebook working together on new advanced camera technology. Who’d have thought it?!

In one of the more surprising announcements we've heard this year, a team up between RED and Facebook has been announced, and things could now become very interesting indeed.

The announcement by RED of its partnership to build a VR camera for Facebook is typically vague. There are no tech specs, no pricing, not even a tentative shipping date. Arguably this hides the real story. For what Facebook along with Microsoft, Apple and Google really want is not VR at all. It is a way to create and display full motion parallax with completely photoreal detail and texture. And that is called a hologram.

I see the tie-up between the maverick camera maker and the social network as a marker on the road to achieving this goal. It’s a zig-zagging route. VR and AR have been unleashed yet consumers have been slow to respond quite simply because the content and the clunky way we experience it are not good enough.

There are fears that the market could wither if the tech giants don’t get it right. Already there have been high-profile casualties with Google going back to the drawing board on Glass, Nokia dropping all development of the Ozo camera, Intel binning the Project Alloy headset then Vaunt AR smart glasses, and Lytro, maker of the Immerge VR camera fading out of business in March.

Content is everything

“There is an enormous amount of content production technology and development in AR and VR but consumer wearable devices are about as compelling as the 3D TVs of ten years ago,” says V. Michael Bove, Principal Research Scientist and Head, Object-Based Media at MIT. “That creates an opportunity for people developing a display tech that provides a good experience but that doesn’t require the wearer to wear anything. There is a belief that if VR hits the market en masse and it’s just not very good it could ruin all possibilities of VR development for another decade.”

Let’s look at this from Facebook’s side. It arguably kick-started the entire recent drive for consumer VR when it paid $2bn for Oculus Rift. Four years on and estimates put sales of the headset at not much more than a million. Facebook CEO Mark Zuckerberg has stated that he wants to get a billion people into VR and to achieve this it has just launched a $199 cut down version of the headset called Oculus Go. Importantly this one doesn’t need to be tethered to a PC or a phone to work but it still won’t solve the fundamental issue which dogs VR/AR of feeling constrained by a device.

RED to the rescue

Facebook will hope its tie-up with RED, which by the way isn’t exclusive, goes some way to improving the other side of the equation - content creation. The early misgivings in 2007 about a company no-one had heard of vowing to revolutionise cinema by inventing a professional quality digital sensor are long gone. The RED One may have been delayed to market but few can doubt that Jim Jannard’s company backs up its rhetoric with serious image science to deliver on the promise.

Although RED does not have a VR rig at present, its cameras are already used for high profile VR work, notably by NextVR which is perhaps the leading VR content creation firm and one primarily focussed on sports and concerts in partnership with NFL and Live Nation. 

It also has strong hooks into Hollywood where filmmakers like David Fincher are converts and RED’s 8K chipped cameras are increasingly being used to acquire for 4K Netflix drama like The End of the Fxxking World and Lost in Space. Low resolution is another bugbear of current VR experiences with 8K experts like Japanese broadcasters like NHK only just now attempting to capture and process 8K images in a VR environment.

Studios themselves are still treating VR as an experiment but the combination of a director like Fincher with RED’s successful track record in innovation could provide new impetus for longer form VR storytelling.

“We’ve been on a quest to build immersive capture technology for years,” said Facebook director of engineering Brian Cabral in an interview with Variety. “We want to enable the best storytellers,” Cabral said.

Cracking the nut

The nut that RED and Facebook have set out to crack – and which has so far eluded the big brains and budgets of companies like Intel – is the viewer’s experience of being able to virtually walk around inside a scene without glitches. In VR terminology this is called six degrees of freedom (6Dof).

To achieve this some form of range finding device is needed to record depth information. Broadly speaking, the more depth information you have at the outset, the higher the fidelity of imagery that can be played back in 3D.

RED and Facebook say they aim to produce a professional VR camera capable of capturing 3D imagery that can then be enhanced with depth reconstruction technology.

With REDCODE, RED’s proprietary compression system, it has the basis of a means to process the data through a set to post production pipeline.

Light fields

What is missing from the basic statement of the two firms is that the best way to achieve this is by capturing a light field, which when it is computationally transformed, can be viewed as a hologram.

If sufficient information about the light field is captured then true full motion parallax is achievable, arguably a step beyond 6DoF, particularly if the holographic video is capable of being displayed without headgear or other unwieldy paraphernalia.

Turns out of course that RED has been developing a means of displaying and possibly capturing holographic content – which it terms 4V or four views – for some time. 

It is on the verge of delivering Hydrogen One, a smartphone device reportedly featuring a screen from LEIA Inc which will show something more advanced than an autostereoscopic image, something perhaps less than a full hologram.

According to RED’s literature, Hydrogen One is designed to fit into the existing modular RED products. There is every reason to suppose that its VR camera, its initial holographic phone display and a RED workflow for processing the 4V (or 3D or VR) media are linked.

It should be noted that while Lytro, which has come closest of all to commercialising light field capture, failed at the attempt, some of its core employees saw the writing on the wall 18 months ago and left to develop a holographic display as Light Field Labs. The remaining light field experts at Lytro have been scooped up by Google which has now taken over the reigns of development in this area.