Most of this set is CGI, but it was all filmed in-camera in realtime

The seamless melding of virtual sets with real cameras is on the rise, and the results are indistiguishable from reality.

From a low budget filmmakers perspective, the idea that you could place your actors in any location and see totally convincing results not only viewed, but also recorded live in-camera is a dream situation. Yet this very thing has now become reality thanks to Unreal Engine.

The concept is simple. There are four LED display walls, one behind the actor, two to either side, and one representing the sky. There is also lighting provided by Arri Skypanels. So far so simple.

What makes this all work is that the Unreal Engine 'virtual set' is displayed on the three LED panels, but the system also controls the Skypanel lighting as well. Camera movement is tracked in realtime, so that the viewpoint of the virtual set is moved according to its position. When you're sat on set it just looks like the virtual set is changing perspective. But look through the camera viewfinder and you discover that the set viewpoint is being moved with perfect positioning according to camera position. The result is that the background moves in perfect parallax.

Witness the video below to see how it all works. Importantly take note of the introduction sequence. If the video didn't go into detail about how the sequence was created it is doubtful any of us would have twigged that it was shot in a studio under controlled lighting.

The project was put together for this year's Siggraph show, and was a collaboration between Lux Machina, Magnopus, Profile Studios, Quixel, ARRI, and Matt Workman.

If you think about it the idea is incredibly simple, but until now it hasn't been possible to create realtime computer graphics that are so photoreal and have them so convincingly linked to the camera movement. The implications of this are pretty profound, especially as displays get larger. Green screen is often lamented in modern filmmaking, not least because half the time it is quite lazily done and looks no more convincing that the backscreen projections of old. This new system is different because the background is directly interacting in realtime with the camera movement, so it isn't just a flat, prerecorded or rendered piece of footage. Not only that, but it doesn't take long to realise the advantage of having actors effectively 'lit' by the environment, with realtime reflections in objects such as sunglasses and metal, that are painstakingly difficult to recreate with pure post production CGI.

Such live interactions with live cameras have been implimented in feature films already. The recent Lion King animated film not only recreated incredibly realistic looking animals, but it also utilised real world camerawork from Steadicams and dollies. Although the physical set was nothing other than a blank space, the camera operators were looking at monitors that gave them realtime feedback into the virtual world, as if they were actually operating the cameras there.

Unlike CGI camerawork added later, this allows for a much more realistic and organic look, which further reinforces the illusion of reality. In the case of the Unreal Engine system, it is almost like being in a real location, as every crewmember can see the shot live, on an actual set, instead of having to imagine it. Additionally the set can be modified in realtime. For example a rock might be in an inconvenient place for a particular shot, so it can be virtually dragged and dropped somewhere else, or scaled down. The entire environment lighting can be changed instantly too. You could change it to a sunny day, or it means that you could film eternally during golden hour rather than having to fight the sun. So you could for example get your wide and establishing shots in a real location, and then move to a studio for your dialogue shots where you have full control over the lighting and the environment. And it will all look totally convincing at the end of it with no need for expensive and time consuming CGI afterwards. It'll all be recorded in-camera and ready for editing and grading.

It's not an answer to everything 

Such technology has its limitations. If you want to shoot a long walk and talk through a desert for example, you would need a big set with some very big and long screens, as well as some very good set dressing for the 'real' ground that the actors are walking on. For a film with a large budget this may not be so much of an issue, althought it would probably make more sense to go to a real place.

On the other hand such techniques could come into their own when it comes to pickup shots, or some SFX shots such as actors in front of explosions, alien attack etc, or simply just making driving shots much more convincing without the need or complication of putting a car onto a low loader. Nobody is for one moment suggesting that an entire film could be shot using such equipment. But it is yet another string to the bow of filmmaking tools that could eventually make a lot of lives much easier in the right circumstances.