Shooting on a virtual set used to be hard. Until now

Ncam's virtual set previews continue to impress. And nowhere are its capabilities better shown off than on Disney's new adaption of The Nutcracker.

No matter how much money you've got, and Disney has a lot of money, there's a limit to how huge a set you can build. And if you're going to produce a film based on The Nutcracker, which takes place in a land made entirely of fairytale tropes – castles, bottomless gorges, spooky forests – you're going to need more set than even the best-appointed movie can possibly provide.

Scene as shot

Solutions to this aren't new, but CG hasn't always been easy to use. “Get out of here,” says Qui-Gon Jinn, staring at a point just above Jar-Jar Binks's head. His sentiment is understandable. His eye-line isn't. Still, that was 20 years ago, and realtime on-set previews have been possible for a while. Even so, as late as Avatar, a lot of custom software engineering was needed to give James Cameron an immediate idea of where things would be in the frame.

With on-set preview

More recent projects have begun to use more off-the-shelf systems as the technology matures. The underlying approach is more or less the same as virtual sets for live production, relying on various combinations of camera tracking and lens encoding techniques to provide a computer with enough information about which direction we're supposed to be pointing to and what's supposed to be in the frame. Anyone who's been in a news studio and seen a bunch of circular barcodes stuck to the ceiling will be familiar with one of these techniques; others rely on tracking markers in the frame or scattered around. Varying degrees of surveying and preparation are often needed, as well as mapping out lenses for focus and distortion behaviour.

Final shot

The Nutcracker and the Four Realms, to give the production its full title, used Ncam's camera tracking system – visible on the photos as the camera bar slung under the matte box rods on DP Linus Sandgren's Panaflex, though there's more to it than just the optical tracking. And yes, a Panaflex. The Nutcracker was shot on film, meaning that on-set previews were based on video tap images. Ncam's system is designed to minimise the amount of setup required, especially in terms of formally surveying the studio, but it has a few other tricks up its sleeve which some people may have encountered at trade shows over the last year or two.

Ncam stereoscopic cameras

Perhaps the most immediately engaging option is what we might call an environment camera, designed to shoot a very wide-angle, almost spherical view of the surrounding world that can be used to simulate reflections and lighting on the surface of rendered objects. That wasn't used for Nutcracker, but it will probably find application in live sports studios and news broadcasting. On one occasion, it was demonstrated with a video wall displaying coloured patterns of light which were accurately simulated in illumination of the virtual object, and it's possible for a presenter to walk up to the virtual object and be reflected in it. There are always issues of delay in a system like this, with at least a frame or so required to get the mathematics done, but these seem well controlled in Ncam's system.

As shot

The most visible part of the system, that two-camera setup attached to the rods, is easily recognisable as a stereoscopic pair. That gives the system some knowledge of how far away objects are in a scene – up to a certain distance limit, presumably – though the cameras are much further apart than human eyes, so that distance will be reasonably long. That sort of depth sensing is an imperfect technology that generally produces fairly noisy data, which is why focus pullers haven't been completely replaced by systems based on depth cameras, but the Ncam system is good enough to allow real world people to walk around behind and in front of virtual objects.

Preview shot

The application of all this to live events is clear. With a feature film, it's a convenience on the set, but with Nutcracker the roughly composited shots were used in the edit and for early test versions of the film, taking out a lot of donkey work. As the photos show, a lot of it was set extension and the sort of backdrops that have been reasonably common for years, but even that's not simple if the camera crew has accidentally cropped out the tallest spire on the Disney princess's castle because it wasn't going to be rendered until six weeks later.

With Panaflex

And – though this doesn't directly involve Ncam – notice that the production went at least sometimes for blue, rather than the more common green backdrops. This was often the approach for film productions, though modern compositing systems are capable of doing a workable job on almost anything. Perhaps the choice was provoked by the colour of the background that would eventually be there, or the need to avoid matching a costume colour.