No matter how successful the big effects movies and spectacular sports broadcasts of the last few decades have been, almost nobody becomes involved in film and television because of a desire to spend hours in a room with bright green walls. That might be why virtual production has become so popular. It might also be that it has contributed so visibly to creatively and financially successful productions. In the end, the draw is that it allows people to shoot as they always have, but with results limited only by imagination that previously took months of visual effects work to create.

But like so many tempting things, there are a few practicalities to bear in mind.

Beginnings & Fundamentals

Most new technologies change rapidly after their introduction. Virtual production was perhaps a little less inclined to that than other things; it started off as a way to leverage the sheer power of LED video walls and that is still its core capability. What has changed is the idea that every production will involve tracked cameras, custom 3D graphics, and image-based lighting.

Still, properly displaying an image on a video wall in such a way that it looks right to the camera involves a lot of different disciplines. The camera’s position and lens configuration is often tracked, and any geometric distortion created by that lens must be characterized. A simulated environment must be prepared, which might mean a matte painting, plate photography, or a fully three-dimensional virtual world with all the considerations of art direction and lighting common to CGI. That environment must be rendered in real time, color-corrected, and displayed with geometric corrections based on where the camera is and where it’s looking.

Even if a production is completely comfortable with the visual effects work involved with custom-built virtual spaces, virtual production moves that work from the end of production to the beginning. Firm decisions must be made.

From the camera crew’s perspective, it’s reasonable to think of virtual production as an evolution of back projection. That’s a technique which goes back at least as far as synchronized sound on film, because the synchronization technologies which made The Jazz Singer sing also made it possible to synchronize a projector and a camera. The advantages for camera operators are hard to overstate: not only is it immediately obvious whether the effect is working or not, but the final framing is immediately visible with no reliance on live compositing, and it makes live, multi-camera switching possible with a full preview of what each camera can see.

Sometimes, virtual production can even work too well. When the Wachowskis chose to use back projection to simulate car journeys through The Matrix’s green-tinted faux cityscape, the intention was to create a sense of faint unreality intended to clue the audience into the fact that the characters lived in a faked world. In the end, the effect is so well done that very few people noticed.

Back Projection Becomes Virtual Production

More recent examples of projection cross over with virtual production. Oblivion used craftily front-projected video based mostly on live action plate photography to create a backdrop around a science-fiction habitat atop an impossible spire. Oblivion’s set was a green screen compositor’s nightmare, filled with gauzy, transparent textiles and specular reflections. It was a good model for the average sports or news studio, many of which could be considered sci-fi adjacent and create similar challenges.

The benefits of working with a backdrop that’s actually visible to the camera can hardly be overstated. Things which would usually be anathema to green screen compositing become desirable ways to sell the effect. Smoke, mist, hair, water, and transparent objects integrate perfectly. Reflections – with caveats – are effortless. The current interest in vintage lenses for their softness, aberration and distortion can make green screen hard work, but all of those things actively help a virtual production by blending the real and the not-so-real.

An LED wall has more sheer power than almost any other video display technology which has ever existed, and the light it casts on the scene is another huge boon to convincing composites. Tron: Legacy was possibly among the first to use video wall panels as lighting devices. Gravity, likewise, used the same technique to depict the fall of light on George Clooney and Sandra Bullock. None of those productions put video wall panels in shot; the technology of the time lacked the resolution to create a convincing backdrop.

As well as being powerful, an LED wall is also far less subject to the problems suffered by either front or back projection when extraneous light falls on the projection screen. A projection screen is white, making stray light hard to deal with. Meanwhile, most of the surface of an LED wall is black, so contrast is much higher and problems harder to see.

The Challenges

It is sensible to accept that there are some caveats to deal with. Despite the huge contrast and high brightness of an LED wall, it is not completely immune to the fall of light; a large proportion of the surface is made not of black paint but of reflective plastic LEDs. Certain kinds of reflective surfaces, particularly hard mirrors, can also break the illusion by making it clear that objects displayed on the video wall are much closer than they realistically ought to be. Other boundaries include frame rate limits. Nobody’s shooting material at a thousand frames per second on a virtual production stage, though in other ways the ability to finely control the timing of each frame displayed makes things like live multi-camera production possible.

While that much-vaunted interactive lighting can be enormously effective, the light cast by an LED wall still mostly doesn’t have the color quality we would demand from production lighting tools. With only red, green and blue emitters on most panels, even a white-looking spectrum is made up only of those three colors, and intermediate hues may look dull or undersaturated. Better options have emerged since the dawn of the technique, with specialist companies building video wall panels with white-emitting diodes, as well as processing electronics to create much better color quality - though this inevitably comes at a cost.

At the same time, image-based lighting techniques allow us to control production lighting devices from individual units to arrays of pixel tubes. The result can be convincing, animated, interactive lighting effects with excellent color quality.

Virtual production involves the collaboration of many technologies, and walking in to a virtual production facility will involve spending some time in test and configuration to ensure things look right on the day. It seems likely that integration efforts will hit some fundamental limits: virtual production is rare enough, and complex enough, that we should not expect to see off-the-shelf integrated systems.

Creating content for the video wall, meanwhile, might have been going on for weeks. As we’ll see, collaboration of exactly that kind is key to successful virtual production.

It’s a mistake to think that it involves less work per se. It does involve different work, crucially, at different times; a post production workload becomes a pre production workload, in stark contrast to the extended post production time of a modern effects movie. That’s not a complicated requirement, but it is outside the experience of much of the film industry. The upside is that, in an ideal world, virtual production requires absolutely no special consideration in post production whatsoever.

It does, however, impose an inviolable deadline on the people who are preparing the material which will be displayed on the video wall, whether that’s footage or a 3D world or a hybrid of the two.

Scale & Budget

Whether virtual production seems expensive depends very much on the intent. It’s perhaps safe to say that it can best save money for productions which were otherwise looking at a considerable spend on moving lots of people to lots of international destinations. It can make expensive things cheaper; whether it can make cheap things even cheaper is a more complicated question.

Still, it’s a mistake to assume that every instance of virtual production involves a permanent facility with a vast video wall. Upscale facilities have huge capability, but it’s just as possible to rent a small section of LED wall, the appropriate people and equipment to display images on it, and to wheel it into position in the background of a specific shot. The resulting setup has limited scope, though it enjoys all the advantages of virtual production in terms of handling difficult reflective and transparent subjects and avoids the expense of a big facility.

Stable Fundamentals

All of these things are becoming well-established as standard techniques available to film and TV productions. The idea that all virtual production must necessarily involve every possible technique is changing: many drama productions will not use camera tracking or 3D scene rendering, instead relying on plate photography and playback. That’s not to say that advanced techniques are not possible, and virtual production can still leverage adjacent techniques such as motion capture, 3D object scanning, and almost any of the things done by visual effects artists.

There’s still far too much variation in the way virtual productions are configured for anyone to create a point-by-point how-to - though that variation does create a certain flexibility. Still, the fundamentals are likely to remain, and that’s what we’ll be discussing as the series continues.