In-camera visual effects ideally allow a camera team to work exactly as it would in a real environment, with the benefits of unlimited space, unrestricted design opportunities, and a short commute between virtual continents. At large-scale live events, where assigned working spaces can be very limited, virtual production can seem particularly relevant, and after set-up, the process can seem effortless. However, that set-up process involves a stack of technologies, and a lot of work that would have traditionally been post-production visual effects moves into pre-production.

Realizing the benefits while keeping the technology out of the way of creativity is mainly a matter of preparation.

Defining The Production

Virtual production can be used to create the world’s most spectacular sports presentation suite, huge sci-fi or fantasy vistas, or help shoot commercials for a dozen couches in a dozen different lounges in less time than ever. The only thing that those productions might have in common is the desire for in-camera visual effects, whether that’s because the environment doesn’t (conveniently) exist; because it must change frequently; because traditional techniques like green screen won’t work; or for the sheer creative convenience of camera operators or gallery staff.

Given such a wide variety of applications, there’s necessarily a lot of variation in the planning process, though as with any production technique the starting point will often be creative. Ideally, the process of designing a studio set or choosing a location is very much de-restricted in a virtual world, although experience suggests that the unlimited potential of virtual production can sometimes mean decisions are hard to make. Advances in techniques have made the build process of a virtual environment faster, although there is still a need for careful management of a process which easily attracts over-revision, especially in committee formats.

Many of the fundamentals arise from where cameras, people and objects will be – the geometry of the studio, including both the real world of the foreground and or the virtual world on the video display. Some of those fundamentals are practical – cameras must avoid photographing the LED wall in such a way that its pixels become visible, which may mean avoiding some combination of deep depth of field, long lenses, or close proximity to the wall.

Much as with the development of a graphics package for sports broadcasting, it can be difficult to anticipate absolutely every possible demand. Late changes are possible, but more complete planning of studio layouts – for pieces to camera, discussion groups, interviews and more – will promote a better, easier, more cost-effective design phase.

Communication

Virtual production specialists consulted in the preparation of this article agreed on many things, but particularly emphasized the need for communication. Virtual production is a multi-disciplinary technique involving disparate technologies, and approaches vary widely between studios. The people involved are very aware of the concerns of camera teams, and equally keen to engage in consultancy before a production has made a firm decision about what to do. Facilities will sometimes donate time and consultancy for testing and planning, and are generally very happy to discuss the right approach in order to avoid future problems.

Studio Geography

It’s easy to think of the LED wall as a background in the same way as a traditional back-projection screen, and at the most basic level that can be perfectly valid. If a production chooses to use entirely two-dimensional background plates, that’s exactly how it will work, albeit with the benefit of the high contrast and brightness of LED display technology. The static nature of the image, though, will necessarily limit the range of camera positions in which the result is convincing.

But where there’s a three-dimensional world beyond the LED wall, it’s more accurate to think of the wall as a window onto that world. From a distance, that might mean almost the same thing as a two-dimensional background plate, but as the taking camera approaches the video wall, more and more of the world beyond will be revealed.

In extremis, the taking camera could reveal almost a 360-degree view of the virtual environment by moving close to the LED wall and viewing it at extreme angles, just as someone might approach a normal window for a wider view of the outside world. Close range and extreme angles create the sort of setup that risks revealing the pixels of the LED wall, but it should be clear that in principle it’s possible to reveal a huge amount of the virtual world outside that virtual window. What’s crucial about this is that it generally won’t be necessary – or practical – to define the entire virtual world in fine detail. Establishing allowable camera positions and general studio geometry lets the virtual art department concentrate its efforts appropriately.

Single-camera drama can sometimes employ advanced techniques to move the scene over huge distances, such as suspending an actor on wires to fly through the virtual world, or using treadmills for walking or running scenes. That can make it possible to create long, continuous walking or running shots over distances limited only by the size of the virtual world. In live studio work, this sort of technique tends to be less relevant, but might still be used with certain caveats. Particularly, that – and other related techniques – can make it easier to avoid having subjects too close to the LED wall.

Keeping The Wall Soft

Many virtual production setups rely on the LED wall being slightly out of focus, which is traditionally easier for larger-sensor cameras than smaller ones. Single-camera drama has often had an easier time with this, and while large-sensor cameras are an increasingly practical option for multi-camera production, lens limitations and focus challenges often make them a less-than-ideal choice for live work.

Ideally, the video wall should be some distance from the live-action subject. Real world subjects, including people, which we want to see in sharp focus, must be far enough away from the LED wall that when they are sharp, the wall is still soft. Wide lenses might reveal the edges of the wall or increase depth of field enough to make pixels or moiré visible, while particularly long lenses might, again, enlarge the wall to the point where pixels become visible. The specifics vary with different setups, but where a particularly huge, particularly high resolution LED wall is used, the need for it to be defocused will be minimized – though not eliminated. With broadcast work often leveraging the ability of virtual production to expand a small space, separation is often limited by practicality

Either way, the wall must be large enough to cover the shot, something that is a particular concern with wide angles, unless there’s an intention to extend the shot with other effects techniques (which will impose other requirements regarding characterization of lenses, and perhaps other factors). Regardless, one of the challenges of blocking and staging for virtual production is to ensure enough physical separation between foreground objects and the LED wall.

Test Sessions

Even before the design of the virtual world is finalized – or before background plates are shot, composited and prepared – a simple test shoot on a virtual production stage is likely to be helpful. Working with a stand-in, alongside the intended camera configuration, setting up shots and evaluating the effects of focal length, camera position, framing and exposure will help establish what’s possible and where the limits are. If the facility is being configured specifically for one production, or where new camera equipment is brought into to an existing facility, this sort of test-and-configuration day can be essential in order to test camera and lenses for tracking, color matching, multi-camera synchronization, and everything else which makes virtual production work.

Where a production has created a new 3D virtual world, a test of that world on the actual rendering servers to be used during the shoot, is a good idea. Lighting of the virtual world must match both the creative intent and the lighting used for the real world, although that will inevitably only be finalized on the day. Tests are also important where the world contains complicated lighting or geometry.

Often, simple changes to virtual worlds can massively increase rendering performance, but it’s clearly preferable to avoid having to make those fixes while a full production waits.

On The Day

Well-planned productions will pre-empt as much of the technical preparation as possible. To arrive on the day of the shoot with a multi-camera OB truck and a virtual world on a hard disk is to invite serious problems. The virtual world data may be many gigabytes in size and must be copied to all of the rendering servers; those servers must then do preparatory pre-rendering work before the real time display can begin. Meanwhile, lenses can be characterized for their distortion, focus and zoom behavior, synchronized, and tracking markers attached to the camera and calibrated for position throughout the working space.

With those tasks complete, it should be possible to move the camera around inside the virtual production stage and view the results, at which point the conventional work of finalizing the staging and lighting can begin. Approaches to lighting for virtual production – whether that’s lighting of the virtual world, or interactive lighting in the real world – are a big subject for another time. Ideally, though, the well-planned virtual production makes most camera techniques work in much the same way they always have, and that’s one of the greatest benefits of working this way.