The best-known virtual productions put people in enormous, spectacular environments full of imaginative production design. The least famous ones might recreate situations as everyday as driving down a street, which has become a particularly valuable application of virtual production in drama. Depending on the application, those environments might be represented by simple video footage, complex 3D renders, or a hybrid of the two. Designs vary, especially between the worlds of drama – where the goal is usually to simulate reality – and broadcast, where virtual production often facilitates deliberately abstract designs, echoing the figurative approach sometimes taken to real-world newsroom sets.

Building The World

The specific usage details of various pieces of software are too huge a subject to cover here, although much of what we discuss will apply to Epic Games’ Unreal Engine, which is commonly used for virtual production, as well as other real time rendering engines like Unity. Most of it also applies to visual effects software which are more usually used for offline rendering.

Creating an environment designed to be viewed through the virtual window of an LED wall is a discipline which sits somewhere between VFX and video game design, especially with respect to the need to design for real time performance. In broadcast applications, the work might be at least as much an expression of graphic design, with input from the people who might otherwise be designing graphics for scoreboards and lower thirds.

Scoping The Project

To the relief of producers everywhere, not every virtual production needs an elaborate three-dimensional environment built from scratch. The right approach will depend on exactly what the production needs to shoot, and that’s a decision best made in consultation with senior creatives, experienced virtual production people, and the facilities that might be involved.

Using a live-action background plate, a single rendered animation, or a hybrid of the two can look convincing from a limited range of angles. Unlike a full 3D environment, using a plate will limit where the taking camera can go, but will be much less work. Hybrid approaches are also possible, where live action material is used as part of a simplified three-dimensional world. Sometimes that’s referred to as “two-and-a-half D,” combining some of the time and work benefits of a background plate with at least some of the freedom of movement as in a fully three-dimensional world.

Since some LED walls can be tens of thousands of pixels across, shooting and compositing adequate plates may not be trivial – though often, the high resolution of the wall is chosen to minimize moiré patterns, and the material need not necessarily match the resolution of the display. Stock footage libraries may be able to provide plates in a ready-to-use (or almost ready-to-use) format, reducing the preparation workload. They tend to concentrate mostly on common applications such as driving or flying environments, though many offer custom work as a service if the proposed environment can realistically be recorded from reality.

These techniques are perhaps less common on live broadcasts, particularly sports and other entertainment productions. These productions may be more likely to pursue virtual production specifically because it can create large, abstract or impossible environments, and multi-camera shooting is likely to reveal the trick of a flat background plate. In that case, at least some 3D graphic design is inevitable, and making that process timely and affordable demands careful planning.

The Construction Process

Either way, many virtual productions will need at least some new 3D assets. Many of the software tools are the same as those used to develop video games and visual effects, ultimately building objects from meshes of triangular polygons and wrapping those meshes in images to define color and texture.

There are many ways to model the shape – the geometry – and assign a material to 3D assets. Many of them will be familiar to anyone with a little experience of modern visual effects. Geometry might be based on manual assembly of simple shapes, scanning real environments with laser scanners or by taking a series of photographs to be interpreted by a computer. Materials are often based on one or more images which might be painted manually, photographed from reality, or a combination of the two. Scanning real objects can be a quick and easy approach, though there are some caveats.

However an asset is created, the virtual production stage must reliably render one animation frame for every frame photographed by the taking camera. The performance of the rendering servers has limits, so assets must be designed for high performance, taking care that exactly the right amount of detail is included. Even though most virtual production facilities use the very best available hardware, there are still limits on the number of polygons, the resolution of textures, and particularly the nature and number of lights in any scene. Skillful artists can optimize all of those things to achieve the best possible combination of performance and complexity.

Material Versus Geometry

Realistic objects are a combination of several things, including realistic geometry, realistic materials, and realistic lighting, as well as animation if there is any. All those things must have enough detail to look convincing to the taking camera, given the way they are used in the virtual environment – where they are, how large they are in frame, how sharply-focused they are, and other factors.

Very often, the geometry of an object can be fairly rough if its materials are good enough. Texturing techniques such as bump mapping and normal mapping can simulate fine surface detail on an object. Bump mapping, for instance, uses a black-and-white image to indicate small variations in the surface of an object, with white areas appearing raised. This can work well enough to reduce the need for fine geometry.

Recent developments in real time ray tracing, which was previously available mainly to non-real time rendering systems, have made more accurate materials and lighting possible, particularly where a material might need to reflect its environment.

Off-The-Shelf Assets

There are already libraries of environments and objects available very affordably for game development and visual effects. Geometry and materials may be packaged in a variety of different formats, often intended for use with different software. Some of those formats can be converted to others, although sometimes some manual intervention will be needed from experienced people.

Until recently, game development has not routinely been able to target photorealism. Certain titles have been able to achieve it under some circumstances, and things change quickly, but many assets built for games often won’t have enough fidelity or even the right art style to work in virtual production. Equally, assets built for visual effects work may look good, but have too much detail to be rendered in real time. Again, qualified people may be able to work on an asset to make it usable, although the practicality of that depends on the specifics of the asset and the intended use.

Lighting

Lighting is key to making things look both realistic and appropriate to the production. Directors of photography are likely to become involved in planning and constructing virtual environments, both because layout will naturally affect blocking and framing, and because the lighting of the virtual world must be compatible with the live action scene.

For much of the history of this technology, the lighting tools available to designers were limited, with restrictions on the type and character of light available and the way that light interacts with objects. For years, objects in virtual worlds didn’t even cast shadows, or when they did, the shadow might not look entirely as it should. Soft light sources – even something as simple as overcast – could often only be approximated as a directionless ambient light, striking every surface in the scene equally. Sometimes, complex lighting might still be pre-calculated, or baked, which may take time after each change to the light, and may mean that a particular light source can interact only with stationary, unchanging objects. Recent innovations in real-time ray tracing are beginning to relax those restrictions, though calculation of multiple fully-interactive soft lights is still intensive.

Virtual Scouting & The Virtual Backlot

Even before a virtual environment has been finished, rough or approximate versions of the layout might be available for rehearsals. The pedigree of virtual production in the world of video games often makes it possible to explore that newly-created environment just like a game, perhaps with a virtual reality headset or on the LED wall itself, in a process that’s sometimes called virtual location scouting. Often, that can take place using a personal laptop or workstation, or using video streamed from a remote location.

Once a virtual production environment has been finalized, its assets can be stored for later use – whether the same production might need to return to a virtual location, or if the same assets might be useful for something else. That might even extend to scanning physical sets for later use in virtual production. Just as there are already commercial libraries of virtual assets, and just as there are already physical backlots and prop stores, anyone owning a particular set of virtual production assets might choose to create a virtual backlot, potentially making more sophisticated environments available to a wider range of productions.

Best Use Of The Virtual Art Department

Productions which might have enthusiastically adopted virtual production are sometimes concerned about the workload of creating a virtual environment. Ever-improving tools and techniques have begun to ease the workload very slightly, although specialists would agree that where a production needs a full, new, custom-made environment, lots of lead time and early decisions help control costs and ensure better results, much as when creating a package of graphics. The hard deadline of a live broadcast can be particularly motivating here. Several of the subjects touched on here affect both the art and the science of virtual production, and we’ll explore them in depth in upcoming chapters.