With cameras naturally at the center of film and TV production work, a big priority for visual effects techniques is to create spectacular results without imposing too much technology on the camera department. At the same time, if there’s a single, overwhelming advantage of virtual production, it’s exactly that: the people behind the camera should be able to create advanced effects without changing their working practices.

Some of the most interesting techniques, however, mean real-time rendering of three-dimensional environments, and displaying them on the video wall with correct perspective for the current camera position. To do that, the system needs to know where the camera is and how the lens is configured. Tracking that information might mean attaching things to the camera, and virtual production facilities must make everything work without compromising the ease and convenience that virtual production can and should give us.

When There’s No Need To Track

There are many different ways to track a camera, although one of the fundamental truths of virtual production is that not every setup actually requires it. In some situations, the phrase “virtual production” can cross over with “back projection” (or, similarly, front projection). Using video as part of a scene, whether projected or displayed on an LED wall, ably handles most types of reflection, the fine details of smoke and hair, and other subjects that green screen may handle less well. It also avoids the considerable work of a 3D environment build. Footage libraries have sprung up to offer a huge variety of driving environments, for instance, which has turned out to be a huge time-saver in comparison to a halting journey through a congested modern city. Services have sprung up – often allied with the libraries – offering photography with array cameras to create custom plates representing any environment.

There are limits to how simple, two-dimensional backgrounds can be shot. Usually, the camera will need to be broadly in front of the video display. Some types of camera motion will reveal the trick, as will specific types of reflection (though that’s true for fully three-dimensional, tracked-camera configurations, too). Material to be shown on the video display will need to have been shot in a manner that’s suitable for the intended purpose, which can be complicated, and considerations such as color correction and synchronization still need to be right.

Tracking Technologies

Sometimes, virtual productions will need the flexibility in camera position that’s only provided by a fully three-dimensional scene, or even a partially three-dimensional scene, perhaps based on modified live-action footage, sometimes termed 2.5D. This is perhaps most common in live multi-camera applications, where the need to show a single scene from different angles would quickly reveal the trick of a 2D or 2.5D background image.

Figuring out where something is in three-dimensional space is something that visual effects and games development people have been doing for decades. It even appears in consumer technology, with virtual reality headsets capable of tracking their own position so that the wearer’s point of view can appear to move through a virtual world. Sometimes, those devices can even be pressed into service as virtual production tools.

There are other ways of knowing the position and rotation of the camera, particularly by mounting it on a motion control crane which will put it in a known position to begin with. Tracking the camera, meanwhile, allows for the use of any conventional camera support equipment. The image of an actor performing in a suit covered in reflective tracking markers is familiar to most people, and similar approaches work for cameras.

Probably the most commonly-encountered approach to any sort of motion tracking is for an array of cameras mounted on the taking camera to observe markers in the environment, called inside-out tracking. Alternatively – and equivalently – cameras in the environment might observe markers on the taking camera, termed outside-in tracking. Some designs are now capable of working in any environment which offers sufficient static, visible, high-contrast subjects to track in the environment. This requires less setup, but is sometimes less straightforward than it might seem, as much of the environment of a virtual production stage will be made up of the video wall itself.

One solution to that involves displaying tracking patterns on the actual video wall, timed to appear only when the taking camera’s shutter is closed – sometimes called subframe techniques. It can interfere with other applications of inter-frame time, especially where there is a need to support more than one camera by displaying the appropriate virtual background for each camera position in quick succession. Even so, the options to use sub-frame timing to display tracking data, alternate backgrounds, or even just a greenscreen creates a lot of opportunities for innovation and requires no physical setup in the studio space.

We will explore the development of subframe technology in more detail in a later article in this series.

Established Technology

Aside from subframe techniques, tracking technology is not unique to virtual production, and many systems are equally suitable for virtual studio, virtual production, and other 3D graphics work. Inside-out tracking has often been used in broadcast news studios, allowing virtual objects to appear in a real space.

Regardless of the configuration, material from the witness cameras will be routed to a computer that’s responsible for interpreting their images and calculating positions, which will then pass that position data on to the rendering system. Many purpose-built motion tracking cameras use Ethernet computer networking, to support a large number of cameras with minimal cabling; the image sent is sometimes nothing more than a cluster of black pixels representing markers on a white field. Cameras may have infra-red lighting distributed around the lens – a ring light – and an infra-red pass filter. That creates a clear reflection from markers with a retro-reflective coating, much as a car’s headlights illuminate a road sign at night.

The more cameras which can see a marker, the better the track. Common systems can locate a marker in a space tens of meters across with a precision of less than a millimeter. 

Lens Data

Knowing where the camera is and where it’s pointing is only part of the information a rendering system needs to draw a three-dimensional screen and display it in a way that it looks correct to the taking camera. It’s instinctive that a wider-angle lens, for instance, will mean displaying an image over a wider area of the wall. A zoom lens might mean that changes over time.

Similarly, the focus position of a lens, as well as its aperture setting, will change the way out-of-focus areas of the image are rendered depending on their distance from the camera. The rendering system must simulate that falloff of focus in the virtual world. All that data might be available from inbuilt encoding systems on some lenses, or from add-on devices such as a remote follow focus. Technical provisions for retrieving that information from the camera and passing it to the tracking system vary, although it has been common for this information to be used for conventional visual effects for some time.

Lens Geometry

Most people have seen an image produced by a fisheye lens, where straight lines appear bowed. While that’s most associated with very wide angle lenses, most have at least some geometric distortion, described using terms such as spherical, barrel or pincushion distortion. Very few (formally, perhaps absolutely no) lenses are entirely rectilinear, or without distortion. That can sometimes be demonstrated by lining up a horizontal or vertical line with the edge of a monitor.

As with many types of visual effects photography, lens geometry can be compensated by shooting test charts, which might mean a chequerboard or grid pattern, allowing a computer to characterize the lens’s distortion and correct for it. While this is often effective, different software has different capabilities, and different lenses may create complex distortions that, while subtle, can make alignment difficult. Lens designers often try to correct for these distortions, and the residual distortion left after those corrections can be more complex than just continuous curves. That’s especially true for anamorphic lenses, which will have different characteristics in the horizontal and vertical dimensions. Some lenses may exhibit geometric distortion which changes as the focus is altered.

Some lenses, in some circumstances, may have characteristics so complex that they’re beyond the ability of the software to correct, potentially creating hard-to-solve alignment problems. Using those lenses in virtual production might be difficult. In general, though, one of the biggest advantages of virtual production is that vintage or unusual lenses with interesting optical characteristics are handled seamlessly. Unusual flares, glow, corner softness, or even optical filtration are a great way to tie the real and virtual parts of the scene together. So, where it’s possible to take the time in prep to make difficult lenses work, there are benefits for both the effects and camera teams.