At its core, virtual production is a development of back projection, and the very value of the technique is that the finished image is captured in-camera. The caveat, at least in comparison to common chroma key, is a loss of flexibility: the same frame cannot be finished in different ways in post production, or with live ad insertion on a sports field where graphics vary for different regions.

At the same time, live multi-camera shoots present a problem: ordinarily, one video wall can only create background images which are correct for one perspective. In principle, a video wall image can be modified to suit the taking camera in real time, displaying the appropriate background image for whichever camera is live from moment to moment. This can work, although each camera operator will not see the final frame while that camera is not live, and the integration of video wall and gallery hardware can be complex.

A video wall can, after all, only display one image at once – but it can display several different images in very quick succession.

Subframe Basics

Subframe techniques take advantage of the fact that cameras do not record light continuously. Usually, they capture the light representing one video frame over a shorter period of time than that frame will be displayed to the audience. At thirty frames per second, the shutter speed is often one sixtieth of a second, so the camera only records light for half the time.

That choice matches the motion rendering of video to that of film at the same frame rate. A mechanical shutter in a film camera closes for half the frame time so that the mechanism has time to advance the film. That has fundamental effects on the amount of motion blur in each frame, and subtly affects the way fast-moving subjects appear.

Therefore, given that most video cameras are only recording light for half the duration of a frame, another camera can be synchronized so that it records in the other half of the time. The two cameras are not genlocked – or synchronized – in the usual way and they do not record frames at exactly the same time. That means the two video signals are not quite simultaneous – one video signal is delayed one half frame against the other – although such a tiny discrepancy is not usually objectionable.

Synchronize changing frames on an LED video wall to the exposure times of the two cameras, and each camera sees a different image while maintaining all the advantages of in-camera virtual production for each one. This will involve coordinating the capabilities of the video wall panels, cameras and synchronizing devices, rendering servers and other aspects of the installation. In person, the image changes so fast that the video wall looks as if all of the images are transparently overlaid.

Productions using more than two cameras can reduce the exposure time below one-half of the frame time, allowing time for more cameras to capture different video wall images. Often, that means exposures of one quarter of the frame period, or 1/120s for a 30-frame production. This allows for four cameras, each of which can see a different image on a single video wall. In theory, even more is possible, although practicalities can intervene.

Applications

While live multi-camera production is a major application for subframe technology, it has many uses. One example involves the tracking systems used to locate cameras in a virtual production studio. Many systems use witness cameras mounted on the taking camera to observe markers placed in the environment. It may be difficult to place markers where they are visible to the witness camera while keeping them out of view of the taking cameras. Subframe displays allow tracking markers to be displayed on the video wall. The witness cameras are synchronized to detect them, while the taking camera does not.

Sometimes, people in front of the camera might need to see cue or positioning markers which should remain invisible to the taking camera. Some cameras might need to see a green screen for live or post-production compositing. With the right choice of camera, it is also possible to derive several different images from a single camera position, allowing different language or commercial content to be shown to different audiences, all with the same foreground subject.

Camera Considerations

Where a subframe display is used to show tracking markers for a single witness camera, the taking camera itself may not need any changes beyond the synchronization measures common to most virtual production stages. Likewise, in the case of a live broadcast using two cameras, both cameras can use conventional settings, other than the required shift in synchronization. The (electronic) shutter is normally open for half the duration of a frame and with two cameras that need not change.

With more cameras, each camera must use a reduced shutter time. Halving the shutter speed allows for four cameras, each recording one-quarter of the frame duration. This must be compensated with either a wider lens aperture or more light, which can become a limiting factor on the practical maximum number of cameras, though limits are also likely to be imposed by the capabilities of the video wall. The reduction in exposure time may also subtly affect motion rendering, especially on sharp, staccato movement, though for many live studio applications this is unlikely to be objectionable.

Meanwhile, creating multiple different images from a single camera position means shooting at a higher frame rate. Where the camera records at 60 frames per second, the video wall can change images on alternate frames, potentially creating two near-simultaneous video images each at thirty frames per second with different video wall images. In this configuration, the camera will record with the longest possible exposure time on each frame and the resulting video will have normal motion rendering.

High-speed cameras can often shoot at four or more times the normal frame rate. In principle, if the video wall can switch images quickly enough, this makes it possible to generate four different video streams, each with effectively the same foreground content but different images on the video wall. This, again, will slightly affect motion rendering and require more light or faster lenses.

Display Considerations

The LED emitters used in video walls can react incredibly quickly. Limits on how many frames a video wall can display are set by the electronics in the panel and processors, as well as the connection between the two.

LED video wall panels are invariably pulse-width modulated, meaning that they simulate changing brightness by switching on and off for variable periods of time. This process generally happens several thousand times per second and there must be enough pulses per video frame for reasonable performance. At the same time, LED video panels are multiplexed, meaning that only one of several rows of LEDs actually illuminates, in repetitive sequence, at any one time. These issues can interact in complex ways and will require case-by-case evaluation to determine the best available performance.

Similarly, subframe displays must generate more image data and transport it to the display. This will place additional load on rendering servers. Links between those servers, the video wall processors, and the panels themselves must be able to handle the workload. Connection via standards such as SDI, SMPTE ST 2110 and DisplayPort is normal, which may impose limits of their own.

Caveats And Future Improvements

The complete configuration of a virtual production system is already a complex task, and subframe techniques inevitably add to that. Usually, a camera engineer will seek to genlock all the cameras on a set so that each frame is photographed at the same instant. Subframe requires something subtly different, although modern camera equipment will mostly offer enough adjustability to make the technique work.

Encouragingly, cameras are increasingly capable of shooting at high frame rate with fewer compromises than ever.  They are often sensitive enough to tolerate reduced shutter timing without demanding the huge apertures which make focus difficult, and without requiring huge amounts of light.

Recent display development has often targeted sheer speed, increasing pulse width and multiplexing rates so that more frames can be displayed with greater fidelity of color and brightness. This involves the most fundamental components of the panel, the chips which directly interface with the LED emitters themselves, as well as the modular receiver in each panel which communicates with the processor. The capability of the overall system will be a matter of sheer bandwidth, settings and firmware, depending on the goal.

The benefits of virtual production remain, and are perhaps most relevant of all to live multi-camera production. Camera operators can work in the full knowledge of what the frame will eventually contain. Gallery staff can see a complete preview of each camera. Everyone on the studio floor retains a full understanding of what will be seen by the audience – and the audience sees a live image with all the benefits of in-camera visual effects.