Blurring the real and the virtual live

The capture of depth information in scenes is an increasingly rich field of development but it has so far remained on the fringes of TV production because of the need to either use physical markers on objects or to render the output in post. Start-up company Zinemath aims to change that with a technology called zLense it claims to be the first real-time 3D depth mapping tool for broadcast.

The zLense houses six sensors in a matte box-style unit attached to the front of a camera to collect information about the depth of objects for each pixel in a frame without need for external tracking sensors or markers. Software then computes the depth (or z axis) alongside camera tracking data and fuses it with the images from the camera. The output is rendered in realtime in an adjacent workstation using multiple GPU cards and SDI IO cards with 4K support to provide the final composite video. The box is light enough to used with handheld or Steadicam.

A studio-based product will launch early 2015 with a starting price of $100,000. A second version is being developed also for release next year and targeting the feature film and location shooting market.

“With this device the gap between the VFX options that a broadcaster has available and those of a film producer will get much smaller,” says Bruno Gyorgy, president and CEO of Zinemath. “TV producers will have the ability to introduce certain VFX in realtime which are not possible without [zLense].”

Examples he cites include the ability to animate a 3D graphic (such as a dolphin - pictured) behind and in front of a presenter or object using just a green screen and lighting effects such as volumetric lighting, global illumination, shadow casting and reflections.

'TV producers will have the ability to introduce certain VFX in realtime which were not possible before' says Bruno Gyorgy

'TV producers will have the ability to introduce certain VFX in realtime which were not possible before' says Bruno Gyorgy

“Current VR technologies are layer based, and cannot provide accurate 3D compositing of the real and virtual worlds,” he says. “Also the lack of full 3D scene information does not allow perfect visual interaction between the virtual and real world, like reflections, shadows and artificial lighting. Also to make real time interactions between the talent and the virtual world, an external motion tracking solution is required.”

Interactions between a presenter and a virtual object can also be achieved without visible markers or a separate tracking system. So a sports presenter could be on the field of play at half time analysing set pieces with virtual graphics rather than being in the studio.

“Installing the rendering engine in an OB or SNG truck can provide real-time augmented reality enhanced broadcast from the field,” he says. “We believe this allows creatives to get certain visual values on the screen which was not done before.”

He suggests unscripted programming such as elements of game shows could benefit from making freer use of graphics to enhance production. Product placement could also be made easier, he suggests.

“You could shoot the show and then get a deal on product placement, or easily insert different branded products into scenes for sale of the programme in different territories.”

While the studio product has a distance of 12 metres, the film version which is primarily intended for pre-visualization, will be optimised for longer range.

“This technology will have a major impact on how films are made by making the creation of certain VFX accessible to smaller budget movies,” claims Gyorgy. “It will allows directors and actors to see in realtime what they are doing in a virtual scene.”

The basic package comes with a Unity3D rendering engine for “cost-effective studio production” but for channels looking for higher performance integration with rendering engines from VizRT and Orad is facilitated.

According to inventor Norbert Komenczi, the focal length of a shot could also be altered in post production. “If you have shot footage with a great depth of field you can shorten it as much you want and put the focus point anywhere.”

In depth

The zLense unit contains two small RGB cameras, an inertial motion unit to record the acceleration of camera movement and three Time Of Flight depth sensors. The latter are developed by Bluetechnix and fire infrared light into a scene to measure the distance between the lens and objects in front of it. According to Komenczi the Time Of Flight sensors do not interfere with the actual lighting setup nor the light hitting the camera sensor. The depth information is recoded by the system in parallel to that of the image which is recorded – in HD, 2K, 4K or beyond – by the camera as normal. Realtime 3D composition can be viewed by a director on a monitor.

Zinemath has its R&D centre in Budapest. Komenczi, a former TV director, initiated development in 2010 and gained investment from Luxembourg's Docler Group in 2012.

Gyorgy is also COO at Docler Entertainment and was a former EVP production services at Budapest’s Korda Studios.

“Depth sensing cameras may well support everyday news and sports broadcasting as a result of their additional dimensions and the simplicity of their use since they do not require a studio environment or any complex calibration,” he added.

“They may allow informative and spectacular presentations as well as a set of graphics applications that bring flat diagrams and explanatory illustrations to life. The border between the real and generated content may be blurred.”

Let us know what you think…

Log-in or Register for free to post comments…

You might also like...

Articles You May Have Missed – August 16, 2017

Virtual Reality (VR). There are basically two camps; you love it or could not care less. The rabid VR fans look for every hit of a new product, technique or movie/clip to try. The reality is that creating VR…

EBU Weighs Up Costs, Challenges and Opportunities of Virtual Reality

The EBU (European Broadcasting Union) has urged the whole video service industry to unite behind Virtual Reality and avoid the mistakes made with previous attempts to deliver new experiences for users, such as 3D.

CES 2017:  The ‘VR’ Reality

CES 2017 has come and gone with most press reporting Virtual Reality landed with a virtual thud. My view of CES is different because I see two very different types of VR.

Sixty Builds Virtual Sets for Norway’s TV2

Norwegian design firm SIXTY conceived all new virtual sets for TV2’s most popular shows using graphics from Vizrt.

Control Upgrade Improves Workflow for Crystal Vision’s Chroma Keyer

​Crystal Vision has further improved the operation of its Safire 3 real-time chroma keyer in a live situation following significant changes to the VisionPanel universal control panel.