Broadcast And The Metaverse - Part 1

From the earliest public radio transmissions in the 1920s to 4K television of today, broadcasters have been working to continually deliver and improve the immersive experience. Television broadcasting has gone from black and white, to color, then HD and 4K, with massive improvements in audio all building on previous technologies to encourage viewer engagement and get closer to the event. The last hundred years, with all the technological advances in television, has taught broadcasters that viewers always want more.


This article was first published as part of Essential Guide: Broadcast And The Metaverse - download the complete Essential Guide HERE.

Now, a new generation of ultra-high-power parallel compute and storage resource is forming the infrastructure to create the Metaverse that will deliver the ultimate in immersive and viewer experience. The Metaverse is rapidly driving adoption by bringing into existence a 3D overlay of the Internet to create a natural human interface that will provide an experience near indistinguishable from the real world.

The Metaverse end game may seem like a futuristic utopia, however, large parts of the enabling technology are actively being used today. And by adopting this technology, broadcasters will be empowered to develop the immersive experience that their viewers are continually demanding, thus driving greater viewing figures and potential growth.

Using todays Metaverse technology will bring viewers closer together for sports, concerts, and public events, and will not only create a greatly improved immersive experience, but also empower community engagement. This technology is significantly more than just enhanced VFX as ML, 3D animation, selective viewports, and avatars, to name but a few, have come together to allow broadcasters to massively improve their live programs and deliver the experience their viewers are demanding.

Virtual Worlds

One of the benefits of digitization of video tape into on-prem and cloud storage is that it has become much easier to process for AI applications. The data is parsed using convolution neural networks that learn aspects of the image to allow them to be classified, such as trees, cars, and highways. Which in turn provides the opportunity for image synthesis through AI technologies including generative adversarial networks (GANs).

In traditional broadcast workflows using green screen or LED wall technology, the designer has had to build the background images using recorded video or digitally generated images. This is incredibly expensive and time consuming as either a crew has to go out and record the GVs, which will need to be edited, or a graphic designer has to painstakingly create a background sequence frame by frame.

Contrast this to an AI method using virtual world technology consisting of high-end GPUs, servers and storage. Instead of creating a complete background, the graphic designer might create a viewport layered mask. For example, a scene with a car driving down the road might have one mask that outlines the road, another mask for the cars, another mask for the trees, and another mask for the building. The graphic designer would then parse the mask sequence through an AI image synthesizer that would replace the masks with the images it has created. This is far more than just replacing the masks with pictures of cars or trees as the images it synthesizes will be based on a library of vector mapped images that exist in 3D. Consequently, the images are represented as vector maps and only become visible at the point of image rendering, that is, when it is turned into a 2D image.

Due to the vector mapped nature of this technology, real-time viewport orientation can be applied. This allows the designer to move the visible rendered image so that it can be anywhere within the scene. This is further enhanced as the vector representation includes depth so that the viewport can be moved into and out of the image as well as around it. All in real time. Not only does this provide incredible 2D rendered images for the home viewer, but also has the potential to allow the viewer to enter into the scene, especially when using headsets, thus greatly improving the viewer immersive experience.

Figure 1 – Vector representation abstracts the image data from the display so that the scene is not dependent on the display (as is the case with raster images). The process of rendering the data will provide an image that can be displayed from the correct viewpoint and optimized for the target display.

Figure 1 – Vector representation abstracts the image data from the display so that the scene is not dependent on the display (as is the case with raster images). The process of rendering the data will provide an image that can be displayed from the correct viewpoint and optimized for the target display.

Broadcasting Future Virtual Worlds

Although the technology may be available now to help broadcasters create virtualized LED walls, the future offers many more interesting opportunities when considering how the metaverse is predicted to progress, especially when considering improving the immersive experience.

The metaverse can be thought of as adding a 3D layer to the 2D internet which in turn will allow viewers to enter the virtualized world that has been created. In broadcasting terms this could be a sports stadium where viewers swap seats and move from their homes to being in the stadium. Viewers then cease to be passive observers and instead become immersed in the game itself, allowing them to explore and experience the event almost at firsthand.

One of the key challenges for broadcasters is how do they take advantage of the future potential of the immersion that the metaverse promises?

In the same way that Google Maps have vehicles traversing roads around the world to provide street views, will broadcasters do the same with stadiums and venues? Thus, allowing them to build a massive library of images that can be used to form the virtualized worlds. Viewers could even choose their own seat in the stadium, go “backstage” to see how the game is being planned and the team strategies being adopted, and even join their friends to increase their enjoyment.

Once the historical limitations of thinking in a single viewport linear-2D environment have been removed, the true potential of moving and exploring within an interactive 3D virtual world becomes clear. And with this comes massive revenue opportunities for broadcasters.

Supported by

You might also like...

NAB Show 2024 BEIT Sessions Part 2: New Broadcast Technologies

The most tightly focused and fresh technical information for TV engineers at the NAB Show will be analyzed, discussed, and explained during the four days of BEIT sessions. It’s the best opportunity on Earth to learn from and question i…

Standards: Part 6 - About The ISO 14496 – MPEG-4 Standard

This article describes the various parts of the MPEG-4 standard and discusses how it is much more than a video codec. MPEG-4 describes a sophisticated interactive multimedia platform for deployment on digital TV and the Internet.

Chris Brown Discusses The Themes Of The 2024 NAB Show

The Broadcast Bridge sat down with Chris Brown, executive vice president and managing director, NAB Global Connections and Events to discuss this year’s gathering April 13-17 (show floor open April 14-17) and how the industry looks to the show e…

Comms In Hybrid SDI - IP - Cloud Systems - Part 1

We examine the demands placed on hybrid, distributed comms systems and the practical requirements for connectivity, transport and functionality.

Audio For Broadcast - The Book

​Audio For Broadcast - The Book gathers together 16 articles into a 78 page eBook which explores the science and practical applications of audio in broadcast.  This book is not aimed at audio A1’s, it is intended as a reference resource for …