The Sponsors Perspective: Capturing Immersive Audio

Strategies for capturing immersive audio for scene and object-based audio.


This article was first published as part of Essential Guide: Immersive Audio Pt 3 - Immersive Audio Objects

For a truly immersive experience, cinematic virtual reality needs spatial sound - 360° spatial audio will make or break the immersive illusion. Just as with any video production, the key to success is correct recording. While traditional microphones still play an important role in virtual reality productions, they need to be augmented with spatial microphones that capture the full 360° ambience.

This is what the Sennheiser AMBEO VR Mic does - a single compact microphone that operates on the Ambisonics principle, allowing you to capture complete spherical audio from a single point in space via its four matched KE 14 condenser capsules in a tetrahedral arrangement. For playback, the audio is rendered binaurally, allowing you to virtually rotate the orientation of the perspective in all directions. Ambisonics is supported by all major post-production and playback tools on the market today. This makes Ambisonics the appropriate tool for Virtual Reality and all other applications involving immersive sound. Basically, you capture exactly what a listener would hear if he or she was standing in that position.

AMBEO on set.

AMBEO on set.

Location Recording

Some care should be taken to record the Ambisonics signal correctly with regard to position and level, as certain errors cannot be corrected during post-production. A field recorder that has an AMBEO VR Mic mode such as the Zoom F8 will help to make this task easier. On location, the AMBEO VR Mic – fitted with the appropriate windshield or hairy cover – should be positioned as close as possible to the 360° camera as you need to patch the microphone out later in post-production, the same goes for the sound bag that accommodates the recorder. The VR Mic will usually be combined with additional conventional spot micrcophones such as wireless lavalier mics. This allows for increased flexibility during post-production, giving the mixing engineer greater control over the final experience.

Mixing

Mixing for cinematic virtual reality can be done in most standard DAWs as long as they support multichannel tracks, i.e. a minimum of four channels in a track. To support you in the mixing process, you should select an Ambisonics tool chain because most deliveries for cinematic virtual reality – including the AMBEO VR Mic recordings – are in Ambisonics. There are many tools to choose from, such as DearReality’s dearVR tool chain or the free Spatial Audio Workstation available from Facebook. As all Ambisonics tool chains operate in B-format, you first need to convert the AMBEO VR Mic’s raw 4-channel output signal to B-format using Sennheiser’s free A-to-B format converter. The converter is available as free download for VST, AU and AAX format for your preferred Digital Audio Workstation for both PC and Mac. B-format is a W, X, Y, Z representation of the sound field around the microphone. W being the sum of all 4 capsules, whereas X, Y and Z are three virtual bi-directional microphone patterns representing front/back, left/right and up/down. Thus, any direction from the microphone can be auditioned by the listener during playback of Ambisonics B.

AMBEO on drumkit.

AMBEO on drumkit.

In mixing, use the recorded Ambisonics signal as the base ambience, then add signals from conventional microphones such as wireless mics and foley sound to emphasize and build your final mix. These additional conventional sound sources need to be spatialized so that they come from the correct point in space and match the video image and the ambience. This step is accomplished by your selected Ambisonics tool chain.

During mixing, it is important to monitor your Ambisonics mix. However, before you are able to listen to it, Ambisonics must be decoded. As cinematic virtual reality will in most cases be delivered over headphones, use a binaural renderer. Best practice is to monitor via the binaural renderer that is used on the platform or device that you will deliver your content to.

It should be noted that Ambisonics is the description of a sound field. Therefore, you should never work on any of the constituent tracks of an Ambisonics signal on its own. Always use Ambisonics mixing and editing tools if you want to modify an Ambisonics signal. When using a standard multichannel mixing tool, you must make sure that changes are applied equally to all four channels – otherwise you risk altering the spatial image of the Ambisonics signal.

Delivery

Ambisonics B-format is the audio format of choice for cinematic virtual reality. All major cinematic VR distribution platforms support Ambisonics B-format, including YouTube and Facebook.

To ensure that a file is viewed properly as a 360 video with 3D immersive audio, every platform requires its own metadata and has its own file format specifications. Please check the documentation of your targeted service. If you plan to distribute to your own app or custom platform, make sure to include support for Ambisonics decoding.

AMBEO end to end workflow.

AMBEO end to end workflow.


Ambisonics In Practice

  • Recording - You can use the AMBEO VR Mic to record full Ambisonics audio, or the AMBEO Smart Headset and the KU 100 dummy head to capture binaural audio. Simple mono microphone recordings or library sounds can also be used but require special encoding.
  • Encoding -  For further processing, the various input formats need to be converted to Ambisonics B-format and rendered binaurally for headphone monitoring. For these purposes, the dearVR AMBI MICRO includes the AMBEO A-to-B and AMBEO Ambisonics-to-binaural conversion libraries. Mono sources can be encoded to Ambisonics with dearVR PRO or dearVR MUSIC.
  • Mixing - dearVR SPATIAL CONNECT enables the user to mix virtual sound sources in VR and to control their position and levels in the dearVR PRO plug-in when in Ambisonics output mode.
  • Integration - By adding dearVR AMBI MICRO to the Ambisonics master bus in the DAW, the user can binaurally monitor the Ambisonics mix with headtracking using a VR headset. This makes for an easy assessment of the Ambisonics track on 360° video platforms or in game engines.
  • Playback - Use any pair of stereo headphones to monitor the binaural soundfield, or use the AMBEO Soundbar, AMBEO Smart Headset or loudspeakers for playback.

Supported by

You might also like...

NAB Show 2024 BEIT Sessions Part 2: New Broadcast Technologies

The most tightly focused and fresh technical information for TV engineers at the NAB Show will be analyzed, discussed, and explained during the four days of BEIT sessions. It’s the best opportunity on Earth to learn from and question i…

Standards: Part 6 - About The ISO 14496 – MPEG-4 Standard

This article describes the various parts of the MPEG-4 standard and discusses how it is much more than a video codec. MPEG-4 describes a sophisticated interactive multimedia platform for deployment on digital TV and the Internet.

The Big Guide To OTT: Part 9 - Quality Of Experience (QoE)

Part 9 of The Big Guide To OTT features a pair of in-depth articles which discuss how a data driven understanding of the consumer experience is vital and how poor quality streaming loses viewers.

Chris Brown Discusses The Themes Of The 2024 NAB Show

The Broadcast Bridge sat down with Chris Brown, executive vice president and managing director, NAB Global Connections and Events to discuss this year’s gathering April 13-17 (show floor open April 14-17) and how the industry looks to the show e…

Essential Guide: Next-Gen 5G Contribution

This Essential Guide explores the technology of 5G and its ongoing roll out. It discusses the technical reasons why 5G has become the new standard in roaming contribution, and explores the potential disruptive impact 5G and MEC could have on…