Standards: Part 18 - High Efficiency And Other Advanced Audio Codecs
Our series on Standards moves on to discussion of advancements in AAC coding, alternative coders for special case scenarios, and their management within a consistent framework.
This article is part of our growing series on Standards.
There is an overview of all 26 articles in Part 1 - An Introduction To Standards.
The previous two articles addressed the evolution of MPEG audio. Starting with MPEG-1 and MPEG-2 for layers 1 to 3. Later, MPEG-2 introduced AAC as a better but non-backwards compatible alternative in Part 7. MPEG-4 improves on the coding and MPEG-D adds a hybrid codec for speech and music.
MPEG-4 Part 3
Where earlier MPEG Audio standards concentrated on general audio coding, MPEG-4 covers a much wider range of target applications. It manages a collection of versatile codecs through a unified interface. The new codecs are sometimes a more efficient choice than AAC:
Category | Codecs |
---|---|
Lossy speech coding | HVXC and CELP. |
General audio coding | AAC, TwinVQ and BSAC. |
Lossless coding | MPEG-4 SLS, Audio Lossless Coding, MPEG-4 DST. |
Text to speech | TTSI. |
Structured Audio | SAOL, SASL and MIDI. |
Audio Synthesis | Wavetable based, Sample based, Algorithmic and Effects. |
Building your own custom player for these special codecs might incur a patent licensing liability.
High Efficiency AAC Coding (HE-AAC/AAC+)
High Efficiency AAC (HE-AAC) refines the coding techniques but remains compatible with MPEG-2 Part 7. It can deliver almost CD-quality sound at 32 Kbps.
HE-AAC is known by a variety of other names:
Alias | Canonical name |
---|---|
AAC+ | HE-AAC v1 |
aacPlus | HE-AAC v1 |
aacPlus v2 | HE-AAC v2 |
eAAC+ | HE-AAC v2 |
These are some important features:
- Spectral Band Replication (SBR).
- Parametric Stereo (SSC).
- Perceptual Noise Substitution (PNS).
- Long Term Predictor (LTP).
- Low Delay (AAC-LD).
- MPEG-4 Scalable to Lossless (SLS).
- AAC Scalable Sample Rate (SSR).
- Structured Audio (SA).
- Text To Speech (TTSI).
- Error Resilience (ER).
Structured Audio and text to speech are extremely compact because they describe a sound that the player renders entirely in the receiving client. This can deliver performance as low as 100bps.
There are a many historical versions with profiles adding more variants:
Version | Description |
---|---|
AAC (Original) | Described in ISO 13818-7:1997. |
AAC (Version 1) | Described in ISO 14496-3:1999. |
AAC (Version 2) | Described in the ISO 14496-3:2000 revision. |
AAC (Current) | Described in ISO 14496-3:2009. |
AAC+ (Version 2) | Version 2 of aacPlus is described in ETSI TS 102 005:2010. |
HE-AAC v1 (AAC+) Profile | Described in the ISO 14496-3:2001 revision. Combines AAC LC with Spectral Band Replication (SBR). |
HE-AAC v2 (aacPlus) Profile | Described in the ISO 14496-3:2005 Revision. Adds Parametric Stereo (PS) to the version 1 features to achieve lower bitrates. Sometimes described as eAAC+. |
xHE-AAC | Fraunhofer introduced Loudness Control and adaptive streaming around 2016 (See MPEG-DASH). Well supported by many players including iOS and Android. |
Extended HE-AAC | ISO 23003-3:2020 add USAC coding to HE-AAC v2, extending the tool set. |
Tools & Technologies
The coding tools were reorganized in MPEG-4 to make them more flexible. New tools were added and older tools were refactored into separate items. They are now described as Audio Objects, each one having a specific identity and purpose. Some of them are containers for descriptive information. This is an additional layer of abstraction facilitating profile definitions.
These are the basic audio objects and the technologies that are used inside them. Refer to the MPEG-4 Part 3 sub-parts for functional descriptions of these tools and how they are mapped into the profiles. The sub-part references are all in the MPEG-4 Part 3 standard:
Terminology | Description |
---|---|
AAC Main | Based on AAC LC. |
AAC LC | The Low Complexity Audio Object combines the MPEG-2 Part 7 Low Complexity profile (LC) with Perceptual Noise Substitution (PNS). See sub-part 4. |
AAC SSR | Scalable Sample Rate is based on the MPEG-2 Part 7 Scalable Sampling Rate profile (SSR) combined with Perceptual Noise Substitution (PNS). See sub-part 4. |
AAC LTP | Long Term Prediction introduces a forward predictor with lower computational complexity. Also uses AAC LC. |
AAC LD | Low Delay, used with CELP, HVXC, and TTSI in the Low Delay Profile. Suitable for real-time conversation applications. |
AAC ELD | Enhanced Low Delay improves the bitrate and latency at the expense of a small increase in computational workload. |
SBR | Spectral Band Replication used with AAC LC in the HE-AAC Profile version 1. |
TwinVQ | Transform-domain Weighted Interleave Vector Quantization is designed for coding audio at extremely low bitrates (8 kbps). See sub-part 4. |
CELP | Speech coding with Code Excited Linear Prediction operates at low bitrates. TwinVQ may be more efficient. Not suited for use with music. See sub-part 3. |
HVXC | Speech coding with Harmonic Vector eXcitation Coding works well with low sample rates around 8 kHz delivering coded output at 1.6 kbps. Latency is very low making it suitable for telephony applications. See sub-part 2. |
SSC | SinuSoidal Coding. The technical underpinnings of Parametric Stereo coding for high quality audio. See sub-part 8. |
PS | Parametric Stereo used with AAC LC and SBR in the HE-AAC v2 Profile. The implementation uses SinuSoidal Coding (SSC). Stereo audio is coded as a monaural channel with two differential channels for the left and right signals. See sub-part 8. |
MP1, MP2, MP3 | MPEG-1/MPEG-2 Audio Layer 1,2 & 3 in MPEG-4 See sub-part 9. |
USAC | Unified Speech and Audio Coding switches the coding strategy mid-stream between low bitrate CELP (for speech) and HE-AAC (for music) as it determines which is more efficient for each segment. See ISO 23003-3. |
BSAC | Bit Sliced Arithmetic Coding is an alternative scalable noiseless coding mechanism providing almost perfect quality at 64 kbps. Used for Digital Media Broadcasting (DMB) services. See sub-part 4. |
HILN | Parametric audio coding with Harmonic and Individual Line plus Noise. Sound can be coded as various harmonics of a sine wave plus a noise component described as a spectral envelope. See sub-part 7. |
PNS | Perceptual Noise Substitution improves efficiency by representing noise-like signal components with a parametric representation instead of coding the exact waveform. The decoder synthesizes the noise component based on the description. |
DST | Lossless coding of oversampled audio with Direct Stream Transfer. Popularized by Super Audio CDs. See sub-part 10. |
ALS | Audio Lossless Coding uses short and long-term predictors to encode sounds that are rich in harmonics. See sub-part 11. |
SLS | Scalable Lossless Coding is based on a layered approach which implements a lossy coding component in AAC with an additional correction layer that enhances it to provide the lossless result. SLS and ALS are not related to one another. See sub-part 12. |
SLS non-core | A lossless audio coder with a single coding stream without the lossy General Audio base layer. |
MPEG Surround | Also known as MPEG Spatial Audio Coding (SAC). Not the same as SAOC. |
SAOC | Spatial Audio Object Coding. See ISO 23003-2. |
SAOC-DE | Spatial Audio Object Coding Dialogue Enhancement. |
LD MPEG Surround | Low Delay MPEG Surround coding. The side channel information is described in ISO 23003-2. |
Audio Sync | Audio synchronization maintains the coherence of multiple content streams in multiple devices. See sub-part 13. |
TTSI | Text to Speech Interface that synthesizes the audio. See sub-part 6. |
SA | Structured Audio describes the audio as components or algorithms. The top level is a scheduler for controlling the construction and playback. See sub-part 5. |
Wavetable synthesis | Uses combinations of waveforms to create virtual instrument sounds. |
Sample based synthesis | Sampled natural sound fragments are combined and mixed to create a track. Based on SoundFont technologies. |
Algorithmic synthesis | Converts a description of a sound and how to play it into a compiled source code format (such as C Language). Then an application can be created to generate the sound. |
Audio effects | Part of the structured audio toolset. |
SMR Simple | Simplified version of Symbolic Music Representation. See ISO 14496-23. |
SMR Main | Main version of Symbolic Music Representation. See ISO 14496-23. |
SAOL | Structured Audio Orchestra Language. Derived from the earlier MUSIC-N language. |
SASL | Structured Audio Score Language. |
SASBF | Structured Audio Sample Bank Format. |
MIDI | Musical Instrument Digital Interface describes sound (predominantly music based) as a series of events (notes), sounds (patches) and modulations (controls). |
General MIDI | A standard set of sounds defined by Roland Corp to provide instrument sound (patch) compatibility across multiple MIDI devices. |
DLS | Downloadable Sounds standardized digital musical instrument sound banks which can be used with data driven sound tracks such as MIDI or SAOL. |
Spectral Band Replication (SBR)
Spectral Band Replication discards redundant harmonic components in the encoder but reconstructs them by replicating the lower frequencies to derive suitable replacements in the player. This can be used with any codec.
A typical stream of audio might be coded to a target bitrate of around 128kbps. This would reproduce all frequencies up to 15kHz with a small reduction in the frequency response at the top end.
SBR cuts off the incoming frequencies at around 7.5kHz. This loses a lot of the detail but reduces the bitrate to 64kbps.
Then the higher band from 7.5kHz to 15kHz is processed through a more aggressive compression tool. This generates a description of the high frequency sounds that can be used in the decoder to reconstruct them from the lower order harmonics. The description is carried in auxiliary segments within the stream and only adds 1.5kbps to the bitstream (65.5 kbps in total).
The player transposes the lower frequencies into the upper band where it can filter and mix them in using the descriptions in the auxiliary segments.
Parametric Stereo (PS) & SinuSoidal Coding (SSC)
Parametric stereo exploits the similarity between the left and right channels to code them more efficiently.
The two channels are mixed down into a single monophonic channel and coded at full resolution. This is a base from which two differential channels can be derived. Those differences can be coded to a 3-kbps bitrate using SinuSoidal Coding.
The player decodes the mono channel and applies the differences to make the left and right outputs.
Instead of delivering two full bitrate channels, the encoder delivers 1 full bitrate channel and two very low bitrate differentials.
Perceptual Noise Substitution (PNS)
The bitrate gains from using PNS are often not worth the computational workload when the audio is of a high quality.
For noisy audio sources, the noise can be filtered out and described as control parameters for a pseudorandom noise generator in the player where they can be recreated.
Scalable Sample Rate (SSR)
Scaling the audio coding by splitting at the sample level is an interesting alternative to using base and enhancement layers.
If we de-interleave CD audio into 3 scalable streams then stream 1 carries the first sample, stream 2 the second and stream 3 the third and perhaps the fourth. The next sample is added to stream 1 and so on. This yields two 11 kHz sample streams and one 22 kHz stream which can be used by the target device in any combination.
A low-quality service can be reconstructed from one stream or all of them can be combined to reconstruct the original sample stream.
Error Resilience (ER)
Some audio objects have Error Resilient counterparts which are indicated with the ‘ER’ prefix. This is useful for transmitting coded audio over unreliable and error prone network links.
Additional error resilience is possible with checksums and Forward Error Correction introduced as the payload is segmented into network packets.
Profiles & Audio Objects
A profile could use a single Audio Object while other profiles stack the tools hierarchically to make more efficient and sophisticated coders. Complexity requires more computational effort and increases the latency:
- MPEG-2 AAC-LC profile just uses the Low Complexity AAC-LC object.
- MPEG-4 AAC-LC adds Perceptual Noise Substitution (PNS).
- MPEG-4 HE-AAC v1 adds Spectral Band Replication (SBR).
- MPEG-4 HE-AAC v2 Adds Parametric Stereo (PS).
Because they are hierarchical, HE-AAC v2 players can decode any of the lower stacked levels.
These are the standardized profiles. Organizations such as Fraunhofer create their own proprietary profiles:
Profile | Introduced by |
---|---|
Low-Complexity | MPEG-2 |
Main | MPEG-2 |
Scalable Sampling Rate | MPEG-2 |
AAC | MPEG-4 |
High Efficiency AAC (v1) | MPEG-4 |
HE-AAC v2 | MPEG-4 |
Main Audio | MPEG-4 |
Scalable Audio | MPEG-4 |
Speech Audio | MPEG-4 |
Synthetic Audio | MPEG-4 |
High Quality Audio | MPEG-4 |
Low Delay Audio | MPEG-4 |
Low Delay v2 Audio | MPEG-4 |
Natural Audio | MPEG-4 |
Mobile Audio Inter-networking | MPEG-4 |
HD-AAC | MPEG-4 |
ALS Simple | MPEG-4 |
Extended High Efficiency AAC | MPEG-D |
(Limited) Scalable Lossless Coding | Fraunhofer HD-AAC |
The Fraunhofer Scalable Lossless Coding (HD-AAC) is not the same as the SLS support defined by the MPEG-4 standard.
Refer to section 1.5 of the MPEG-4 Part 3 standard for a detailed description of the Audio Objects and how they are mapped to the profiles.
Additional MIME Type
Because HE-AAC is coded differently to classic AAC, a new MIME type is needed so that browsers can distinguish between the two formats:
MIME type | Description |
---|---|
audio/aac | Use this for Standard AAC format content. This is the most widely supported. |
audio/aacp | Describes AAC+ content but is not as widely supported by web browsers. |
Related Standards
The version date indicates the most recent base standard, corrigenda or amendment. Although the latest versions are indicated, earlier versions may contain relevant information that is removed from later standards. Some devices may be compatible only with an earlier version and you should use that when developing your services for them.
There is a gradual refactoring of the MPEG standards so they can benefit from reusing supporting technologies without needing to repeat them. The MPEG-D and Coding Independent Code Points standards are examples of that as are the ISO 23XXX group of MPEG related standards which provide additional infrastructural support outside of the individual coding specifications.
Standard | Version | Description |
---|---|---|
ISO 11172-3 | 1996 | MPEG-1 Part 3 - Audio. |
ISO 13818-1 | 2023 | MPEG-2 Part 1 - Systems. |
ISO 13818-3 | 1998 | MPEG-2 Part 3 - Audio. |
ISO 13818-7 | 2007 | MPEG-2 Part 7 - Advanced Audio Coding (AAC). |
ISO 14496-1 | 2014 | MPEG-4 Part 1 - Systems. Currently being revised. |
ISO 14496-3 | 2009 | MPEG-4 Part 3 - Audio coding. Released in 2001 and amended in 2003 and 2004. |
ISO 14496-4 | 2019 | MPEG-4 Part 4 - Conformance bit-streams specification. |
ISO 14496-5 | 2019 | MPEG-4 Part 5 - Reference Software. |
ISO 14496-11 | 2015 | MPEG-4 Part 11 - Scene description and application engine. |
ISO 14496-23 | 2008 | Symbolic Music Representation. |
ISO 23091-3 | 2022 | MPEG-CICP - Coding Independent Code Points for delivering out of band metadata. |
ISO 23001-8 | n/a | Withdrawn and replaced by ISO 23091. |
ISO 23003 | n/a | MPEG-D is a group of standards for audio coding. |
ISO 23003-1 | 2017 | MPEG-D Part 1 - MPEG Surround (a.k.a. Spatial Audio Coding). |
ISO 23003-2 | 2018 | MPEG-D Part 2 - Spatial Audio Object Coding (SAOC). |
ISO 23003-3 | 2021 | MPEG-D Part 3 - Unified speech and audio coding (USAC). |
ISO 23003-4 | 2023 | MPEG-D Part 4 - Dynamic Range Control. Currently being revised. |
ISO 23003-5 | 2020 | MPEG-D Part 5 - Uncompressed audio in MPEG-4 File Format. |
ISO 23003-6 | 2022 | MPEG-D Part 6 - USAC Reference Software. |
ISO 23003-7 | 2022 | MPEG-D Part 7 - USAC Conformance specification. |
DVB-H | 2004 | Handheld mobile TV services. |
DVB-SH | 2008 | Handheld mobile TV services delivered via a satellite link. |
ETSI TS 101 154 | 2019 | HE-AAC and HE-AAC v2 audio coding for DVB applications. |
ETSI TS 102 005 | 2010 | Video and Audio Coding in DVB services delivered directly over IP protocols. |
ETSI TR 102 377 | 2009 | DVB-H Implementation guidelines |
ETSI TS 103 466 | 2019 | DAB audio coding (MPEG Layer II). |
ETSI TS 126 401 | 2017 | Enhanced aacPlus general audio codec. |
ETSI EN 302 304 | 2004 | Describes DVB-H. |
3GPP TS 26.401 | 2024 | Describes the use of Enhanced AAC+ for mobile services. |
General MIDI | 1999 | Developed by Roland to allow MIDI devices to sound similar when music sequences are played through them. |
DLS | 1998 | The MIDI Downloadable Sounds Specification by the MIDI Manufacturers Association. |
MIDI 1.0 | 1996 | The Complete MIDI 1.0 Detailed Specification by the MIDI Manufacturers Association. |
MIDI 2.0 | 2020 | Extends MIDI 1.0 with additional capabilities. |
ITU Rec H.223 | 1998 | Annexe C describes a Multiplexing Protocol for Low Bitrate Multimedia Communication Over Highly Error-Prone Channels. |
ITU Rec H.222.0 | 1995 | See ISO/IEC 13818-1 - Systems. |
Patent Licenses
Patents for MPEG-4 Audio coding are managed by Via Licensing. Contact them for a license if you design and sell an Encoder or Decoder (Player) of your own.
Content owners do not need a license to distribute their MPEG Audio content.
Patents for AAC baseline technologies expire in 2028 and some newer extensions will have active patents until 2031.
Conclusion
Audio and video compression is a complex subject. We balance it here at a level sufficient to explain the fundamentals whilst avoiding a deep dive into the ‘Rabbit Hole’. Consult the MPEG-4 Part 3 standard if you need to explore MPEG Audio coding in greater detail.
The AAC audio standard is increasingly being used with High-Definition TV services (HDTV). This is supported by the DVB standards that are managed by ETSI. HE-AAC is particularly relevant for mobile TV using DVB-H.
Digital radio services such as DAB+ and Digital Radio Mondiale are also adopting HE-AAC
Our recent articles have traced the history of MPEG audio coding from MPEG-1 Layer 1 up to the latest hybrid USAC codec. We are not quite finished with Audio yet. ST 2110-3x and AES are on the horizon.
These Appendix articles contain additional information you may find useful:
Part of a series supported by
You might also like...
Broadcasters Seek Deeper Integration Between Streaming And Linear
Many broadcasters have been revising their streaming strategies with some significant differences, especially between Europe with its stronger tilt towards the internet and North America where ATSC 3.0 is designed to sustain hybrid broadcast/broadband delivery.
Microphones: Part 2 - Design Principles
Successful microphones have been built working on a number of different principles. Those ideas will be looked at here.
Expanding Display Capabilities And The Quest For HDR & WCG
Broadcast image production is intrinsically linked to consumer displays and their capacity to reproduce High Dynamic Range and a Wide Color Gamut.
C-Suite Insight: The Broadcast Bridge In Discussion With MainStreaming CEO Tassilo Raesig
Tassilo Raesig became CEO at MainStreaming this year straight from being CEO of the German Streaming Service Joyn (part of ProSieben). We sat down with him to discuss his unique perspectives on the state of the streaming industry from the…
Standards: Part 20 - ST 2110-4x Metadata Standards
Our series continues with Metadata. It is the glue that connects all your media assets to each other and steers your workflow. You cannot find content in the library or manage your creative processes without it. Metadata can also control…