About The AVC Standard

The MPEG-4 Part 10 standard is very large and complex with approximately 900 pages of densely concentrated detail.

The support for profiles and levels is fundamental to successfully deploying your content using the H.264 format. There have also been some important extensions (SVC, MVC and MFC) to the original codec design. These are included as Annexes to the main body of the standard and require a high degree of focus to read and interpret correctly.

Section 3 briefly describes the terminology and abbreviations used throughout the standard. Understanding them makes the rest of the standard much easier to comprehend.

The notational conventions in Section 5 are relevant if you want to understand the mathematical and logical concepts described later on. These will be most useful to codec developers.

The structure of the Network Abstraction Layer packets (NAL units) is described in Section 7. The coverage of the NAL unit payload includes descriptions of how the profile and level parameters are formatted. Read this in combination with the Annexes to glean the specific values and locations for the profile_idc, profile_ iop and level_idc bytes in the NAL unit.

Profiles are described in Annex A for the core AVC compression standard. More profiles are introduced in Annex F which describes Scalable Video Coding (SVC). Annexes G, H and I address various multi view coding techniques for stereoscopic and 3D viewing. The additional profiles needed to constrain them and signal the client-player are also described there.

Levels are addressed comprehensively in Sub-section A.3 of Annex A.

Annex B describes the Byte Stream Syntax as opposed to the bitstream syntax in Section 7. It also explains how a decoder can resynchronize itself to the incoming stream. The decoder frames the bitstream into 8-bit bytes to unpack the payloads in the NAL units.

Understand the decoding process with the Hypothetical Reference Decoder described in Annex C.

Supplemental Enhancement Information (SEI) is described in Annex D. This additional metadata describes the content in the stream. Decoders have some discretion in how they respond to this.

Annex E describes Video Usability Information (VUI) which parameterizes aspect-ratio, picture size, over-scanning, color gamut ranges and their associated transfer functions. The client-player uses this to present the video canvas correctly.

The rest of this chapter will focus on Profiles and Levels. This is an area of some complexity and low-level explanations of how it works are scarce and hard to find.

Profiles & Levels

Profiles and levels facilitate deployment of content delivery systems and receiving client-player designs by reducing the range of choices when interpreting how the standard works.

Choose the profile and level that best suits your needs. Encoders transmit the details to the client which interprets the bitstream accordingly.

Profile settings manage the encoding process and select appropriate sub-sets of the individual coding tools. This is a huge benefit and reduces the complexity of encoder configurations. The decoder has counterparts for each of these tools.

Levels are important in the receiving client-player and are concerned with the display size and color depth of the decoded images.

Do not confuse the container profiles defined by the MPEG4 systems layer with Part 10 video compression profiles. They are not the same thing.

Signaling The Profile & Level

The profile and level signaling mechanism in AVC has become very complex because the standard has been revised multiple times while retaining the necessary backwards compatibility with many millions of previously deployed devices.

The profile and level values are located near the start of a NAL unit (packet) payload. Unpack it carefully to reveal three bytes representing these properties:

The profile_iop value uses individual bits as flagging indicators. Conventional Boolean notation applies with the value 1 representing TRUE and the value 0 representing FALSE.

Byte 1 contains the profile_idc which identifies the foundation profile.

The same profile_idc value may be used to identify several different profiles because they are uniquely distinguished by appending the profile_iop value. For example, the same profile_idc is used for Baseline and Constrained Baseline profiles but IOP constraint bit-flag 1 determines which is selected.

Byte 2 is the Interoperability Profile (IOP) described as the profile_iop. It carries 5 constraining individual bit-flags which alter the behavior of the profile specified in the profile_idc. It also affects the behavior of the level_idc value. To unambiguously select a profile, Bytes 1 and 2 must be combined. The meaning of these individual constraint flags depends on the context. Refer to Section 7.4.2.1.1 for details and cross-references to the applicable annex descriptions.

Byte 3 the level_idc describes the level at which the chosen profile is operating so the client can reconstruct the images correctly.

Profile Categories

Many of the profiles are derived from the same common Baseline and High ancestors. This has implications when the behavior of level_idc values are examined. This diagram illustrates the inheritance:

H.264 profiles can also be grouped according to which part of the ISO standard they are described in:

Current List Of Profiles

The annexes at the end of ISO 14496 Part 10 are the authoritative source. Table 5 in IETF RFC 6184 is also helpful.

Levels

The levels describe picture resolutions and frame-rates for the client-player to use when presenting the decoded output. Within any given bitrate, there is a trade-off between frame-rate and picture size. If you have a higher frame-rate, the pictures must be smaller. Decoding speed is also affected and so is the number of frames that can be buffered. The level limits defined in Table A.1 describe how the client must be able to support this.

A decoder must support the maximum level limit values defined for a level and all lower levels beneath it.

File Name Extensions

There are a variety of file extensions which describe containers that carry AVC video content. The 3GP files are intended for use with mobile devices. These would be somewhat rare in a broadcast environment but they may show up in user contributed content.

There are other file extensions for MPEG-4 audio files which are listed elsewhere.

Media Type Identifiers

The rules governing the parameter values and their formatting are complex and different for each codec.

Applying AVC

Standards compliance does not guarantee interoperability. Make sure the profile and level you are encoding with is consistent with your target client-player.

For example, if company A makes a video codec that processes the picture size at high definition and company B makes a video player that expects to play content that is strictly standard definition these are incompatible even though they may both claim to be (and are) 100% standards compliant.

Bear in mind also that H.264 is not a lossless codec. It does have some features that make regions within a frame lossless but it cannot make the entire frame or sequence of frames entirely lossless.

Patent license fees are levied for commercial use of the AVC codec. Free content delivered via the Internet is license free however. This has spawned an open-source project to develop a patent free codec which is known as X264. You will most likely see this as a codec option in compression tools that have it installed.

These Appendix articles contain additional information you may find useful: