Production & Post Global Viewpoint – January 2016

Video Quality Evaluation: Methodology and Verification Testing of HEVC Compression Performance

The media industry sometimes gets ahead of itself, especially with standards. A solution is picked, it then takes a while for university-level research to catch up proving the decision was correct.

Such might be considered the case with the High Efficiency Video Coding (HEVC) standard (ITU-T H.265 and ISO/IEC 23008-2). The standard was developed with the goal of providing significantly improved video compression compared with its predecessors and most readers would concur that it does. But, by how much and can you “prove” it?

Early tests on HEVC coding were initially conducted by the Joint Collaborative Team on Video Coding of ITU-T SG 16 WP 3. This month, a group of researchers released their re-evaluation of HEVC versus AVC encoding performance and developed more precise results of coder performance.

These results are presented in the paper, Video Quality Evaluation Methodology and Verification Testing of HEVC Compression Performance, and printed in the recent issue of IEEE Transactions: On Circuits and Systems For Video Technology. The material contained in this article is taken from that paper.

The paper presents both the subjective and objective results of a verification test in which the performance of the new standard High Efficiency Video Coding (HEVC) is compared with its highly successful predecessor, the Advanced Video Coding (AVC) video compression standard (ITU-T H.264 and ISO/IEC 14496-10).

The test used video sequences with resolutions ranging from 480p up to ultra-high definition (UHD), encoded at various quality levels using the HEVC Main profile and the AVC High profile. In order to provide a clear evaluation, this paper also discusses various aspects for the analysis of the test results.

The tests showed that HEVC saves about 59% on average to provide the same perceived video quality, which is higher than a bit rate saving of 44% demonstrated with the PSNR objective quality metric. However, it has been shown that the bit rates required to achieve good quality of compressed content, as well as the bit rate savings relative to AVC, are highly dependent on the characteristics of the tested content.

HEVC provides significantly improved encoding

The authors call today’s digital coding capability a “once-in-a-decade event in the world of video compression”. The mid-1990s saw the introduction of the Moving Picture Experts Group (MPEG)-2 video coding standard (ITU-T Rec. H.262 and ISO/IEC 13818-2), the first compression standard to be widely adopted in broadcasting and entertainment applications.

Advanced Video Coding (AVC) (ITU-T Rec. H.264 and ISO/IEC 14496-10) appeared in the mid-2000s, offering the same subjective quality at approximately half the bit rate. Now HEVC (ITU-T Rec. H.265 and ISO/IEC 23008-2), has been developed and it promises a further factor of two improvement in compression efficiency for the mid-2010s,

Improved testing

The HEVC standard has been jointly developed by the same two standardization organizations whose previous collaboration resulted in both MPEG-2 and AVC: 1) the ISO/IEC MPEG and 2) the ITU-T Video Coding Experts Group (VCEG), through the Joint Collaborative Team on Video Coding (JCT-VC). HEVC version 1 was ratified in 2013 as H.265 by the ITU-T and as MPEG-H Part 2 by ISO/IEC.

This first version supports applications that use conventional (single-layer) encoding of 4:2:0-sampled video with 8- or 10-bit precision. A second edition was completed in July 2014, and an additional extension was completed in February 2015.

These extend the standard to support contribution applications with tools that enable 4:2:2- and 4:4:4-sampled video formats as well as 12- and 16-bit precision, and multilayer coding enhancements for efficient scalability and stereo/multiview and depth-enhanced 3D compression. A further amendment is currently being developed to enable more efficient coding of screen-captured graphics and text content and mixed-source content .

While this paper reports an extended set of results of HEVC verification tests, it also summarizes the details of tools that can be used in the analysis of the results, pointing to a number of factors an evaluation should consider. Compared with the initial results, those presented in this paper are based on the use of more viewers for subjective tests, the objective results are presented and compared with the subjective results, and additional analysis of the coding gains versus the bit rate is provided.

Ultimately, the verification test showed that the key objective of HEVC had been achieved—i.e., providing a substantial improvement in compression efficiency relative to its predecessor AVC.

For the convenience of video coding performance assessment, the most commonly used objective metric is peak signal-to-noise ratio (PSNR). However, it is commonly acknowledged that PSNR has the disadvantages of disregarding the viewing conditions and the characteristics of human visual system perception. In addition, the PSNR for a given video sequence can be computed in different ways, depending on how the picture components (e.g., luma and chroma) or individual picture PSNR values are combined. Nevertheless, for a particular content item and small variations of coding conditions, the changes in PSNR values for an overall video sequence can typically be reliably interpreted.

Subjective Quality Evaluation

For the HEVC verification test that includes a wide range of visual quality points, a degradation category rating (DCR) test method was selected. For this purpose, it was used to evaluate the quality (and not the impairment) with a quality rating scale made of 11 levels, ranging from 0 (lowest quality) to 10 (highest quality), which may be interpreted as in Table I. The numerical scale helps avoid misinterpretations associated with the use of category adjectives (e.g., excellent or good), especially in cases where the tests are performed across different countries and including nonnative English speakers.

Frame grabs of the 1080p test sequences used in the evaluation tests.

Frame grabs of the 1080p test sequences used in the evaluation tests.

The basic results of the subjective test are evaluated in terms of the average rating, which is called the Mean Opinion Score (MOS), and the associated confidence interval values that are computed for each coding point, after having verified the reliability of each viewer.

Editor’s Note: The Broadcast Bridge has published an article addressing the use of MOS in QoE and QoS environments. QoE: What Does it Really Mean Today?

Screenshots of the video sequences are shown in Figures 1 and 2. The sequences were selected from different sources and have different spatio-temporal characteristics, which leads to different behavior of compression algorithms. This is the first formal test of video compression standards where a wide range of resolutions including content with UHD resolution with high frame rate has been evaluated. The format, as specified in ITU-R Rec. BT.2020, has a number of extended features compared with ordinary HD video. In addition to containing more pixels per frame, it specifies support for higher frame rates, wider color gamut, and higher bit depths. However, for compatibility with available playout and display systems, all tested video sequences have 8 bits per component per sample and are in the Y’CBCR color space defined by ITU-R Rec. BT.709.


Frame grabs of the UHD test sequences used in the evaluation tests.

The test sequences were compressed using HEVC Main profile and AVC High profile encoding. Either a random access (RA) or low delay (LD) configuration (Cfg) was used (similarly configured for both HEVC and AVC, with a refresh period of approximately 1 s and hierarchical referencing for RA and with no periodic refresh and no reordered referencing for LD.

For each test sequence, four test points using different fixed quantization parameter (QP) settings were selected so that the tested HEVC bit rates are approximately half of the AVC bit rates. This bit rate ratio was chosen because it is already well established that the quality of HEVC is much better than the quality of AVC at the same bit rate. These test conditions can identify whether a bit rate saving of 50% or more is achieved for the majority of the tested video sequences.

The human factor

Subjective tests for different categories of video sequences were conducted in separate sessions. Each subjective test session of the DCR method consisted of a series of basic test cells (BTCs). Each BTC was made of two consecutive presentations of the video clip under test.

First, the original version of the video sequence was displayed, followed by the coded version of the video sequence, with a gap of 1 s. Then, a message asking the viewers to vote was displayed for 5 s. Each test session was designed with 45 BTCs in total: the first three BTCs represented the stabilization phase and were selected to show to the viewers the whole range of quality they would see during the test.

Table 1. This table illustrates the image resolution, frame rate and the number of frames shown to the viewers.

Table 1. This table illustrates the image resolution, frame rate and the number of frames shown to the viewers.

By considering the positions of MOS points for AVC and HEVC, it can be observed that HEVC achieves the same subjective quality as AVC while typically requiring substantially lower bit rates.

The results show that for 74 out of the 80 pairs of test points (or 92.5%), HEVC has a bit rate saving compared with AVC greater than or equal to 50%. Only six pairs of test points (or 7.5%) show a bit rate saving of less than 50%. Four of the six pairs of test points were contributed by one video sequence (SVT04a), where the HEVC encoder did not perform as well as in the other cases.


The data convincingly show that HEVC is achieving bit rate saving that is at least 50%. However, the granularity of the above bit rate saving estimation was limited by the test design, where in each pair of test points, the HEVC bit rate was selected as approximately half the AVC bit rate.

In this more rigorous analysis of the subjective test results, the data shows that HEVC test points at half or less than half the bit rate of the AVC reference were found to achieve a comparable quality in 92.5% of the test cases.

It was found for the investigated test cases that the HEVC Main profile can achieve the same subjective quality as the AVC High profile while requiring on average approximately 59% fewer bits.

The PSNR-based BD-rate average over the same sequences was calculated to be 44%. This confirms that the subjective quality improvements of HEVC are typically greater than the objective quality improvements measured by the method that was primarily used during the standardization process of HEVC.

It can therefore be concluded that the HEVC standard is able to deliver the same subjective quality as AVC, while on average (and in the vast majority of typical sequences) requiring only half, or even less than half, of the bit rate used by AVC.

This means that the initial objective of the HEVC development (substantial improvement in compression compared with the previous state of the art) has been successfully achieved.

Researchers and authors: Thiow Keng Tan, Rajitha Weerakkody, Vittorio Baroncini, Marta Mrak, Naeem Ramzan, Jens-Rainer Ohm and Gary J. Sullivan

This presentation is based on material obtained from the paper;"Video Quality Evaluation: Methodology and Verification Testing of HEVC Compression Performance", IEEE Transactions on circuits and systems for video technology, Vol. 26, No. 1, January, 2016.

Copies of the complete paper and results can be obtained from the IEEE.

Let us know what you think…

Log-in or Register for free to post comments…

Related Editorial Content

HEVC Demystified: A Primer on the H.265 Video Codec

The most recent non-proprietary video compression standard, High-Efficiency Video Codec (HEVC), also known as H.265, was placed into final draft for ratification in January 2013 and is expected to become the video standard of choice for the next decade. As with…


High Efficiency Video Coding (HEVC) is the latest and certainly an efficient compression standard targeted at today’s high pixel count images.

Open Media Alliance Throws Down the Gauntlet to HEVC

Seven Internet giants have joined forces to build an open source codec to challenge HEVC for compressing high definition video content and especially Ultra HD (UHD) or 4K. The new Open Media Alliance Source project has the stated intention of…

​For IP, H.264/AVC is Key

The broadcast industry is once again going through both business and technology changes. At the heart of these changes is the need to move to IP-based infrastructure. There are many different codecs and standards being proposed but one that is…