Testing ABR streams requires comparing source and system output video. That is easier said than done without the right test equipment.
Adaptive-bit-rate technology is a boon to multi-channel delivery, in part because of reduced bandwidth requirements. A downside is that ABR signals need specialized testing. Fortunately, specialized test solutions are available to easily and objectively make the needed analysis.
Distributing multiple video resolutions via IP at adaptive bit rates is a major part of any entertainment-delivery operation today. The nature of adaptive-bit-rate (ABR) services means there will be downstream deliverables in multiple profiles derived from a single HD or UHD source — all of which require test and measurement to verify their quality. And the most accurate way to do that is through full-reference testing.
The basic idea behind full-reference testing is to take a short video clip, send it through a system to be tested, and then compare the system’s output to the original (See figure). If the system causes any differences in any of the video frames, those differences can be measured using a variety of objective metrics to yield numerical results.
Today most traditional television signals are maintained at specific resolutions and frame rates from production through transmission. This consistency makes reference-based testing relatively easy to do in the broadcast domain because you can do a straight comparison between the processed video and its source. However, in the IP domain, it’s not so simple.
Why It’s Challenging to Test ABR Content
The foundation of full-reference testing is that the source and processed sequences must be at the same resolution and frame rate. In other words, you must start out comparing apples to apples in order to accurately measure changes in quality between the source and processed video.
With IP content, you can’t simply compare the processed video to the source, because the source will very likely be at too high a resolution. That means you first have to reduce the resolution/frame rate of the source material in order to create an acceptable reference for each delivery profile. Then, as you create new tests for new downstream profiles, you must create similarly processed streams to use as references during each test.
Furthermore, network operators rely on ABR delivery in order to accommodate a range of network conditions and end devices, which means they must be prepared to deliver multiple profiles (resolutions, bit rates, frame rates) for every asset. ABR delivery makes the testing process even more complex because you must perform a test for every one of those profiles.
With these complicating factors, it’s easy to see why testing the quality of IP-based ABR video can be so troublesome. So what’s the best way to do it?
Video Clarity's ClearView makes ABR measurements easy to complete.
A New, More Accurate Process for Measuring ABR Video in the IP Domain
In order to get the most accurate results from full-reference testing, you want to start with the most pristine reference possible. When testing IP streams that are delivered at different resolutions than the source video, the best way to arrive at a pristine reference is to use the same encoder to create the reference as you do to create the downstream deliverables. Doing so minimizes artifacts and/or degradations of using different video scaling techniques that could distort the results.
Therefore, the new methodology relies first and foremost on creating the references and test sequences with the same encoder/transcoder. Then, once you’ve created the identically formatted streams, you use the same measurement system and scoring index throughout the entire testing process to ensure repeatable results. Finally, after the tests have run, an experienced video engineer considers the resulting score while visually comparing the reference and the processed streams side by side.
To apply this test method, follow these steps:
- Generate a mezzanine-quality reference for each profile using the highest possible bit rate and optimal encoding parameters.
- Generate each profile’s test signal using the application’s encoding parameters.
- Calculate quality using the chosen algorithm and scale, comparing each test-profile signal to the appropriate reference-profile signal.
- Analyze results against visual comparisons of source- and downstream-profile test segments.
Repeat this process for each stream that needs testing, creating a new version of the reference for each profile or device under test, and making incremental adjustments to the variables one at a time in order to test the effect on quality.
Because manually repeating the test for every variation and then visually comparing the results can be a daunting task, the methodology might call for an automated tool to manage the testing phase (steps 3 and 4). Such a tool would measure video quality in the many different combinations of bit rates, frame rates, and lower-than-broadcast resolutions using an algorithm that yields numerical results based on human-perception scales. At the same time, it would automatically create measurement charts and synchronized, side-by-side pictures so that video engineers can see the differences between the reference and the processed video.
Video Clarity’s ClearView A/V quality analyser records the references created at high bit rates at the target resolutions from the device under test. The user then matches them up with the corresponding profiles created by that same device for testing.
The graph shows an estimated quality trend based on the averages of many sequence-scores. (In this case, a lower score indicates better picture quality). Blue and red lines show a potential difference in quality averages between two encoders. In general, quality will increase as bit rates increase, signified by downward-sloping curves on the graph.
For example, the figure above illustrates the testing process using Video Clarity’s ClearView video quality analyzer.
- An HD/UHD video source feeds an uncompressed signal to the encoder's input.
- The output of the transcoder goes into ClearView, where it is captured in real time and simultaneously decoded to form one sequence in a set of full-reference test sequences.
- Additional outputs are then captured and added to the set by making incremental changes in the transcoder bit rate. More sets of full-reference test sequences can be generated by using different source content and repeating the process of capturing and storing the encoder output sequences.
- In this example, ClearView performs several types of measurements on a complete set of full-reference test sequences. The test results create a single quality score per frame of video on any given sequence, and then presents results using the MS-SSIM measurement on the DMOS scale (or MS-SSIM scale). It can also test using the Sarnoff JND —Just Noticeable Differences, or MOVIE Temporal, MOVIE Spatial and overall MOVIE measurements.
- The output is a set of automatically aligned the sequences, each with measured quality, comparison charts, and a set of the uncompressed reference and processed test streams side-by-side on HDTV or UHDTV monitors for visual inspection.
A Proven, Applied Method for ABR Streaming
This method is a proven, mathematically accurate means of objectively comparing different profiles and equipment. In fact, major broadcast and cable networks around the world use this method to develop and refine their OTT delivery quality. For example, a major U.S. television network relied on this method to perfect multiple ABR profiles for streaming a recent major sporting event.
When using ABR services to deliver low-resolution video over IP networks, this test method gives operators the truest possible picture of how their downstream IP deliverables match up to the source video. The result? An accurate way of deciding how to deliver ABR content and where to invest equipment dollars.
Adam Schadle, vice president Video Clarity
You might also like...
CDNs are much more than just high-speed links between Origins and ISP (Internet Service Provider) networks. Instead, they form a complete ecosystem of storage and processing, and they create new possibilities for highly efficient streaming at scale that will likely…
At the moment it is far from clear exactly how the OTA TV landscape will evolve in the US over the next few years… the only sure thing is that we are in a period of rapid change.
Why keeping control of wi-fi and other devices within a broadcast facility to ensure there is no interference with critical devices is essential.
Part 7 of The Big Guide To OTT is a set of three articles which examine the pivotal role of CDN’s, how they are evolving and how Open Caching aims to support broadcast grade streaming.
Here we look at codecs and encoding for digital RF modulation such as ATSC 3.0, DVB and other digital OTA standards, and network requirements for encoding and delivering streaming internet video.