Because this Panasonic DX900 television has the Ultra HD Premium logo, consumers are assured it meets certain specifications.
As standards move forward, vendors wanting to sell into those spaces need to ensure their products “work”. A simple task you say. In the case of ATSC 3.0, one could run some images through an encoder and see what comes out. If only it were that simple.
A test suite is a package of digital streams, files, databases, whatever it takes to test and measure the performance or compliance of a device against a standard. There is a marketplace of vendors who supply what are known as “Test Suites” for any number of technologies. In this industry, the test suite may focus on H.264/AVC video decoder compliance or DVB-T, DBV-3D, IPTV, ISDB-Tb and MPEG-TS systems. In the example of an H.264/AVC test suite, the user may want to examine syntax streams or performance streams.
A syntax stream test ensures that decoders abide by all features of H.264 compression standards. Performance streams can help determine if a decoder is able to decode worst-case streams in real-time. Both types of tests are important if you hope to sell into this particular market.
Why use test suites?
Conformance testing can be critical to the successful introduction of new technology into a marketplace. This test procedure helps ensure that devices are compliant with whatever standards have developed. Success here drives the adoption and success of that technology.
With this in mind, the provision of a test suite – including its design, development, distribution, support and maintenance – needs to be a high priority to any organisation seeking to successfully deploy a new technology.
This aspect is particularly relevant to standards bodies and industry consortia, which too often place most their focus on developing a device or protocol specification but give insufficient attention to how it will be validated. Hoping the consumer will “figure it out” is not a successful strategy.
Here are two examples of how test suites might be used. On the left, each factory or company purchases and uses a separate test suite. On the right, each company or factory performs all of their compliance testing through a single test suite, often sourced via the internet. Click to enlarge. Image: Eurofins Digital Testing
What should a test suite accomplish?
A company introducing a technology or a standards group should consider conformance testing with a standard test suite. See the figure above. The test suite should perform exhaustive tests not just simple ones. There is little value in just testing the easy to-do-stuff.
A first step is to determine what group will manage the test suite. Usually that requires choosing to either let the industry players voluntarily provide the test equipment and set parameters or participants could identify a company that specializes in this task. Once a test suite has been selected, it must be equally available to all payers.
The five test scenarios
First, let us set up some definitions.
The test suite is a collection of test cases with a defined test method for systematically testing a specification. Each test case verifies conformance or interoperability for a particular requirement in the technology specification.
Sometimes a test suite will require a particular test harness – i.e. a framework for executing the test cases and a repository for storing test results. These may or may not be included as part of the test suite.
Most test cases will have corresponding test materials. These are often data files that exemplify the standard and may include; audio-video streams, images, HTML pages, or protocol simulations. You probably have seen image test streams at IBC and NAB without realizing what you were viewing. Early in HD development, vendors would use the same set of source videos to highlight their products. That is one reason the same video appeared in so many booths.
There are tens of thousands of files, digital streams and other source content that can be used to test various standards and compliance. One test suite company claims a database of 10TB of files and more than 21,000 test streams.
This process can verify that products meet the technology specifications. It is based on the use of a standard set of test procedures covering the specification, and corresponding test materials, which together form a test suite. Combined, the tests can help certify that the products conform to a desired standard. This test is typically used once the products are close to being finalized, perhaps in a beta version.
This is where multiple products from a variety of companies are compared. For instance, the test might use an encoder from vendor A feeding a decoder from vendor B. Do they work together? Are there instances when they do not work together properly?
This is based on testing with available test material and implementations. Often the testing focuses on user interoperability. The test can be performed throughout the development of technology specifications and via interoperability events, sometimes called plug-fests.
A recent IT/IP InterOp or Plug-fest was conducted in Austin, TX. A second one took place during IBC 2016 at the Interoperability Pavilion. More information on those tests can be found in The Broadcast Bridge article, “Making IP Networks Work For Video”.
At such public events, multiple vendors test to ensure that their devices work together. The gatherings are often ad-hoc, but are significantly more effective when test materials are agreed upon prior to the event. In the early specification-drafting phases, plug-fests typically achieve some validation of a technical approach amongst the parties developing the technology.
Later plug-fests help drive out interoperability challenges as the ecosystem matures and the standard starts to be used. Such testing may be as simple as suggesting a particular type of connector be used or the adoption of common definitions for a feature or function. These events are usually voluntary and tend to focus on a specific feature of the technology, as opposed to exhaustively covering all cases and failure modes.
Ultra HD Premium branding logo. Only equipment meeting standards set forth by the UHD Alliance may display this logo on their products.
Sometimes referred to as conformance tests, this process is often used by industry bodies to manage and foster adoption of a technology. They are often combined with the permitted use of a logo that helps consumers and the industry identify and have confidence in the technology.
This practice is common in the consumer electronics industry where vendors want the specific branding that accompanies a standard to be available to their products. Most notably is how the Consumer Electronics Association (CEA) controls television set branding logos.
At last January’s Consumer Electronics Show, the UHD Alliance, an industry consortium of content creators, distributors, and hardware manufacturers, announced that manufacturers that wanted their TV sets to carry the "Ultra HD Premium" label had to meet certain specifications. Required performance parameters covered; resolution, bit depth, color gamut, high-dynamic-range imaging and rendering required for Ultra HD (UHDTV) content and displays. In Japan for example, only television sets providing 8K UHDTV performance can be branded as “Super Hi-Vision”.
The Consumer Electronics Association issued a similar required certification back in 2012. To help consumers evaluating new TV sets, CES said the terms, "Ultra High Definition", or "Ultra HD", could be used only for displays that had an aspect ratio of 16:9 or wider and at least one digital input capable of carrying and presenting native video at a minimum resolution of 3840×2160 pixels.
Why should I care?
End users seldom have a direct case use for a test suite. Therefore, no, you probably do not need to purchase a test suite. Fortunately, the manufacturers of the gear you buy will (likely) already have measured their gear against an official test suite.
However, remember that all the costs associated with bringing a product to market end up in the final selling price. So yes, you may actually help pay for controlled testing.
What about IT/IP test suites, are any being planned? Here is where things may get a bit foggy. There are currently multiple proposed standards. I have not learned of any uniform test suite that has been applied to any of those proposed standards, or that one is planned for use.
As mentioned above, at least one IT/IP plug-fest has occurred with encouraging results. However, that is not the same as requiring manufacturers to clear their solutions through an industry association-approved standard test. Editors at The Broadcast Bridge will monitor industry actions and keep you informed.