AMWA Begins Development Of Automated Controller Testing For Professional Media Networks

As manufacturers develop new devices for TV production networks, they must also write specific code that allows that tool to be recognized and controlled over an IP network. While most companies follow common open source API specifications in their software controllers, they sometimes deviate from each other, causing major interoperability issues for network designers and system integrators of such networks.

The answer is critical testing to ensure that these device controllers comply with the Advanced Media Workflow Association (AMWA)’s NMOS IS-04 and IS-05 protocols. This helps ensure that the controller is making the correct API calls to the network registry (04) and to the nodes to set up the correct connections between them (05).

Up until this year, such testing was conducted manually, which was expensive and time consuming. Thus, the NMOS Client Testing Work Group, within AMWA, was formed in May of this year to develop a suite of automated test (software) tools. The idea is to test every controller before it is deployed on a network to ensure it will work with other products on the network. These tests are vital to keeping an IP network up and running smoothly.

“Broadcasters want better products out on the market as quickly as possible,” said Rob Porter, Project Manager within the Advanced Technology Team at Sony. He’s based at Sony’s European Professional Engineering facility in Basingstoke, UK, and has led a team of AMWA members that included major broadcasters like the BBC and CBC in Canada as well as vendors of broadcast equipment, like EVSImagine CommunicationsMatrox Video and Pebble.

Sony’s Rob Porter gave a presentation on the new test tools during the recent virtual IP Oktoberfest 2021 event, sponsored by AIMS and AMWA.

Sony’s Rob Porter gave a presentation on the new test tools during the recent virtual IP Oktoberfest 2021 event, sponsored by AIMS and AMWA.

Porter, together with colleague Jonathan Thorpe, gave a presentation on the status of this “Automated NMOS Controller Testing” at the IP Oktoberfest 2021 virtual event last month, sponsored by the Alliance for IP Media Solutions (AIMS) and AMWA. AMWA has made this new test suite available for free at AMWA’s GitHub website (https://github.com/AMWA-TV) for anyone to review and implement.

“We need good testing to ensure these new products will interoperate correctly on the network,” he said. “Simply put, automated testing will make everyone’s life a bit easier, when it comes to testing, and could lead to facilities being designed and implemented faster and more efficiently.”

There is an existing test suite developed by AMWA over recent years, but it is only able to test the media nodes on the network (the cameras, servers, etc.) to make sure that they are conforming to the NMOS specifications and communicating over the APIs. It ensures that the devices register with the network registry correctly, for example. This test suite is also able to test the registry itself and make sure it’s able to process those API calls correctly, both by registering devices that attempt to register with the registry and also by telling a controller that queries the registry about what is available on the network.

In a Semi-Automated testing workflow, the developers have replaced the human tester on the left with what they call an automated “testing tool” (right). In the middle is the testing façade that provides specific questions about the controller and the instructions on how to implement it.

In a Semi-Automated testing workflow, the developers have replaced the human tester on the left with what they call an automated “testing tool” (right). In the middle is the testing façade that provides specific questions about the controller and the instructions on how to implement it.

What was missing from the first generation test tools was the ability to test each manufacturer’s controller. As with much of AMWA’s work, the specialized test software does the work.

“Testing controllers was something that was always done manually in the past,” said Porter. “Vendors were self-testing their controllers, which sometimes resulted in widely different results and interoperability issues. Now we have a new suite of automated tools that make the process easier and, I’d say, more genuine and fairer to everyone involved.”

As part of the automated test tools development process—which began in May of this year with the Working Group meeting every week for an hour over four months—the group studied the Joint Task Force on Networked Media (JT-NM) certification event from March of 2020, during which participating vendors connected their controller to a test VPN network set up at the CBC. Over a secure connection the broadcasters instructed each vendor to carry out various operations with their controller to discover what’s on the registry and tell them what they saw on their end. Or they had them try to make a connection between the sender and receiver in the correct way. They would then verify that those connections were indeed happening as they should.

It was a very time consuming (over several days) and expensive process for everyone involved. But they learned a lot and it all worked very well. The six vendors participating were then able to claim that their controllers were “JT-NM Tested”, which is something to brag about to customers.

“After an experience like this, it was clear that an automated test tool was required, that would basically replace that single engineer doing the testing at a broadcast facility with software tools and an automated process that could run in the background and perform the job quickly and accurately,” said Porter. “On the vendor side, we are also excited about speeding up the process of testing and releasing new products, and are committed to working on this area of interoperability in the long run.”

What the Working Group has come up with is a suite of tools that can test in either Semi-Automated or Fully Automated ways.

Semi-Automated still requires a human to drive the controller’s GUI according to specific instructions provided by the new testing tool. Because you are connecting your controller to a set of mocked up resources (mock nodes and mock registry) on a network, not only is the testing tool verifying that those API calls have been made correctly, but it is also using a test façade (with a webpage GUI) that gives the engineer instructions on what steps to follow.

In Manual Testing mode, the operator on the left represents the vendor’s in-house testing and on the right side is the broadcaster’s verification workflow upon receiving NMOS compliant product from the vendor.

In Manual Testing mode, the operator on the left represents the vendor’s in-house testing and on the right side is the broadcaster’s verification workflow upon receiving NMOS compliant product from the vendor.

For example, the operations engineer looks at his interface that’s connected to the mock registry and they see a list of “senders”, devices waiting to join the network. The engineer then selects from a list of multiple-choice answers that they can see on the testing façade. The test tool then verifies that they have correctly answered the questions, while at the same time checking it has received the correct API calls over the network from the controller.

So, it’s testing at two levels: one, that the operator is seeing the right device on their user interface and two, that the controller has made the right API calls.

The Fully Automated Testing mode is designed for the benefit of the vendor to speed up the process of product testing—ensuring the code is written correctly—in their labs. This method allows the user to perform a new set of testing using a Selenium robot framework to automatically operate the controller and run it against the AMWA test tool. The test tool is the same as the Semi-Automated one, but the way you operate the controller to test against it, is different and requires no human interaction, allowing it to be used as part of vendors’ automated continuous integration processes for software development.

The full test suite is now available in Beta for review and real-world testing on AMWA’s GitHub site. It performs the same level of testing that was performed at the JT-NM Tested event a year ago, but in an automated way, saving time and effort in the process. More development work and additional capability is sure to follow.

“Interoperability issues are expensive, not just for the end user but also for the system integrator and for all of the vendors that are involved in making these complex systems work,” said Porter. “If we can minimize those issues, we’ll all be in a good place.”

You might also like...

REMI On A Budget - A Remote Switching Paradigm Shift

Remote Integration Model (REMI) production is more than remote cameras. It’s a new way of thinking and working. This tale of trying to implement a REMI production model within tight financial constraints highlights some of the operational challenges involved.

NextGenTV Mid-2022 Progress Report

Some call it the Broadcast Core Network, or Broadcast Internet, or One-to-Many Private Datacasting. Others simply call it datacasting.

Vendor Spotlight: Blackmagic Design

Founded in 2001, Blackmagic Design is based in Melbourne, Australia, with offices in Fremont and Burbank, Calif. in the U.S., as well as Manchester, England, and Amsterdam, The Netherlands.

Essential Guide: Flexible Contribution Over IP

Outside Broadcast connectivity using managed and unmanaged networks is delivering opportunities for employers that enhances productivity through flexibility, scalability, and resilience.

Information: Part 5 - Moving Images

Signal transducers such as cameras, displays, microphones and loudspeakers handle information, ideally converting it from one form to another, but practically losing some. Information theory can be used to analyze such devices.