Care and Feeding of an IP Facility: Part 1

A few years ago at a technical conference, the CTO of a major sports entity was challenging the storage vendors on how much storage capacity and throughput architecture would be needed to meet his needs. What they neglected to tell him was the cost, technology and resources required to monitor, maintain and manage it.

The IP change is a “YUGE” problem

While the industry argues over which standard or standards will rule the world (for at least 15 minutes), the people that need to make systems work on a daily basis, the engineering and tech support folks, are being ignored. And the more standards, protocols, API’s, middleware and “open” everything means such systems can be a nightmare to maintain for even the best trained personnel.

Let’s fantasize for a minute that we can actually build a working IP production system. Now what?

As a designer and project executive, I always say the building part is easy, it’s running these systems that is hard. I’m not diminishing the challenges, skills and knowledge that constructing a media infrastructure requires. However, when building a system, there is the luxury of being able to solve problems without worrying that those systems are live.

But once a system gets turned on and goes live or online, there is no turning it off. The integrators have completed their punch lists and moved on. Now its operations and engineering staff’s responsibility to assure everything works and stays that way.

Every time there is a major technology change, in comes the glue and black boxes that enable the transition to occur seamlessly. As the transition matures, the need for these little helpers diminishes because devices and systems begin to connect directly with each other.

The new MCR

The transmission or master control center is responsible for maintaining quality control over all inbound and outbound signals. This room is equipped with the tools to confirm the signals meet certain technical requirements. In the equipment rooms, there are dedicated stations that enable engineers to monitor the systems in a non-disruptive way. Devices and systems can be patched around, or if there are spares, taken to maintenance and/or to the engineering shop. The shop has the required tools and test equipment to get things working properly again.

Monitor, maintain and manage IP

However, the introduction of IP-centric software and hardware technology has blown up this entire model.

Just how does a staff monitor, maintain and manage these IP systems? Great question. Let’s examine how these tasks are performed in an IP environment.

Engineers view the IP transition as posing more technical than cultural obstacles. Source: Imagine Communications, “2016 Focus Forward Technology Trends Report”.

First, monitoring needs to be done in a non-invasive and non-disruptive way. For the purposes of this article we are not going to discuss the type of the knowledge and skill sets needed. We will assume that properly trained video and IT staffs are available and work well together.

The program origination center, aka Master Control, now looks more like a data center, often called a Network Operations Center, or NOC. The operators here monitor multi-view displays, which are managed by an application. The displays show all program feeds and the playlist for each delivery platform (applications), which are controlled by automation (another application). All of these applications run on servers. These servers are connected via a network and they pull both content and playlists from shared storage.

Content quality control is handled by an application doing file or stream checking. Content contribution arrives as files over a dedicated network or open internet using cloud services and/or proprietary software. As the content become streams, the network needs additional types of monitoring and management.

New monitoring solutions required

IP and file-based technology has introduced a new level of management to broadcast and production systems. Devices are monitored using SNMP (Simple Network Management Protocol) traps. The servers are monitored with tools that display CPU utilization, running processes, memory and storage, and this happens before we even get to the application.

Many devices and systems have their own management tools. These might be accessible using a web browser. Does this mean a dedicated computer, or is this a good application for virtual machines (VM)? If content is playing from a server how is it monitored? From where are the servers, storage and network going to be monitored and managed? While KVM (Keyboard, Video Mouse) extenders or matrix might be a first thought, using KVM to remotely manage is highly disruptive of operating services. This is where sharing applications like VNC (Virtual Network Computing) are appropriate solutions.

Transition to IP timeline. Ninety-four of survey takers put a decade-long cap on the duration of the transition to IP. This reflects widely held assumptions that media companies will transition, not in lockstep, but in accordance with numerous company-specific factors over a window of time consistent with most major technology evolutions and in line with traditional technology refresh cycles. Source: Imagine Communications, “2016 Focus Forward Technology Trends Report”.

Transition to IP timeline. Ninety-four of survey takers put a decade-long cap on the duration of the transition to IP. This reflects widely held assumptions that media companies will transition, not in lockstep, but in accordance with numerous company-specific factors over a window of time consistent with most major technology evolutions and in line with traditional technology refresh cycles. Source: Imagine Communications, “2016 Focus Forward Technology Trends Report”.

IP system maintenance can be a Pandora’s Box

What about network bandwidth or congestion? This needs to be monitored. What about managing the storage itself? There are applications that manage storage retention policies. But, what happens when utilization peaks and it’s time to load balance? Or, consider a worst case scenario where the storage system fills to capacity.

Maintenance for IP systems opens a Pandora’s Box! First, both hardware and software come with Service Level Agreements (SLAs) that are typically a percentage or the purchase price. Before we even get to the dedicated application, regular updates must be applied to the operating system. Then there is threat protection (Anti-Virus, intrusion protection, security), which must be kept current. Also, what about firmware updates to firewalls and switches?

Checking for updates, determining whether they will create a problem or interfere with a running application are all part of maintenance. What happens when vendor A upgrades their software? Will those changes effect the middleware and API’s that are used by external systems? What if a new codec or IP standard comes out? Is the update considered a patch or does it require some custom code and who does the install?

Remote access

The shop is now a NOC and if there’s a problem, accessing the media systems can be done remotely. IP introduces remote access as a way to monitor, manage and maintain the systems. Monitoring systems can be configured to detect many errors. They can also send alerts and messages over email and text (SMS). Remote access is the ability to use either open internet or dedication access to connect into the network or troubled system. Once connected, the engineer has the ability to diagnose and correct the issue.

As I stated earlier, this access needs to be non-invasive and non-disruptive. Taking control away from an operator or automation system and then not having the ability to restore control is not a healthy way to solve a problem. It only creates more issues.

This is the first in a series that will look at the tools, methods and technologies to monitor, maintain and manage an IP facility. 

​Gary Olson has written a book on the conversion to IP, “Planning and Designing the IP Broadcast Facility – A New Puzzle to Solve”. It is available from major booksellers.

​Gary Olson has written a book on the conversion to IP, “Planning and Designing the IP Broadcast Facility – A New Puzzle to Solve”. It is available from major booksellers.

Some additional The Broadcast Bridge resources written by Gary Olson are listed below.

You might also like...

5G Gaining Momentum In Broadcast, But Universal Infrastructure (And Experience) Lagging

The recent news that NTV has become the first Russian TV channel to experiment with 5G broadcast, one of many such transmission tests that have been conducted over the past 18 months, illustrates that broadcasters see a bright future in the…

The Sponsors Perspective: PRISM Waveform Monitors: Form Is Temporary, But Class Is Permanent?

In the beginning, there was television. And whenever people tried to make television programmes effective video signal monitoring was an essential pre-requisite.

Timing: Part 3 - Early Synchronization

Synchronizing became extremely important with the growth of AC power systems, which ended up being used to synchronize all sorts of equipment, from Radar to television.

Essential Guide: Cloud Microservice Workflow Design

The power and flexibility of cloud computing is being felt by broadcasters throughout the world. Scaling delivers incredible resource and the levels of resilience available from international public cloud vendors is truly eye watering. It’s difficult to see how a…

Practical SDI And IP - Part 2

In this second instalment of our extended article looking into the practical applications of SDI and IP we look at how SDI is naturally plug-and-play and works with well-defined formats whereas IP is dynamic and more versatile but requires greater…