Care and Feeding of an IP Facility: Part 1

A few years ago at a technical conference, the CTO of a major sports entity was challenging the storage vendors on how much storage capacity and throughput architecture would be needed to meet his needs. What they neglected to tell him was the cost, technology and resources required to monitor, maintain and manage it.

The IP change is a “YUGE” problem

While the industry argues over which standard or standards will rule the world (for at least 15 minutes), the people that need to make systems work on a daily basis, the engineering and tech support folks, are being ignored. And the more standards, protocols, API’s, middleware and “open” everything means such systems can be a nightmare to maintain for even the best trained personnel.

Let’s fantasize for a minute that we can actually build a working IP production system. Now what?

As a designer and project executive, I always say the building part is easy, it’s running these systems that is hard. I’m not diminishing the challenges, skills and knowledge that constructing a media infrastructure requires. However, when building a system, there is the luxury of being able to solve problems without worrying that those systems are live.

But once a system gets turned on and goes live or online, there is no turning it off. The integrators have completed their punch lists and moved on. Now its operations and engineering staff’s responsibility to assure everything works and stays that way.

Every time there is a major technology change, in comes the glue and black boxes that enable the transition to occur seamlessly. As the transition matures, the need for these little helpers diminishes because devices and systems begin to connect directly with each other.

The new MCR

The transmission or master control center is responsible for maintaining quality control over all inbound and outbound signals. This room is equipped with the tools to confirm the signals meet certain technical requirements. In the equipment rooms, there are dedicated stations that enable engineers to monitor the systems in a non-disruptive way. Devices and systems can be patched around, or if there are spares, taken to maintenance and/or to the engineering shop. The shop has the required tools and test equipment to get things working properly again.

Monitor, maintain and manage IP

However, the introduction of IP-centric software and hardware technology has blown up this entire model.

Just how does a staff monitor, maintain and manage these IP systems? Great question. Let’s examine how these tasks are performed in an IP environment.

Engineers view the IP transition as posing more technical than cultural obstacles. Source: Imagine Communications, “2016 Focus Forward Technology Trends Report”.

First, monitoring needs to be done in a non-invasive and non-disruptive way. For the purposes of this article we are not going to discuss the type of the knowledge and skill sets needed. We will assume that properly trained video and IT staffs are available and work well together.

The program origination center, aka Master Control, now looks more like a data center, often called a Network Operations Center, or NOC. The operators here monitor multi-view displays, which are managed by an application. The displays show all program feeds and the playlist for each delivery platform (applications), which are controlled by automation (another application). All of these applications run on servers. These servers are connected via a network and they pull both content and playlists from shared storage.

Content quality control is handled by an application doing file or stream checking. Content contribution arrives as files over a dedicated network or open internet using cloud services and/or proprietary software. As the content become streams, the network needs additional types of monitoring and management.

New monitoring solutions required

IP and file-based technology has introduced a new level of management to broadcast and production systems. Devices are monitored using SNMP (Simple Network Management Protocol) traps. The servers are monitored with tools that display CPU utilization, running processes, memory and storage, and this happens before we even get to the application.

Many devices and systems have their own management tools. These might be accessible using a web browser. Does this mean a dedicated computer, or is this a good application for virtual machines (VM)? If content is playing from a server how is it monitored? From where are the servers, storage and network going to be monitored and managed? While KVM (Keyboard, Video Mouse) extenders or matrix might be a first thought, using KVM to remotely manage is highly disruptive of operating services. This is where sharing applications like VNC (Virtual Network Computing) are appropriate solutions.

Transition to IP timeline. Ninety-four of survey takers put a decade-long cap on the duration of the transition to IP. This reflects widely held assumptions that media companies will transition, not in lockstep, but in accordance with numerous company-specific factors over a window of time consistent with most major technology evolutions and in line with traditional technology refresh cycles. Source: Imagine Communications, “2016 Focus Forward Technology Trends Report”.

Transition to IP timeline. Ninety-four of survey takers put a decade-long cap on the duration of the transition to IP. This reflects widely held assumptions that media companies will transition, not in lockstep, but in accordance with numerous company-specific factors over a window of time consistent with most major technology evolutions and in line with traditional technology refresh cycles. Source: Imagine Communications, “2016 Focus Forward Technology Trends Report”.

IP system maintenance can be a Pandora’s Box

What about network bandwidth or congestion? This needs to be monitored. What about managing the storage itself? There are applications that manage storage retention policies. But, what happens when utilization peaks and it’s time to load balance? Or, consider a worst case scenario where the storage system fills to capacity.

Maintenance for IP systems opens a Pandora’s Box! First, both hardware and software come with Service Level Agreements (SLAs) that are typically a percentage or the purchase price. Before we even get to the dedicated application, regular updates must be applied to the operating system. Then there is threat protection (Anti-Virus, intrusion protection, security), which must be kept current. Also, what about firmware updates to firewalls and switches?

Checking for updates, determining whether they will create a problem or interfere with a running application are all part of maintenance. What happens when vendor A upgrades their software? Will those changes effect the middleware and API’s that are used by external systems? What if a new codec or IP standard comes out? Is the update considered a patch or does it require some custom code and who does the install?

Remote access

The shop is now a NOC and if there’s a problem, accessing the media systems can be done remotely. IP introduces remote access as a way to monitor, manage and maintain the systems. Monitoring systems can be configured to detect many errors. They can also send alerts and messages over email and text (SMS). Remote access is the ability to use either open internet or dedication access to connect into the network or troubled system. Once connected, the engineer has the ability to diagnose and correct the issue.

As I stated earlier, this access needs to be non-invasive and non-disruptive. Taking control away from an operator or automation system and then not having the ability to restore control is not a healthy way to solve a problem. It only creates more issues.

This is the first in a series that will look at the tools, methods and technologies to monitor, maintain and manage an IP facility. 

​Gary Olson has written a book on the conversion to IP, “Planning and Designing the IP Broadcast Facility – A New Puzzle to Solve”. It is available from major booksellers.

​Gary Olson has written a book on the conversion to IP, “Planning and Designing the IP Broadcast Facility – A New Puzzle to Solve”. It is available from major booksellers.

Some additional The Broadcast Bridge resources written by Gary Olson are listed below.

Let us know what you think…

Log-in or Register for free to post comments…

You might also like...

eBook:  Preparing for Broadcast IP Infrastructures

This FREE to download eBook is likely to become the reference document you keep close at hand, because, if, like many, you are tasked with Preparing for Broadcast IP Infrastructures. Supported by Riedel, this near 100 pages of in-depth guides, illustrations,…

Practical Broadcast Storage - Part 3

Artificial Intelligence (AI) has made its mark on IT and is rapidly advancing into mainstream broadcasting. By employing AI methodologies, specifically machine learning, broadcasters can benefit greatly from the advances in IT infrastructure innovation and advanced storage designs.

IP Control Uncovered

As broadcasters continue to successfully migrate video and audio to IP, attention soon turns to control, interoperability, and interconnectivity to improve reliability and efficiency. In this article, we investigate IP control as we move to IP infrastructures.

Practical Broadcast Storage - Part 2

Broadcast systems are renowned for their high speed and high capacity data demands. Up to recently, they relied on bespoke hardware solutions to deliver the infrastructure required for live real-time uncompressed video. But new advances in IT data storage have…

Practical Broadcast Storage - Part 1

Broadcast and IT technology collaboration is continuing its journey with the cross-over between them becoming ever clearer. Storage is a major growth area with advanced methods achieving greater efficiencies and resilience. And as broadcast repeatedly demands more and more capacity,…