The transition to IP is having an impact on all aspects of broadcast from business to technical operations, workflow engineering and maintenance. One of the dramatic changes is how a system is documented. Gone are the good old days when an engineering drawing and wire run list and some service manuals were all the engineer required to maintain, diagnose and troubleshoot issues.
Engineering drawings told the whole story. The device type, number and type of inputs and outputs, what they did and where they went. Device A Video out (wire number#) to Video Patch (Panel x, Row x, Position x) to Device B Video in (wire number), the wire list showed the device name (sysname, make & model) for source and destination, signal type, port type, connector type and drawing number. There are unique drawings for audio, video, intercom, reference, management and control. An engineer could easily identify points in the system to test the quality or integrity of a signal and the signal path is straight forward to follow as it moves across the system between devices.
Traditional system block diagram. With today's IP system, such drawings provide little useful information.
On to IP – A whole new world of Documentation
IP introduces the need for more and different documentation. Line drawings do not provide the level of detail and information an engineer needs to operate, maintain, and troubleshoot the system. For IP systems, there is a single drawing type for network and basically two types of cable; either copper or fiber. There is a single or sometimes more than one network connection. However because the different signal types are on VLAN’s it’s the same connection type and IP is a duplex system so the port on the device can be both an input or an output. So in IP, the line drawings are limited in the information they provide. The same is true of the wire list. The basic information is the same, device ID (sysname, make and model), the port on the device and the port on the switch. Does the wire list transition to include the IP address, port configuration, VLAN ID for signal type and Subnet assignments?
What about the VLAN and subnet matrix showing which ones communicate with each other and which ones do not? This VLAN matrix shows which VLAN’s need to interoperate with each other. This is configured and manged using Access Control Lists (ACL) or Trunking within the IP network.
This VLAN matrix shows which VLAN’s need to interoperate with each other. This is configured and managed using Access Control Lists (ACL) or Trunking within the IP network. Chart courtesy Dave Schwarz, Diversified. Click to enlarge
In my article How Coax became a VLAN, we looked at how a unique signal transitions from a unique cable and connector to a VLAN on the network.
- Where are the IP addresses, port configuration and other information documented?
- What about unicast and multicast port addresses?
- Where are the VLAN’s and subnets listed and mapped?
- On the network, what about documenting the settings for each VLAN (QoS, packet shaping, path routing, bandwidth, spanning tree, duplex setting, etc.)
- What about Firewall rules and connection to the enterprise?
- What about the applications and services running on each of the devices?
- Or where is the documentation showing the integration between applications if there are API’s or middleware.
Then there are the configuration settings for each device and/or application. Oh yeah, and those pesky config files, now where did I put them? Is there an inventory of them? What about version control or SLA management?
What new documentation does the engineer need to solve a problem?
On a recent project, the LTO (Tape Archive) seemingly stopped writing to the library and the monitoring application started sending out critical alerts. All good- right? Well, maybe! So first we sat at our trustee workstation and using the KVM start toggling between all the devices and systems in the archive workflow. This was across three manufacturers' devices plus some intermediate devices and applications on various servers and accessed by the KVM over the network.
As we logged onto the LTO manager, it said everything is fine. Then we look at the asset manager and saw the assets pending to archive, So, what is the problem?
Well, there’s the LTO monitor and the LTO manager. The monitor showed all systems working normal, the manager showed the tapes in the LTO were all full therefore it needed new tapes loaded and configured for the system.
None of this was documented, and, we discovered the person who was trained on the system wasn’t on the project anymore.
The monitoring point was the KVM, accessed via a workstation. But, where is this documented? The tech needed to know what applications are in the archive system and how do they interact with each other and where do you look to diagnose the problem?
In the same project we have one tool (application) to monitor the network and the basic computer operations ie: OS, disk full, on/off, network connectivity. In addition, the two main product manufacturers each have their own SNMP tools that provide monitoring and some control of only their devices. And we have remote access for solving problems, but where is this documented?
And what happens when the mirrored redundant system you thought you had is really a 2-channel system using the same database and common storage? That was only discovered when a hard drive failed and took the whole system down. Again, documentation is key.
Back to Documentation
In addition to the “equipment list”, there needs to be a list or inventory of each device and its configuration including disk type, configuration (RAID, SAN, NAS)) and storage size, memory and OS. The applications running on each server should be included in this inventory.
It seems we need to modify the line drawing to show how applications communicate, show what language they use (XML, JSON). If IP glue is middleware and API’s, how do we show them in a document? What about file format changes as media moves, is that documented? If house format is ProRes and a file enters as XDCAM or DNxHD, it is just as important to know not only what folder it arrives in, but how the transcoder gets it, where the media is stored after the transcode and how the transfer was made; simple transfer, FTP, FileZilla?
Such information is key so when all the right things don't happen someone can figure out why. Think about the type of documentation a technician might need in order to find and correct such problems.
As an industry that has always prided itself on comprehensive documentation of the systems we build, shouldn’t we revisit this process as a critical component in IP design and documentation?
The transition to IP is impacting all aspects of the industry and because the industry depends on the engineers to keep it operating, let’s give them the tools they need.
Editor’s Note: Gary Olson has a book on IP technology, “Planning and Designing the IP Broadcast Facility – A New Puzzle to Solve”, which is available at bookstores and online.
Related Editorial Content
There have been many significant changes in the core technologies that make up the infrastructure of the broadcast facility. The cable plant is or was the heart of the broadcast facility. And there are many different types of cables. Now…
Video engineers and managers understand broadcast signals and their correct operating parameters. Yet when it comes to moving those signals over IP, technical answers are not easy to find. The result is that many video engineers are looking for help…
Media Streams and files have special needs when it comes to monitoring and maintenance. Contribution has one set of formats, production another and distribution its own set. Even so, we still need to monitor all of them.