At the start of 2013, BCE at RTL City was a hole in Luxembourg’s ground and in less than four years they were on air broadcasting 35 different channels across Europe and Singapore. Costas Colombus is BCE’s Special Projects Manager and gave The Broadcast Bridge a unique insight into how they made this mammoth installation work, including describing the issues and how they overcame them along the way.
Transitioning to IP is a major challenge for any broadcaster, especially when moving an existing facility. Greenfield sites have it difficult too, and learning how they successfully made their installations work will help us all.
BCE’s wish was to be format agnostic and ready for future technologies such as 1080P, 4K and HDR. Preliminary requirements dictated the system should be flexible, scalable, redundant and reliable. Aspirations for us all, but often difficult to achieve. Cable installation was identified as a major area for cost savings so fiber was used extensively. To de-risk reliance on a single supplier, multiple vendors and COTS equipment was used wherever possible.
Waiting for ST2110
During the design phase ST2110 had not been ratified so Costas and his team needed to decide on the specification of the system that was available at the time, but would allow for future upgrade to ST2110. SMPTE 2022-6/-7 was chosen for SDI video and AES67 for audio over IP, and DANTE was used for RTL Radio.
From a hole in the ground, BCE at RTLCity was on air in less than four years
To reduce the amount of copper cable required, fiber was used wherever possible, to the extent where all connections between buildings, floors and rooms were fiber, and the only cables carrying video that exist are intra connections within rooms and racks.
And this is where the challenges began.
Fiber is an established technology and has been in active use since the 1970’s. However, the combination of interfacing it with SDI, and the variable quality of fiber available, caused Costas no end of issues. It soon became apparent that fiber and the associated connectors, SFP and QSFP’s vary wildly in quality and more attention had to be paid to the procurement stage. Systems could fail in a non-predictable manner.
During investigations into the issues Costas stumbled across another challenge, the differing attitudes and working practices of broadcast and IT engineers. Broadcast engineers are used to working in new, dynamic, fast paced environments where equipment often goes wrong and creative, out-of-the-box solutions must be found.
IT engineers have their training and influence in the well-ordered world of the banking industry. Many of the working practices IT engineers work to are well established in the finance market, and their ITIL processes exist for good reason as decimal points can make an enormous difference in the finance sector.
Information Technology Infrastructure Library (ITIL) is a set of detailed procedures used in the IT industry and are designed to bring order to a system to stop engineers jumping into systems and fixing them without proper authorization. This method of working is usually alien to most broadcast engineers, but in the IT world there must be an audit trail for every change or modification. As networks typically affect so many different services, one change on one part of the network could have unintended consequences on another. ITIL aims to stop these actions but the system only really works in an established working environment.
IP television is at the cutting edge of technology and driving IT to its limits. Engineers are finding obstacles nobody had predicted or expected. Consequently, when something goes wrong it must be fixed immediately, there is no time to raise a help desk ticket or have five meetings about the proposed fix.
Forward With ITIL
Broadcast television will eventually move to the ITIL model and we have already witnessed this in SDI playout facilities. The last thing anybody wants is an unscheduled software upgrade causing outage on fifty pay-per-view channels. But we must have working systems before these processes can be instigated.
ITIL is used to maintain, install and modify systems that are understood, well-ordered and defined. Broadcast IP is not there yet. But it’s fair to say that if we had stuck with SDI playout systems we would probably be using ITIL extensively now. However, to facilitate the new business models of flexibility, scalability and COTS, we must move to IP and we will continue to find solutions to complications we do not yet know we have.
Improve Equipment Monitoring
One of the areas Costas highlighted for equipment design improvement is monitoring the SDI-IP interface where television turns from copper coax to fiber. He also identified varying quality in SFP/QSFP light sources causing packet loss. Using light power meters from the telco industry, BCE’s engineers could prove that fiber channels were failing, but in a further twist this always wasn’t due to light transmitters.
Terminating fiber proved to be fraught with the same cost issues affecting fiber. Inferior quality connections don’t tend to just fail, they start by creating unpredictable erratic packet loss. Channels would work one minute but not the next, again the origin of the failure proved difficult to find. Was packet loss due to issues in the SDI domain or IP? A failing SDI connection can cause packet loss on an IP connection due to the amount of erratic data being created.
BCE solved the dilemma by assigning a small team of wiremen (fiber-men) to running and terminating or connecting fiber exclusively, a task requiring great skill, patience and experience, and giving highly specified fiber specifications to his procurement department ensuring the use of high quality fiber. Dirt is the fiber’s greatest enemy as even small amounts of debris can cause major headaches for video-IP installations running at the limits of their capabilities.
In article two of “BCE Going Deeper Part 2 – Choosing IP Routers” we continue our conversations with Costas and his colleagues.
You might also like...
As the wider broadcast industry picks up the pace with virtualized, cloud-native production systems we take a look at what audio vendors currently have available and what may be on the horizon.
The Sponsors Perspective: Proactively Monitor IP Video Networks & Essences With Inspect 2110 & PRISM
For over two decades Telestream has streamlined the ingest, production, and distribution of digital video and audio. Today, compared to its SDI/AES-based predecessors, IP video adds exciting new challenges to these workflows.
IP connectivity delivers flexibility and scalability but making the theory work often requires integrated solutions that are adaptable, open, and promote interconnectivity.
In the previous article in this series, we looked at layer-2 switching and layer-3 routing. In this article, we look at Software Defined Networks and why they are so appealing to broadcasters.
It was late in 2018 when a major public broadcaster in the UK came to London-based 7FiveFive, a technology solutions provider, with a growth challenge. Their postproduction department had about 75 edit positions throughout the building working off a shared storage SAN…