TelevisaUnivision is the world’s leading Spanish-language media company. Powered by the largest library of owned Spanish-language content and a prolific production capability, TelevisaUnivision is the top producer of original content in Spanish across news, sports and entertainment verticals.

As part of its digital transformation initiative, TelevisaUnivision sought to migrate its international broadcast operation from a traditional satellite-based network to an all-IP workflow. To achieve this vision, TelevisaUnivision built a best-in-class ecosystem of services and solutions to ensure high-quality broadcast operations.

TelevisaUnivision’s engineering team understood that a successful IP operation hinged upon the ability to overcome the inherently unpredictable behavior of IP networks (e.g., transient events), as well as the complexity of transporting live video over IP and the cloud.

To tackle this challenge head on, TelevisaUnivision equipped its state-of-the-art network architecture with a dedicated toolset that would provide real-time visibility into unexpected network behavior that could potentially impact video quality.

State-of-the-Art IP Network Architecture For Video Transport

To maximize reliability and relying on known industry best practices, TelevisaUnivision built a dual x dual redundant IP architecture for video distribution. The primary site has a dual encoder platform sending the content over SRT to two cloud availability zones. A secondary on-prem site (for site diversity) does the same. All network paths include switching and redundancy in case a router breaks down or needs maintenance (e.g., so.ware upgrade or replacement).

The availability zones included the same VPC, and each availability zone included a transcoder receiving a feed. The output was fed to four entry points in the cloud for content preparation and distribution across the USA to designated destinations. SRT was chosen for reliable video transport over IP.

Unexpected Quality Issues During IP Network Testing

Aware of the intrinsic challenges and complexities of video delivery over IP, TelevisaUnivision’s engineering team conducted comprehensive network testing to detect and address any possible issues. During this phase, the engineers noticed that the video wasn’t always stable, and the redundancy scheme fell short of their initial expectations.

All efforts to react to workflow alarms coming from ETR290 events, as well as periodic attempts to run industry-standard IPERF/Traceroute tests, did not reveal the smoking gun.

In line with best practices, TelevisaUnivision’s remediation efforts began with a number of IT fine-tuning activities such as MTU expansion, setting ACL rules, DNS update and setting QoS for each stream.

An in-depth network evaluation was then performed, which revealed several network problems:

• The cloud SRT transcoder kept stopping for a few minutes at a time.
• RTT tests performed by ping and IPERF showed inconsistent values.
• There was a 2-4% packet loss on the path to the cloud.
• Frame errors were reported with no network error indication by traditional network monitoring tools.
• The Operations team ran IPERF from time to time, collecting ETR 290 alarms and trying to correlate them to IPERF-detected events.

Following this evaluation, it became clear to TelevisaUnivision that in order to pinpoint the source of the problem, it needed better visibility into changing network behavior.

AlvaLinks’ Cloudrider™ Observability Suite Solution

To resolve this mission-critical challenge, TelevisaUnivision turned to AlvaLinks for a solution that would allow it to deep dive into the network behavior. Specifically, TelevisaUnivision wanted more insights into the basic network parameters of their MPLS and fiber circuits, such as latency, jitter, packet loss, and RTT.

Such an understanding would help its engineering team configure systems and equipment to handle the intricacies of IP networks, as well as adapting the flow to TelevisaUnivision’s specific network topology. They realized that better information would allow them to make faster decisions and avoid potential issues that could impact video quality.

“AlvaLinks was the only company we could find that provides a specialized video IP network observability solution. Their Cloudrider technology addressed our visibility issues head on, giving us the real-time data and historical information we need to understand network behavior and ensure quality of service.” Ralf Jacob, EVP Broadcast Operations & Technology at TelevisaUnivision.

Helping TelevisaUnivision And Its Partners Pinpoint Root Cause

AlvaLinks sofware agents were deployed on each on-prem encoder node and at the transcoder node in the cloud. Within 60 minutes of the CloudRider activation, the system detected RTT and latency anomalies that correlated to packet loss events. The CloudRider buffer emulation showed that the minimal buffer latency was higher than the SRT setting (during loss events). This misconfiguration made it difficult to support the network transient events that are “invisible” to typical IP monitoring tools.

Moreover, CloudRider discovered that all links exhibited similar routing and error events in the same time frame (a few seconds apart), while no external tra£c was detected – hinting at a synchronization problem. This behavior was highly unexpected as the links were perceived to be completely diverse from one another for redundancy purposes. Cloudrider also showed the network’s ability to reach 500mbits, but with a base floor packet loss of 4%.

“IPERF and MTR are great but make no sense to me. AlvaLinks gives me full visibility across all sources so I can understand my network.” Marcos Obadia, SVP Network Engineering at TelevisaUnivision.

Bottom Line

AlvaLinks technology uncovered network changes and behavior beyond those detected using traditional tools. Examples include the time correlation of events occurring in parallel on unrelated network paths and their impact on ETR 290 alarms, as well as incorrect bandwidth limits. AlvaLinks also captured high packet loss events (15- 22% packet loss) that were not registered using IPERF.

Armed with this knowledge, TelevisaUnivision reconfigured the encoder/transcoder SRT for better cloud contribution stability, which eliminated the recurring service interruptions. In addition, working together with AlvaLinks, the cloud provider was able to optimize traffic flows to ensure smooth service operations.