But these features rely on dynamic and complex IP-based video delivery platforms, which in turn makes quality assurance extremely challenging. What does quality assurance look like for OTT services, and how are network-side monitoring solutions supporting the continued drive towards excellent QoE (Quality of Experience)?

Network-Side

Before discussing quality, it is necessary to clarify the scope of “network-side”.

Network-side is inherently focused on service-level measurements and the impacts and actions that span multiple streams to multiple end customers simultaneously. This contrasts with client-side that is inherently focused on an individual customer experience (albeit data can be aggregated for an entire audience).

One of the primary challenges with OTT quality assurance is that different consumers can have very different service experiences for a wide range of reasons. The dynamic nature of networks and internet-based delivery is a general risk to streaming video quality. To manage quality, it is therefore imperative to understand network paths between origination points and the receiving devices (see Figure 1). And to proactively protect QoE, it is necessary to focus on network-side measures and network-side corrective actions.

But don’t think that “network side” means it is impossible to understand individual customer experience. On the contrary, from the network side it is possible – and very necessary – to understand quality of service (QoS) to all as well as quality of experience (QoE) to individuals (see below).

Because of the complex OTT delivery network reality (see Figure 1), OTT operators have a tough job to deliver excellent customer service. Not only should they be able to monitor QoS and QoE, but they should also be able to pinpoint incidents, understand root cause, and assure quality, all in real-time. OTT operators who are most sensitive to customer dissatisfaction from poor service should also be able to predict potential future incidents and avert them. 

The good news is that lessons have been learned about how to approach this problem by IPTV and Cable TV operators. Like OTT operators deploying Apps or websites, these operators deploy set-top-boxes. Over many years, it’s become clear that set-top-boxes can only tell you something has gone wrong. They cannot tell you exactly where the problem is in the delivery chain. Measuring at the OTT App is the equivalent of the set-top-box. But OTT is more complex in this regard because to know what went wrong requires monitoring and corrective action over multiple third-party network domains.

OTT operators deploy client-side monitoring for good reason – to know the activities and experiences of their customers. The capabilities of client-side monitoring cannot be fully replaced by network-side monitoring. But network-side capabilities must take the lead in assuring quality of service and quality of experience at the consumer’s device. OTT operators need to be able to see inside the delivery network and directly supervise QoS and QoE.

Quality In OTT

Quality for this article is grouped into three distinct categories – quality of content, quality of delivery and quality of experience. Each can individually affect a consumer’s decision to continue watching an OTT operator’s content.

Quality of content – including topics like image resolution, subtitles, audio loudness and more – is normally measured at the point of ingest to the delivery platform. In OTT, this typically means the file or live encoders. Today, as important VOD content is stored in various caching servers and OTT operators grapple with QoE subjects like stream start-up time, there is growing pressure for quality control on the content that is stored inside CDNs.

Quality of delivery is concerned with the actual delivery of content in the required bitrates, package-types and language versions into the locations where the OTT service is provided. This can be measured at an individual stream level or at an aggregate audience level. Of growing importance is the ability to predict and avoid quality-affecting situations, such as network congestion, whether they are under the OTT operator’s direct control or not. Quality of delivery is important for both live streaming services and VOD services.

Quality of experience is the ultimate measure, reporting how individual consumers are experiencing the video playout. Measures here include bitrates played, bit-rate consistency/changes, rebuffering rates, and start-up times. Individual consumers can have a very different experience of the same OTT service for a wide range of reasons – device performance, LAN performance and network contention are some examples. The OTT operator needs to manage this situation above all else to assure their end customer’ssatisfaction.

QoS And QoE Measures

Network-wide measures are often referred to as QoS measures, simply because the network-side can see performance of the whole service – i.e., every stream that passes through the network. This compares with client-side measures which can be grouped under the heading of QoE, because QoE is an individual experience measure. But this loose categorization hides the fact that QoE is very measurable from the network-side.

While there are good examples of network-side and client-side integrations in the industry, they are few. Today, the most typical situation for OTT operators to manage quality is to monitor stream quality at the delivery platform (i.e., the encoder or the origin) and to use client-side monitoring for understanding QoE and to balance traffic between CDNs. But as the cable and IPTV operators learned, to fix or prevent an incident you need to know exactly where the root cause is. Normally, in the multi-dimensional pull/push/cache system of OTT, the root cause is somewhere inside the complex set of network interconnections between the encoder and the device.

“Monitoring vendors need to cater for the current needs of multi-supplier OTT delivery, while adapting to the new world of hybrid on-prem/cloud architectures,” states Stuart Newton, VP Business Development at Telestream. “Solutions and deployment methods are evolving very quickly, and the real-time data from the monitoring will also provide a critical feedback mechanism for management of dynamic network architectures to assure quality for both live and on-demand OTT services”.

On the network-side, it is essential to measure performance of the network in a customer-centric manner. In other words, to measure the network using QoE metrics. It is already proven from integrations between network-side and client-side tools that it is possible to measure QoE from the network. It is also clear that this provides an early-warning tool for managing service quality. By interpreting network-side data correctly, OTT services can avert customer experience incidents. A classic example relates to the sustained bit-rate metric (Figure 2), which can be detected early by network-side tools.

In our next article (The World Of OTT - Quality Assurance Pt2), we look at how service monitoring and platform monitoring combine forces for OTT quality assurance, what the relative priorities are for efficiently monitoring OTT service quality, and how the growth of OTT is driving collaborative multi-network information sharing.