Test, QC & Monitoring Global Viewpoint – August 2016

Measuring OTT QoE Requires Different Thinking: Part 1

Measuring viewer Quality of Experience (QoE) requires different and, perhaps, unfamiliar test and measurement methods.

A recent Forbes magazine article predicted that OTT will have the largest market growth rate and dominate the video streaming market from 2016 to 2021, due to the increasing use of digital platforms for branding and marketing of products.The popularity of such video is responsible for two fundamental shifts in consumer behavior: higher peak bandwidth levels and heightened subscriber sensitivity to video quality.

During periods of peak utilization, resources on the network are more prone to congestion and from the viewers’ perspective, congestion can visibly be see as a degradation in video streaming quality. Therefore, measuring the customer’s quality of experience (QoE) requires measurement of video QoE.

OTT video is delivered in two primary streaming mechanisms: progressive video, in which sections of a single file are delivered in bursts; and adaptive video, in which chunks of differing display quality are delivered based upon the network’s transport capabilities.

From a transport layer perspective, HTTP over TCP is the dominant transport mechanism. This can introduce two primary dimensions to video quality: display quality (fidelity) and transport quality (stalling). Perhaps counter intuitively, video QoE is difficult to correlate to TCP performance metrics (e.g., jitter and packet loss).

For a representative assessment of QoE from the viewer’s perspective, the video QoE solution must use different methods to measure QoE of progressive video and QoE of adaptive video, and must take into account both display quality and transport quality. To fulfill these requirements, the solution must be both protocol- and container aware.

Introduction to Internet Video

Third-party video applications that run over-the-top (OTT) of a communications service provider’s(CSP) transport layer are the dominant drivers of bandwidth on today’s consumer Internet. In regions where it is available, Netflix either has emerged or is emerging as the largest single source of Internet traffic, while YouTube is the largest globally.

This popularity of video in general, and OTT video in particular, is responsible for two fundamental shifts in consumer behavior:

1. Higher peak bandwidth levels: because OTT video is an “on demand” application, it drives traffic when it is viewed, which typically peaks in the range of 8pm to 11pm. Previously, video content was often acquired in bulk via P2P networks for later consumption.

2. Viewers have a heightened sensitivity to quality. Video is a sensory experience with rapidly changing sights and sounds, so shifts in quality (e.g., stalls, pixelization, compression artifacts, shifts up or down in resolution, changes in frame-rate) are now instantly recognized by the viewer.

From the CSP, perspective, OTT video is decreasing network efficiency in the macro sense, in that the peak-to-trough ratio is increasing and there is a large amount of available but unutilized capacity throughout the day. During these high peaks, the network is more prone to congestion. From the viewers’ perspective, congestion can visibly manifest as degradation in video streaming quality.

To understand how to measure the video quality of experience from the viewers’ perspective, one must first understand how OTT video is delivered.

There are two primary forms of streaming video on the Internet today: progressive and adaptive. While there are other video methods available e.g. RTP (Real-Time Protocol) multicast over UDP (User Datagram Protocol), they are not in common use today as a delivery mechanism for OTT video.

Progressive Video

In the progressive method, a single large file is ‘burst-paced’ into the network, with no consideration for available bandwidth. In this scenario, the media asset is a single video file stored on a server and HTTP ‘byte-ranges’ are used for seeking through the video. Each byte range may either come in the same TCP connection or a new connection.

For situations in which videos are available with different display quality choices (each of which corresponds to a different bitrate), the video client (or user) manually selects a particular resolution, which corresponds to a different file on the server. The result is that two users watching the “same” video at different resolutions are actually watching different files.

Once chosen, this display quality is constant, regardless of the network’s ability to transport sufficient data to maintain stall-free playback.

If one were to look at a cross-section of a progressive video stream, it would resemble Figure 1. The container includes information relevant to the video file and content, and the elementary streams correspond to video, audio, sub-titles, etc.

Click to enlarge.

Click to enlarge.

More specifically, the following information relevant to determining QoE is available from each layer:

● IP: Subscriber, CDN, Border Gateway Protocol (BGP) AS Path 

● Subscriber: physical location on network, service plan, device type

● TCP: nothing of relevance to QoE

● HTTP: asset (used to link multiple chunks together), duration, stall information (transport quality)

● Container: codec, resolution, bitrate (display quality), CDN

● Elementary stream: bytes transferred

Adaptive Video

Quite different from the progressive method, adaptive video modulates the display quality based on the network’s available transport capacity (i.e., bandwidth). It achieves this effect by fetching ‘chunks’ of the video in a piece-wise manner; the chunk chosen is of the maximum deliverable display quality. At the beginning, the video client requests the first chunk, and starts playing it. If this chunk takes too long to deliver, then the next chunk will be requested at lower display quality; if this initial chunk delivered especially quickly, then the next chunk will be requested at a higher display quality.

An illustration of an adaptive video flow ‘in flight’ is provided in Figure 2. In this example, the viewer gets two chunks of low  definition video, followed by two chunks of standard definition, then two of high definition, and finally two again of standard, all in response to the dynamically-changing transport capacity of the network.

Click to enlarge.

Click to enlarge.

The most common adaptive video methods are Microsoft Silverlight, Apple HTTP Live Streaming (HLS), and Adobe Real-time Media Protocol (RTMP). They each operate in a similar fashion: the video asset is stored on disk in multiple target bitrate /resolutions. A chunk is generally between 10 MB and 100 MB in size, depending on display quality.

In the same fashion as progressive video, each chunk may either come in the same TCP connection or a new connection. From a cross-section perspective, adaptive video is similar to progressive, with one key difference. In this case, an additional ‘protocol’ layer provides parameters for seeking chunks, obtaining meta-data, etc. More specifically, the following information relevant to determining QoE (and also additional business intelligence) is available from each layer:

Click to enlarge.

Click to enlarge.

● IP: Subscriber, CDN, BGP AS Path

● Subscriber: physical location on network, service plan, device type

● TCP: nothing of relevance to QoE

● HTTP: asset (used to link multiple chunks together), ‘protocol’, CDN

● Protocol: duration, stall information (transport quality)

● Container: codec, resolution, bitrate (display quality)

● Elementary stream: bytes transferred

With this tutorial as background, we will continue our discussion in Part 2 of this series, which will focus on how to make to make accurate QoE tests. That article will be published on The Broadcast Bridge August 15th, 2016.

Editor Note: This article is based on the white paper,“Internet Video Quality of Experience From the Viewer’s Perspective” from Sandvine, all rights reserved. It is available at the highlighted link.

Let us know what you think…

Log-in or Register for free to post comments…