Witbe recently released the third edition of its “QoE Snapshots”. Quality of Experience (QoE) Snapshots test digital services to make information available on the true QoE delivered internationally. The QoE Snapshots should not serve as benchmarks, nor as rankings of operators by service, or by device. Rather, the goal of these QoE Snapshots is to provide a global overview of digital services, with multiple configurations and in various environments.

Test Environment

These OTT video streaming apps were used: YouTube, Netflix, Tou.TV, CBC Gem and Crave. Witbe Robots monitored QoE KPIs, such as App Launch Time, Login Time, Video Initial Buffering Time, and measured the Witbe VQ-MOS Score, a Witbe proprietary algorithm for video quality.

The environment for this Snapshot consisted of a Witbe “Video & Media” Mini Robot hosted in the Witbe Montreal Office and connected to four Apple iPhone 7 Plus running iOS version 12.1.2.

The four mobile operators tested were Bell Mobility, Fido, Telus Mobility and Videotron (in 4G LTE). Fido and Videotron advertise 60Mbps download speed, Bell and Telus advertise 85Mbps download speed.

In order to test the mobile operators, we used five of the most popular OTT video streaming apps in Canada: YouTube, Netflix, Tou.TV, CBC Gem, and Crave. Each mobile operator was tested with every OTT app. Each OTT app was playing the same video across network.

KPIs Measured and Verified

App Launch Time: Time between the Robot clicking on the app icon and the app being ready to be used. Apps are “force-closed” after every run.

Login Time: Time between the Robot clicking on the login button after credentials were input, and it being successfully logged-in. The apps are logged-out before every run.

Video Initial Buffering Time: Time between the Robot clicking on the play button and the video starting to play.

Witbe VQ-MOS Score: A score representing the video quality of the first 30 seconds of video, measured with Witbe’s Video Quality Mean Opinion Score algorithm.

The KPIs were measured every 12 minutes on every mobile operator, over a 24-hour period, for a total of more than 100 viewing sessions per mobile operator.

All graphs and other visual representations of KPIs in this QoE Snapshot are generated with Witbe Datalab, a new restitution interface for fault finding, root cause analysis and advance analytics.

Performances KPIs (App Launch Time, Login Time, and Video Initial Buffering Time) were all within the optimal range determined based on our expertise. No major discrepancies were noted between the four mobile operators.

This shows that in the center of Montreal, all tested mobile operators are providing an equivalent level of Quality of Experience. Still, it is interesting to observe that even though all the operators are advertising a download speed above 50Mbps, the Witbe VQ-MOS score is averaging at 3, which is decent but not at the value of around 4 that would be expected for HD content.

It would be interesting to perform measurements in a drive test condition by using our Witbe “Video On-the Go” Robots. It would allow evaluation of operators’ performances in various environmental conditions, such as in a lower population density.

Some Problems are Bugs

While analyzing video availability for each mobile operators, we were surprised that Crave was the only app without a perfect 100% Video Availability Rate. 4% of Crave’s viewing sessions revealed a problem during the start of the video. The problem was not due to a network or service issue but to a bug in the app where a parental controls popup is mistakenly displayed at random when requesting a video to be played.

This type of error is the perfect example of why it is so important to test video apps as close as possible to real end-user conditions: with real apps, on real devices, and testing the service from one end to the other. Replicating users’ behaviors with active but non-intrusive Robots has been Witbe’s methodology since the beginning.

As a side note, by focusing on the performance for every mobile operator on the most popular Apps (Netflix and YouTube), we can see that YouTube has the fastest Video Initial Buffering Time and the lowest Witbe VQ-MOS score, and Netflix has the slowest Video Initial Buffering Time and the highest Witbe VQ-MOS score. This highlights once again the challenges faced by OTT app’s engineers when balancing load times and video quality on unmanaged networks. The same observation was noted during the first Witbe QoE Snapshot, comparing mobile OTT apps in the UK.

Witbe VQ-MOS is a ten-year R&D effort in psycho-acoustic and psycho-visual analysis of video streams. It works without referential, or previous knowledge of the video, in real-time. It is based on three main artifacts: jerkiness, blurriness and blockiness. Because the tests on Netflix and YouTube were using the same video content (a Bird Box Trailer), it was a good opportunity to compare VQ-MOS scores on the very same frame.

The result of this Snapshot is quite encouraging for mobile operators in Canada, with good and constant performances. One area where they could improve appears to be the overall video quality which is lower than the one measured (on WiFi) in the UK, during Snapshot #1.

The next Witbe QoE Snapshot will compare channel change times on live TV channels.