A comparison reveals a QoE difference between the heavyweights.
After focusing on the best video mobile app in the United Kingdom, and on the best OTT apps for Apple TV in the US, Witbe’s third QoE study looked at the video apps of four Canadian mobile operators; Bell Mobility, Fido, Telus Mobility and Videotron.
Witbe recently released the third edition of its “QoE Snapshots”. Quality of Experience (QoE) Snapshots test digital services to make information available on the true QoE delivered internationally. The QoE Snapshots should not serve as benchmarks, nor as rankings of operators by service, or by device. Rather, the goal of these QoE Snapshots is to provide a global overview of digital services, with multiple configurations and in various environments.
These OTT video streaming apps were used: YouTube, Netflix, Tou.TV, CBC Gem and Crave. Witbe Robots monitored QoE KPIs, such as App Launch Time, Login Time, Video Initial Buffering Time, and measured the Witbe VQ-MOS Score, a Witbe proprietary algorithm for video quality.
The environment for this Snapshot consisted of a Witbe “Video & Media” Mini Robot hosted in the Witbe Montreal Office and connected to four Apple iPhone 7 Plus running iOS version 12.1.2.
The four mobile operators tested were Bell Mobility, Fido, Telus Mobility and Videotron (in 4G LTE). Fido and Videotron advertise 60Mbps download speed, Bell and Telus advertise 85Mbps download speed.
In order to test the mobile operators, we used five of the most popular OTT video streaming apps in Canada: YouTube, Netflix, Tou.TV, CBC Gem, and Crave. Each mobile operator was tested with every OTT app. Each OTT app was playing the same video across network.
KPIs Measured and Verified
App Launch Time: Time between the Robot clicking on the app icon and the app being ready to be used. Apps are “force-closed” after every run.
Login Time: Time between the Robot clicking on the login button after credentials were input, and it being successfully logged-in. The apps are logged-out before every run.
Video Initial Buffering Time: Time between the Robot clicking on the play button and the video starting to play.
Witbe VQ-MOS Score: A score representing the video quality of the first 30 seconds of video, measured with Witbe’s Video Quality Mean Opinion Score algorithm.
The KPIs were measured every 12 minutes on every mobile operator, over a 24-hour period, for a total of more than 100 viewing sessions per mobile operator.
All graphs and other visual representations of KPIs in this QoE Snapshot are generated with Witbe Datalab, a new restitution interface for fault finding, root cause analysis and advance analytics.
According to the medians of the four KPIs that were measured, the overall QoE when viewing videos through mobile networks in Canada is good. Click to enlarge.
Performances KPIs (App Launch Time, Login Time, and Video Initial Buffering Time) were all within the optimal range determined based on our expertise. No major discrepancies were noted between the four mobile operators.
This shows that in the center of Montreal, all tested mobile operators are providing an equivalent level of Quality of Experience. Still, it is interesting to observe that even though all the operators are advertising a download speed above 50Mbps, the Witbe VQ-MOS score is averaging at 3, which is decent but not at the value of around 4 that would be expected for HD content.
It would be interesting to perform measurements in a drive test condition by using our Witbe “Video On-the Go” Robots. It would allow evaluation of operators’ performances in various environmental conditions, such as in a lower population density.
Drilling down into the App Launch Time KPI for Videotron and grouping the KPI by OTT app, revealed a problem with Netflix only and not with the other apps. Click to enlarge.
Some Problems are Bugs
While analyzing video availability for each mobile operators, we were surprised that Crave was the only app without a perfect 100% Video Availability Rate. 4% of Crave’s viewing sessions revealed a problem during the start of the video. The problem was not due to a network or service issue but to a bug in the app where a parental controls popup is mistakenly displayed at random when requesting a video to be played.
This type of error is the perfect example of why it is so important to test video apps as close as possible to real end-user conditions: with real apps, on real devices, and testing the service from one end to the other. Replicating users’ behaviors with active but non-intrusive Robots has been Witbe’s methodology since the beginning.
As a side note, by focusing on the performance for every mobile operator on the most popular Apps (Netflix and YouTube), we can see that YouTube has the fastest Video Initial Buffering Time and the lowest Witbe VQ-MOS score, and Netflix has the slowest Video Initial Buffering Time and the highest Witbe VQ-MOS score. This highlights once again the challenges faced by OTT app’s engineers when balancing load times and video quality on unmanaged networks. The same observation was noted during the first Witbe QoE Snapshot, comparing mobile OTT apps in the UK.
Witbe VQ-MOS is a ten-year R&D effort in psycho-acoustic and psycho-visual analysis of video streams. It works without referential, or previous knowledge of the video, in real-time. It is based on three main artifacts: jerkiness, blurriness and blockiness. Because the tests on Netflix and YouTube were using the same video content (a Bird Box Trailer), it was a good opportunity to compare VQ-MOS scores on the very same frame.
Performances measured over a 24-hour period in Montreal were not significantly impacted by peak traffic hours and were relatively constant. Click to enlarge.
The result of this Snapshot is quite encouraging for mobile operators in Canada, with good and constant performances. One area where they could improve appears to be the overall video quality which is lower than the one measured (on WiFi) in the UK, during Snapshot #1.
The next Witbe QoE Snapshot will compare channel change times on live TV channels.
You might also like...
Whenever I’m asked about my opinion on the transition to IP, I always state that the impact can’t be appreciated until its history is understood. This brings into context the need for broadcasters to educate and surround themselves wit…
As broadcasters accelerate IP migration, we must move from a position of theory to that of practical application. Hybrid solutions to integrate SDI, AES, MADI, and IP will be needed for many years to come, even with green field sites,…
Thanks to Over-the-Top (OTT) streaming video, content owners and broadcasters have a very different relationship with the end consumer – often a direct one.
OTT distribution is worlds apart from traditional unidirectional broadcasting in terms of its fundamental operation and viewing preferences. The internet is a rapidly expanding collection of service providers, many in direct competition, transferring broadcaster video and audio streams alongside many…
In the last two articles in this series we looked at why we need to monitor in OTT. Then, through analysing a typical OTT distribution chain, we sought to understand where the technical points of demarcation and challenges arise. In…