Field Report: Video QoE Snapshot

After focusing on the best video mobile app in the United Kingdom, and on the best OTT apps for Apple TV in the US, Witbe’s third QoE study looked at the video apps of four Canadian mobile operators; Bell Mobility, Fido, Telus Mobility and Videotron.

Witbe recently released the third edition of its “QoE Snapshots”. Quality of Experience (QoE) Snapshots test digital services to make information available on the true QoE delivered internationally. The QoE Snapshots should not serve as benchmarks, nor as rankings of operators by service, or by device. Rather, the goal of these QoE Snapshots is to provide a global overview of digital services, with multiple configurations and in various environments.

Test Environment

These OTT video streaming apps were used: YouTube, Netflix, Tou.TV, CBC Gem and Crave. Witbe Robots monitored QoE KPIs, such as App Launch Time, Login Time, Video Initial Buffering Time, and measured the Witbe VQ-MOS Score, a Witbe proprietary algorithm for video quality.

The environment for this Snapshot consisted of a Witbe “Video & Media” Mini Robot hosted in the Witbe Montreal Office and connected to four Apple iPhone 7 Plus running iOS version 12.1.2.

The four mobile operators tested were Bell Mobility, Fido, Telus Mobility and Videotron (in 4G LTE). Fido and Videotron advertise 60Mbps download speed, Bell and Telus advertise 85Mbps download speed.

In order to test the mobile operators, we used five of the most popular OTT video streaming apps in Canada: YouTube, Netflix, Tou.TV, CBC Gem, and Crave. Each mobile operator was tested with every OTT app. Each OTT app was playing the same video across network.

KPIs Measured and Verified

App Launch Time: Time between the Robot clicking on the app icon and the app being ready to be used. Apps are “force-closed” after every run.

Login Time: Time between the Robot clicking on the login button after credentials were input, and it being successfully logged-in. The apps are logged-out before every run.

Video Initial Buffering Time: Time between the Robot clicking on the play button and the video starting to play.

Witbe VQ-MOS Score: A score representing the video quality of the first 30 seconds of video, measured with Witbe’s Video Quality Mean Opinion Score algorithm.

The KPIs were measured every 12 minutes on every mobile operator, over a 24-hour period, for a total of more than 100 viewing sessions per mobile operator.

All graphs and other visual representations of KPIs in this QoE Snapshot are generated with Witbe Datalab, a new restitution interface for fault finding, root cause analysis and advance analytics.

According to the medians of the four KPIs that were measured, the overall QoE when viewing videos through mobile networks in Canada is good. Click to enlarge.

According to the medians of the four KPIs that were measured, the overall QoE when viewing videos through mobile networks in Canada is good. Click to enlarge.

Performances KPIs (App Launch Time, Login Time, and Video Initial Buffering Time) were all within the optimal range determined based on our expertise. No major discrepancies were noted between the four mobile operators.

This shows that in the center of Montreal, all tested mobile operators are providing an equivalent level of Quality of Experience. Still, it is interesting to observe that even though all the operators are advertising a download speed above 50Mbps, the Witbe VQ-MOS score is averaging at 3, which is decent but not at the value of around 4 that would be expected for HD content.

It would be interesting to perform measurements in a drive test condition by using our Witbe “Video On-the Go” Robots. It would allow evaluation of operators’ performances in various environmental conditions, such as in a lower population density.

Drilling down into the App Launch Time KPI for Videotron and grouping the KPI by OTT app, revealed a problem with Netflix only and not with the other apps. Click to enlarge.

Drilling down into the App Launch Time KPI for Videotron and grouping the KPI by OTT app, revealed a problem with Netflix only and not with the other apps. Click to enlarge.

Some Problems are Bugs

While analyzing video availability for each mobile operators, we were surprised that Crave was the only app without a perfect 100% Video Availability Rate. 4% of Crave’s viewing sessions revealed a problem during the start of the video. The problem was not due to a network or service issue but to a bug in the app where a parental controls popup is mistakenly displayed at random when requesting a video to be played.

This type of error is the perfect example of why it is so important to test video apps as close as possible to real end-user conditions: with real apps, on real devices, and testing the service from one end to the other. Replicating users’ behaviors with active but non-intrusive Robots has been Witbe’s methodology since the beginning.

As a side note, by focusing on the performance for every mobile operator on the most popular Apps (Netflix and YouTube), we can see that YouTube has the fastest Video Initial Buffering Time and the lowest Witbe VQ-MOS score, and Netflix has the slowest Video Initial Buffering Time and the highest Witbe VQ-MOS score. This highlights once again the challenges faced by OTT app’s engineers when balancing load times and video quality on unmanaged networks. The same observation was noted during the first Witbe QoE Snapshot, comparing mobile OTT apps in the UK.

Witbe VQ-MOS is a ten-year R&D effort in psycho-acoustic and psycho-visual analysis of video streams. It works without referential, or previous knowledge of the video, in real-time. It is based on three main artifacts: jerkiness, blurriness and blockiness. Because the tests on Netflix and YouTube were using the same video content (a Bird Box Trailer), it was a good opportunity to compare VQ-MOS scores on the very same frame.

Performances measured over a 24-hour period in Montreal were not significantly impacted by peak traffic hours and were relatively constant. Click to enlarge.

Performances measured over a 24-hour period in Montreal were not significantly impacted by peak traffic hours and were relatively constant. Click to enlarge.

The result of this Snapshot is quite encouraging for mobile operators in Canada, with good and constant performances. One area where they could improve appears to be the overall video quality which is lower than the one measured (on WiFi) in the UK, during Snapshot #1.

The next Witbe QoE Snapshot will compare channel change times on live TV channels.

You might also like...

Wi-Fi Gets Wider With Wi-Fi 7

The last 56k dialup modem I bought in 1998 cost more than double the price of a 28k modem, and the double bandwidth was worth the extra money. New Wi-Fi 7 devices are similarly premium-priced because early adaptation of leading-edge new technology…

NAB Show 2024 BEIT Sessions Part 2: New Broadcast Technologies

The most tightly focused and fresh technical information for TV engineers at the NAB Show will be analyzed, discussed, and explained during the four days of BEIT sessions. It’s the best opportunity on Earth to learn from and question i…

Chris Brown Discusses The Themes Of The 2024 NAB Show

The Broadcast Bridge sat down with Chris Brown, executive vice president and managing director, NAB Global Connections and Events to discuss this year’s gathering April 13-17 (show floor open April 14-17) and how the industry looks to the show e…

NAB Show 2024 BEIT Sessions Part 1: ATSC 3.0 And TV RF

A full-time chief engineer in good relationships with manufacturer reps and an honest local dealer should spend most of their NAB Show time immersed in BEIT sessions. It’s an incredible opportunity to learn from and personally question indisputable industry e…

Essential Guide: Network Observability

This Essential Guide introduces and explores the concept of Network Observability. For any broadcast engineering team using IP networks and cloud ecosystems for live video production, it is an approach which could help combat a number of the inherent challenges…