Maintaining a high quality of experience for viewers requires knowledge of both QoS and QoE measurement capability.
There are some immutable laws for success in any consumer-oriented industry. Potential customers need to first know the product or service exists, and then they have to be persuaded that it’s desirable or necessary.
That is but the beginning. Customers have to be satisfied that a service or product is affordable and represents good value, and in this respect the product has to stand comparison with competing offerings. It’s fairly easy to compare value in a field where there’s a lot of competition, for example in the purchase of a house or a car. And although there’s no way to road test a house before you buy it, you can certainly do that with a car – or you can read road tests and reviews, as you can with most other consumer goods. In the case of services rather than goods, there isn’t the same history of ‘road testing’ information, but increasingly customers make their experiences known on the internet, so the astute buyer is able to make a more informed choice than before.
Once committed to a contract for a service such as phone, internet or digital media, a consumer is generally inclined to stick with the choice unless there are compelling reasons to change. The most likely reason to look elsewhere is the failure of the service to live up to expectations. Because a consumer chooses the service on the basis of the visible benefits – number of channels, type of content, bandwidth, and so on – any disappointment will probably arise from poor service quality.
Today’s consumers have a wide choice of available media. Capturing those eyeballs for a test may be easy. Holding them for the longer term may be much harder.
Digital media businesses are highly completive in a market where the customer can switch quickly if not satisfied. This means providers need to know when their viewers feel they are getting a satisfactory service, and when they aren’t. In strict terms the only way to evaluate how customers feel about a service is by asking them, in much the way that market research does. Good market research measures a lot of factors that are influential on consumer opinion and these include economic pressures and social trends that have nothing to do with the quality of the product or service itself.
Importance of proper metrics
For those whose responsibility it is to ensure that the service itself is of unimpeachable quality, the idea of measuring ‘consumer satisfaction’ has come to be embodied in the concept of (QoE) monitoring. Conventional wisdom says that QoS (Quality of Service) monitoring can provide diagnostic information on the devices in the delivery chain and the transport streams. The QoE element is intended to provide a user-centric evaluation of the service, pointing to any faults that have an impact on the user’s experience.
The established concept of QoE was really rooted in the assumption that QoS alone is insufficient to guarantee good service quality at the consumer end. And when QoE first originated in the telecoms industry this was definitely true. This lead to the development of a new measurement methodology called MOS (Mean Opinion Scoring). It was deployed when telcos were developing new infrastructure or, more recently, services such as VoIP.
A panel of expert assessors would listen to the service and note their evaluation of quality while key phrases such as ‘You will have to be very quiet’, ‘There was nothing to be seen’, and ‘They worshipped wooden idols’ were transmitted over the line. The experts would record their scores, assessing any impairments on a scale of 1-5, from ‘imperceptible’ to ‘very annoying’. The scores would then be averaged and weighted using the kind of statistical manipulations common in social sciences and market research.
Tools for monitoring digital media
The same basic concept of QoE monitoring later took hold in digital media monitoring and MOS spawned ‘VideoMOS’ and ‘AudioMOS’ – criteria which brought opinion-scoring methodologies into the media arena.
Historically telcos used a common phrase when asked about quality problems, “Trouble’s leaving here okay.” While a humorous response, today’s content deliverers cannot afford to adopt such a cavalier response to consumer concerns.
But media providers need to know what real users are experiencing, rather than panels of expert assessors. So MOS evaluation became ‘robotized’ and algorithms were developed to simulate those subjective reactions from ‘imperceptible’ to ‘very annoying’. The robots ‘watch’ the service and the data is fed into the algorithmic engine, which attempts to simulate the subjective reaction of the viewer, with scores for factors such as ‘jerkiness’, ‘blurriness’ and ‘blockiness’.
But subjectivity is complicated, and simulation is an unreliable way to assess service quality. A human viewer watching a top-quality 1080i transmission of the Super Bowl, followed on the same channel by the 1940s B&W movie Casablanca might be fully satisfied. Yet in a robotized QoE ‘subjective’ assessment based on MOS criteria, the Super Bowl would score high, while Casablanca would be marked way down for ‘blurriness’, scratches and other artefacts, because of its lack of resolution and color. Distorted by such results, the data from this kind of QoE system becomes more or less meaningless to the provider.
QoS can do more
We are now in an era where integrated end-to-end monitoring systems for QoS can do much more to ensure quality for the end user. By going far beyond mere verification that the equipment is functioning correctly, a proper solution can give the service provider total visibility and control over the integrity of the signal throughout the entire network—or almost the entire network.
This is because with OTT services the content delivery chain includes the internet. The ability to check the quality of the service at the point of reception by the consumer is still essential if we are to have confidence that the user experience is as good as it should be.
Combining QoS and QoE can result in a more complete monitoring system, which can help ensure a higher level of customer satisfaction.
QoS and QoE—part of same continuum
But the traditional concept of separate domains for QoS and QoE is now out of date. Most of today’s advanced monitoring systems consider QoS and QoE as part of the same continuum, both capable of delivering objective, empirical data without the artificial and spurious distraction of simulated ‘opinion’.
QoS and QoE data often come from different points in the delivery chain. QoE data arrives in large part from the point of consumption and is represented in different ways to the monitoring staff. But it’s the close correlation between QoS and QoE data within the same integrated environment that allows operators to reach and maintain a higher quality level.
If your monitoring system can correlate and compare data on the factors that affect quality throughout the whole delivery chain, from ingest at the head end to viewing on the smartphone, then you have a completely trustworthy objective set of data on which to assess – in real time – both quality of service and of experience.
And beyond that, if you want to measure subjective opinion, there’s always market research.
You might also like...
Distributing error free IP media streams is only half the battle when building reliable broadcast infrastructures. SDP files must match their associated IP media essence or downstream equipment will not be able to decode it. In this article we dig…
Modern broadcast facilities adopting video and audio over IP have found themselves working with thousands, or tens of thousands, of IP streams. Expecting humans to keep track of all these flows, monitor their quality and efficiently fault find is a…
Delivering determinant latency is more important than fighting variable latency, even if it is small. In this article, we look at how codec design and JPEG-XS can scale to make the best use of network bandwidth while keeping latency predictable.
At this year’s IBC Show in Amsterdam, finally in-person after two years, remote production solutions were scattered throughout the exhibition floors, to no real surprise. Reduced costs, travel and shipping expenses, scalable infrastructure and efficient use of resources were a…
The more digital TV technology advances, the more the fundamental elements of TV remain the same.