Quality of experience (QoE) in multimedia applications

One of the standard quality assessment methods for networked delivery of multimedia presentations is called quality of service (QoS), which specifically measures the throughput and reliability of the data transmission network. However, QoS disregards encoding/decoding behavior, the context in which the content is consumed, and most other characteristics of the transmitting and receiving endpoints. In recent years, convergence of the digital media industry and the information and communications technology (ICT) industry has led to a paradigm shift away from QoS and toward a broader concept: the end-to-end metric known as quality of experience (QoE).

Figure 1. The relationship between quality of service (QoS) and quality of experience (QoE) in a networked media handling system. EC: Error correction.

QoE considerably extends the concept of QoS (see Figure 1). While QoS primarily measures the accuracy of networked data delivery, QoE encompasses additional factors affecting the user's perception of multimedia presentation quality. Naturally, quality assessments differ, but as users have come to consume ever richer digital media on a progressively wider variety of devices and networks,¹ their expectations have tended toward increasingly higher quality, dependability, and security. Furthermore, certain elements that are important to users (including context, network conditions, and end-user device capabilities) remain largely outside the control of the creators, owners, and providers of content.

Our research has focused on addressing these conflicting issues, many of them non-trivial, with the goal of developing a formal definition of QoE. For example, one of the earliest works on QoE defines it as a measure of the impact of content on a specific user in a specific context.² This impact can be measured through a subjective assessment, or alternatively it can be estimated through a model based on parameters of content, specific user, and specific context. Another early approach at modeling QoE follows a traditional methodology (see Figure 2), in which user perception is measured by formal subjective evaluations. The results are taken as ground truth and are used as a basis for developing highly correlated objective metrics.³

Figure 2. Objective metrics for assessing QoE from the user's point of view rely upon formal subjective evaluations. A/V: Audio/visual.

But because of the importance of distinguishing artifacts of capture, media processing, and network conditions from the aesthetic quality actually intended by the creator, the recently developed formal definition of QoE defines, models, and measures quality from the point of view of both users and providers. We captured the results in a 2012 white paper.⁴ Though driven by the media representation and delivery community, the definition incorporates elements of psychology, the social sciences, and other fields (see Figure 3).

Figure 3. Components of QoE.

The new target of our research is immersive media, as it is our contention that QoE is fully applicable to new digital media and to increasingly immersive experiences. Immersive media technology experience (IMTE) is an emerging field that describes and defines this work. Its intent is to support natural interactions between the user and environment by enriching traditional interactive audio and visual presentations with novel modes of interaction, such as haptics, olfactory, and taste. The goal is the digital recreation of real-world presence. Achieving this goal will require a multidisciplinary effort that draws from such disciplines as media technology, ICT, and media studies, and incorporating core competencies from fields as diverse as communications, information retrieval, entertainment, and social networks. In the context of IMTE, which follows content from capture and representation through delivery to the user and into the business model, QoE is seen as an overriding tool for monitoring and managing the user experience at every interface between adjacent model layers (see Figure 4).

Figure 4. The QoE toolbox affects each layer interface in the experience model. FPS: Frames per second. GUI: Graphical user interface. HDR: High dynamic range.

The IMTE platform provides a common architecture for studying not only network media handling but also QoS mechanisms for dynamic networks, quality assessment, and the interactions between all these factors. Our research relies on a cyclic design and development process, in which each stage of the cycle both depends on and influences the next. As might be expected, controlling quality modeling and measurements within the IMTE framework frequently requires the design of novel experimental content. The resulting work often bridges the gap between technology and art. An example of this is ‘Chroma Space,’ in which the experimental results were published as a scientific paper⁵ while the content was exhibited as a piece of art.⁶

The complete media technology ‘ecosystem’ considers content not merely as a bitstream for consumption but also in terms of its social, economic, and cultural impact. QoE has become a critical planning and operations target for providing environments, both on the move and at home, that are surrounding, immersive, multisensory, interactive, always connected, and seamless. Industry has taken notice, as evidenced on the exhibition floors of NAB2012 and IBC2012 (the world's two major broadcast conventions) and in the workings of Qualinet.⁴ We expect that users will be most attracted to those providers who can ensure the highest possible QoE in all devices, service, and applications. Our ongoing research encompasses the creative process, the user, the context, and the delivery system, with special emphasis on resource representation, digital item adaptation, media conversion, and error resilience.

This work was supported by the Centre for Quantifiable Quality of Service in Communication Systems, a Norwegian Centre of Excellence appointed by the Research Council of Norway and funded by it and by the Norwegian University of Science and Technology and UNINETT.