A Complete Approach for Quality and Service Assurance W H I T E P A P E R
Introduction Video service providers implement new technologies to maximize the quality and diversity of their entertainment program offerings to their subscriber base in order to generate additional revenue, attract new subscribers, and maintain brand loyalty. The goal of service providers is to deliver this gamut of high quality video services cost-effectively through efficient operations. The quality of the viewer experience depends upon the quality of the MPEG stream from the source, throughout processing and its distribution points. There are many factors that can degrade video quality: source video quality, transport network characteristics and capacity and the configuration of processing equipment. In addition, various impairments, both static and dynamic, can originate from video sources (servers or feeds) and the network infrastructure. With such a complex infrastructure and many points for failure, many issues arise that result in visible anomalies such as black screen, tiling, freeze frame, blocking, blurring, lip sync, and slow user response. Achieving the desired high service availability in such a dynamic system requires both careful system design and continuous Quality of Service (QoS) and Quality of Experience (QoE) monitoring and performance analysis. In a traditional environment, video and network performance are verified with independent tools across the entire network. Multiple performance metrics are utilized across the network with no efficient way to pinpoint the problem source. This time-consuming process requires many manhours of inter-group interaction and often leads to incorrect resolution paths. Broadcaster Headend Core Edge Subscriber Baseband compliance QoE - Picture quality MPEG verification Server evaluation MPEG monitoring Feed verification IP monitoring / testing Optical network evaluation Continuity tests Attenuation, noise Eye/jitter performance QAM tests Visual QoE inspection STB verification The latest video assurance techniques implement uniform metrics across the network with a management node where performance correlation is analyzed. Furthermore, leading cable, telecommunications and satellite companies pursue joint QoE and QoS assurance techniques to combine the advantages of both methods. This whitepaper will discuss these techniques, their application and benefits. 1
Historical Evolution of Video Monitoring Approaches Approach 1: The Old Method Verify video as voice and data Historically, the health of voice and data distribution networks has been evaluated in terms of the transmission parameters of the network. Based on this method, some video technology companies have focused on basic transport parameters, such RMON statistics, RF performance, IP packet loss and jitter. This approach is not adequate for assuring video quality because voice and data network metrics cannot ensure video delivery. Additional metrics specific to video delivery are necessary to present a full view of video transport health. These metrics, such as Media Delivery Index (MDI) will be discussed in detail in a later section of this paper. Additionally, transport health alone does not assure a high quality video experience for the viewer. QoE verification at the source and at later network events, such as aggregation, encoding, rateshaping and grooming, is also necessary to ensure that the processed content meets the quality requirements for both video and audio. Approach 2: Trying Something Different Video assurance through perceptual quality As the industry realized that video content cannot be monitored like voice or data signals a new monitoring trend began to evolve. Video monitoring vendors utilized QoE parameters, such as perceptual quality, as the best approach to monitor the video network. Supporters of this philosophy argued that QoE and QoS were not closely related. Rather, only QoE verification was necessary because it better represents the viewer s perception and thus the overall video quality. Ultimately, this is an incomplete perspective of video performance because a monitoring system based solely on QoE will only detect problems when they become visible (when the customer sees the poor video feed). By then, it is too late. In addition, the analysis of video must include verification of metadata. The metadata contained on special MPEG PIDs contribute to the video user experience by providing audio control information (such as dialnorm), off-line information (such as closed caption), and interactive capabilities (such as EBIF widgets). However, the verification of metadata is absent from QoEonly methods of video monitoring. 2
Approach 3: An Intelligent Approach Correlating QoE and QoS performance Since transport analysis or perceptual quality alone cannot provide a comprehensive and proactive understanding of video performance there is a necessity for a combined view that associates network performance and delivered content quality. The recommended approach is a balanced application of QoS and QoE monitoring methods throughout the network. This method confirms that video processing nodes deliver a satisfactory level of video quality, video quality is maintained throughout the network, and degradation trends occurring at the transport plane are addressed before their cumulative effect has visible impact on content quality. Correlation: what went wrong where? Validation point: Wherever content is processed, it must be verified Initial transport Integrity verification QoS QoE Initial content verification Transport Integrity verification QoS Downstream verification: As content is not modified, only transport can impact quality encoding Rateshaping, Grooming Aggregation core Sources Processing Distribution Screen The diagram above shows the content from diverse sources being aggregated at the headend (as in the case of ad or promo insertions, switching from local to national content, etc.). Here, aspect ratios and presentation formats may be modified and video and audio are encoded (using diverse codec types and compression rates), groomed and rate-shaped. At these points it is necessary to verify that the produced content meets the QoE goals of the service provider. Farther along the network, the video is transported through the core, to the edge, and up to the screen location. Throughout these nodes, the MPEG packets can suffer loss, misplacement, or delay. If left unattended, these issues will cause a perceivable impairment of video. At these points, continuous QoS monitoring is a proactive way to deliver service and quality assurance. 3
Meaningful QoE and QoS Evaluation QoS Monitoring SCTE-168 Recommended Practice and MDI Metric Usage Quality of Service, or the implementation of guarantees to meet specified data transmission rates and error percentages, is used as a measurement of transport plane quality. At the network level for video transmission, QoS can be evaluated by monitoring network performance metrics, such as delay, jitter, and packet loss rate. This allows service providers to control and measure data transmission rates and error rates. In a nutshell, the SCTE-168-4, -6 and -7 recommendations cover the principle that once video streams have left the head-end and their structure is verified and intact, the main IP issues are not transport stream structure-related (as the structure is not modified within the IP network), but mainly packet loss and jitter-related events. With video s inherent sensitivity to delivery time distortions and packet loss, it is demanding on packet switched networks, especially those that have had little provision for assuring quality of service. The importance of measuring QoS of a video distribution becomes increasingly important because of the growing interest in video delivery over new channels, such as wireless networks and the Internet. This focus on the stable operation of IP infrastructure for video delivery is where the importance of the MDI is found. The MDI measurement described on the informational specification IETF RFC 4445 is used to monitor networks that are sensitive to arrival time and packet loss. It provides key indicators of video network performance: traffic jitter, measurement of deviation from nominal flow rates, and instant data loss measurements for a particular flow. Too much bursting of video IP packets (mainly video content PIDs) causes high network delays or jitter, which is a major challenge for video delivery. In turn, this causes an element of the network or a set top box buffer to overflow or under-run, resulting in lost packets (macroblocking) or dry buffers (black-screen/freeze frame) conditions, respectively. Too much jitter can take a heavy toll on video delivery quality if not held in check. Thus, actively monitoring jitter can highlight a major cause of video-related problems. In dealing with jitter, the MDI provides a relative indicator of needed buffer depths at the consumer node as well as an indication of lost packets. By probing a streaming media service network at various nodes and under varying load conditions, it is possible to quickly identify devices that introduce significant jitter or packet loss to the packet stream. By monitoring a network continuously, deviations from nominal jitter or loss behavior can be used to indicate an impending or ongoing fault condition such as excessive load. MDI provides the necessary information to detect all network-induced impairments for streaming video applications and enables the measurement of network jitter on constant and variable bit rate IP flows. 4
As it relates to SCTE 168, errors impacting QoS affect program availability. The program availability goal defined on SCTE-168 is to maintain less than 6 errored seconds a day for HD video, or an equivalent of 99.99% program availability. QoS monitoring is an integral piece in meeting this goal. QoE Monitoring Perceptual Quality and other QoE Metrics Quality of Experience is a subjective measurement used to determine how well a vendor is satisfying the customer s requirements. QoE takes into consideration factors that contribute to overall user value such as flexibility, mobility, cost, and choice. The video quality experience is further influenced by the viewer s display device, environment, and expectations. Video QoE basically has two components: transaction timing and video quality. The Mean Opinion Score (MOS) is an objective measurement used for assessing the quality of video services in a limited form of QoE measurement process. The MOS score uses mathematical algorithms to emulate human perception of content. VQmon, is a widely deployed MOS algorithm using packet/frame based zero reference, with fast performance that enables real time analysis on the impact that loss in I, B, and P frames has on the content, both encrypted and unencrypted. This algorithm emulates the content playout and buffer behavior. Supplementing the algorithm, video decoding for unencrypted MPEG allows for analysis and visualization of the content for black screen, frozen frames, and audio issues. MPEG Demux I/B/P frame detection Playout Buffer Emulator Per Frame Quality PAT/PMT Decode MPEG TR101290 Metrics Content Analysis Blank, Frozen Image Detection Codec/ Bandwidth Model Motion Blockiness Model Perceptual Quality Model Key QoE metrics provided by the algorithm are: 5
Absolute MOS-V High definition video provides better image resolution, and the absolute MOS value will show a higher score for a perfect video in HD than a perfect SD for the same scene. Relative MOS-V To compensate the natural difference of quality associated with image resolution, the relative MOS is a size independent score. In this case, an equivalent video on SD and HD will have the same score. Relative MOS-A This metric describes the quality of audio using MOS score. The image below shows an application where the real-time MOS score is shown for every channel monitored along with QoS metrics and Program Availability. This type of correlation delivers a complete understanding of video service and quality performance. 6
Application Cases This section shows a simplified network consisting of a source, headend, core and edge network segments and a subscriber node. In each, there are several cases of correlated content and transport monitoring that show how diverse impairments can be seen and interpreted from the QoE or QoS perspective, thus demonstrating the benefits of a joint QoS and QoE approach. Case 1 In the picture shown below, QoS monitoring reports no problems in the transport plane, but viewers experience poor quality. In a complete QoE and QoS monitoring system, it is easy to detect content quality problems at the source. When the root cause of problems is identified by the video management system, repair tasks can be more efficient and the number of viewers impacted can be reduced. Content Source Video Headend Core Edge Subscriber CONTENT TRANSPORT Case 2 Before QoE is impacted, the QoS monitoring capability uses MDI to detect a warning condition about potential buffer underrun issues. With this information, the issue can be addressed proactively before having an impact on QoE. CONTENT TRANSPORT Case 3 In this case, the bitrate of a program has been constant for 3 seconds. The QoS monitoring reports the condition and the frozen frame is reported by QoE monitoring. Correlating both planes, the operator has a unified report for the transport and content issue. CONTENT TRANSPORT 7
Summary Distributed continuous video QoE and QoS monitoring provides the infrastructure to establish detailed, reliable feedback on quality and service assurance. This is a necessary component in establishing a culture of quality delivery and revenue assurance. Even the most capable video operator cannot succeed in controlling operational expenses and improving quality without accurate QoE and QoS metrics. Continuous real-time video delivery must use continuous real-time feedback metrics that reflect the viewing nature of subscribers. Continuous real-time metrics obtained from thousands of video streams generate large volumes of data. Visualizing the key trends and locating specific faults in this data requires a centralized intelligent video management system to facilitate and close the feedback loop for quality improvement. With such a system, the operations staff has the tools to produce results visible on the bottom line. Combined with distributed continuous monitoring and visualization tools for comprehensive fault detection and location capabilities, delivery systems can grow to support virtually unlimited subscribers while assuring system quality and preventing ballooning operations costs. IneoQuest Technologies, Inc. 170 Forbes Boulevard Mansfield, MA 02048 Toll free: 1 866-464-4636 Fax: 508-339-4727