ABSTRACT 2. BACKGROUND AND MOTIVATION. 2.1 IPTV Content Distribution System. Keywords IPTV, Quality of Experience, Quality Assessment

Transcription

1 A DISCRETE PERCEPTUAL IMPACT EVALUATION QUALITY ASSESSMENT FRAMEWORK FOR IPTV SERVICES Mu Mu, Andreas Mauthe, Francisco Garcia Computing Department, Lancaster University, United Kingdom Agilent Technologies, Edinburgh, United Kingdom {m.mu, ABSTRACT Network impairments are unpredictable and highly destructive to the perceptual quality of media content in the IPTV content distribution networks. As a result the existing network QoS based assessment methodologies are not adequate for the IPTV QoE assessment which determines the overall acceptability of a service as perceived by the end user. A discrete perceptual impact evaluation quality assessment framework (DEQA) is introduced in this paper. The proposed framework enables a real-time, nonintrusive assessment service by efficiently recognising and assessing the individual IPTV quality violation events in the distribution network. The discrete perceptual impacts to a media session are aggregated for the overall user level quality evaluation. With its deployment scheme, the DEQA framework also facilitates efficient network diagnosis and QoE management. Recent work on realising the functions of the framework is also presented. Keywords IPTV, Quality of Experience, Quality Assessment 1. INTRODUCTION Packet-based networks have become one of the main content distribution media for emerging multimedia services such as IPTV and VoIP. Multimedia content with various genres such as news, football matches and even video telephony are becoming the popular services within IPTV content delivery networks (CDN). Due to the link quality and congestions in packet networks, packets of encoded multimedia streams can be damaged or dropped, which results in perceptual quality degradation to the delivered content. Although the network impairments are rare in the managed IPTV networks, they are highly destructive to the perceptual experience. The impact of a specific packet loss depends on a number of factors related to the video content characteristics, encoding, packet transmission scheme and decoding process. An identical amount of impairments over a certain time can create significantly different impact to the perceptual quality. Several user level metrics have been defined as the QoE requirement of IPTV services [1, 2]. However, the deterioration of video content in CDN is still widely evaluated by IPTV quality assessment methods with network performance (NP) and network QoS (NQoS) metrics such as packet loss rate (PLR) or total amount of losses that occur [3-5]. Although considerable efforts have been taken to map the NP and NQoS to Quality of Experience (QoE) with comprehensive mapping functions and guidelines, user level quality metrics such as one visible error per two hours cannot be fully adopted by the existing IPTV QoE frameworks. In this paper, we propose a discrete perceptual impact evaluation quality assessment framework (DEQA) to offer high performance in-service quality evaluation for the IPTV service. The DEQA recognizes the individual perceivable network impairments and evaluates their perceptual impact separately. The overall quality assessment is then evaluated based on the impact level and distribution of the detected impairment events. The rest of this paper is organized as follows. In Section 2, background of the distribution system and quality requirement of the IPTV services are presented, followed by the motivation of a new quality assessment framework. Section 3 introduces the DEQA framework with details of the key function components. A user case is presented in Section 4 to demonstrate the effectiveness of the proposed framework. Recent work on realising the functions of the framework is presented in Section 5. Section 6 summarizes and concludes the paper, giving an outlook on future work. 2. BACKGROUND AND MOTIVATION 2.1 IPTV Content Distribution System Within a video distribution service, the original video content from the content provider is encoded and encapsulated by the service provider. The packetized video streams are then distributed by the network provider to the destination. The end devices decode and repair the video stream before the content is displayed to the end user (Figure 1 [6]). Although emerging distribution mechanisms such as peer-to-peer file sharing are popular in consumer networks, the more reliable unicast and multicast are still the most favourable distribution methods in IPTV content networks to guarantee a high standard of packet delivery. Figure 1 Video content distribution service With the increasing demand for different qualities of video transmitted over the Internet, H.264/AVC codec, taking into account the specific requirements of emerging high quality video applications has been standardised. The packet-based format of the content unit is commonly chosen for video streaming. It defines the data packets that are framed by the system transport protocol, without use of start code prefix to increase the coding efficiency for applications of native RTP/UDP/IP streaming [7]. Although other encapsulation methods such as MPEG-2 TS are also available, the native RTP method shows many advantages such as higher encapsulation efficiency, better error detection/resilience, /10/$ IEEE ICME 2010

2 and good integration with other internet protocols [8]. Along with the native video content, parameters about the compression, encoding and encapsulation procedures are also included in the encoded content for distribution and decoding purposes. Recently these parameters have also been considered for the advanced quality assessment models. 2.2 QoE requirements of IPTV Quality of user experience (QoE) is defined in [9] as the overall acceptability of an application or services as perceived subjectively by the end user. The encoding/compression process and the transmission impairments in the delivery network are two sources of the inevitable QoE violations in the delivery chain. The encoding process is usually well managed by the service provider to minimise the visual artifacts of video compression. In contrast, network impairments are highly unpredictable and destructive. In [10] the quality requirement for IPTV services are specified. It is concluded that the video streams are highly sensitive to information loss and QoE impact is in turn correlated to a number of variables including: type of data lost, codec, encapsulation and packetisation mechanism, loss distance, loss profile and decoder concealment algorithms. In the ITU Focus Group on IPTV, quality target metrics have been extended to consider the user perceived quality. User level quality metrics such as Maximum one visible artifact per x hours and Mean time between payload artifacts were defined to evaluate the delivery of IPTV services [1]. The Broadband Forum has also defined a set of user level QoE metrics in [2]. For instance, a criterion of one impairment event per 12 hours or better is defined for HDTV services. Because of the duration of the visual artifacts and the error concealment in decoders, not all network errors will result in visible impairments. Thus a value of four hours as the minimum visible loss distance for HDTV services was proposed. 2.3 Motivation It has been recognised that the network level abstract metrics are not adequate for high quality video distribution services [11, 12]. To fill the gap between network corruptions and the actual user perception, the user level quality metrics have been proposed as the industrial standard of the IPTV quality level measurement. However, there is still no quality assessment framework describing how the user level metrics can be utilised in an end-to-end quality assessment service. 3. DISCRETE PERCEPTUAL IMPACT EVALUATION QUALITY ASSESSMENT FRAMEWORK 3.1 Design goals Four major design goals distinguish the proposed framework from the traditional QoS approach. Firstly, the framework is driven by the user level quality metrics. Any network impairment must be mapped to user level impact before the overall quality evaluation is carried out. Secondly, impairment events and QoE evaluation results should be maintained throughout the session lifetime so that the status of the target service is constantly available for network or application management. Thirdly, the root-cause (e.g. location) of each QoE violation event will be registered to facilitate prompt QoE diagnosis and management. Lastly, the assessment methodology must be light-weight so that the in-service quality monitoring can be achieved on different assessment points along the IPTV service path with minimal effort. 3.2 Framework overview The main function blocks of the framework are shown in Figure 2. The shallow/deep packet inspection and header inspection constitute the packet analysis (PA) function set. This function set is responsible for all packet level operations on IP packets and media encapsulation layers. The perceptual impact assessment model and the impact aggregation model are the quality evaluation function set which maps the packet analysis results to the actual user perceived quality. The impact aggregation function also requires reference to the service quality agreement. The session state registry (SSR) maintains the stateful quality evaluation results during the session lifetime. The signalling system exchanges analysis and control information between functions of the DEQA assessment framework and the QoE management functions. Figure 2 DEQA framework 3.3 Discrete network analysis methodology Currently, network status is summarised by the network operator and then utilised by the service provider for the quality assessment (Figure 3a). The summary based analysis is ideal for system survey and statistics. However, recent researches have shown that the same level of network impairment can have significantly different visual impact to the video content [11, 13]. Thus a summarised network status is not sufficient to accurately model the user perception. Evaluation Summary (percentage, total number) Evaluation Assessment Assessment Assessment (a) (b) Figure 3 Network analysis strategy Service Quality Survey NQoS Summary Packet Stream Impact Aggregation Perceptual Impact Assessment Packet Inspection Packet Stream

3 To better model the perceptual quality of deteriorated video service, the DEQA framework performs a discrete assessment based on the impact of individual quality violation events. As it is shown in Figure 3b, the packet inspection function observes the packets of a media stream and recognises the network impairments such as packet loss. It then conducts multilayer packet header analysis to provide necessary content and encapsulation parameters for the quality evaluation functions. 3.4 Evaluation Function Set Perceptual impact assessment function When a network impairment event is detected, the discrete assessment framework enables the perceptual impact assessment function to evaluate the impact level of the event. The assessment function is realised by an objective parameter-based video quality model. In order to maximise the assessment performance, comprehensive metrics are provided for the objective model. Figure 4 shows three levels of metrics derived from packet inspection functions. The network metrics give network level parameters of the impairment such as the RTP packet number of the missing packet. Details about the nature of the multimedia content and the encapsulation mechanism are available as the content bitstream metrics. Terminal metrics provide information about the user devices such as error concealment mechanisms and buffer size. With these multilayer metrics, the perceptual impact assessment function benefits from the advanced knowledge of the effect of certain network impairment on the actual media content (e.g. the spatial extent and position of a visual deterioration on the video frame) and how the deteriorations would be repaired and displayed by the end devices. An impact level score is derived to express the perceptual impact of a given network error (Figure 4). Figure 4 Perceptual impact assessment function must be deployed. This means that the relatively deep packet inspection (e.g. deep to the slice header level) has to be conducted repeatedly (Figure 5 (a)). There are two issues with this deployment method. The computing power may not be sufficient on each monitor point especially when large amount of streams are involved. Deriving the identical information repeatedly along the service path is not an efficient utilisation of the network resource Impact aggregation function The overall quality of the delivered media content is determined by the cumulative effect of the individual perceptual impact level as estimated by the impact assessment function. The aggregation function approximates the user s satisfaction and gives a qualitative overall score based on the level of severity and the distribution of the individual impairment impacts (Figure 3b). The estimation model is manually configured by service provider referring to the quality agreement with network providers or customers. For instance, if it is agreed and promised to the customers that a maximum number of two visual errors are allowed for a three hours of VoD session, the aggregation function would indicate the service quality as unsatisfactory when two network impairments with high impact level have been detected. In practice, even the most severe deteriorations might be neglected by the end users while some light imperfection in high definition content service can annoy customers with high quality expectation. A complex aggregation model can be customised by the service provider. 3.5 Advanced packet inspection and header extension A video service may cross multiple network/service domains along the delivery path. Operators at domain borders have the option to perform monitoring which, when taken together, forms an end-toend monitoring topology with several monitoring points [6] (Figure 1). On each of these monitoring points, an assessment function Figure 5 Two packet analysis schemes We have proposed a packet analysis deployment scheme to improve the practicality of the quality monitoring with parametric assessment function. With this method, three functions are introduced to the framework: deep packet inspection, shallow packet inspection and header extension. The deep packet inspection (DPI) function is conducted only at the beginning of the delivery path or at the entry point of the target network segment. The DPI results are written back to the upper layer packet header before the packets are forwarded. By doing so, the subsequent monitoring points along the service path can acquire all the key deep header information by shallow packet inspection functions ( Figure 5 (b)). The proposed method greatly reduces the computation intensity of the following monitoring points. In practice, the DPI and header extension should ideally be embedded with the encapsulation process when the video content is being prepared for delivery. Thus the impact to the delivery network is minimised. We will further analyse the issues of computational load with our prototype system in a future study. 3.6 Session State Registry

4 The stateful session management is becoming a key component of IPTV services. It enables the interactivity, mobility as well as quality assurance of media sessions. Although the RTCP offers basic control mechanisms along RTP media streaming, the IP Multimedia Subsystem (IMS) defined by 3GPP and TISPAN is widely considered as one of the ideal architectures for IPTV session management [14, 15]. Thus, instead of stressing the session management system, we discuss in the following how media Session State Registry must be maintained and managed to support QoE assessment and assurance under the DEQA framework. Two session states are managed throughout the session lifetime. Event Log state records the details, impact and location of each network impairment violation events. Table 1 gives an example of how packet loss events are recorded. The ID identifies the RTP packet number of the missing packet. This is done by examining the RTP number discontinuity at each inspection point. The network location of each event is also maintained to support prompt network diagnosis. The impact level is derived by assessment models and then used by both assessment and management functions. The overall quality of a media stream is evaluated by examining the impact level and correlation of the quality violation events. When the error recovery mechanism is available, the impact level helps the management process to determine the recovery options effectively. For instance, the packet loss recovery [16] of a packet with impact level of 5 would be more appreciated than that of a packet with impact level of 1. Table 1 Session State Registry Session Quality Level (SQL) is a global metric which is updated by any monitoring points along the service path every time an event is detected. It indicates the overall quality level of the target media session against the corresponding service level agreement between service providers, network providers and end users. Although the quality of the delivered media stream can only be measured at the receiver point properly, the SQL score produced at assessment points within the distribution network enables an instantaneous in-service management before more content packets are affected. For instance, if the SQL is reaching an alerting level, network or application management functions can be triggered for remedy before the user experience is further deteriorated, (e.g. by increasing the transport priority or engaging network resource) 3.7 Signalling System Service Quality Level Level 1 Event Log ID Impact Location AP AP5 The signalling system exchanges analysis and control information between functions of the DEQA assessment framework and also to the QoE management function. Several signalling management schemes have been proposed in the past for QoE measurement and management [17, 18]. The signalling protocol is commonly customised according to the specific network infrastructure. Although the signalling protocol is not stressed in this paper, the signalling systems must be compatible with the underlying management systems such as the resource allocation controllers within large-scale IPTV networks. 4. DEQA RUNTIME USE CASE This section gives a use case of the DEQA framework to demonstrate how functions of the framework operate and collaborate for an end-to-end QoE assessment and management service across providers and domains (Figure 6). Step1: During the content preparation progress, service provider embeds the video coding and encapsulation information into the transport packet header for future quality assessment. If such an information appending service is not available during the content preparation, deep packet inspection is conducted over encoded media content at the entrance of the first domain where DEQA framework is applied. After the first progress, the required assessment parameters are available in the low level packet header such as the RTP header. Shallow packet inspection is adopted by the following assessment services. Figure 6 DEQA runtime use case Step 2: Real time packet analysis is then conducted at each assessment point. To detect the packet loss, RTP packet number is examined on each packet. When packet number discontinuity event is detected, the packet analyser registers the event and starts to collect the related information about content involved in this packet. However the impact evaluation function is only triggered if the error is not repaired after a predefined tolerance time. The tolerance time is set according to the terminal metrics or the application scenario. If the missing packet is eventually repaired or retransmitted on time, the record in the SSR is then removed. Step 3: The evaluation function gives the impact level of each packet loss and register the event details in the event log. An overall quality level is also evaluated with all the previous events registered. The SQL part of the SSR is updated. If the SQL has reached an alerting level, a remedy procedure of the management service will be called with the signalling functions.

5 5. REALISATION To apply the DEQA framework in practice, functional components must be realised. The discrete impact assessment function and packet analysis function are two key modules that have not been systematically studied in the past. In this section, we introduce the related work and our recent development on realising the key functions of the DEQA framework. 5.1 Impact Assessment Function The current objective video models rely mostly on payload and/or bitstream analysis [19]. The payload analysiss method requires decoding of the target video streams, after which the quality assessment is carried out by picture/video analysis in either full- mode. reference, reduced-reference or no-reference assessment Although payload analysis usually produces good results [20] it suffers from several drawbacks such as high computational complexity due to image signal processing. This has made real time evaluation impossible in consumer networks. Furthermore, no payload of the packets will be available for processing if the content encryption is enabled on customer s services. In contrast, the emerging parametric method offers non-intrusive objective assessment which estimates the user perceived quality based on the delivery of IP packets without decoding the target video streams. The ongoing ITU standard P.NAMS [21] describes an objective parametric quality assessment model that predicts the perceptual impact of network impairments on IPTV applications. The prediction is based on packet header information, prior knowledge on the media stream and information about the client devices. With no requirements of accessing the content payload, P.NAMS model is ideal for in-service real-time quality monitoring on multiple media streams. ITU-T SG12 has also initiated a computational model (G.OMVAS) for video and audio-streaming over IP networks which assess the combine effects of content and network parameters [22]. In parallel to the P.NAMS model, P.NBAMS (Non-intrusive bit-stream model for assessment of performance of multimedia streaming) has also been proposed [23]. Compared with P.NAMS which only accesses the header information, P.NBAMS (bitstream model) gains extra access to the content payload to acquire content native information such as motion vectors. Combining the no reference (NR) picture analysis and the bitstream analysis, a hybrid model - J.BITVQM was also proposed in ITU-T [24]. The idea of the hybrid model is to improve the performance of NR models with extra information such as GOP, quantification, bit rate and the DCT coefficients. Research on the parametric models has also been carried out in the academia. In [25], authors have proposed a parameter-based model for predicting the perceived quality of IPTV applications based on factors like the coding bit-rate, packet loss rate and error concealment. Another parametric packet-layer model was proposed in [26] utilising the metrics of packet loss rate and bit-rate as well as loss distance. Packet loss pattern is specifically considered in parametric model introduced in [27]. However the current research in the field are still based on simple network QoS metrics, although in some recent work [11, 13] researchers have verified the significant variance on visual impact of individual network errors. Some initial studies have be carried out to explore the visibility of content loss with various slice sizes, loss distribution, frame type and impact from region of user s interest on screen. A dedicated testbed has also been developed to enable researchers to produce test sequences according to every specific test pattern to model the perceptual impact of network error capturing all relevant factors [28]. An example of our recent user study result (Figure 7) shows the perceptual impact of packet loss with different frame type, spatial location, spatial extent and content characteristics. It clearly demonstrates how content and network parameters affect the visibility of network impairments. A conventional QoE assessment framework would not differentiate between packet losses. Although some advanced designs in the past take frame type into consideration [29], the test result (Figure 7) has proven that even packet losses in the same type of frame lead to dissimilar perceptual impact. The discrete assessment methodology is considered as the ideal approach to capture the users actual experience of network error. test video 1 Figure 7 Subjective user study results 5.2 Packet Inspection and Header Extension Depending on the requirements of the given parametric model, the packet inspection (PI) can provide header information by examining the RTP header, NAL unit header, Sequence Parameter Set (SPS) unit, Picture Parameter Set (PPS) unit and Slice header (Figure 8). However, access to the content payload (slice content) is prohibited due to the nature of the parametric model. For instance, by examining the RTP header, the timestamp and specific priority of each packet can be captured.. Looking into the header of a NAL unit, we are able to distinguish between SPS, PPS, IDR slice or other RBSP units. Parsing the SPS and PPS, basic information of video stream such as frame size and number of slice groups are available. When a data slice is presented, PIs look into the slice header for deeper metrics such as the slice type and the spatial position of the slice. RTP header NALU header SPS PPS Slice header Figure 8 Hierarchical Header Information To realise the proposed assessment methodology as described in Section 0, we have proposed the RTP packet header extension solution referring to the RFC 5285 [30]. The one byte header extension supports extension of up to 16 bytes which is sufficient to support 1080p HD video content (Figure 9). Figure 9 RTP Header Extension test video 2 Data

6 Figure 10 shows the information which the packet inspection derives from the packet IPTV media stream to support the perceptual impact assessment function in our test system. The XML report file contains information about how video content was prepared and delivered prior to the inspection point. Figure 10 XML Report File (excerpt) 6. CONCLUSION A high performance quality assessment service which grasps the actual perceptual quality is essential to the quality assurance in IPTV content delivery networks. User level service quality metrics have been widely recognised in the academic community. However, current QoE assessment methodologies are not able to provide user level quality evaluation due to the nature of conventional network analysis mechanisms. A discrete perceptual impact evaluation quality assessment framework (DEQA) is introduced in this paper. The proposed framework enables service providers to perform a non-intrusive QoE assessment service on large quantities of concurrent video streams. This is achieved by efficiently recognising and assessing the individual IPTV quality violation events in the distribution network. The DEQA framework also facilitates a quick network diagnosis and QoE management. Under the DEQA framework, we have initiated research and development to realise the key function modules. We will continue our study on the operational and practicality issues in future work. 7. REFERENCES [1] "Application layer reliability solutions for IP TV services," ITU. FG IPTV-ID-0097, [2] T. Rahrer, et al., "Triple-play Services Quality of Experience (QoE) Requirements and Mechanisms - For Architecture & Transport," Broadband Forum, [3] N. Farber, et al., "Analysis of error propagation in hybrid video coding with application to error resilience," in Proc. IEEE Int. Conf. on Image Processing (ICIP), ed, pp [4] K. Stuhlmuller, et al., "Analysis of video transmission over lossy channels," in IEEE J. on Selected Areas in Communications vol. 18, ed, 2000, pp [5] I. M. Kim and H. M. Kim, "A new resource allocation scheme based on a PSNR criterion forwireless video transmission to stationary receivers over Gaussianchannels," vol. 1, ed, 2002, pp [6] "Performance monitoring points for IPTV," ITU Draft Recommendation G.IPTV-PMP, [7] S.-k. Kwon, et al., "Overview of H.264/MPEG-4 part 10," Journal of Visual Communication and Image Representation, vol. 17, [8] A. MacAulay, et al., "WHITEPAPER IP Streaming of MPEG-4: Native RTP vs MPEG-2 Transport Stream," ed. [9] "Appendix I to P.10/G.100: Definition of QoE," ITU-T P.10/G/100. [10] "Quality of experience requirements for IPTV services," ITU- T FG IPTV G [11] M. Mu, et al., "Visibility of Individual Packet Loss on H.264 Encoded Video Stream A User Study on the Impact of Packet Loss on Perceived Video Quality," in MMCN'09, San Jose, California, USA, [12] M. Mu, et al., "A Utility-based QoS Model for Emerging Multimedia Applications," in First IEEE Future Multimedia Networking (FMN 08) Workshop, Cardiff, UK, [13] A. R. Reibman, et al., "Visibility of individual packet losses in MPEG-2 video," in IEEE ICIP, [14] C. Riede, et al., "Quadruple play session management enabler for multimedia streaming," in 16th IST Mobile & Wireless Communications Summit, Budapest, Hungary, [15] J. Maisonneuve, et al., "An Overview of IPTV Standards Development," IEEE Transactions on Broadcasting, vol. 55, p. 315, [16] J. Asghar, et al., "Preserving Video Quality in IPTV Networks," IEEE Transactions on Broadcasting, [17] M. Mu, et al., "Quality of Experience management framework for real-time multimedia applications," International Journal of Internet Protocol Technology, vol. 4, pp , [18] E. Cerqueira, et al., "Multi-user session control in the next generation wireless system," in Proceedings of the 4th ACM international workshop on Mobility management and wireless access, 2006, pp [19] S. Winkler and P. Mohandas, "The evolution of video quality measurement: from PSNR to hybrid metrics," in IEEE Transactions on Broadcasting vol. 54, ed, 2008, pp [20] "Final Report of VQEG's Multimedia Phase I Validation Test," VQEG, [21] "A non-intrusive parametric model for the assessment of multimedia streaming (P.NAMS)," ITU-T Draft Recommendation. [22] "Opinion model for video and audio streaming applications (G.OMVAS)," ITU-T Draft Recommendation. [23] "Non-intrusive parametric bitstream model for the assessment of performance of multimedia streaming (P.NBAMS)," ITU-T Draft Recommendation. [24] "Hybrid perceptual bitstream video quality assessment technique in NR model (J.BITVQM)," ITU-T Draft Recommendation. [25] A. Raake, et al., "TV-MODEL: Parameter-based prediction of IPTV quality," Proc. of ICASSP08, pp , [26] K. Yamagishi and T. Hayashi, "Parametric packet-layer model for monitoring video quality of IPTV services," [27] F. You, et al., "Packet Loss Pattern and Parametric Video Quality Model for IPTV," in Proceedings of the 2009 Eigth IEEE/ACIS International Conference on Computer and Information Science, 2009, pp [28] M. Mu, et al., "LA1 TestBed: Evaluation Testbed to Assess the Impact of Network Impairments on Video Quality," in The 5th International Conference on Testbeds and Research Infrastructures for the Development of Networks and Communities (TRIDENTCOM 2009), Washington D.C., [29] C. H. Ke and N. Chilamkurti, "A new framework for MPEG video delivery over heterogeneous networks," Computer Communications, [30] D. Singer and H. Desineni, "A general mechanism for RTP Header Extensions," RFC 5285, July 2008.