Evaluating the Impact of Network Performance on Video Streaming Quality for Categorised Video Content

Evaluating the Impact of Network Performance on Video Streaming Quality for Categorised Video Content 1 Evaluating the Impact of Network Performance on Video Streaming Quality for Categorised Video Content Seán Murphy, Michael Searles, Cyril Rambeau, and Liam Murphy Abstract This paper describes an empirical study that examines the relationship between network performance and video streaming quality for different types of video content. Three classes of video content newsclips, music and movie trailers are studied in order to generate appropriate maps from network performance to a quality score with correlates with perceived video quality. More specifically, a mapping from measured packet loss and delay jitter to one of a number of levels of approximate video quality is obtained. This mapping information can be used in the context of dynamic server selection to provide QoS support to a distributed RTP-based multimedia delivery system. The focus of the study is on short clips of low bit-rate video which is suitable for transmission to mobile devices. The results illustrate that the three classes of content react quite differently to changes in the network conditions. The movie trailers, which are typically characterized by high motion content suffer considerably more than the other two categories when there is jitter or loss on the network path between client and server. The music video content behaves similar to the high motion content for low packet loss and jitter, but tends to behave more like the low motion news content in the presence of high loss and jitter. There is sufficient difference between the performance of the different video content classes to warrant different maps for each of the content types which can be used to support server selection decisions in a distributed video content system. Index Terms Video quality assessment, Real-time video, Video streaming, QoS support, Server selection, Distributed multimedia D I. INTRODUCTION IGITAL distribution of video content has grown enormously in the last few years in the broadcast industry. Packet based video distribution has also experienced considerable growth, partly facilitated by the increased penetration of broadband access technologies. Packet based The financial support of Enterprise Ireland is gratefully acknowledged. S. Murphy and L. Murphy are with the Department of Computer Science, University College Dublin, Dublin 4, Ireland. (S. Murphy is the corresponding author; e-mail: sean.murphy@iname.com). M. Searles was with the Department of Computer Science, University College Dublin, Dublin 4, Ireland while some of this work was being carried out. C. Rambeau was an intern with the School of Electronic Engineering, Dublin City University, Dublin 9, Ireland while some of this work was carried out. video, however, still has enormous growth potential as it offers much greater flexibility than broadcast mechanisms. At the same time, solutions to issues relating to digital distribution of audio content have been found and the recording industry has recently taken significant steps to enable legalized access to online digital audio content. Companies such as Apple, Roxio and Sony now offer large databases of audio content online. With increasing penetration of broadband access technologies, it is natural to envisage a similar path being followed by digital video. Video, however, has extra complexity due to the very large capacities required for storage and transmission. A standard approach to designing scalable video distribution systems is to use replicated content distributed over a number of servers [2]. In such a system, however, there are issues relating to the approach used to map a client request for video content to a stream from a particular server. This issue, largely, motivates the research described in this work. One issue which impacts the quality of the video content received from a particular server is the state of the network path between client and server; if this path suffers significant loss or jitter, this has an impact on the perceived video quality at the client. The focus of this work, then, is to determine how measured network performance impacts the quality of the video received at the client. In this paper, the objective is to examine the relationship between network performance parameters and video streaming quality in order to make server selection decisions that translate into good viewing quality for the end-user. Specifically, we want to determine variation in video quality, measured using Peak Signal to Noise Ratio (PSNR), as a function of packet loss and delay variation and codify this into a map. This information is then used to choose the best video server from amongst a group of replica servers for particular video content. An experimental approach is used here to examine the relationship between network parameters and video quality: large amounts of video content were streamed on a testbed which can emulate different loss and jitter characteristics. The motion characteristics of different media content can influence the sensitivity of streaming video quality to network perturbations [1]. This may result in significant variations of

Evaluating the Impact of Network Performance on Video Streaming Quality for Categorised Video Content 2 video quality for different types of video content that are streamed under the same network conditions. If this variation is found to be large, then a single mapping strategy may not be sufficient to support a range of media content types. For this reason the video content used in these experiments is classified into one of three classifications based on the nature of the content. The utility of such a classification scheme is also considered here. The rest of this paper is organised as follows. Section 2 discusses the issues relating to server selection schemes for continuous media, associated probing methodologies and the metrics that they measure. The map from network performance parameters to some quality score is motivated and described in Section 3. In Section 4, the experiments used to derive this map are described. Three aspects to the experiments are discussed: the experimental methodology, the network test-bed and a video quality measurement tool developed throughout the course of this work. Results and analysis are presented in section 5. The paper is concluded in section 6. II. SERVER SELECTION ISSUES FOR CONTINUOUS MEDIA A. Server Selection A number of server selection algorithms have been proposed in the literature, they can be classified into three groups: static, statistical, and dynamic. Static approaches are based on metrics which do not change rapidly over time, such as hop count. Statistical approaches analyze responses to previous requests for service and use this as the basis for deciding where to route the next request. Dynamic approaches perform active measurements by inserting probe traffic into the network to ensure that recent information is available for the server selection decision. We focus here on dynamic algorithms because they have been shown to provide better performance [3][4]. Many existing dynamic server selection algorithms have been designed to select among web-based information sources that mainly consist of static images and textual information. These algorithms are not suitable for multimedia content selection because voice and video services have more restrictive QoS requirements on delay, delay jitter, and loss compared to the aforementioned class of applications. The few multimedia server selection studies that have been conducted to date, focus on balancing the load between a group of distributed media servers in the Internet. The above approaches do not address all the problems associated with streaming delay-sensitive content. For example, none of the above approaches takes into account the impact of the variation in network performance on the quality of the streamed video. This issue is the focus of this work. B. Probing Here we divide the probing methodologies into one of two specific categories, namely, advanced probing methodologies which report sophisticated metrics such as bottleneck link capacity and available bandwidth on a path, and simple probing methodologies which report more rudimentary packet-level metrics such as delay, jitter and loss [6]. Simple probing methodologies inject simple periodic probe streams into the network in order to detect network faults and monitor basic network performance metrics. These basic metrics are simple, easy to implement and result in relatively low cost measurements. Advanced probing methodologies, on the other hand, typically require a large number of probe measurements, sophisticated probe scheduling, or complex statistical analysis [5]. Using such techniques, they can be used to determine parameters such as the available bandwidth on a path. However, current approaches to measure such parameters suffer from two significant drawbacks: the overhead associated with them can be significant and the resulting measurement is often not very accurate. Network performance parameters such as available bandwidth, packet delay, jitter and loss are known to have a significant impact on video streaming quality and therefore may be used to assess path suitability for video streaming. Available bandwidth is a useful metric for video server selection because it can be used directly to make selection decisions. This is because data-intensive applications such as multimedia streaming directly benefit from improved transmission performance associated with higher available bandwidth. Unlike bandwidth measurements, raw packet-level metrics such as delay, jitter and loss are difficult to use directly for server selection. While it is reasonable to say that streaming video quality will improve as loss and delay decreases, the exact relation between these network performance parameters and streaming video quality is unclear. Due to the difficulties associated with using the advanced probing schemes, it was decided to use the simpler schemes here. Consequently, much of this work is focused on determining the relationship between these parameters and the resulting streaming video quality. More specifically, we focused on the use of loss and jitter as metrics on which to base the server selection decision as these are relatively simple to measure and are known to have a significant impact on video quality [7][8]. III. QOS MAPPING A fundamental component of this server selection system, then, is the empirically-derived map which maps measured loss and jitter parameters to some indicator of resulting video quality. It is known that video quality is difficult to measure objectively; the best test of video quality is performed using a set of subjects. However, it was not practical for us to use subjective testing in our experiments and hence an alternative had to be used. The well known PSNR video metric is known to have some correlation with perceived video quality and it is quite simple

Evaluating the Impact of Network Performance on Video Streaming Quality for Categorised Video Content 3 to calculate. Further, sophisticated objective measurement methodologies based on the human visual system have been shown to provide little or no advantage over simpler objective schemes [9]. Hence, PSNR was chosen as the most suitable means of determining video quality in this work. between them. Conversely, if the number of bands is too small, it is difficult to differentiate between the different servers reducing the utility of the map in the server selection process. It is known that different types of video content have Video Quality SVQM TOOL Disk Video Source YUV Format Darwin Server MP4 RTP UDP IP Quick Time MP4 Encoder RTSP RTCP RTP Nist-Net IP Network Emulator Delay + Std. Dev. % Packet Loss Quick Time MP4 Decoder RTSP RTCP RTP Disk MPlayer Client MP4 RTP UDP IP Figure 1: Configuration of testbed. Bandwidth Capacity Sub-Net 1 Sub-Net 2 RTSP: Real-time Streaming Protocol, RTCP: Real-time Control Protocol, RTP: Real-time Transport Protocol While it is known that there is some correlation between PSNR and perceived video quality, this relationship is certainly non-linear and in many cases it can be dependent on some characteristics of the particular video content. Hence, it is not possible to develop a meaningful map which maps from network performance parameters to a single PSNR value. The approach that is used here, then, is to develop a map from network performance parameters to some range of PSNR values or PSNR bands. Given that the PSNR values for video typically range from 40dB for excellent quality video to very poor quality at less than 20dB it is reasonable to think of bands lying in this range. Also, since a difference of 1-2dB in PSNR for particular video content can often be imperceptible, the size of the bands needs to be reasonably large. Differences of the order of 5dB, however, are much more likely to be perceptible and such differences are used to determine the bands in the map. This results in a map from packet loss and jitter to some coarse indicator of resulting video quality. A key objective, here, is to arrive at a map between network performance parameters and video quality which strikes a good balance between complexity and utility. Thus, it is desirable that the map should be quite simple but yet sufficiently powerful to produce a result which can be used effectively in a server selection scheme. A map in which there are a large number of the aforementioned bands will require considerable effort to generate. Moreover, the greater the number of bands, the smaller the difference between them and the more difficult it is to identify differences in video quality different characteristics. In particular, different types of video content often have quite different motion characteristics. This can result in different perceived performance for different types of content under the same network conditions [10]. This observation led us to believe that it may be appropriate to have different maps for different categories of video content. This idea is explored further below. IV. METHODOLOGY AND EXPERIMENTAL SET-UP This section describes the main elements of the experimental set-up: the network testbed, the experimental methodology and a small tool that had to be developed to aid analysis of the video content. A. Network Test-bed The quality assessment of received video clips was performed using the network test-bed architecture shown in Figure 1. The test-bed is an isolated LAN in which all hosts are connected through 100Mb/s links. The testbed consists of a client, a streaming server and a network emulator. The client used the MPlayer software to stream content from the server. MPlayer was used because unlike most other players it provides functionality to enable the received video content to be written to a file in uncompressed format. Moreover, the information that is written to the file is exactly what would have been displayed to a user, i.e. MPlayer stores erroneous frames in this file as well as those that are received error-free. MPlayer requests the streamed content using RTSP and receives content transmitted using RTP.

Evaluating the Impact of Network Performance on Video Streaming Quality for Categorised Video Content 4 The Darwin streaming server was used in these experiments. This free, open source server provides functionality to stream Quicktime and MPEG-4 compatible content which are stored locally. The server accepts requests for content using RTSP and sends content to the client using RTP. The NistNet network emulation tool was used to emulate different loss and jitter conditions in the network. The system operates on a PC under the Linux operating system and enables a number of different parameters, such as delay, loss, jitter and available bitrate, to be controlled on the path between client and server. An important issue when considering streaming of video content is the size of the client buffer to use as it has a significant impact on the operation of the system. A large client buffer affects the interactivity of the application, while a small client buffer makes the system more sensitive to variations in network conditions. In this work, a client buffer size of 100ms was used. This value was chosen as it has been argued that response times of 100ms or less are required to support interactive streaming. B. Experimental Methodology Three different categories of video content were used in these experiments: movie trailers, news content and music videos. Each category content had 5 different pieces of content and all of these pieces of content were streamed through the testbed a number of times. Each test video clip was approximately 150s in duration. This duration was selected because we envisage streaming such short clips to, say, a mobile device. Each video clip was encoded using Apple s QuickTime MPEG-4 compression codec using the default parameters shown in Table 1. The test clips were uploaded to the streaming server and streamed to the MPlayer client. A preliminary set of experiments to establish an appropriate range of test conditions was performed. The objective of these experiments was to determine what values of path loss and jitter result in such low video quality that the streamed content is effectively useless. It was found that loss rates in excess of 6% and jitter values of over 100ms resulted in very poor quality video. For this reason, the remainder of experiments focused on that part of the state-space in which loss is in the range 0-6% and jitter is in the range 0-100ms. TABLE 1 MPEG-4 CODEC COMPRESSION SETTINGS Compression Parameters Compression Quality Frame Rate Key Rate Image Size (width, height) Data Rate Packet Size Settings Best 5 frames/s Every 5 frames 176*144 pels 15 KB/s 1024 bytes An important point to note here is that the losses were distributed uniformly in time. Thus, there was no correlation between consecutive packet loss events. While it is known that this is atypical of Internet loss patterns, the NistNet tool did not provide for more sophisticated models. Consequently, the standard uniform distribution was used. A natural way to progress this work is to investigate the impact of more sophisticated loss models on these mappings. Each of the compressed video clips was then streamed from the server, via Nist-Net, to the MPlayer client under the predetermined range of test conditions. Each piece of video content was streamed from the server to the client 5 times and the results obtained were averaged. In each case, the content received at the MPlayer client was stored for post-processing. C. Streaming Video Quality Measurement (SVQM) Tool There are a number of existing tools for calculation of PSNR of video data. These tools calculate PSNR using the original and streamed/reconstructed video sequences as input. However, they are not robust enough for use in the context of video streaming. Typically, when video is being streamed, there is a very real possibility that a frame may be lost. If the received data is being written to a file, as in this case, the file containing the received data will not contain the same number of frames as that of the transmitted data. One problem, then, with determining the PSNR for such a sequence is that there can be a loss of synchronization between the transmitted video sequence and the received sequence. Some measures must be taken in order to ensure that each frame in the received sequence is compared with the appropriate frame from the transmitted sequence. Since none of the available tools have functionality to cater for this scenario, we developed our own tool throughout the course of this work which can determine when there is a loss of synchronization between the received and transmitted video data and effects a resynchronization between the two sequences. The tool determines the overall PSNR for the received sequence of video data. Figure 2: Graph of video quality against packet loss for news content. The delay jitter was 0ms in this case. V. RESULTS AND DISCUSSION In the first set of experiments that we performed, we were

Evaluating the Impact of Network Performance on Video Streaming Quality for Categorised Video Content 5 interested in determining how much variation there is between the results obtained for different types of video within a single category. To this end, separate graphs relating the PSNR to the packet loss and delay jitter for each piece of video content were generated. In the case of the packet loss graphs, the jitter was set to 0ms and in the case of the jitter graphs, the packet loss was set to 0%. These graphs are shown in Figure 2-Figure 7 below. Figure 5:Graph of video quality against delay jitter for news content. The packet loss was set to 0% in this case. Figure 3: Graph of video quality against packet loss for music content. The delay jitter was 0ms in this case. From these figures, it is clear that there is some quite homogeneous behaviour for each of the different categories: for each content category, the shape of the loss and jitter curves is quite similar for each of the video clips. This led us to believe that, even though the video content can be quite different, as, say, in the case of the movie trailers, the characteristics of the video content are quite similar within a particular category. Figure 6: Graph of video quality against delay jitter for music content. The packet loss was set to 0% in this case. The next stage of the analysis involved comparing the results obtained across different content categories. Since there is considerable homogeneity within each category, it is reasonable to perform averaging over the set of video clips used in each category. Such averages were then used as the basis of the comparison. The average values of video quality for different values of loss and jitter were obtained for each video category. These are shown in Figure 8 and Figure 9. The graphs show that there is some quite different behaviour for each of the types of content. Figure 4: Graph of video quality against packet loss for trailer content. The delay jitter was 0ms in this case. Figure 7: Graph of video quality against delay jitter for trailer content. The packet loss was set to 0% in this case. Of the three categories of content, the behaviour of the high

Evaluating the Impact of Network Performance on Video Streaming Quality for Categorised Video Content 6 motion movie trailer content stands out as being most unlike that of the other two types. It suffers considerably more for higher packet loss ratios and delay jitter values than the other two types of video content. Poor network performance has a more pronounced negative effect on this type of content than on the other two categories. The other two types of video content do converge in a similar way as the loss and jitter values get larger. However, the graphs still show a significant discrepancy between the two content types for packet losses in the region 0-2% and jitter values in the region of 20-80ms. Indeed, for smaller values of loss and jitter, the medium motion content behaves similar to the high motion content, but for larger values of delay and jitter its behaviour is more like that of the low motion content. Consequently, the results obtained so far would seem to suggest that there is not sufficient homogeneity between the different categories of content to use a single map from network performance to perceived video quality. Figure 8: Graph of video quality against loss averaged for each category of content. Inall of the above cases, the jitter introduced was 0ms. Figure 9: Graph of video quality against delay jitter averaged for each category of content. In all of the above cases, the packet loss introduced was 0%. The results discussed so far focused on parts of the lossjitter state-space. However, we did obtain results for much more of the state space in order to determine an appropriate map. A graph depicting an average of the results obtained for the music content category is shown in Figure 10. This is averaged over the results obtained for all pieces of content for all streaming runs. Figure 10: 3D graph showing variation of video quality with packet loss and delay jitter for the music content. These results were averaged over 5 pieces of music content, each streamed 5 times. The graph illustrates that, for the experiments performed, a large jitter value can have a very significant impact on the resulting video quality. Indeed, it can have a greater impact than packet loss percentages of 5-6%. Similar graphs can be shown for the other content types, but they do not exhibit any further interesting characteristics and hence are omitted here. While the above result is moderately interesting in its own right, the main motivation for generating such graphs is to enable the QoS map to be determined. To determine this map, then, it was necessary to decide on a the size of the bands. Above, we argued that the size of the band should be of the order of about 5dB and in fact this is exactly the size that was used to obtain the maps illustrated below. However, there is certainly scope for experimentation with this and future work will focus on this. Choosing a band size of 5dB, then, naturally results in the following 5 levels of video quality: excellent (35-40dB), good (30-35dB), mediocre (25-30dB), poor (20-25dB), very poor (<20dB). Hence, the map that we developed can map between measured packet loss and jitter and one of the above 5 levels of video quality. This choice of mapping appears to provide a reasonable compromise between complexity and utility: 5 bands is a sufficient number to enable differentiation between the paths to the servers and yet the band size of 5dB is sufficiently large to enable, in most cases, a reasonable differentiation between the resulting video qualities. A map for each of the different content categories is shown in Figure 11-Figure 13. Visually, it is very evident from these maps that there is a considerable difference between them. For example, the excellent quality band is much smaller in the case of the movie trailer map than that of the news content map. The size of the excellent quality band in the music video map falls somewhere between those of the other two. The same applies to the other bands in the maps.

Evaluating the Impact of Network Performance on Video Streaming Quality for Categorised Video Content 7 Figure 11: Map of loss and jitter to video quality band for news content. Figure 12: Map of loss and jitter to video quality band for music video content. Figure 13: Map of loss and jitter to video quality band for movie trailers. Clearly, then, there is a significant difference between each of the maps that have been obtained, which would appear to further support the hypothesis that different maps are appropriate for different content categories. Further work, however, is required to perform greater validation of this hypothesis. Specifically, it is necessary to compare the behaviour of a system operating using a single map with one which operates using different maps for different content categories. The results obtained here strongly support the contention that these two systems will exhibit considerably different behaviour and that the the system which utilizes multiple maps will perform better, but it would still be interesting to measure the difference in perceived performance. VI. CONCLUSION An experimental study of the impact of network performance on perceived video quality for streamed content was performed. Three different categories of content were used in the experiment: movie trailers, music videos and news content. The results showed that the different types of content were affected differently by the network performance. The high motion content the movie trailers was most adversely affected by poor network performance, while the other two types of content suffered less under the same network conditions. Using the above results, three maps were generated which can be used to map measured network performance parameters packet loss and jitter, specifically to an indicator of expected video quality. Using this map, then, it is possible to predict how the prevailing network conditions on a path between a server and a client will impact RTP-based streamed video content. This can then be used in the context of a distributed video content system to support the server selection process. The resulting maps were quite different for the different categories of video content which leads us to believe that a number of different such maps would be appropriate in the above application. There are still quite a number of open issues which require further investigation in this work. There are issues regarding the dependence on the client buffer size, the encoding scheme that was used and the size, frame rate and bitrate of the content. It is not clear how these results are dependent on any or all of the above parameters. There are issues relating to the number of content categories required. Further, it may make sense to classify content based on temporal and/or spatial complexity rather than classifying video based on content type; for example, content could be classified as high, medium or low motion video. In this case, it would be necessary to devise some suitable criteria or thresholds for differentiating between the different content categories. Finally, it could be argued that the number and size of the bands requires further analysis. Indeed, it may be the case that a non-linear approach to defining these bands gives the best performing trade-off between complexity and utility. REFERENCES [1] S. Winkler, A. Sharma, D. McNally, Perceptual video quality and blockiness metrics for multimedia streaming applications, in Proceedings of the International Symposium on Wireless Personal Multimedia Communications, pp. 547-552, Alborg, Denmark, September 2001. [2] Dapeng Wu, Yiwei Hou, Whenwu Zhu, Ya-Qin Zang, Jon Peha, Streaming Video over the Internet: Approaches and directions, IEEE Transactions On Circuits And Systems For Video Technology, Vol. 11, No. 3, March 2001. [3] R. L. Carter, M. E. Crovella, Server selection using dynamic path characterisation in wide-area networks, in Proceedings of IEEE INFOCOM 97, April 1997. [4] S. Dykes, K. Robbins, and C. Jeffery, An empirical evaluation of clientside server selection algorithms, in Proceedings of IEEE INFOCOM 00, pp. 1361 1370, March 2000.

Evaluating the Impact of Network Performance on Video Streaming Quality for Categorised Video Content 8 [5] C. Dovrolis, R.S.Prasad, M.Murray, K.C.Claffy, Bandwidth Estimation: Metrics, Measurement Techniques, and Tools, IEEE Network, November-December 2003. [6] L. Ciavattone, A. Morton, and G. Ramachandran, Standardized Active Measurements on a Tier IP Backbone, IEEE Communications, vol.41, no.6, June 2003. [7] L. Zhang, L. Zheng, K. Ngee, Effect of delay and delay jitter on voice/video over IP, Computer Communications Vol. 25, PP. 863-873, Sept. 2002. [8] Jill M. Boyce, Robert D. Gaglianello, Packet Loss Effects on MPEG Video Sent Over the Public Internet, ACM Multimedia 1998. [9] VQEG, Final report from the video quality experts group on the validation of objective models of video quality assessment, http://www.vqeg.org/, Mar. 2000. [10] M. Claypool and J. Turner, The effects of jitter on the perceptual quality of video, in Proceedings of the 7 th ACM International Conference on Multimedia, Orlando, Florida, Nov. 1999.