Institute of Computer Science Chair of Communication Networks Prof. Dr.-Ing. P. Tran-Gia The Memory Effect and Its Implications on Web QoE Modeling Tobias Hoßfeld 1,2, Sebastian Biedermann 1, Raimund Schatz 2, Alexander Egger 2, Sebastian Egger 2, Markus Fiedler 3
Trend towards Quality of Experience Increasing competition among Telco s and ISPs, among application and service providers, among cloud providers Keep customers happy, attract new customers Quality as key differentiator, but only as experienced by end user Shift from Quality of service (QoS) to Quality of Experience (QoE) QoS: packet loss, delay, jitter, QoE: subjective experience/satisfaction of users of a service Example: VoIP user interested in speech quality web user interested in short page load times What are relevant QoE influence factors How to integrate key influence factors in appropriate QoE models Tobias Hossfeld 2
QoE Model for Web Browsing Importance of web traffic is increasing Related work on QoE mainly covers multimedia applications Scenario: User downloads several web pages within a session But: Random test sequences according to standardization Lack of web QoE models including quality changes over time Contribution: tra affic increase (normalized to 2010) 8 7 6 5 4 3 2 Internet video Online gaming Web, email, and data File sharing VoIP Mainly web traffic 1 2010 2011 2012 2013 2014 2015 year Source: Cisco Visual Networking Index: Forecast and Methodology, 2010-2015 Subjective user study on web browsing with quality changes Identification of memory effect as relevant QoE influence factors Integration of key influence factors in appropriate QoE models Tobias Hossfeld 3
Agenda Conducted Subjective User Study QoE Influence Factors, Design of Study Implementation and Measurement Setup Statistical Analysis of User Ratings Page Load Times, Memory Effect Key QoE Influence Factor via Support Vector Machines Implications on QoE Models Iterative Regression Model Hidden Memory Markov Model Conclusions and Outlook Tobias Hossfeld 4
QoE Influence Factors for Web Browsing Context Social and cultural background Task/purpose e.g. time killing, information retrieval Online and crowdsourcing tests Usage History QoE User Expectations Memory Effects Quality change per page System Transmission network, e.g. bandwidth, delay, jitter, loss Client Device Emulate page load time Web browser Content Type of web site, e.g. YouTube, Google, Facebook Implementation of web site Simple Photo Page Tobias Hossfeld 5
Methodology: Subjective User Studies Subjective user rests required due to lack of existing studies Picture is downloaded with predefined time. Test run over several web pages (with changing network conditions) User rating on 5-point ACR scale Laboratory tests to get first insights, delay via traffic shaper Online tests to reach more users, local applet with def. page load times Tobias Hossfeld 6
Quantifying Quality of an Experience Mean Opinion Score (MOS): numerical indication of the perceived quality of received media after compression and/or transmission Fair = 3 Excellent! MOS Quality Impairment 5 Excellent Imperceptible Poor! Bad! 4 3 2 Good Fair Poor Perceptible Slightly annoying Annoying Good! Fair! 1 Bad Very annoying Tobias Hossfeld 7
Agenda Conducted Subjective User Study QoE Influence Factors, Design of Study Implementation and Measurement Setup Statistical Analysis of User Ratings Page Load Times, Memory Effect Key QoE Influence Factor via Support Vector Machines Implications on QoE Models Iterative Regression Model Hidden Memory Markov Model Conclusions and Outlook Tobias Hossfeld 8
Impact of Page Load Times on QoE Models from literature for mapping current QoS to QoE IQX hypothesis when using network parameters like bandwidth [Hoßfeld, Fiedler, Tran-Gia, ITC 2007] QoE(x) = a exp(-b x) + c Weber-Fechner law from psychophysics when using page load time as QoS parameter [Reichl et al., ICC 2010] QoE(x) = a ln(b x) But: strong deviations from model observable Only current QoS is considered Temporal effects have to be included Tobias Hossfeld 9
The Memory Effect Same QoS but different QoE Bad QoS before Good QoS before Web pages with same page load time have different QoE, depending on previous QoS conditions Exponential decays/increase after quality changes Tobias Hossfeld 10
Is Memory Effect a Relevant QoE Influence Factor Investigation with correlation analysis and machine learning Support vector machine decide two-class problems user ratings are separated into good quality and bad quality Implementation of SMO (Sequential Minimal Optimization) in WEKA machine learning software for analysis QoE from previous web site has relevant impact on QoE Memory effect as dominant as technical influences (PLT) Only last QoE has to be considered Tobias Hossfeld 11
Agenda Conducted Subjective User Study QoE Influence Factors, Design of Study Implementation and Measurement Setup Statistical Analysis of User Ratings Page Load Times, Memory Effect Key QoE Influence Factor via Support Vector Machines Implications on QoE Models Iterative Regression Model Hidden Memory Markov Model Conclusions and Outlook Tobias Hossfeld 12
Memory has to be included in QoE Models Support Vector Machines Consider previous experiences as own variables Weighting factors indicate importance of variables Exponential iterative regressions Weber-Fechner law yields f(plt) Considering previous QoE via iterations Markov models are memoryless, but can include memory by appropriate state space Tobias Hossfeld 13
Hidden Memory Markov Model Describe the QoE with a Hidden Markov Model (HMMM) Page load time as hidden state QoE i.e. user ratings as emission H i, H i-1 transition prob. Sequence of web pages with page load times x i is extended to series of pairs (x i,x i-1 ) Page load times are discretized emission prob. 2D-State space of HMMM is (H i,h i-1 ) Transition and emission probabilities derived from user studies Tobias Hossfeld 14
Conclusions Time dynamics of human perception for web QoE analyzed Designed and conducted subjective user study on web browsing Identification of memory effect as relevant QoE influence factors Integration of key influence factors in appropriate QoE models Support vector machines with additional past variables Weber-Fechner law with iterative exponential regressions Hidden Markov model by increasing state space Consequences QoE model available for performance evaluation, measurement studies, subjective user surveys In case of unpredictable QoS: avoid memory effects (e.g. for QoE based traffic management, development of apps, etc.) In general: Interdisciplinary research required Tobias Hossfeld 15
Institute of Computer Science Chair of Communication Networks Prof. Dr.-Ing. P. Tran-Gia Questions
Local Tests in Laboratory Environment Tobias Hossfeld 17
Online Tests with Net-Sim-Applet Tobias Hossfeld 18
Statistics about the Conducted Experiments Exp. Id #test users X-point scale #web pages #chang es of PLT Min PLT (ms) Mean PLT (ms) Max PLT (ms) 0 29 3 93 21 348 2594 8184 1 12 3 40 2 240 420 720 2 72 5 40 7 240 660 1200 3 30 3 25 4 240 336 480 4 26 5 40 9 240 600 960 5 15 5 25 4 240 528 720 Tobias Hossfeld 19
Local Test Run (Exp. 0) Page download times measured via TCPDump / HTTPFox 9 8 local test run 7 me (s) page download ti 6 5 4 3 2 1 0 0 20 40 60 80 100 web page Tobias Hossfeld 20
Online Test Runs page downlo oad time (s) 1.5 1 exp. 1 0.5 0 0 10 20 30 40 1.5 1 exp. 2 0.5 0 0 10 20 30 40 1.5 1 exp. 3 0.5 0 0 5 10 15 20 25 1.5 1 exp. 4 0.5 0 0 10 20 30 40 web page Test memory effect Test several effects Test. exp. regressions Test several effects Tobias Hossfeld 21
Agenda Existing QoE Models for Web Traffic Measurement Settings QoE Characteristics Identification of User Groups QoE Models Tobias Hossfeld 22
Existing QoE Models for Web Traffic Fidel Liberal, Armando Ferro, Jose Oscar Fajardo: PQoS based model for assessing significance of providers statistically (2005) MOS = 6 - log2(t), t: total download time [s] R.E. Kooij, R.D. van der Mei, R. Yang: TCP and web browsing performance in case of bi-directional packet loss (2009) MOS = 4.75-0.81log(t), t: total download time [s] ITU-T Recommendation G.1030: Estimating end-to-end performance in IP networks for data applications (2005) MOS = 4.298 exp( 0.347 t)+1.390, t: weighted session time [s] J. Shaikh, M. Fiedler, D. Collange, Quality of Experience from user and network perspectives (2010) MOS = 1.15 + 1.50 ln(r), R: delivery bandwidth [Mbps] MOS = 5.50 exp( 0.2L), L: packet loss ratio [%] What s missing Psychological and temporal effects! Tobias Hossfeld 23
QoE Related to Page Download Time 5 4.5 4 3.5 LiFeFa05 KoMeYa09 G.1030 MOS 3 2.5 2 1.5 1 0 5 10 15 total download time [s] Tobias Hossfeld 24
QoE Depending on Bandwidth and Loss ShFiCo10 20 0 10 5 delivery bandwidth [Mbps] packet loss ratio [%] 0 10 1 2 3 4 5 MOS Tobias Hossfeld 25
Overview on Test Runs Tobias Hossfeld 26
Exponential Regression Exponential decay / increase (similar to Raake, A. (2006a) Shortand long-term packet loss behavior: towards speech quality prediction for arbitrary loss distributions) Tobias Hossfeld 27
Simple Cluster Analysis For each user, the average rating and the coefficient of variation of the user ratings is calculated Cluster analysis with k-means algorithm (Matlab, RapidMiner) As input parameter, only these two parameters are used Simple approach is already sufficient to detect the different clusters coefficient of variance 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 Cluster 1 Cluster 2 Cluster 3 0 1.5 2 2.5 3 3.5 4 4.5 5 average rating of each user Tobias Hossfeld 28
QoE Ratings for Different Clusters average user rating of three groups, download time 5 4 3 2 1 0 5 10 15 20 25 30 35 40 sequence of web pages Cluster 1: bored users Cluster 3: normal users Cluster 2: hectic users Tobias Hossfeld 29
Implicit QoE Model: Hidden Markov Model Describe the QoE with a Hidden Markov Model (HMM) Download time as hidden state QoE / user ratings as emission State transition matrix describes system dynamics (in terms of QoS) Emission probabilities for perception categories (MOS 1,, 5) according to actual state and user group One-dimensional HMM fails, since memory effect is not taken into account Tobias Hossfeld 30
2D Hidden Memory Markov Model Enhance the HMM by one dimension wrt. memory effect State of the system is a tupel (actual download time dt_i, previous download time dt_i-1) QoE / user ratings as emission Tobias Hossfeld 31
Notes about the 2D-HMMM Relevant outcome of subjective tests are emission probabilities (for given, i.e. tested, network tupels) Underlying network model (i.e. hidden states) can be changed for a proper description of a system wrt. QoE, it is important to describe it as 2D MMM (due to memory effect) then emissions probabilities remain the same and can be applied Problem is to get measurement data for all N 2 states, when N download times are observed Tobias Hossfeld 32
Discussion Discrete States of HMM Weber s law from psychophysics (1840): The just-noticeable difference ( R) of the change in a stimulus s magnitude is proportional to the stimulus s magnitude (R), rather than being an absolute value. R / R = k and states of HMM are defined accordingly State of the system previous download time vs. average download time (using exponential moving average and discretization of download times) Similar to Oliver Rose: A Memory Markov Chain for VBR traffic with strong positive correlations, ITC 16, Jun 1999. Tobias Hossfeld 33
Test Run 2: Complete User Survey Simulated QoS settings in terms of page download time are colored according to MOS Complete CDF gives overview on actual user experience, not on average users (MOS1+MOS5 ~= MOS3+MOS3) 1 MOS1 MOS2 MOS3 MOS4 MOS5 0.8 ratio of users 0.6 0.4 0.2 0 0 10 20 30 40 web page Tobias Hossfeld 34
Weber-Fechner Law Web traffic experiments@uniwue 5 Weber-Fechner-Law for web traffic 4.5 4 3.5 QoE 3 2.5 2 exp.0, R 2 = 0.968 1.5 exp.2, R 2 = 0.939 exp.4, R 2 = 0.854 1 10 2 10 3 10 4 page load time [s] Tobias Hossfeld 35