Application and segment driven QoS measurements and benchmarking

Transcription

1 ITU Workshop on Monitoring and Benchmarking of QoS and QoE of Multimedia Services in Mobile Networks (Dubai, UAE 2-3 November 2014) Application and segment driven QoS measurements and benchmarking Esa Vesterinen Vice President Omnitele

2 Topics discussed Application QoS Be the customer assessment Case studies: Commuting traffic in Finland Dutch football experience Omnitele Ltd

3 Quality perception is application and segment dependent Location Different segments have different QoS needs: service location time Average QoS hides the most important - user segments customer experience There is no average customer, nor average quality Omnitele Ltd

4 Be-the-Customer QoS Assessment T A R G E T E D one segment at a time R E A L U S A G E Segment dependent M E A N I G N F U L service level KPIs Omnitele Ltd

5 Our recent case examples... WWW browsing in commuting trains Marketing Benchmark Telia-Telenor challenge WWW Browsing for Dutch football fans International Roaming Business travellers Omnitele Ltd

6 CASE PASSENGER: WWW Browsing Experience in Train Omnitele Ltd

7 Segment example: The Passenger HOME (HYVINKÄÄ) WiFi 1 HOUR IN TRAIN Cellular Data WORK (HELSINKI) WiFi Passenger relies on mobile data only while commuting to work 2 hours/day: Hyvinkää-Helsinki-Hyvinkää. On WiFi at home & office. For Passenger, macro-cellular average QoS is irrelevant Omnitele Ltd

8 The Passenger is not alone #Travels annually < 0.1 M M M M M M >5.0 M Omnitele Ltd

9 Location and time dependent usage Passenger QoE lost in noise The passengers visible in 0.3% sites ~6000 sites example network in total ~20 sites on Passenger rail route The passengers counts for 0.14% time 24 h: typical aggregation in statistics 2 min: one passing train in typical cell area But daily ~ Passengers only see the in-train QoS Omnitele Ltd

10 Cluster average could be OK even if the in-train QoS was horrible! Omnitele Ltd

11 Be-the-Customer Measurements 3 Operators benchmarked: Location: Train 100km Hyvinkää-Helsinki-Hyvinkää Time: to/from work morning 7:30-9:00 / evening 16:00-18:00 Service: WWW Browsing, commercial SIM cards & iphone 5s Omnitele Ltd

12 Results: Poor WWW Success Only 77% of all WWW page retrievals are successful (N=1090) 100% 80% 60% 40% WWW Page Success Ratio (%) 99% 98% 99% 87% 67% 77% 20% 0% Operator X Operator Y Operator Z Reference Cluster In-train Omnitele Ltd

13 Results: Slow WWW Surfing In train experience is 2-3 times slower than in network reference clusters 10 WWW Page Waiting Time (s) Operator X Operator Y Operator Z Reference Cluster In-train Omnitele Ltd

14 Poor in-train WWW success not location specific Hyvinkää WWW success, Operator Y Successful WWW retrieval Drop / Time-out Helsinki Omnitele Ltd

15 Case take-out Different user segments have very different needs and usage patterns Average QoS easily hides the segment s experience Need to address targeted segments with meaningful application KPIs Omnitele Ltd

16 CASE FOOTBALL FAN: WWW Browsing Experience WC2014 in Amsterdam Omnitele Ltd

17 Web Browsing Customer Experience in Amsterdam - Football WC2014 experience CEM s o l u t i o n CHALLENGE best mobile WWW browsing experience? HOW WC2014 related tests with Anite s CEM Observer WHERE locations where football fans watch the games Omnitele Ltd

18 Benchmark Overview 38 test locations in Amsterdam where fans watch the WC2014 games Measurement profile built on football experience browsing of WC2014 related pages: results, stories, statistics, interviews in-brazilie.htm Omnitele Ltd

19 Benchmark Overview CEM s o l u t i o n 38 test locations in Amsterdam where fans watch WC2014 games Measurement profile built on WC2014 results, stories, statistics, interviews Omnitele Ltd

20 Results: the delusion of bitrate importance 15 end-user bitrate [Mbps] If bitrates were benchmarked, T-Mobile has the best service 0 6 WWW page waiting time [s] but in customer experience operators are neck and neck 0 Omnitele Ltd

21 who offers the best customer experience? Highest throughput on average Fastest browsing on average Best in most locations % 26% % Omnitele Ltd

22 KEY MESSAGES Bitrate not self-sufficient to represent Customer Experience Different result aggregations may yield different conclusions Customer Experience measurements give means to focus network optimisation Omnitele Ltd

23 Optimising Mobile Web Customer Experience Omnitele Ltd

24 10 Average throughput tells nothing - domain performance varies greatly WWW page waiting time [s] - domain performance 8 6 Operator average s always poor? always good? Omnitele Ltd

25 websites differ but expectations don t #content elements data volume [kb] customer expectations are same Omnitele Ltd Test sites:

26 Frustration level of web browsing very fast acceptable a bit too slow too slow, annoying out of my tolerances >5s often frustration expectations vary among markets, segments, time WWW page waiting time [s] [QoE results from Omnitele case study 2013] Omnitele Ltd

27 Focus your optimisation effort to right areas CEM s o l u t i o n WWW page waiting time [s] - domain performance frustration threshold Focus: internet connectivity Focus: Local optimisation Omnitele Ltd

28 www waiting time [s] Operator X: Identifying the locations for RAN optimisation CEM s o l u t i o n 20 4, 22, 24, 30, 32: telegraaf.nl OK probably little or no gain with radio optimisation 36, 38: fifa.com & telegraaf.nl correlates, potential gain from radio optimisation location ID fifa.com telegraaf.nl Omnitele Ltd

29 Example: Long web browsing loading time for operator X 1: Due to poor LTE coverage, the mobile is camping on UMTS : When trying to visit a website, the mobile attempts to access the RACH, but fails. Only after 5 seconds, an attempt is successful. The mobile sends measurement event 4a and is immediately sent from carrier to carrier 10811, where the data transfer starts. 3: Even though radio condition are good and download of the website is not yet complete, a physical channel reconfiguration reduces the throughput from 7M Mbps to 700 kbps. This reconfiguration combined with the RACH access problems create an extremely long web page loading time. Solution: Investigate the capacity of the UMTS cell 1. on carrier (RACH failure) and (dropping throughput). Omnitele Ltd

30 Case take-out Dutch football fans have certain expectations where and how they 1 can access their interest sites www waiting time was measured and 2 variations were analysed network performance was measured simultaneously and root cause 3 analysed and solved Omnitele Ltd

31 Questions? For any further info, kindly refer Omnitele Ltd

32 Omnitele QoE management services ASSESS customer experience benchmark IMPROVE customer experience optimisation CONTROL QoE driven performance management Omnitele Ltd

33 maximised customer experience minimised network cost Omnitele Ltd w w w. o m n i t e l e. c o m

34 why we exist Omnitele is an international telecommunications consulting and engineering company. We provide services for telecom operators and regulators in domains of network strategy, design and quality assurance. Our mission is to maximise mobile subscriber quality of experience and minimise operator network cost. track record We have solid track record and over 25 years of experience in telecoms industry. The company was founded in 1988 to set up world s first GSM operator and network. Since then we have completed projects in 80+ countries. presence Our headquarters is located in Helsinki, Finland. We have local presence in the Netherlands and the Hashemite Kingdom of Jordan. Our company is owned by Finnish telecom investors and we are independent of all international operators and telecom network infrastructure equipment vendors. omnitele way Our unique way of working sets us apart from the competition. We call this the Omnitele Way: being Straightforward, Trusted and Intelligent. We deliver tangible results and ensure excellent Omnitele Experience. Omnitele Ltd w w w. o m n i t e l e. c o m

35 our services Technology Strategy Design and Optimisation Audit and Benchmark Performance Management maximised customer experience minimised network cost Omnitele Ltd