Testing & Assuring Mobile End User Experience Before Production. Neotys

Transcription

1 Testing & Assuring Mobile End User Experience Before Production Neotys

2 Agenda Introduction The challenges Best practices NeoLoad mobile capabilities

3 Mobile devices are used more and more At Home In 2014, more people will connect to the Internet from mobile devices than from PCs 1.2 billion new smartphones enter the market in the next 5 years In 2015, 98 billion mobile applications will be downloaded In the Office 30% of companies have or develop applications for their employees. 88% of employees use their personal devices in their work.

4 Different type of mobile testing Functional testing Device testing Performance Testing Functional Is this application does what it is supposed to do? Equipment How is the application works on the device? Rendering CPU, Battery Performance Is 1000 users can use this application simultaneously? What is the user experience when the server is loaded? What is the behavior of the application with different network conditions?

5 Agenda Introduction The challenges Best practices NeoLoad mobile capabilities

6 Mobile Performance Testing is Challenging for Enterprise Organizations For 64% Mobile Performance Testing is #1 Testing Priority Don't have the right processes or methods 34% But Don't have the right tools 65% Don't have the devices they need readily available 52% 6

7 Mobile Project are shorter with less budget Back End services Common services for all devices Don t need the mobile application to start testing Testing driven development Mobile application Use the service calls to retrieve the data Last mile of the development life cycle

8 Validate the Performance of a Mobile Application is complex! Device Network Back-End Client APP App Native Mobile Mobile Browser Hybrid EXECUTION PLATFORM iphone 4, 5, 6 Samsung Blackerry (7,000 different device types are used to access Facebook) 3G, 4G / LTE, Wifi RIA/FRAMEWORKS Database Video OS Screen size CPU Memory Throughput Latency Packet loss Server Infrastructure HTTP Adaptive Streaming

9 Impact of the Network

10 Latency and bandwidth The Most important network constraints are : Latency Bandwidth From a certain level of bandwidth, increasing the bandwidth will have a minor effect.

11 Packet size on a 3G Network What is the average packet size on mobile networks? Number of packet b 1kb 10 kb 100 kb 1 Mb 10 Mb More Traffic recorded on a 3G Network by an European Operator during 1 week in October 2010

12 Packet LosT influence Packet loss consequence Packet Retransmission End user experience degraded because Packet loss will increase jitter Errors could occur if Packet loss are in pure data transmission Packet loss is inevitable in Mobile Network Mobility/Fast switch to next cell/temporary loose the signal for congestion of the current cell Mobile protocols try to reduce the packet loss rate

13 Wrap-up

14 Different kind of application

15 Mobile Context Lots of Devices Device Challenges Operating Systems - ios - Android - Windows Technologies used in the app - HTML 5 - Web Socket - Streaming Video/audio - Google SPDY Screen Sizes - Different content for different devices - Design of what content is above/below the fold - When that content gets delivered

16 Mobile Context High, Distributed Load Lots of users Mobile applications often have large user bases High growth in popularity can create the need for highly scalable load generation infrastructure Geographical distributed load Users are accessing apps from locations around the globe Distance and location can greatly affect performance Mix of Web & Mobile Users Many mobile apps have a corresponding web app with a shared backend High load on the web app can affect performance on mobile

17 CDN principles

18 CDN advantages example

19 How to test CDN-based architecture 2 kinds of CDN Caching CDN Streaming CDN Tests must be realistic Spread the load among different geos and network Cache usage Streaming emulation CDN Backbone are hosted by Cloud providers: must choose appropriate Cloud provider(s) DNS resolution is mandatory in order to avoid stress a cdn node Measure response times from different locations Cloud or multiple load generators are needed to stress several node of the CDN

20 Agenda Introduction The challenges Best practices NeoLoad mobile capabilities

21 Constructing Scenarios Creating Real World Conditions Application under realistic load Real devices for capturing actual application performance on the device Real mobile CDN paths for testing the whole delivery chain

22 Include Network Constraint during your test Select most represented location and carries Measure the latency /packet Loss from the major locations Store your measurements into a network Matrix Involve the Management to get statistics Ratio of user connected on Wifi/4G/3G/Edge Ratio or number of user per locations Include the Network constraint in your load policy Organize your load policy depending on the network constraints Combine the usage of Cloud Based Load generator and Wan emulation

23 Take profit of the Cloud Based Load Generator Test the entire chain Firewall DNS ISP Network Traffic generated from different geos Different networks Multiple Cloud providers Response time comparison Save Time Deployment time is reduced Administration time is reduced No maintenance No Upgrades Save Cost Pay only for what you use No machines acquisition No Maintenance

24 Analytics to Enable Decision-making Actionable Insight Monitoring on all infrastructure layers (App, Web, DB) Stats on backend and frontend to pin point root cause of performance problems Shift Left Test integration with CI systems to get feedback to development teams earlier Fix problems earlier when they re less expensive

25 Mobile Performance Testing Product What are you really measuring?

26 Complete Mobile EUE Load Rendering Time App Error Rate Real Device Testing Tool Real Device Users 00:00:00 00:00:30 00:01:00 00:01:30 NeoLoad Virtual Users Load Application Server

27 Report proper metrics Pie I have not yet eaten Pie I have eaten

28 Agenda Introduction The challenges Best practices NeoLoad mobile capabilities

29 NeoLoad s Mobile Capabilities Record from any device Browser-based application Native application Simulate any device or browser Ensures the right content is delivered for that device or browser Simulates parallel connections for each specific device and browser Emulate realistic network conditions Bandwidth Latency Packet Loss Generate load from the Cloud Geographic realism Rapid scalability for large tests Monitor Cloud Load Generators Command Network H+ 3G Emulation 4G WiFi & Device Simulation APP App Server Database Server On-premise Load Generators Load Web Server

30 NeoLoad + MobileCloud Real Device Metrics Under Load Automatically trigger MobileCloud test with the start of a NeoLoad test See real device rendering times and vitals while the backend of the application is under load Results are correlated in real time for accurate and fast analysis On-premise Load Generators NeoLoad Controller Command Cloud Load Generators Simulated H+ Devices 3G w/ Network 4G WiFi Emulation Load Real Device Metrics Command 4G MobileCloud H+ 3G Real Devices & Networks Monitor APP App Server Database Server Web Server

31 Questions? Next Steps Visit Neotys booth Download NeoLoad Free Edition from NeoLoad PoC Assistance: Visit Neotys Community at answers.neotys.com Evaluation support at /support/contact.html