Testing & Assuring Mobile End User Experience Before Production Neotys
Agenda Introduction The challenges Best practices NeoLoad mobile capabilities
Mobile devices are used more and more At Home In 2014, more people will connect to the Internet from mobile devices than from PCs 1.2 billion new smartphones enter the market in the next 5 years In 2015, 98 billion mobile applications will be downloaded In the Office 30% of companies have or develop applications for their employees. 88% of employees use their personal devices in their work.
Different type of mobile testing Functional testing Device testing Performance Testing Functional Is this application does what it is supposed to do? Equipment How is the application works on the device? Rendering CPU, Battery Performance Is 1000 users can use this application simultaneously? What is the user experience when the server is loaded? What is the behavior of the application with different network conditions?
Agenda Introduction The challenges Best practices NeoLoad mobile capabilities
Mobile Performance Testing is Challenging for Enterprise Organizations For 64% Mobile Performance Testing is #1 Testing Priority Don't have the right processes or methods 34% But Don't have the right tools 65% Don't have the devices they need readily available 52% 6 http://www.capgemini.com/thought-leadership/world-quality-report-2012-13
Mobile Project are shorter with less budget Back End services Common services for all devices Don t need the mobile application to start testing Testing driven development Mobile application Use the service calls to retrieve the data Last mile of the development life cycle
Validate the Performance of a Mobile Application is complex! Device Network Back-End Client APP App Native Mobile Mobile Browser Hybrid EXECUTION PLATFORM iphone 4, 5, 6 Samsung Blackerry (7,000 different device types are used to access Facebook) 3G, 4G / LTE, Wifi RIA/FRAMEWORKS Database Video OS Screen size CPU Memory Throughput Latency Packet loss Server Infrastructure HTTP Adaptive Streaming
Impact of the Network
Latency and bandwidth The Most important network constraints are : Latency Bandwidth From a certain level of bandwidth, increasing the bandwidth will have a minor effect.
Packet size on a 3G Network What is the average packet size on mobile networks? 350000 300000 250000 200000 150000 Number of packet 100000 50000 0 100b 1kb 10 kb 100 kb 1 Mb 10 Mb More Traffic recorded on a 3G Network by an European Operator during 1 week in October 2010
Packet LosT influence Packet loss consequence Packet Retransmission End user experience degraded because Packet loss will increase jitter Errors could occur if Packet loss are in pure data transmission Packet loss is inevitable in Mobile Network Mobility/Fast switch to next cell/temporary loose the signal for congestion of the current cell Mobile protocols try to reduce the packet loss rate
Wrap-up
Different kind of application
Mobile Context Lots of Devices Device Challenges Operating Systems - ios - Android - Windows Technologies used in the app - HTML 5 - Web Socket - Streaming Video/audio - Google SPDY Screen Sizes - Different content for different devices - Design of what content is above/below the fold - When that content gets delivered
Mobile Context High, Distributed Load Lots of users Mobile applications often have large user bases High growth in popularity can create the need for highly scalable load generation infrastructure Geographical distributed load Users are accessing apps from locations around the globe Distance and location can greatly affect performance Mix of Web & Mobile Users Many mobile apps have a corresponding web app with a shared backend High load on the web app can affect performance on mobile
CDN principles
CDN advantages example
How to test CDN-based architecture 2 kinds of CDN Caching CDN Streaming CDN Tests must be realistic Spread the load among different geos and network Cache usage Streaming emulation CDN Backbone are hosted by Cloud providers: must choose appropriate Cloud provider(s) DNS resolution is mandatory in order to avoid stress a cdn node Measure response times from different locations Cloud or multiple load generators are needed to stress several node of the CDN
Agenda Introduction The challenges Best practices NeoLoad mobile capabilities
Constructing Scenarios Creating Real World Conditions Application under realistic load Real devices for capturing actual application performance on the device Real mobile CDN paths for testing the whole delivery chain
Include Network Constraint during your test Select most represented location and carries Measure the latency /packet Loss from the major locations Store your measurements into a network Matrix Involve the Management to get statistics Ratio of user connected on Wifi/4G/3G/Edge Ratio or number of user per locations Include the Network constraint in your load policy Organize your load policy depending on the network constraints Combine the usage of Cloud Based Load generator and Wan emulation
Take profit of the Cloud Based Load Generator Test the entire chain Firewall DNS ISP Network Traffic generated from different geos Different networks Multiple Cloud providers Response time comparison Save Time Deployment time is reduced Administration time is reduced No maintenance No Upgrades Save Cost Pay only for what you use No machines acquisition No Maintenance
Analytics to Enable Decision-making Actionable Insight Monitoring on all infrastructure layers (App, Web, DB) Stats on backend and frontend to pin point root cause of performance problems Shift Left Test integration with CI systems to get feedback to development teams earlier Fix problems earlier when they re less expensive
Mobile Performance Testing Product What are you really measuring?
Complete Mobile EUE Load Rendering Time App Error Rate Real Device Testing Tool Real Device Users 00:00:00 00:00:30 00:01:00 00:01:30 NeoLoad Virtual Users Load Application Server
Report proper metrics Pie I have not yet eaten Pie I have eaten
Agenda Introduction The challenges Best practices NeoLoad mobile capabilities
NeoLoad s Mobile Capabilities Record from any device Browser-based application Native application Simulate any device or browser Ensures the right content is delivered for that device or browser Simulates parallel connections for each specific device and browser Emulate realistic network conditions Bandwidth Latency Packet Loss Generate load from the Cloud Geographic realism Rapid scalability for large tests Monitor Cloud Load Generators Command Network H+ 3G Emulation 4G WiFi & Device Simulation APP App Server Database Server On-premise Load Generators Load Web Server
NeoLoad + MobileCloud Real Device Metrics Under Load Automatically trigger MobileCloud test with the start of a NeoLoad test See real device rendering times and vitals while the backend of the application is under load Results are correlated in real time for accurate and fast analysis On-premise Load Generators NeoLoad Controller Command Cloud Load Generators Simulated H+ Devices 3G w/ Network 4G WiFi Emulation Load Real Device Metrics Command 4G MobileCloud H+ 3G Real Devices & Networks Monitor APP App Server Database Server Web Server
Questions? Next Steps Visit Neotys booth Download NeoLoad Free Edition from NeoLoad PoC Assistance: Email sales@neotys.com Visit Neotys Community at answers.neotys.com Evaluation support at /support/contact.html