Mobile Testing: Actual Results from Nationwide Insurance How we could have tested the same application with the latest tools available

Transcription

1

2 Contents Introductions Petar Puskarich Todd DeCapua Mobile Testing: Introduction / How To Testing our latest mobile application with limited tools and procedures Mobile Testing: Actual Results from Nationwide Insurance How we could have tested the same application with the latest tools available Questions and Discussion The time to ask pertinent questions on mobile performance or anything else that interests you! 2 Copyright 2013 Shunra Software, LLC. Information contained herein is subject to change without notice. 2

3 Introductions Petar Puskarich [Nationwide] Sr. Performance Engineer 16 years in the industry LoadRunner from 4.51 to PC Founding member of the NW Performance Engineering Team Shunra Cloud 1.0 through Shunra for HP Performant Deep Diagnostics 3.0 through HP Diagnostics 9.22 Todd DeCapua [Shunra] Expert: Mobile / Performance / Cloud / Agile Technology Integrations, Channels Management, Customer Advisory Board Executive Leadership in both Applications and Infrastructure Past: Developer, Tester, Performance Engineer ScrumMaster, and Scrum Practioner; transormed 3 organizations Author, Speaker, and Technology Visionary Vivit Worldwide Chapter and SIG Leader HP Performance CAB 3 Copyright 2013 Shunra Software, LLC. Information contained herein is subject to change without notice. 3

4 Mobile Testing: Introduction & How To 4 Copyright 2013 Shunra Software, LLC. Information contained herein is subject to change without notice.

5 State of the testing market Testing Remains Challenging (World Quality Report, Sept. 2012) Organizations struggle with the challenges of mobile only 31% formally test mobile apps Lack of testing tools and knowledge persists 59% characterize QA teams ability to effectively test as merely average ; two-thirds do not have the right tools and over one-third lack appropriate testing methodologies Performance Awareness (Computerworld UK, May 2012) Performance problems are highly prevalent 74% cite performance issues with critical business applications Performance problems are getting worse 82% report speed and responsiveness problems in the past year and nearly half report issues occurring at greater frequency 5 Copyright 2013 Shunra Software, LLC. Information contained herein is subject to change without notice.

6 The network s increasing impact Mobile 70% of mobile transaction response time (TRT) depends on the network % of impact on performance Cloud Cloud Composite Apps Composite Apps Composite Apps Web Web Web Web LAN WAN WAN WAN WAN WAN Application Development Trends 6 Copyright 2013 Shunra Software, LLC. Information contained herein is subject to change without notice.

7 The challenge Deliver high performing applications, on all platforms; In an increasingly complex application and deployment landscape; While significantly reducing costs, and increasing speed to market. Customers Employees The average application connects to 38 discreet services 3 rd Party Cloud Services Major ISP Load Balancers Web Servers App Servers Private or Public Cloud Services DB Servers Mainframe Storage Browsers & Devices Mobile Carriers & ISPs Content Delivery Networks Web Services Mobile Components 7 Copyright 2013 Shunra Software, LLC. Information contained herein is subject to change without notice.

8 When customers struggle with an app In the time it takes to blink an eye, 400 milliseconds to be exact, online and mobile users will have begun to abandon an application. App Provider Competitor 43% 16% 14% 12% Abandon the transaction, try later on computer Become more likely to prefer a competitor Log a complaint with customer service Go directly to a competitor Source: Harris Interactive Mobile Transactions Survey for TeaLeaf, March Copyright 2013 Shunra Software, LLC. Information contained herein is subject to change without notice.

9 When customers impact with an app Basic Assumptions: (Variables Underlined) Current data center utilization is 60% Projected future mobile users are 5% Broadband User has 30msec of latency Mobile User has 300msec of latency Simple Calculation Example: 5% of your users at 10x latency = 50% increased server resource time; and current data center utilization = 60%; therefore, projected future utilization = 50% + 60%, or 110% of infrastructure capacity. Pictorial Representation: Broadband User Mobile User Dynamic Mobile Network Wifi-2.5g-3g-4g 9 Copyright 2013 Shunra Software, LLC. Information contained herein is subject to change without notice.

10 Complete Mobile Testing Solution 10 Network Virtualization for Software Testing

11 End-to-End Enterprise Mobility Testing Strategy Load / Performance Testing: Multi-user and Multi-location Broadband Users Home, 15% Mobile Users 3G, 15% Mobile Users 2.5G 10% Enterprise Users HQ, 50% Real-World Network Conditions & Performance Backend Monitoring Service Virtualization Enterprise Users Distribution, 10% Mobile Functional / Device Testing 11

12 Complete End-to-End Enterprise Mobile Testing Matrix Example: HP, Shunra, and Mobile Device Capabilities Complete Mobile Solution HP Shunra Mobile Device Tool End-to-End Automated Solution for Development & Operations X X X Real-world Load and Capacity Testing X X Virtualized Network Conditions Unlimited Points of Presence X Mobile Application & Services Performance X X End User Application Experience on Mobile Devices X Interoperability Testing with Mobile and other Applications/Traffic X X Industry leading tool for performance, load, and capacity testing X X Industry leading tool for mobile functional test automation X X Record mobile scripts from browser-based applications X X Record mobile scripts from native applications X X Automated Remediation and Optimization Recommendations X Waterfall Performance Results X Transaction Response Time X X Application Turns per Transaction X Bandwidth Utilization per Transaction X Total Data per Transaction X X Average KB per Application Turn X Average / Min / Max of BPS (bits per second) X X 12

13

14 Mobile Testing: Actual Results from Nationwide Insurance (confirming the concepts) 14

15 I & O Performance Engineering Mobile Application Performance Testing: Will your applications PERFORM in PRODUCTON? June 12, 2013

16 Mobile Testing First Attempts lstandard Agency Application Mobile Application Native Android Mobile Web Common Business Transactions Login View Agent Info View Policy Card Directions to your Agent's Office View Current Bill View Account Info Logout Hybrid Hosted Application Infrastructure Internet to Service Provider Service Provider to Local Mobile Engine Local Mobile Engine to Local Database I&O Performance Engineering June 12,

17 Mobile Testing First Attempts ltools Available HP Performance Center p5 HP Diagnostics Android SDK Emulator Corporate Proxy Shunra VE Appliance Mobile Platform Vendor Tech Team AppDynamics Java Visibility Custom monitoring lprocess Available Business Transaction Volumes FrontEnd Native and Web 600 global sessions per hour Mobile Engine Web Services Unknown calls per hour Database Unknown service projections 1 script needed for Mobile Web 1 script needed for Native App Android 3G latency bandwidth chosen for average vuser test latency 100 vusers with global iteration times of 10 minutes chosen I&O Performance Engineering June 12,

18 Mobile Testing First Attempts lwhat Happened? First Phase Mobile Engine Testing 5 scripts generated to simulate different web services calls supported Scenario setup initially with 600 invocations/hr to match known projections Ran scenario for 2 hours Excellent response times Local Java Servers with HP Diag probes showed healthy jvms Healthy backend db Ran new scenario for 2 hours 2X rate Same Success as 1x level Ran new scenario for 2 hours 4x rate Same Success as 1x level Break test! Mobile Engine services finally slowed down to 2x response times 24x level 24x600=14400 invocation/hr At least we now have a performance signature! I&O Performance Engineering June 12,

19 Mobile Testing First Attempts lwhat Happened? Second Phase Mobile Web Page Version Released for Testing 1 script with multiple transactions created via Vugen recording through IE Browser Scenario setup with no latency for baseline Ran for about 1 hour Remote JVM OOM Hard Fall Mobile vendor insists its a SCRIPT issue Instead of using VUGEN standard http scripts we are forced to rerecord and correlate with mobile http protocol this time to satisfy objections of vendor Enter Android SDK Emulator Corporate proxy is NOT supported well in simulator Ended up recording on laptop via open retail wifi hotspot... Mobile HTTP script was finally generated New scenario with Mobile HTTP scripts and Native App Ran for 1 hour SAME SYSTEM FAILURE I&O Performance Engineering June 12,

20 Mobile Testing First Attempts lwhat Happened? Back and forth with Vendor Used HP Diagnostics to prove there was a memory leak Turns out memcached not configured properly Retest for 4 weeks coordination with OffShore Tech Staff Finally SUCCESS Setup Shunra to simulate 3G and 4G links Scenario still runs with success! Validation complete! I&O Performance Engineering June 12,

21 Mobile Testing Next Project New Mobile Native App for ios and Android Standard Auto Policy focus Use HP Mobile Proxy New Feature l l l Use Shunra for HP Integration for in-scenario latency generation and response Validate Total end to end performance in approximately half the time as first attempt! l l l I&O Performance Engineering June 12,

22 Questions I&O Performance Engineering June 12,