Performance Testing for Mobile and Cloud Applications May 2014

Shane Evans Senior Product Manager, HP Software Product Manager for industry-leading HP LoadRunner Career Performance Engineer ScrumMaster, ITIL v3 certified 13 years experience in IT ranging from Technical Support to Quality Assurance, Performance Management and Engineering Joined HP in 2009 2

A radically different kind of user Mainframe Client/Server Web Devices System-centric User-centric Users born after 1980 3

Change is accelerating Mobile Pace of Change Cloud Cloud Composite Apps Composite Apps Composite Apps Web Web Web Web 4 WAN WAN WAN WAN WAN LAN Application Development Trends

After the last couple of updates, it became useless Sooo slow, Unavailable, Tx not updated 6

When customers struggle on a mobile device In the time it takes to blink an eye 400 milliseconds to be exact online users will have begun to abandon your application You Your competitor 43% 16% 14% 12% 7 Abandon the transaction, try later on computer (Harris Interactive Mobile Transactions Survey for TeaLeaf, March 2011) Become more likely to prefer a competitor Log a complaint with customer service Go directly to a competitor

Modern Apps are complicated mdev: Data Caching Network: Latency & Loss Network: Encrypt, Compress. Network: Load Bal. Algorithm Dev: Resource Usage SysAdmin: Server Config. Back-end: Pooling & Threading DBA: DB Indexes & Logs Platforms & Versions 2G,3G,4G Cellular 3 rd Party Cloud Services Firewalls VPNs Web Servers App Servers DB Servers Storage Device Features Screen Sizes Carriers ISPs Load Balancers Internet Browsers LANs Web Services Mobile Components CDNs & CoLo. 8

The Challenges Waterfall: Perf. test Steps: Server setup Scripting Milestones: ready ready Operations Developers DBAs SysAdmins Perf. test Data load Config. & Monitors Runs & Reports Agile: Product increment 1 Product increment 2 Product increment 3 Preparation 9

So what can we do? Copyright Scott Adams. http://dilbert.com/strips/comic/2005-11-16/

HP delivers ROI in under 90 days* TechValidate Survey (February 2013) A Global 500 bank reduced performance incidents by over 70% A large, enterprise financial services company reduced testing time and accelerated application delivery by 30-40% A computer software company improved application performance / response time by 51-70% HP is helping customers save between $468,000 (for an incident that requires 1 day to remediate) to $2.8 million (for an incident that takes 6 days to remediate) per month. On average, customers save over $514,800 annually in remediation costs alone. * *Source: 2011 Customer Survey Improve application response times by 51-70%++ Accelerate application delivery cycles by up to 40% Reduce performance issues in production by up to 70% 11

What can we do? Performance Testing in Agile in 3 basic steps Improve time to value Test smarter Provide actionable data 12

Improve Time to Value Develop to performance standards Performance Budgets Best Practices Metrics and reporting Leverage Unit Tests during Development for early performance feedback Automate Performance Test execution, analysis and reporting CI-driven testing Automated analysis and trend reporting Virtualize dependencies for an early integrated view of application performance Network Services/Backend 13

standards are Unicorns Programming Sucks http://stilldrinking.org/programming-sucks

Mobile Development Standards Which of the following testing and release management practices does your development team currently use? Unit testing Performance/load testing Continuous integration with multiple weekly check-ins Automated tests used to verify quality Exploratory testing Successful builds deployed to virtual environments of VM Automated deployment and provisioning process for Use feedback management tools to automate beta testing Code instrumented with analytics Dedicated refactoring time against the existing project Other (please specify) None of the above 1% 20% 17% 14% 11% 10% 7% 14% 32% 30% 38% 58% Base: 698 Professional Developers, Game Developers, IT Developers, Consultants Source: Forrsights Developer Survey, Q1 2013 15

Performance modeling is essentially a "budgeting" exercise. Budget represents your constraints and enables you to specify how much you can spend (resource-wise) and how you plan to spend it. Constraints govern your total spending, and then you can decide where to spend to get to the total. You assign budget in terms of response time, throughput, latency, and resource utilization. Improving.NET Application Performance and Scalability April 2004, MSDN http://msdn.microsoft.com/en-us/library/ff647767.aspx

Performance Budgets Identify Key User Stories Identify/Estimate Workload Define Performance Objectives Breakdown Processing Steps Allocate Budgets Evaluate/Validate 17

Performance Budgets Up to 70% of app performance! 350-450 milliseconds 200 milliseconds 100 milliseconds 150 milliseconds 50 milliseconds 3 rd Party Customers Cloud Services Major ISP Load Balancers Web Servers App Servers DB Servers Mainframe Employees Storage Browsers& Devices Mobile Carriers & ISPs Content Delivery Networks Web Services Mobile Components 1 second *Watch Breaking the 1000ms second time to glass mobile barrier at http://bit.ly/1mx7ohq 18

Network Misconceptions It s All About Bandwidth Bandwidth No Impact No Impact No Impact Some Impact 10Mbps 20Mbps 2Mbps 500kbps Latency 25ms Packet Loss = Further Impact All Traffic Impacted 19

Network Misconceptions It s All About Bandwidth Less Critical Due to 4G (5G, 6G ) Carrier Frequency: Increased Interference 2G 3G 4G 5G 20% of all 4G Transfers Drop to 3G or Lower Cisco Mobile Forecast: Mobile data traffic will grow 7-fold from 2013 to 2018 = compound annual growth rate of 50% Ofcom: Current UK Mobile Data capacity will be exhausted by 2018 Shannon-Hartley Theorem: Every Mobile Cell can only support a maximum channel capacity 20

Network Misconceptions It s All About Bandwidth Less Critical Due to 4G (5G, 6G ) It Doesn t Make Much Difference 20x to 35x slower Baseline London Web Dublin Web Web Users Average Transaction Response Time (s) 21

Network Misconceptions It s All About Bandwidth Less Critical Due to 4G (5G, 6G ) It Doesn t Make Much Difference 80x to 400x Slower Functional Failures Baseline London 3G Dublin 3G London 2.5G Mobile Users Average Transaction Response Time (s) 22

Network Misconceptions It s All About Bandwidth Less Critical Due to 4G (5G, 6G ) It Doesn t Make Much Difference It s Only About Performance VUsers Local Load Test VUsers Load Test with NV Transaction Time Transaction Time Capacity > 500+ Users Response < 10 seconds Capacity < 200 Users Response > 40 seconds 23

Network Misconceptions It s All About Bandwidth Less Critical Due to 4G (5G, 6G ) It Doesn t Make Much Difference It s Only About Performance 50% CPU Increase No Network With Remote With Web With Mobile Thread Count 22% Thread Count Increase No Network Remote Web Users 24

Network Misconceptions It s All About Bandwidth Less Critical Due to 4G (5G, 6G ) It Doesn t Make Much Difference It s Only About Performance NV is Only For Mobile Remote Office Web Cloud Deployments All Other Connectivity (Wifi, Satellite, BT) Back-End Splits DCR Wherever Applications Meet Networks 25

Virtualize Application Dependencies 3 rd Party Customers Cloud Services NV SV Major ISP Load Balancers Web Servers App Servers DB Servers Mainframe Employees Browsers& Devices NV Mobile Carriers & ISPs Content Delivery Networks NV Web SV Services Mobile Components SV Storage Performance testing + virtualization reduces risk and total cost of ownership 26

Test within the Sprint Feedback Code check-in Build & Test Discover defects earlier Change is fresh in developer s mind. Context switching and re-familiarization not needed. Changes are isolated; no need to analyze what caused the defect. Constant testing, ensures stable code and a higher-quality release. 27

Performance test during Development 1. Build Load Test scripts during Development 2. Run Unit Tests in Load Test Scenarios for early feedback to Developers 3. Add Performance Tests as build steps in Jenkins Test 4. Rinse and repeat 28

The ideal picture: continuous testing, including performance testing A relentless assault on project risk Sprint 1 Sprint 2 1 Sprint 3 Sprint 4 Sprint 5 29 ET C1 ET C1 ET C1 ET C1 ET C1 G1 G1 G1 G1 G1 CI CI C2 Regression 1 CI C2 G2 Regression 2 CI C2 G2 C3 Regression B1 3 CI C2 G2 C3 B1 Regression 4 End to End Performance Testing focused on load(independent of sprint cycles)

Test Smarter Eliminate waste Scripting time Data creation/management Environment configuration/validation Maintenance Remember Production! Isolate device and carrier performance Gather production usage data Agile Delivery Deliver faster IT Operations Change is evil Never stop testing Performance Testing 30

Test Efficiently Unicorns aren t just for Developers 31 Build Tests that are modular to reduce maintenance

Testing components and workflow End User Client(s) Internet/WAN Application Server(s) Database 1 Author Scripts 3 Virtualize Network 2 Virtualize Services 4 Generate Load 5 Measure Impact Performance Center 6 Analyze, Compare, Report Centralized Assets 2 Trend Reporting Test Scenario Performance Lifecycle Plan and Execute Tests Global Collaboration 32

Testing can be expensive and risky Dedicated Hardware Network Exposure AUT AUT Scalable Test Environment Secure Configuration LG LG LG Hosting LG 33

Cloud Testing / Maximum Performance and Scale Customer-centric enhancements to performance testing -- in the cloud Scale without the overhead using automated, worldwide provisioning of load generators in the cloud HP LoadRunner HP Performance Center The world s most widely adopted performance testing solutions Save time/effort by leveraging and sharing cloud testing configurations and resources across teams Simple, Elastic, Realistic Simplify administration by defining cloud accounts, manage cloud host usage 34

Cloud Testing / Maximum Performance and Scale Scale without the overhead HP Performance Center and HP LoadRunner seamlessly leverage public Cloud infrastructure to deploy Load Generators (LGs) LoadRunner or Performance Center Connect securely on standard ports AUT AUT Scale up and down based on test requirements Geographical distribution Share cloud environments in between tests LG LG LG LG Value: Elastically scale up to meet demands reducing cost and overhead of managing dedicated infrastructure 35

Cloud Testing / Maximum Performance and Scale Share cloud testing configurations and resources across teams Add multiple Cloud accounts and manage Network Profiles for connecting to your various LGs Manage cloud accounts Provision multiple LoadGenerators (LGs) Run test with Cloud LGs Review provisioning reports Value: Significantly reduces provisioning time of performance testing, while maintaining security and control 36

Cloud Testing / Maximum Performance and Scale Simplify cloud administration with My Performance Center Easy administration of a shared cloud performance test environment using My Performance Center Allow PC testers to create test sets within MY PC Easier Test Creation via Test Express Provide visibility into LAB Management Ease of access of ALM feature set Value: Reduce time of administration of performance testing shared environment 37

We needed to simulate -prior to tax return day- 80,000 users all logging in at the same time to see if the tax system could cope. Thanks to HP SaaS, we were able to make critical changes and tweaks to our system before the big day and have the confidence that it would take the strain. Service Manager, Government Institution

Provide Actionable Insight Monitoring should be baked into the environment configuration, not an afterthought Test the positive AND negative scenarios Profile under load 39

Development Analytics Check In Security Code Scan Test results Tear down environment Monitoring & Diagnostics Run test cases Provision & deploy environment with monitoring 40

Production Feedback Loops Know thy User 4 Leverage test automation, SLOs and Monitoring insights from test in Production 1 Dev-test with lab management automation 2 Automated deployment across development and operations Provision dev and test environments Deploy test app Monitor Test results Run test cases Provision and deploy to staging Provision and deploy to prod 41 3 Production patterns, snapshots, and monitors back to dev and test

Isolate End-User Performance by Carrier 42

Isolate End-User Performance by Device 43

Develop to Perform better Profile application performance under load, not only in isolation. Method taking the bulk of time 44

How to Analyze the Performance of a Mobile App Waterfall Analysis Waterfall chart for the preview step Blocking downloads CDN resources Still slow for mobile access 45

5 Principles of Agile Performance Testing 1. Test within the Sprint 2. Leverage Developer assets (i.e. Unit Tests) 3. Test what has changed compared to baselines 4. Automate test execution from the CI 5. Virtualize all dependencies 46

Accurate Performance Testing Environments for Seagate Mobile CRM, HP LoadRunner and Network Virtualization Provide an Integrated Solution Business Challenge Seagate had application performance issues that required significant time to troubleshoot. Seagate needed the ability to inject real world mobile network conditions into performance tests to avoid performance incidents from the applications they deliver. They had incomplete test coverage. The most critical mobile CRM application that the field sales team utilizes has high visibility attached to it and must perform well, as it has a direct impact on revenue. Further, there are additional mobile applications that are critical for internal organizations and performance is top of mind. Solution HP LoadRunner and Shunra NV for HP provides a full end-to-end mobile performance testing solution that enables Seagate to conduct more accurate testing by incorporating real world mobile network conditions into tests within HP LoadRunner much earlier in the SDLC, and confidently deploy mobile applications into their production environment. Benefits Cost avoidance and cost reduction with the ability to proactively test under realistic mobile network conditions and identify and resolve performance issues early in the SDLC. Seagate avoided a significant amount of cost that would otherwise be invested fixing application performance problems post production. HP and Shunra gave me the ability to inject real world mobile network conditions into our performance tests to avoid performance incidents from the applications that we re delivering. We now have a full end-to-end mobile performance testing solution. Manager, Software Quality Assurance, Seagate 47

Strengthened Client Relationships & Brand Equity for KMD Mobility ELA, Leveraging HP Performance Center & LoadRunner and Network Virtualization Business Challenge KMD needed the ability to avoid performance incidents from the applications they deliver. Performance issues impact their brand, their client relationships and their future abilities to sell. The competitive landscape is becoming increasingly challenging and KMD needs to provide better quality to retain customers. Solution HP Performance Center, HP Load Runner integrated with Shunra NV for HP together enable KMD to predict and tune performance issues very early in the SDLC, by building performance into their mobile and web applications. Benefits The capabilities to predict and overcome performance incidents helped KMD increase brand equity and revenue while improving the service offerings provided by the quality center team. Eliminated severe performance issue delays of 4-6 months. Demonstrated a significant performance increase with Mobile conditions with some transaction response times jumping by up to 10 times. As a software house, the performance of our applications is critical to our business. HP and Shunra gave us an integrated solution to predict and overcome our performance incidents, enabling us to deliver exceptional digital solutions to our customers. Head of Operation Quality Center, KMD 48

Dramatically Reducing Performance Incidents for BNP Mobile and Web, HP Performance Center and Shunra NV for HP Integrated Solution Business Challenge BNP estimated that projects not thoroughly tested were costing the company in excess of 500K annually due to issues occurring in production. Solution BNP expanded the use of its Center of Excellence with Shunra NV for HP Software, ensuring that all projects (including mobile) could be accurately and reliably tested without compromising project schedules. Benefits BNP dramatically reduced the number of performance incidents requiring remediation, resulting in a savings of 720K. An additional 500K investment is expected related to mobile app testing, with an expected ROI of 1.5M. Production issues cost us more than 500K annually. The integrated HP and Shunra solution enables us to reliably test our environments without compromising our project plans or schedules. Quality Assurance Director, BNP 49

SONY Europe Mitigates Data Center Move Risk with HP and Shunra Data Center Move, HP Performance Center and Shunra NV for HP+, >$10M Saved in 3 years Business Challenge Major high risk Data Center Move (DCM) initiative. Sponsor shared that apps must continue to perform after DCM. Had previously experienced an outage on web systems which cost Sony Europe $100 million. Must avoid similar failures. Solution NetworkCatcher allowed the performance team the ability to efficiently gather production network metrics and replay them with confidence with both Shunra NV for HP PC and Shunra NV for desktop. Shunra NV for HP PC allowed performance testing to become realistic end to end. Using in combination with the Shunra Wildcat appliance, we were able to create multiple network streams, replicating both the end users and split between datacenters. Using Shunra NV Desktop, gave customer ability to recreate the production network characteristics on their testers local machines from measurements pulled from Shunra NetworkCatcher. Benefits ROI quantified to be >$10 million in three years, in avoiding performance incidents and costly production based remediation. Also accelerating the time to complete the project will save on Data Center management and lease costs. With HP and Shunra we were able to mitigate the risk (estimated at $100 million) associated with our Data Center Move initiative. We are now working to adopt these best practices and modern tools into all of our testing. General Manager, Testing, SONY Europe 50