The Business Case for Performance Testing. By Joseph Ours

Transcription

1 The Business Case for Performance Testing By Joseph Ours

2 Introducing Joseph Ours About Joseph Owns development and delivery of Cohesion's Quality Assurance and Testing Services More than 15 years of IT experience spanning several industries and roles Contacting Notable success managing multi-million dollar initiatives while introducing innovative solutions for Fortune 1000 companies, resulting in improved quality, increased client satisfaction, cost avoidance, on-time delivery, and establishing a true team culture

3 Agenda Why poor performance is worse than a full blown production outage How better performance drives actual business revenue How performance affects operating expenses Typical stakeholder expectations of performance

4 Performance Issues to Production Outage The slippery slope

5 2007, 2009, 2011 iphone Activations take 3 to 36 hours to complete Complaints trended as top topic on Twitter, made CNN and other new outlet top stories AT&T reputation cemented as one that cannot handle iphone Servers overwhelmed due to concurrent activation requests and length of activation process *iphone-4s Buyers Complain of AT&T Activation Issues, CNET

6 days of network outages globally $50MM in lost sales due to outage Free Tech Support and Free Apps offered to customers Network overwhelmed with processing backlog of data after momentary switch failure *RIM Global Outage Caused By Core Switch Failure, Fix Underway, ComputerWorld

7 Amazon Cloud Service North America - East Region unavailable for over 11 hours Sites like Reddit, Foursquare, and NYTimes affected Customers offered 10 days of free credit Traffic incorrectly switched to backup network. Data stores overwhelmed network with read/write operations *Amazon Outage and the auto-immune vulnerability's of resiliency, Gartner, Lydia Leong

8 JP Morgan k transactions held up worth $132MM Customers charged late fees that had to be reversed 60 hours to resolve Resulted from code update *Chase Website Outage Aftermath, Brian O Connell

9 2010, 2011(twice) 90%+ of users received page time-outs on home page Users could not access online accounts All attributed to routine code changes *Bank of America Site Suffers, Washington Post, 2011

10 Why are Performance Issues Worse Than Production Outages? Production Outages Immediately Known Outages tend to have singular root cause Customers know when its down Usually fixed within hours Break-Fix is appropriate, it breaks, it gets fixed Performance Outages Not always immediately known Often gives a false sense of being available Multiple causes due to interdependent impacts Often takes days or weeks to resolve Not broken, but not working

11 Impact of making good sites, better

12 Improve Page Load By 100 ms Increased Revenue by 1% *Source: Amazon

13 9% Come see presentation Traffic increased when 400ms Come see presentation Page time decreased *Source: Yahoo!

14 Improve Page Load By 100ms-400ms Come see presentation Increased Traffic by.2-.6% Come see presentation *Source: Google

15 Average Impact of Performance Improvement Of the 178+ companies evaluated, those who improved application performance saw Revenue Increase 9% Come see presentation *Application Performance Management: Getting IT on the C-Level's Agenda, 2009, Aberdeen

16 Revenue Impact of Poor Performance 1 second delay 11 % fewer views The bottom line 16 % drop in satisfaction A site generating $100k/day That incurs a 1 Come second see delay Will lose $2.5MM/year presentation *Application Performance Management: Getting IT on the C-Level's Agenda, 2009, Aberdeen

17 Impact of making good sites, better

18 Slow Sites Generate Calls 56% of online bank customers would move to offline channels to ask a general account question Come see presentation Slow is a page taking longer than 4 seconds to load 54% of online bank customers would move to offline channels to complete transactions *Application Performance Management: Getting IT on the C-Level's Agenda, 2009, Aberdeen

19 The numbers behind the things we know

20 Internal and External Customers 58% Respondents said poor application performance lowered employee satisfaction 47% indicated that they had decreased responsiveness to the needs of external customers 57% of online consumers will abandon a site after waiting 3 seconds for a page to 39% customers rank speed as more load. important than functionality Come see presentation 8 out of 10 people will not return to a site after a disappointing experience 60% external users are not satisfied with the performance of business critical applications 32% of companies experienced damage to their brand reputation as a result of poor performing applications Primary Source: Application Performance Management: Getting IT on the C-Level's Agenda, 2009, Aberdeen Alternate Sources: Multiple

21 Executive Customers The C Level 60% report inability to identify performance issues before end users are impacted Organizations are planning to increase the number of business critical applications by 67% over the next 12 months Come see presentation 41% of CIO s have issues with rolling out new applications without them being performance tested 31% believe virtualization would further challenge application performance management Primary Source: Application Performance Management: Getting IT on the C-Level's Agenda, 2009, Aberdeen Alternate Sources: Multiple

22 When you don t know what your users want - find out what the public wants

23 User EXPECTATIONS External Web Page Load Times: <2 seconds Abandonment: 33% abandon between 1-5 seconds, 56% after 4 seconds 100 percent abandon sites near 1 minute Internal performance response times: 5 seconds Web site performance is second only to security in user expectations Come see presentation 39% users say speed is more important than functionality Mobile users expect better performance than full sites 70% of mobile sites are slower than full sites Baby Boomers are more impatient than Gen-Y Primary Source: Application Performance Management: Getting IT on the C-Level's Agenda, 2009, Aberdeen Alternate Sources: Multiple

24 User Perceptions of Performance Recalled (1 week later) Perceived Load Time Immediate Perceived Load Time 35% slower than actual Come see 15% presentation slower than actual Actual Load Time User s perception of performance degrades with time *Source: Psychology of Performance, Stoyan Stefanov

25 Joseph Ours, MBA,