Andreas Grabner. Your place in DevTOps is not about finding more bugs, but problem solutions

Transcription

1 Andreas Grabner Your place in DevTOps is not about finding more bugs, but problem solutions

2 That s why I ended up talking at BTD

3

4

5

6 The stuff we did when we were a Start Up and we All were Devs, Testers and Ops

7 Reduce Lead Time Coding Unit Tes2ng Acceptance Tes2ng Performance Tes2ng Release Improve Process

8 700 Deployments / Year 10+ Deployments / Day Deployments / Day Every 11.6 seconds

9 Inside the Amazon Numbers! Deploying every 11.6s 75% fewer outages since % fewer outage minutes ~0.001% of deployments cause a problem Instantaneous automa2c rollback

10 Deploy Faster!!

11 Fail Faster!!?

12 Failures happen!! Nobody likes it when

13 Unless you work for Google or Microsoft J

14 or this

15

16 .. as it leads to this

17 The War Room Facebook December 2012

18 and potentially to this

19

20 And this isn t helping either

21

22

23

24

25 Tester Developer

26

27

28 This ATTITUDE is the main PROBLEM as it LEADS to

29 80% $60B

30 Insufficient Focus on Quality 150x 100% 80% 50% 50x 1x 10x 25x 40% 40% Coding Unit Tes2ng Acceptance Tes2ng Performance Tes2ng Release % of Bugs in SoTware Rela2ve Cost of a Bugfix Barry Boehm: EQUITY Keynote Address, March 19 th, 2007

31

32 80% 20%

33 Focus on Well Known problems! Automated! 150x 100% 80% 80% 1x 40% 50% 10x 20% 25x 50x 40% 40% 10% 5% Coding Unit Tes2ng Acceptance Tes2ng Performance Tes2ng Release % of Bugs in SoTware Rela2ve Cost of a Bugfix Barry Boehm s numbers adjusted by Andreas Grabner

34

35 4 Pillars of DevOps + Quality Focus Collabora2on Automa2on Measuring Sharing

36 Level- Up Skills Browser Diagnos2cs JBoss, Tomcat, JConsole IIS, ASP.NET, PerfLib Wireshark, Fiddler Oracle, SQL Garbage Collec2on Thread Dumps

37 Architecture # of Calls between Components # of Services # of Servers # of Databases

38 Performance Page Load Time Render Time SQL Query Time Service Call Time

39 Scalability Component Roundtrips Load Distribu2on # Connec2ons Memory Usage per User Cache U2liza2on

40 4 Pillars of DevOps + Quality Focus Collabora2on Automa2on Measuring Sharing Level- Up Skills Architecture Performance Scalability

41 4 Use Cases: HOW WHY, & METRICS

42

43 Bloated Websites

44 Mobile Web Site: This SHOULDN T happen! Total size of ~ 20MB 434 Resources in total on that page: 230 JPEGs, 75 PNGs, 50 GIFs,

45 Not following Web Performance Best Practices 9.68MB Page Size 282! Objects on that page Most objects are images delivered from your main domain 8.8s Page Load Time Very long Connect 2me (1.8s) to your CDN

46 Lessons Learned NO Excuse for Developers not using the browser built-in diagnostics tools Testers not doing a sanity checks with the same tools Some tools for you (Quick Demo) Built-in Inspectors via Ctrl-Shift-I in Chrome and Firefox YSlow, PageSpeed, SpeedTracer Dynatrace Level-Up: Automate Testing and Diagnostics Check

47 # Resources # of Domains Usage of CDNs Page Load and Size

48

49 Not every Architect makes good decisions

50 Project: Online Room Reservation System Symptoms HTML takes between 60 and 120s to render High GC Time Developer Assumptions Bad GC Tuning Probably bad Database Performance as rendering was simple Result: 2 Years of Finger pointing between Dev and DBA

51 Developers built own monitoring void roomreservationreport(int officeid) { long starttime = System.currentTimeMillis(); Object data = loaddataforoffice(officeid); long dataloadtime = System.currentTimeMillis() - starttime; } generatereport(data, officeid); Result: Avg. Data Load Time: 45s! DB Tool says: Avg. SQL Query: <1ms!

52 #1: Loading too much data 24889! Calls to the Database API! High CPU and High Memory Usage to keep all data in Memory

53 #2: On individual connections 12444! individual connec2ons Individual SQL really <1ms Classical N+1 Query Problem

54 #3: Putting all data in temp Hashtable Lots of 2me spent in Hashtable.get Called from their En2ty Objects

55 Lessons Learned Don t Assume you know what code is doing Challenge the developers Explore Tools that might seem out of your league! Built-In Database Analysis Tools Logging options of Frameworks such as Hibernate, JMX, Perf Counters, of your Application Servers Performance Tracing Tools: Dynatrace, NewRelic, AppDynamics,

56 # SQL ExecuZons # of SAME SQLs Conn. AcquisiZon Time

57

58 Deployment Gone Bad!

59 Test Environment Produc2on Environment 8x slower 3x more SQL

60 Test Environment Produc2on Environment That s Normal: Having I/O for Web Request as main contributor Hibernate, Classloading, Hibernate, XML Classloading, The Key Hotspots XML I/O for The Web Key Requests Hotspots doesn t even show up!

61 Top Contributor Class.getInterfaces Called from Hibernates FieldIntercep2onHelper These calls all originate form thousands of calls to find item by code

62 Top Methods related to XML Processing Classloading is triggered through CustomMonnkey CustomMonkey and the Xalan Parser

63 Lessons Learned Plan enough time for proper testing Anticipate changed user behavior during peak load Only test what really ends up in Production

64 Time Spent in API # Calls to API

65

66 Incorrect Sizing of Pools and Queues

67 1.69m (=101s!) To Check Balance! Online Banking: Slow Balance Check 87% spent in IIS 600! SQL Execu2ons

68 #1 Time really spent in IIS? Tip: Elapsed Time tells us WHEN a Method was executed! Tip: Thread# gives us insight on Thread Queues / Switches Finding: Thread 32 in IIS waited 87s to pass control to Thread 30 in ASP.NET

69 #2 What about these SQL Executions? Finding: EVERY SQL statement is executed on ITS OWN Connec2on! Tip: Look at GetConnec2on

70 #2 SQL Executions! continued #1: Same SQL is executed 67! 2mes #2: NO PREPARATION because everything executed on new Connec2on

71 Lessons Learned! ASP.NET Worker Thread Pool Sizing! DB Connec]on Pools More Efficient SQL

72 How to Monitor: Web Server Idle vs. Busy Threads

73 How to Monitor: Application Server App Server Threads

74 Idle vs. Busy Threads # SQLs / Request # GetConnecZon

75 More of these Use Cases?

76

77 Remember: New Metrics When Testing Apps # Images # Redirects Size of Resources # SQL Executions # of SAME SQLs # Items per Page # AJAX per Page Time Spent in API # Calls into API # Functional Errors 3rd Party calls # of Domains Total Size

78

79 Build 17 testpurchase OK testsearch Build 20 testpurchase OK Putting it into Test Automation Test Framework Results Build # Test Case Status # SQL # Excep CPU OK Build 18 testpurchase FAILED testsearch OK Build 19 testpurchase OK testsearch OK testsearch We identified a regresesion OK Problem solved Now we have the functional and architectural confidence Let s look behind the scenes Architectural Data ms ms ms ms ms ms ms Exceptions probably reason for failed tests Problem fixed but now we have an architectural regression ms

80 #1: Analyzing each Test #3: Detec2ng Regression based on Measure #2: Metrics for each Test

81 #1: Test Status Overview based on our new Metrics #2: Lets the build fail

82 Every 11.6 seconds Unit Tes2ng Acceptance Tes2ng Performance Tes2ng Release Monitor Tests Analyze Results Quality Gate in your Build Tool

83 Deploy Faster!! With Be_er Quality

84 Slides: slideshare.net/grabnerandi Get Tools: bit.ly/dttrial YouTube Tutorials: bit.ly/dttutorials Contact Me: Follow Read More: blog.dynatrace.com Andreas Grabner Your place in DevTOps is not about finding more bugs, but problem solutions