Evidence based performance tuning of

Transcription

1 Evidence based performance tuning of enterprise Java applications By Jeroen Borgers

2 Evidence based performance tuning of enterprise Java applications By Jeroen Borgers

3 Bad performance can lead to complete failure 3

4 Bad performance: hard to fix Call center agent took 20 minutes to work through five screens with customer Customer un-friendly High cost Speedup needed before rush of number porting Major app redesign was needed Result very slow system when up, mostly system down damage to company image loss of thousands of customers and $100,000,000 company crash 4

5 Goal Provide an answer to the question: How can I speedup my enterprise Java app? 5

6 Who is Jeroen Borgers? sr. consultant with Xebia helps customers on enterprise Java performance issues instructs Java performance tuning courses doing Java since 1996 various roles in several industries developer, architect, team lead, quality officer, auditor, performance tester and tuner

7 Who is Xebia? international IT organization NL, FR, IN specialized in enterprise Java consultancy and projects 5 years old ±100 employees and growing share knowledge focus on quality

8 Agenda Big picture architecture vs. implementation How to achieve bad performance Evidence based tuning for high speed Tools overview Largest Dutch web shop case Conclusions 8

9 Enterprise Java performance Java is not slow anymore big picture Importance how you do it: algorithms Typical enterprise app: most time spent in calls out of Java process Remote calls Database access for (int i = 0; i < size; i++) { order = orderdao.getorder(i); if (order.isdelivered()) { DB ~0,1µs >~10 ms 9 factor !

10 Architecture enables performance and scalability Key choices Application distribution Persistent data access approach: O-R mapping Data conversion: XML, encodings Presentation tier technologies Use of proprietary features of db, app server Data caching It is all about: Remote calls Database access More general: data conversion between representations 10

11 Architecture enables, implementation makes it happen Minimize remote calls and other I/O Speed-up data conversion Code optimizations involving remote calls and data conversion 11?

12 Architecture enables, implementation makes it happen Minimize remote calls and other I/O Direct JDBC: use of batch updates Hibernate: use of query caching Speed-up data conversion XML usage: use JiBX instead of JAXB URL encoding: use commons codec instead of java.net Code optimizations involving remote calls and data conversion Algorithms, collections Threading and synchronization 12

13 Architecture enables performance The architecture & scalability 13

14 Architecture enables performance & scalability The architecture is scalable 14?

15 Architecture enables performance & scalability The architecture is scalable Add horse power: more load with same speed 15

16 Architecture enables performance & scalability How one implements the architecture makes a difference But the requirement of 120 km/h will never be met! Major redesign needed 16

17 Architecture enables performance & scalability A more suitable architecture for 120 km/h: 17

18 Architecture enables performance & scalability More demanding requirements can be met with the right implementation Bugatti EB Veyron 16.4 Top 407 km/h in 14 s. 18

19 How to achieve bad performance Many JavaEE apps have hard to solve performance problems How to achieve this? Architecture not suitable for requirements Ignore the problems too long Don t follow best practices Optimize counterproductively 19

20 Architecture not suitable Architect: Most important is a pure, flexible, scalable architecture: we distribute our application components and loosely couple with XML and web services. Nonsense! Those who pay the bill care about efficient working, not purity; Speed is traded for scalability: remote calls and data conversion can be killing; How is flexibility gained? 20

21 Ignore problems too long Project leader: First we get the functionality right, then we start worrying about performance. Performance is often ignored until the final test, a week before production, or later! To meet requirements you may need a major redesign: costly Solved by: Verify the architecture in a POC: test a vertical slice Test performance continuously 21

22 How to achieve bad performance How to achieve this? Architecture not suitable for requirements Ignore the problems too long Don t follow best practices Optimize counterproductively 22

23 Don t follow best practices Best practices are like: Use Map instead of List for lookups Use I/O buffering: InputStreamBuffer Use SQL batch updates Use db table indexes Use StringBuffer for String concatenation? 23?

24 Don t follow best practices Best practices are like: Use Map instead of List for lookups Use I/O buffering: InputStreamBuffer Use SQL batch updates Use db table indexes Use StringBuffer for String concatenation? Be aware: a best practice may not work in your situation 24

25 Best practices can be premature optimizations with trade-offs Developer: String performs badly, we use StringBuffer for concatenation everywhere. Many books: use StringBuffer for tostring() { return [ + getname() + ] ; } faster is tostring() { StringBuffer buf = new StringBuffer( [ ); buf.append(getname()).append( ] ); return buf.tostring(); } However: same bytecode! Maintainability and developer time is traded for an assumed, nonexisting performance gain Premature optimization is the root of all evil - Hoare and Knuth 25

26 Optimize counterproductively Updating state remotely by sending only delta s like RemoveAttribute, AddAttribute many small objects with much serialization and communication overhead Object and thread pooling depending on JVM Optimizations are very time and environment dependent So: don t trust your knowledge or intuition, but measure that your optimization solves the problem! 26

27 Bad performance Architecture not suitable for requirements Ignore the problems too long Don t follow best practices Optimize counterproductively How to speedup? 27

28 Evidence based tuning for high speed Don t make assumptions but measure Have quantified requirements, prove they are met or not Prove the architecture performs in a POC Continuously test performance during development to get a first impression for quick feedback on changes Test representatively in a dedicated environment with production-like load on a production-like environment with production-like data Fix problems when found Prove the effect of each optimization Monitor the performance of your app in production 28

29 Evidence based tuning cycle

30 Tools for measuring OS profilers to see resource usage: CPU, memory, I/O, threading Database analyzers to see query behavior in the database Network analyzers to see network traffic App server monitors to see app server resource usage: heap, connection pool, stmt cache,.. J2EE analyzers to see J2EE behavior under load Instrumentation monitors to see timing at the instrumented locations Java profilers to see detailed CPU and memory behavior in JVM Continuous test tools to quickly get timing feedback of code changes Load test tools to generate usage load Web site analyses tools to optimize page resource usage 30

31 Largest Dutch web shop case

32 Case: speeding up a web shop 32

33 Wehkamp TRC architecture DB web server: ASP/.Net Tuxedo TRC-appsrv: Java DB DB Services here 33

34 Speedup results at Wehkamp Several services optimized to meet requirements Most expensive: voegklanttoe (addcustomer); verwerkennieuweorder(processneworder): from 7 s. to 1.5 s. 45% db-queries, 45% tux-calls, 10% java code Top optimizations: Reduce number of db queries Optimize most expensive db queries Reduce number of Tux/mainframe calls Optimize most expensive Tux/mainframe calls App orchestrates: many Java code changes needed for this 34

35 Tools used Load test tool: Apache JMeter App level monitor: JAMon API History and trends reporter: JARep Continuous test tool: cruisecontrol+maven+jmeter Java profiler: Quest JProbe App server monitor: Tivoli Performance Monitor J2EE analyzer: Quest PerformaSure Database analyzer: Quest Toad, Oracle Statspack Unix OS profiler: vmstat, nmon Network analyzer: Wireshark 35?

36 JMeter for generating load and black box testing 36

37 JMeter mail to stakeholders 37

38 JAMon for monitoring statistics added service name 38

39 Instrumenting with JAMon API JAMon API call 39

40 40 JARep reporting for history

41 41

42 42

43 43

44 deployment of JARep your app ShowJamon Counters.jsp jamon.war jamonadmin.jsp JVM / app server node 4 JVM / app server node 3 jamonapi.jar JVM / app server node 2 JVM / app server node 1 every 5 minutes JaReporter.jsp DataFetcher DataPersister JVM / off-line box RDBMS

45 PerformaSure for analyses under load 45

46 Analysis tools compared Profiler JProbe Pro: High detail: code line level Con: High overhead, cannot deal with load JAMon API + JARep Pro: Low overhead < 1% Pro: Enables comparing over time, see trends Con: No call tree, cannot drill down Con: Limited to built-in measure points PerformaSure Pro: reasonable overhead, slick UI, measures all on JVM Con: not cheap, configuration, wait time 46

47 Conclusions & recommendations Architecture and implementation both determine performance Consider whole application life cycle: requirements, POC, continuous testing, representative testing, monitoring Biggest gains are in Remote calls and other I/O Database access or other data conversions Tuning based on evidence is key measure and prove by using the right tools JMeter, JAMon and JARep can help 47

48 48 Meten is weten (measuring means knowing)

49 References

50 50 Q & A