Realizing Continuous Performance Management

Transcription

1 Your logo here Realizing Continuous Performance Management Steven Haines Quest Software Java Domain Expert February 14, 2008

2 Agenda State of the Market Cost of Mistakes Test-Driven Development Continuous Integration Continuous Performance Management CPM in Practice Business Value Next Steps Q&A

3 Industry Experts Say Forrester Research reports nearly 85 percent of companies with revenue of more than $1 billion reported incidents of significant application performance problems. Survey respondents identified the architecture and deployment as the primary causes of these problems. -Forrester Research Medium-sized businesses* are losing an average of 1% of their annual revenue, or $867K, to downtime. Application outages and degradations are the biggest sources of downtime, costing these companies $213K annually. -Infonetics Research *101 to 1000 employees

4 Impact of Poorly Performing Applications Business-to-consumer Direct loss of customer revenue Loss of confidence Business-to-business Loss of reputation Potential loss of business relationships Intranet Loss of productivity Longer time-to-market = loss of competitive advantage

5 Cost of Mistakes Performance problems found later in production cost almost 10x more to fix Test Early, Test Often

6 Why is the Current State of Testing Failing? Test suites are written to out-of-date artifacts Test suites are not comprehensive enough Test suites are not automated Not performed regularly No regression testing

7 Test-Driven Development Test cases must be written prior to writing code Test cases written by developers Each code component must include a test suite

8 Test-Driven Development Test-Driven Development Code Test Code Test Code Test Continuous Integration Test Suite Source code Repository Continuous Performance Management Picture to show profiling of a test: JProbe or PAS type of screen Profile Analyze React

9 Test-Driven Development Lifecycle Add a Test Test Suite

10 Test-Driven Development Lifecycle Prove Failure Add a Test Test Suite

11 Test-Driven Development Lifecycle Prove Failure Add a Test Test Suite Implement New Functionality

12 Test-Driven Development Lifecycle Prove Failure Add a Test Test Suite Implement New Functionality Prove Success

13 Test-Driven Development Lifecycle Business Value: Prove Failure Add a Test Re-factor the Code Implement New Functionality Prove Success Limited Debugging Faster Triaging Test Suite Shorter Development Cycles Clearly Solved Business Problems Confidence in Deployment

14 Continuous Integration The Challenge The Solution With complicated applications, integration can take longer than development Integrate after each code check-in The Tool Continuous Integration Server

15 Test-Driven Development + Continuous Integration Test-Driven Development Code Test Code Test Code Test Continuous Integration Build CI Server Test Suite Source code Repository Test Publish Continuous Performance Management Picture to show profiling of a test: JProbe or PAS type of screen Profile Analyze React

16 Continuous Integration Lifecycle Monitors SourceCode Repository Continuous Integration Server

17 Continuous Integration Lifecycle Monitors SourceCode Repository Detects New Check-Ins Continuous Integration Server

18 Continuous Integration Lifecycle Monitors SourceCode Repository Detects New Check-Ins Continuous Integration Server Compiles Entire Application

19 Continuous Integration Lifecycle Monitors SourceCode Repository Detects New Check-Ins Continuous Integration Server Runs Exhaustive Test Suite Commit Build Secondary Builds Compiles Entire Application

20 Continuous Integration Lifecycle Business Value: Monitors SourceCode Repository Detects New Check-Ins Publishes Build Results Continuous Integration Server Compiles Entire Application Runs Exhaustive Test Suite Commit Build Secondary Builds Shorter integration times Integration issues resolved as they occur Higher quality end product

21 What is Continuous Performance Management? Challenge in Performance Testing Time consuming and error prone because manual Problems discovered at the end of development effort often affect assumptions made at the beginning of a development effort Requires re-factoring the architecture Solution Performance Management in a Continuous Integration Environment Proactive approach Execute performance tests on every developer commit On every code check-in

22 Continuous Performance Management (CPM) Create additional Secondary Builds that implement performance tests Secondary Build 1 1. Performance Unit Tests 2. Performance Integration Tests Business case Use case scenario 3. Performance Stress/Load Tests CPM Secondary Build 2 Secondary Build 3

23 Test-Driven Development + Continuous Integration + CPM Test-Driven Development Code Test Code Test Code Test Continuous Integration Build CI Server Test Suite Source code Repository Test Publish Continuous Performance Management Profile Analyze Repair

24 Performance Unit Tests Memory Lingering Object References (memory leaks) Object cycling Performance Identify slow running code Coverage Quantify the percentage of classes, methods, lines of code and conditions exercised by unit tests

25 Performance Integration Tests Performance analysis of a single user Identify request response times that exceed service-level agreements Triage to offending methods and SQL statements Follow requests across tiers (and across JVMs) Identify application problems prior to load Find problems in isolation Easier to pinpoint root cause

26 Performance Stress/Load Tests Performance analysis under load (multiple users) Identify request response times that exceed service-level agreements Triage to offending methods and SQL statements Follow requests across tiers (and across JVMs) Validate environmental configuration Identify resource contentions Identify scalability problems prior to deployment Identify environmental configuration issues Resolve scalability and reliability problems

27 Continuous Performance Management Toolset Continuous Integration Server CruiseControl Build Environment Apache Ant Test Bed JUnit Load Generator Apache JMeter Performance Testing: Unit tests: Quest JProbe Integration/Load tests: Quest PerformaSure

28 CPM Too Arduous a Task? It is a common misconception that CPM is too risky and time consuming of an undertaking We ve gone through painstaking efforts to make CPM as easy as possible to implement Only requires a handful of build script additions

29 CPM In Practice Performance Unit Tests Additions to Ant build script to profile all your JUnit tests Easy to do can paste on a single slide <!-- Set location of the JProbe Enterprise Tools --> <property name="jprobe.enterprisetools.home location="c:\jprobe-ent" /> <!-- Define the Pre-processor CLASSPATH --> <path id="classpath.preprocessor"> <fileset dir="${jprobe.enterprisetools.home}" includes="lib/*.jar"/> </path> <!-- Execute the preprocessor --> <java classname="com.javasrc.anttools.jprobepreprocessor"> <sysproperty key="src.dirs" value="${src};${src.test}" /> <sysproperty key="jprobe.home" value="c:\program Files\JProbe 7.0" /> <sysproperty key="jprobe.build.dest" value="${jprobe.enterprisetools.home}\jprobe.xml" /> <classpath refid="classpath.preprocessor"/> </java> <!-- Import the Generated Script --> <import file="${jprobe.enterprisetools.home}\jprobe.xml" />

30 CPM In Practice Unit Test Report

31 CPM In Practice Unit Test Report

32 Performance Unit Testing Quest JProbe Three analysis enablers: Memory Performance Coverage

33 CPM In Practice Integration / Load Tests Additions to Ant build script to launch JMeter Easy to do can paste on a single slide <!-- Import the JMeter task --> <taskdef name="jmeter classname="org.programmerplanet.ant.taskdefs.jmeter.jmetertask"/> <!-- Execute the test script --> <jmeter jmeterhome="c:\lib\jakarta-jmeter-2.2" testplan="${basedir}/mytests/mytest.jmx" resultlog="${basedir}/mytests/mytestresults.jtl"> <property name="jmeter.save.saveservice.output_format" value="xml" /> </jmeter> <!-- Generate an HTML Report --> <xslt in="${basedir}/mytests/mytestresults.jtl" out="${basedir}/mytests/mytestresults.html" style="${basedir}/mytests/jmeter-results-report.xsl" />

34 CPM In Practice Integration / Load Tests Additions to Ant build script to record a performance session <property name="pas.home" location="j:\performasure5.0" /> <target name="pas" description= "Start a PerformaSure recording" > <!-- Execute nexusctl --> <exec executable="cmd"> <arg value="/c" /> <arg value="${pas.home}/scripts/nexusctl.cmd" /> <arg value="start-recording" /> <arg value="-user" /> <arg value="user" /> <arg value="-l" /> <arg value="30m" /> <arg value="-t" /> <arg value="10s" /> <arg value="-f" /> </exec> </target>

35 Performance Integration/Load Testing Quest PerformaSure Quest Management Server Tag-and-Follow

36 Performance Integration/Load Testing Quest PerformaSure

37 Business Value Benefits Reduce R&D and QA time in performance testing Better performing applications Results Faster time-to-market = more time to add features that give you a competitive advantage over your competition Higher quality applications = higher customer satisfaction

38 Next Steps Download one of my whitepapers from under Application Management, solutions for Java: Extending the Continuous Integration Environment with Performance Testing (Part 1) Achieving Continuous Performance Management (Part II) Best Practices in Java Environment Performance Testing A Formal Performance Tuning Methodology: Wait-Based Tuning Sign up for my upcoming Java Performance Winner s Circle webcast series on performance testing, tuning, and management at under Application Management, solutions for Java (starts in March 2008) Watch my Continuous Integration OnSoftware Video Podcast on (or itunes) Read my forthcoming digital shortcut book, Agile Java Development with Test-Driven Development and Continuous Integration, on Read my latest book, Pro Java EE 5 Performance Management and Optimization, at your favorite book store

39 Next Steps cont d Quest Software Product Information: JProbe PerformaSure Other Resources: JProbe Live Product Demo PerformaSure Product Demand Contact Quest Software Sales:

40 Agenda State of the Market Test-Driven Development Continuous Integration Continuous Performance Management CPM in Practice Business Value Next Steps Thank You -- Q&A