Your logo here Realizing Continuous Performance Management Steven Haines Quest Software Java Domain Expert February 14, 2008
Agenda State of the Market Cost of Mistakes Test-Driven Development Continuous Integration Continuous Performance Management CPM in Practice Business Value Next Steps Q&A
Industry Experts Say Forrester Research reports nearly 85 percent of companies with revenue of more than $1 billion reported incidents of significant application performance problems. Survey respondents identified the architecture and deployment as the primary causes of these problems. -Forrester Research Medium-sized businesses* are losing an average of 1% of their annual revenue, or $867K, to downtime. Application outages and degradations are the biggest sources of downtime, costing these companies $213K annually. -Infonetics Research *101 to 1000 employees
Impact of Poorly Performing Applications Business-to-consumer Direct loss of customer revenue Loss of confidence Business-to-business Loss of reputation Potential loss of business relationships Intranet Loss of productivity Longer time-to-market = loss of competitive advantage
Cost of Mistakes Performance problems found later in production cost almost 10x more to fix Test Early, Test Often
Why is the Current State of Testing Failing? Test suites are written to out-of-date artifacts Test suites are not comprehensive enough Test suites are not automated Not performed regularly No regression testing
Test-Driven Development Test cases must be written prior to writing code Test cases written by developers Each code component must include a test suite
Test-Driven Development Test-Driven Development Code Test Code Test Code Test Continuous Integration Test Suite Source code Repository Continuous Performance Management Picture to show profiling of a test: JProbe or PAS type of screen Profile Analyze React
Test-Driven Development Lifecycle Add a Test Test Suite
Test-Driven Development Lifecycle Prove Failure Add a Test Test Suite
Test-Driven Development Lifecycle Prove Failure Add a Test Test Suite Implement New Functionality
Test-Driven Development Lifecycle Prove Failure Add a Test Test Suite Implement New Functionality Prove Success
Test-Driven Development Lifecycle Business Value: Prove Failure Add a Test Re-factor the Code Implement New Functionality Prove Success Limited Debugging Faster Triaging Test Suite Shorter Development Cycles Clearly Solved Business Problems Confidence in Deployment
Continuous Integration The Challenge The Solution With complicated applications, integration can take longer than development Integrate after each code check-in The Tool Continuous Integration Server
Test-Driven Development + Continuous Integration Test-Driven Development Code Test Code Test Code Test Continuous Integration Build CI Server Test Suite Source code Repository Test Publish Continuous Performance Management Picture to show profiling of a test: JProbe or PAS type of screen Profile Analyze React
Continuous Integration Lifecycle Monitors SourceCode Repository Continuous Integration Server
Continuous Integration Lifecycle Monitors SourceCode Repository Detects New Check-Ins Continuous Integration Server
Continuous Integration Lifecycle Monitors SourceCode Repository Detects New Check-Ins Continuous Integration Server Compiles Entire Application
Continuous Integration Lifecycle Monitors SourceCode Repository Detects New Check-Ins Continuous Integration Server Runs Exhaustive Test Suite Commit Build Secondary Builds Compiles Entire Application
Continuous Integration Lifecycle Business Value: Monitors SourceCode Repository Detects New Check-Ins Publishes Build Results Continuous Integration Server Compiles Entire Application Runs Exhaustive Test Suite Commit Build Secondary Builds Shorter integration times Integration issues resolved as they occur Higher quality end product
What is Continuous Performance Management? Challenge in Performance Testing Time consuming and error prone because manual Problems discovered at the end of development effort often affect assumptions made at the beginning of a development effort Requires re-factoring the architecture Solution Performance Management in a Continuous Integration Environment Proactive approach Execute performance tests on every developer commit On every code check-in
Continuous Performance Management (CPM) Create additional Secondary Builds that implement performance tests Secondary Build 1 1. Performance Unit Tests 2. Performance Integration Tests Business case Use case scenario 3. Performance Stress/Load Tests CPM Secondary Build 2 Secondary Build 3
Test-Driven Development + Continuous Integration + CPM Test-Driven Development Code Test Code Test Code Test Continuous Integration Build CI Server Test Suite Source code Repository Test Publish Continuous Performance Management Profile Analyze Repair
Performance Unit Tests Memory Lingering Object References (memory leaks) Object cycling Performance Identify slow running code Coverage Quantify the percentage of classes, methods, lines of code and conditions exercised by unit tests
Performance Integration Tests Performance analysis of a single user Identify request response times that exceed service-level agreements Triage to offending methods and SQL statements Follow requests across tiers (and across JVMs) Identify application problems prior to load Find problems in isolation Easier to pinpoint root cause
Performance Stress/Load Tests Performance analysis under load (multiple users) Identify request response times that exceed service-level agreements Triage to offending methods and SQL statements Follow requests across tiers (and across JVMs) Validate environmental configuration Identify resource contentions Identify scalability problems prior to deployment Identify environmental configuration issues Resolve scalability and reliability problems
Continuous Performance Management Toolset Continuous Integration Server CruiseControl Build Environment Apache Ant Test Bed JUnit Load Generator Apache JMeter Performance Testing: Unit tests: Quest JProbe Integration/Load tests: Quest PerformaSure
CPM Too Arduous a Task? It is a common misconception that CPM is too risky and time consuming of an undertaking We ve gone through painstaking efforts to make CPM as easy as possible to implement Only requires a handful of build script additions
CPM In Practice Performance Unit Tests Additions to Ant build script to profile all your JUnit tests Easy to do can paste on a single slide <!-- Set location of the JProbe Enterprise Tools --> <property name="jprobe.enterprisetools.home location="c:\jprobe-ent" /> <!-- Define the Pre-processor CLASSPATH --> <path id="classpath.preprocessor"> <fileset dir="${jprobe.enterprisetools.home}" includes="lib/*.jar"/> </path> <!-- Execute the preprocessor --> <java classname="com.javasrc.anttools.jprobepreprocessor"> <sysproperty key="src.dirs" value="${src};${src.test}" /> <sysproperty key="jprobe.home" value="c:\program Files\JProbe 7.0" /> <sysproperty key="jprobe.build.dest" value="${jprobe.enterprisetools.home}\jprobe.xml" /> <classpath refid="classpath.preprocessor"/> </java> <!-- Import the Generated Script --> <import file="${jprobe.enterprisetools.home}\jprobe.xml" />
CPM In Practice Unit Test Report
CPM In Practice Unit Test Report
Performance Unit Testing Quest JProbe Three analysis enablers: Memory Performance Coverage
CPM In Practice Integration / Load Tests Additions to Ant build script to launch JMeter Easy to do can paste on a single slide <!-- Import the JMeter task --> <taskdef name="jmeter classname="org.programmerplanet.ant.taskdefs.jmeter.jmetertask"/> <!-- Execute the test script --> <jmeter jmeterhome="c:\lib\jakarta-jmeter-2.2" testplan="${basedir}/mytests/mytest.jmx" resultlog="${basedir}/mytests/mytestresults.jtl"> <property name="jmeter.save.saveservice.output_format" value="xml" /> </jmeter> <!-- Generate an HTML Report --> <xslt in="${basedir}/mytests/mytestresults.jtl" out="${basedir}/mytests/mytestresults.html" style="${basedir}/mytests/jmeter-results-report.xsl" />
CPM In Practice Integration / Load Tests Additions to Ant build script to record a performance session <property name="pas.home" location="j:\performasure5.0" /> <target name="pas" description= "Start a PerformaSure recording" > <!-- Execute nexusctl --> <exec executable="cmd"> <arg value="/c" /> <arg value="${pas.home}/scripts/nexusctl.cmd" /> <arg value="start-recording" /> <arg value="-user" /> <arg value="user" /> <arg value="-l" /> <arg value="30m" /> <arg value="-t" /> <arg value="10s" /> <arg value="-f" /> </exec> </target>
Performance Integration/Load Testing Quest PerformaSure Quest Management Server Tag-and-Follow
Performance Integration/Load Testing Quest PerformaSure
Business Value Benefits Reduce R&D and QA time in performance testing Better performing applications Results Faster time-to-market = more time to add features that give you a competitive advantage over your competition Higher quality applications = higher customer satisfaction
Next Steps Download one of my whitepapers from www.quest.com under Application Management, solutions for Java: Extending the Continuous Integration Environment with Performance Testing (Part 1) Achieving Continuous Performance Management (Part II) Best Practices in Java Environment Performance Testing A Formal Performance Tuning Methodology: Wait-Based Tuning Sign up for my upcoming Java Performance Winner s Circle webcast series on performance testing, tuning, and management at www.quest.com under Application Management, solutions for Java (starts in March 2008) Watch my Continuous Integration OnSoftware Video Podcast on www.informit.com (or itunes) Read my forthcoming digital shortcut book, Agile Java Development with Test-Driven Development and Continuous Integration, on www.informit.com Read my latest book, Pro Java EE 5 Performance Management and Optimization, at your favorite book store
Next Steps cont d Quest Software Product Information: JProbe www.quest.com/jprobe PerformaSure www.quest.com/performasure Other Resources: JProbe Live Product Demo www.quest.com/events PerformaSure Product Demand www.quest.com/performasure Contact Quest Software Sales: www.quest.com/company/contact-us.aspx
Agenda State of the Market Test-Driven Development Continuous Integration Continuous Performance Management CPM in Practice Business Value Next Steps Thank You -- Q&A