The Tester's Role in Continuous Integration

Transcription

1 W8 Track 10/3/2012 The Tester's Role in Continuous Integration Presented by: Roi Carmel HP Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL

2 Roi Carmel HP Software With twelve years of experience in software development, engineering management, and product management, Roi Carmel leads product management for ALM at Hewlett-Packard s Application Transformation business unit. Roi is responsible for the company s mobile ALM agenda (including mobile testing), testing products such as Unified Functional Testing, Business Process Testing and Sprinter, and the offerings for developers such as Application Lifecycle Intelligence (ALI), Systinet, and others. With almost his full career spent in quality and ALM products, Roi is now working with the HP team to plan the next generation of ALM tools for the modern challenges of the market.

3 Velocity and quality: continuous testing for an agile world Speaker: Date: A unified view of modern delivery Cloud ELASTICITY Mobile ACCESSABILITY Agile VELOCITY RESPONSIVENESS BETTER, FASTER OUTCOMES MODULARITY DURABILITY Composite REUSABILITY RIA USABILITY 2 DELIVERY CONSUMPTIO N 1

4 A unified view of modern delivery Cloud ELASTICITY Mobile ACCESSABILITY Agile VELOCITY RESPONSIVENESS BETTER, FASTER OUTCOMES MODULARITY DURABILITY Composite REUSABILITY RIA USABILITY 3 DELIVERY CONSUMPTIO N Traditional waterfall delivery Defects found too late in the cycle for effective resolution R i s k Jan Feb Mar Apr May Jun Code / unit test System test Time 4 2

5 Agile: as it should be Jan Feb Mar Apr May Jun RQMT DESGN CODE / UT TEST Time-boxed for focus Hands-on with stakeholders Surfaces issues sooner Continuous testing ed for change True measure of progress 5 Agile: as it too often is Jan Feb Mar Apr May Jun RQMT DESGN CODE / UT TEST Time-boxed for focus Hands-on with stakeholders Surfaces issues sooner Continuous testing ed for change True measure of progress 6 3

6 Scrummerfall. n. The practice of combining scrum and waterfall so as to ensure failure at a faster rate than with waterfall alone. Instability & pre-automation Exploratory test each new element as it s introduced C Component (non-gui) G GUI element B Business process Stable Sprint 1 Sprint 2 1 Sprint 3 Sprint 4 Sprint 5 8 4

7 Why explore? The best way to mine sweep Mine Fix Repeatability alone won t clear the field Repeatability and variability are key 9 Any tests that can be scripted should be automated. We don t need humans doing something that a machine can do. What we do want the human testers doing is exploratory testing. Exploratory testing is a creative endeavor in which human testers explore the behavior of the system in an attempt to characterize Dr. Robert C. ( Uncle its behaviors, Bob ) Martin both documented and undocumented. 5

8 The s+1 rule of test automation Automation never lags by more than one sprint C Component (non-gui) G GUI element B Business process Stable Sprint 1 Sprint 2 1 Sprint 3 Sprint 4 Sprint 5 11 Solving for multiple app layers Multi-layered testing allows automation even without GUI C Component (non-gui) G GUI element B Business process Stable Sprint 1 Sprint 2 1 Sprint 3 Sprint 4 Sprint

9 Solving for multiple app layers Service virtualization eliminates dependencies C Component (non-gui) G GUI element B Business process Stable Sprint 1 Sprint 2 1 Sprint 3 Sprint 4 Sprint 5 13 External components The ideal picture: continuous testing A relentless assault on project risk Sprint 1 Sprint 2 1 Sprint 3 Sprint 4 Sprint 5 14 ET ET ET ET ET CI CI Regression 1 CI Regression 2 CI Regression 3 CI Regression 4 Longevity testing (independent of sprint cycles) 7

10 Terrific. Now how do you do it? How do you do it? Exploratory testing Aim Reality Solution 16 Advance understanding of app behaviour Test in realtime Haphazardness and wasted cycles Inability to replicate defects Lack of GUI thwarts testers HP Sprinter & Service Test Automatic logging creates a retroactive test plan Built-in screen capture & annotation for defect replication Spot Testing to interrogate service components 8

11 How do you do it? Continuous integration Aim Reality Solution 17 Smoke test builds hourly / nightly Reduce app volatility, maximize test cycles Test lab setup, infrastructure provisioning, app configuration, test scheduling is laborious & complex Limited nonfunctional verification HP Lab Mgmt. Automation Test results Code check-in Run test cases HP Fortify Execute build Provision environment Deploy Test app How do you do it? Regression testing Aim Demonstrate binary progress Attack risks, course correct early Reality Arguments over what done means Component instability & external systems make system testing impossible in sprinttime Solution HP Unified Functional Testing, HP Service Virtualization, HP Agile Accelerator Real-time Agile dashboard based on automated test results Virtualization instead of stub creation 18 9

12 How do you do it? Longevity testing Aim Reality Solution 19 Evaluate app resilience Identify performance decay Regression against dependent apps System of systems : composite nature of apps confuses test effort & root cause Complex performance variables to modern UI (AJAX, etc) HP Business Process Testing, HP Performance solutions Orchestrate cross-layer, cross-app tests and correlate results Verify performance across range of technologies, speed bottleneck diagnosis Tying it all together HP ALM 20 10

13 How to avoid the scrummerfall trap Remember Exploratory testing is perfectly suited to the evolving, unstable components in an Agile project 2. Time must be explicitly allocated for test automation in each sprint. 3. Automation should begin as early as possible and should never lag by more than one sprint. 4. When assessing the results of a sprint, the automation goals must be considered as vital as the development objectives. 5. Multi-layered testing allows for test automation in advance of GUI stability, and furthers test coverage. 6. Legacy tools won t deliver modern velocity. Solutions for acceleration, virtualization and advanced automation are crucial. 21 Thank you 11