T15 INCREASE THE VALUE OF YOUR TESTING WITH BUSINESS- ORIENTED METRICS. Claude Fenner Arsin Corporation BIO PRESENTATION

Transcription

1 BIO PRESENTATION T15 Thursday, October 30, :00 PM INCREASE THE VALUE OF YOUR TESTING WITH BUSINESS- ORIENTED METRICS Claude Fenner Arsin Corporation International Conference On Software Testing Analysis & Review October 27-31, 2003 San Jose, CA USA

2 Claude Fenner Claude Fenner has over 20 years experience in the software industry as an engineering manager, quality analyst, and software designer. His experience includes enterprise application software, mission-critical system software, and commercial and military embedded software. Across these industries he has transformed organizations to deliver high-quality software at lower cost through automation and globalization Arsin Corporation is a software reliability services and product company, specializing in enterprise testing, quality, and readiness assessment. Arsin provides expertise and tools to several Fortune 100 companies. Arsin also partners with major tool vendors such as Mercury Interactive and has a wholly-owned subsidiary Arsin Limited in India. Claude Fenner is responsible for driving Arsin s tools and processes into market-ready product solutions. During his career, he has driven large-scale quality and testing programs for major qualityoriented companies including Compaq, Tandem Computers, and SAIC. At Tandem, he built and led the quality teams that delivered Tandem s high availability database and transaction processing technologies used for stock exchanges and ATM networks. At Compaq he defined and drove the Y2K process that successfully updated Compaq s more than 1000 NonStop branded products. Recent enterprise experience includes high volume B2B and B2C middleware for financial transactions. Beyond the enterprise arena, his experience includes developing processes and building tools for assuring the reliability of commercial nuclear safety systems and embedded military systems. He earned degrees in EE/CS from the University of California Berkeley.

3 ARSIN Corporation Powering Collaborative e-business Increase the Value of your Testing with Business-Oriented Metrics Claude Fenner StarWest 2003 October 30, 2003 ARSIN Corporation

4 A lot has changed in the last decade Then Brand Brand Now Your systems Your business are the reputation hands of is consumers the line and like partners never before like never before Arsin Company Confidential 2

5 Good News Business owners care about quality Arsin Company Confidential 3

6 Not everyone s perspective on quality is exactly the same Owner Producer Customer thinks: thinks: I want I ve customer been paid told for a satisfaction what service, customers so and give I want, want me what so to produce that s you promised what value I ll build for business stakeholders Business-value Quality Business Owner Customer Producer Service Quality Process Quality Arsin Company Confidential 4

7 Ok, you re visible to business owners Now what!?!? Bad News: Traditional metrics don t resonate with business managers Arsin Company Confidential 5

8 Let s take a closer look at a business owner Here are things they think about Are our customer being served? What is the best mix of people, capital, geographical locations? What things are assets? What is an investment vs. expense? They say things like Are these tests really assets? Are they worth it? Why? Can I squeeze more out of them? Can I get by with less? Are things good enough for now? Can I wait later to invest? What about assets needed for future demand? Arsin Company Confidential 6

9 You d better get ready to think like a business owner Customer Satisfaction Threat/Risk to service quality Business Value TCO (Total Cost of Ownership) of testing Test asset investment and potential asset gaps over time Hint: Quality Theory is not the answer Answer: Business-Oriented Metrics Business-oriented metrics bridge the gap between the activities your test organization performs and the benefits to the business Arsin Company Confidential 7

10 First, a little introduction to business-oriented metrics Sadly for him, The he face didn t of know a business about business-oriented owner metrics accountable to stakeholders and customers. Arsin Company Confidential 8

11 Here are some metrics. Are they useful? California Electric Supply & Demand for one day Maximum supply Predicted potential supply Predicted demand Actual demand These metrics serve the customer pretty well but not the producer or the business owner Arsin Company Confidential 9

12 How well do these metrics serve their audience? Customer A+ Good for predicting daily outage risk and when outage might occur. Producer D- Doesn t indicate anything useful about the operation of a powerplant (generation efficiency, power quality (brownouts), TTF) Business Owner F Too tactical: Only tells owner supply and demand for ONE DAY Big gaps are uninformative: Sense of false security. Disaster could strike without warning! (unplanned failures, demand spike, plant decommissioning) Worse, a big gap could erroneously suggest over-investment Arsin Company Confidential 10

13 And look what happened Lesson: Gray Davis didn t see how increasing demand was threatened by under-investment in assets and lack of asset health. Arsin Company Confidential 11

14 How to transform these into business-oriented metrics oil coal hydro oil nuclear 1. Show assets individually 2. Indicate asset growth and decay (health) 3. Add time (past and future) 4. Indicate future demand for assets time Revised metrics show business owner the demand for assets and asset health Arsin Company Confidential 12

15 But don t forget, business owners also care about value Efficiently run businesses run close to the edge. Business value is always critcal. No such thing as quality at any cost! Arsin Company Confidential 13

16 Business-Oriented metrics in the Testing World Before starting, remember what you re trying to do Talk to business owners in terms they understand Influence decisions by demonstrating and defending testing value Get them to participate (be a stakeholder) in your process Compete against others for scarce resources and win Always remember the basics of metrics: It s a sequence of steps from objectives to actions. 1.Define Objectives and Metrics 2.Collect Data 3.Reveal Trends 4.Make Decisions 5.Take Actions Arsin Company Confidential 14

17 Business-Oriented testing metrics defined Test Asset Trajectory metric (benefit-side) Identifies testing assets and measures how adequate they are at meeting testing demand. Also shows asset benefit, health, and risk. The term trajectory is used to indicate that the metric is always measured over time. Test Asset Leverage metric (cost-side) Identifies the costs associated with the testing assets. The term leverage is used to indicate that the metric subdivides cost into components that are useful for identifying best utilization of resources. Together they form the total asset value picture. asset-value = benefit-side + cost-side Note: One simple building block is key to both metrics Arsin Company Confidential 15

18 Test Asset Trajectory metric Assets & Types Today Demand Timeframe & Events Stress Completeness & Health End-to-end End-to-end End-to-end Stress TIme Single, clear statement of asset composition, benefit, and risk Arsin Company Confidential 16

19 Test Asset Leverage metric Assets Asset Cost by component Total Cost Stress 3 End-to-end End-to-end End-to-end = 11 Stress People Machines Dollars Time People Machines Dollars Time People Machines Dollars Time People Machines Dollars Time Single, clear statement of the asset cost by component TIme Arsin Company Confidential 17

20 Test Asset metrics building block Every building block contains the same information. Demand, asset trajectory and asset leverage metrics are computed by aggregating the information from all the building blocks. Asset Type Definition Name: manual, performance, automated, etc Demand (e.g. function points, list of test items, test scenarios) Asset benefit %completed (degree to which demand is met by asset) %healthy (how much of asset is fully fit for service) Asset cost (in all-or-nothing chunks) # People (by geography, by FTE/Contractor, Outsourcer) # Machines (clusters, labs, in some cases single units) Dollars for tools or testing supply Duration asset is tied up (generally one quarter) Arsin Company Confidential 18

21 Case Study Software Product Company Profile Web services product for handling B2C and B2B financial transactions. Major business events related to Testing New ity Rapid Adoption Technology Changes Acquisition Automation Customer-driven Releases Workforce Redistribution Tests Scalability & Regression Geographical Distribution Test Conversion Rapid Regression Employees, Contractors, Outsource Actions and Results Team transformed from handling a single product line to three product lines with 75% less people! Scalability for multiple thousands of users proven. Outsourcing option enabled. Arsin Company Confidential 19

22 Case Study Test Asset Trajectory (3 products 2 years) Timeframe & Events Assets & Types Demand Completeness & Health A B End-to-end End-to-end Stress C TIme Product C release Product A dot releases Product A scalablity 5k Product B acquisition Product A scalablity 10k Technology change Financial misfortune Product A release Workforce reorg Outsourcing Arsin Company Confidential 20

23 Case Study Test Asset Leverage (3 products 2 years) Trends: Start: 8 people for 1 product End: 5 people for 3 products Due to automation investment A People Machines Dollars Stress B End-to-end End-to-end TIme 5k 5k 10k 10k 10k 5k 5k 5k Arsin Company Confidential 21 C

24 Roadmap - Suggestions for how to proceed immediately Collect Data (step 2) Choose a small, well-differentiated set of asset types. Trade-offs are easier to make when options are distinct. Build your model around events that create discontinuities Upgrades, new customer deployments, configuration changes and hardware expansions, shifting organizational responsibilities, outsourcing Predict demand accurately Include project managers, product managers, service planners, outsourcers Reveal Trends (step 3) Don t present too many options. Make sure your top 3 are visible. Make points business-owners care about, especially using events Always anticipate tension and prepare to defend yourself right to the edge Appreciate your audience. KISS Some really good news These aren t weekly metrics. Generally quarterly although sometimes nonquarterly when there are significant events or discontinuities Arsin Company Confidential 22

25 Questions Arsin Company Confidential 23