Measuring the Cost of Software Quality of a Large Software Project at Bombardier Transportation

Transcription

1 Ingeniería para la Industria Measuring the Cost of Software Quality of a Large Software Project at Bombardier Transportation Presented by Claude Y. Laporte, Eng., Ph.D. Professor Department of Software and IT Engineering g École de technologie supérieure, Québec, Canada Project Editor of ISO/IEC for Very Small Entities Professor Normand Séguin, Ph.D. Universitad de Lima, Peru September, 13 th

2 Ingeniería para la Industria Content Introduction Business Rationale for Software Quality Assurance and the Measurement of the Cost of Quality Undergraduate/Graduate Software Quality Assurance Courses and the Measurement of the Cost of Quality Measurement of the Cost of Software Quality A Case Study Conclusion 2

3 École de technologie supérieure 3

4 École de technologie supérieure Over 6,3000 students Each year 2,200 paid industrial internships in over 900 companies. Students are paid about 36,500$ for 3 internships of 4 months Undergraduate Programs 600 students (400 in Software Eng.) Software Engineering g 20 Professors in the department thave a IT Engineering mean industrial experience of 10 years. Construction Engineering Production Engineering Electrical Engineering Mechanical Engineering Logistics and Operations Engineering Graduate Programs Software Engineering Information Technology Other programs 175 students 4

5 Overview of my Background Oerlikon Aerospace National Defense Defence Nationale ( ) ( ) ( ) Department of Software and IT Engineering i (2000 ) 5

6 Project Editor of ISO/IEC Standards and Guides Joint Committee for IT Sub committee (SC) 7 Working Group (WG) 24 Standardization of processes, supporting tools and supporting technologies for the engineering of software products and systems. Mandated to develop Standards and Guides for Very Small Entities * Very Small Entity - An entity (enterprise, organization, department or project) having up to 25 people. 6

7 ISO/IEC Standards and Guides for Very Small Entities (VSEs) Entry Basic Advanced Intermediate Entry - Targets VSEs typically developing 6 person-month projects and start-up VSEs; Basic - Targets VSEs developing only one project at a time; Intermediate Targets VSEs developing multiple projects within the organizational context; Advanced Targets VSEs which want to sustain and grow as an independent competitive software development business. Very Small Entities are enterprises, projects or departments having up to 25 people. (ISO/IEC 29110) 7

8 Ingeniería para la Industria Content Introduction Business Rationale for Software Quality Assurance and the Measurement of the Cost of Quality Undergraduate/Graduate Software Quality Assurance Courses and the Measurement of the Cost of Quality Measurement of the Cost of Software Quality A Case Study Conclusion 8

9 Software Development What the customer wants Best Approach = Better Faster Cheaper 9

10 Why Software Fails? Studies have shown that software specialists spend about 40% to 50% of their time on avoidable rework rather than on what they call value-added work, which is basically work that s done right the first time. Once a piece of software makes it into the field, the cost of fixing an error can be 100 times as high as it would have been during the development stage. Software managers and software engineers must select and use appropriate practices to reduce rework (i.e. waste or scrap ) Adapted from (Charette 2005) 10

11 Components of Project Cost Project Cost Cost of Quality Cost of Conformance Cost of Performance Generation of plans Software Development Cost of Non Conformance Appraisal Costs Reviews Inspections Testing IV&V Audits Prevention Costs Training Methodologies Tools Data gathering Fixing defects Re testing Re reviews Updating source code Modifying documents Adapted from (Haley et al., 1995) 11

12 Cost of Quality Data from Professional Software Engineers Site A American Engineers (19) Site A American Managers (5) Site B European Engineers (13) Site C European Engineers (14) Site D European Engineers (9) Course A 2008 (8) Course B 2008 (14) Course C 2009 (11) Course D 2010 (8) Course E 2011 (15) Course F 2012 (10) Cost of performance Cost of rework Cost of appraisal Cost of prevention 41% 44% 34% 31% 34% 29% 43% 45% 45% 34% 40% 30% 26% 23% 41% 34% 28% 29% 30% 25% 32% 31% 18% 14% 32% 21% 26% 24% 18% 14% 20% 27% 20% 11% 16% 11% 8% 7% 14% 10% 11% 10% 8% 9% Quality Quality = Number of Defects/ 1,000 Lines of Code 12

13 Defects (%) Defect Injection During Development We must identify if defects as soon as possible to reduce cost and schedule slippage. Development Phases Adapted from (Selby 2007) 13

14 Defects Detected/ Defects Injected Defect Detection During Development We must implement software engineering practices to detect and correct 90% of defects as early as possible to reduce cost and schedule slippage. Development Phases Adapted from (Selby,2007) 14

15 % 45 of ftotal Project Cost 40 41% Cost of Quality Start of Initiative Cost of Non Conformance (Rework) Appraisal & Prevention Costs % % 5 0 Mt Maturity Level % 5% Adapted from (Haley et al., 1995) 15

16 Ingeniería para la Industria Content Introduction Business Rationale for Software Quality Assurance and the Measurement of the Cost of Quality Undergraduate/Graduate Software Quality Assurance Course and the Measurement of the Cost of Quality Measurement of the Cost of Software Quality A Case Study Conclusion 16

17 Software Quality Assurance Course at ETS Fourteen Topics Chapter 1: Basic Software Quality Assurance Knowledge Chapter 2: Quality Culture and Cost of Quality (COQ) French Textbooks Vol. 1 Chapter 3: Quality Requirements Chapter 4: Software Engineering Standards and Models Chapter 5: Reviews and COQ Chapter 6: Software Audit Chapter 7: Verification and Validation and COQ Chapter 8: Tests and Software Quality Assurance English Textbook Vol. 2 Chapter 9: Software Configuration Management and COQ Chapter 10: Policies, Processes and Procedures and COQ Chapter 11: Measurement and COQ Chapter 12: Management of Suppliers and Contracts Chapter 13: Risk Management and COQ Chapter 14: Software Quality Assurance Plan Publication in

18 Software Quality Assurance Course Laboratory Software Development Project with ISO/IEC Put in practice the software quality practices learned in class, Each team manages and executes a project using ISO/IEC A process is a component of the COQ (i.e. Prevention) Customer Statement Software of Work Configuration Project Management Process Planning Control Execution Closure Organizational Management Implementation Process Initiation Analysis Design Construction Tests Delivery Measure the Cost of Software Quality ISO/IEC Standard for Very Small Entities 18

19 Graduate Software Engineering Program Twelve 3-credit courses and a 9-credit Project in Software engineering Students are typically from industry with a minimum of 2 years of experience Students typically have an undergraduate degree in either computer science or management information systems and work for small and medium-sized i dorganizations. Software Quality Assurance Course Students, in teams of 3, have to do a Project in an Organization Measurement of the Cost of Quality 19

20 Ingeniería para la Industria Content t Introduction Business Rationale for Software Quality Assurance and the Measurement of the Cost of Quality Undergraduate/Graduate Software Quality Assurance Courses and the Measurement of the Cost of Quality Measurement of the Cost of Software Quality A Case Study Conclusion 20

21 Measuring the Cost of Software Quality of a Large Software Project at Bombardier Transportation in Canada Laporte, C. Y., Berrhouma, N., Doucet, M., Palza Vargas, E. (Laporte et al. 2012) 21

22 Challenges Facing Railway Manufacturers Better, Faster, Cheaper Criticality of software Financially, environmentally or for human safety. Multi-disciplinary system development, Integrator-Suppliers Relationships, Multi-country development, Multi-cultural teams, Downsizing/Merger/Turnover, Off shoring. ERTMS / ETCS (European Rail Traffic Management System / European Train Control System) 22

23 Overview of Bombardier Workforce of some 80,000 people in over 24 countries. Bombardier Aerospace World ldleader in the manufacture of business jets and regional transport, Bombardier Transportation Leader in the manufacture of rail transport equipment., Manufactures locomotives, freight cars, and propulsion and control systems, Provides systems and signalling equipment. Modern trains and subways are increasingly complex, and more and more subsystems are computer-controlled, such as propulsion and braking systems. At the time of this case study, there were over 30 software development centers within Bombardier Transportation, for a total of about 950 software engineers. (Laporte et al. 2012) 23

24 Media Converter _ 5VDC. 1A + UP LINK LINK PWR LINK SD RX TX The Software Development Group of Bombardier Transportation ti in Québec (Canada) A group of 30 software engineers whose role is to design, develop, and maintain embedded software for trains and subways Software monitoring system used for collecting software maintenance information, Software for controlling car inclination. (Laporte et al. 2012) 24

25 Software Engineering Process of the Software Engineering ggroup * * Located in Québec, Canada (Laporte et al. 2012) 25

26 ISO/IEC Systems and Software Engineering Measurement ProcessStandard Measurement Requirements Information Needs Technical & Management Processes User Feedback Information Products Core Measurement Process Establish Plan Perform Evaluate Commitment Measurement Plan Measurement Measurement Performance Information Experience Base Evaluation Results Scope of ISO/IEC Improvement Actions (ISO/IEC 15939) 26

27 Objectives of the Measurement Project of the Software Development Group 1. Identify a project to measure the Cost of Software Quality (CoSQ), 2. Collect data on costs, 3. Categorize costs related to software quality, 4. Develop a data model for the measurement of CoSQ, 5. Analyze data collected, on the selected project, using the data model, 6. Present the CoSQ report to senior management, 7. Expand the measurement of the CoSQ to other software projects in the Software Development Group. (Laporte et al. 2012) 27

28 Overview oftheprojectselected Development of the software to control the subway of a large American city, Team of 15 software engineers, Large number of software tasks 1,121 tasks Total effort of the completed project 88,598 hours (Laporte et al. 2012) 28

29 The Five Stages of the Measurement Project using the ISO/IEC Standard 1. Identification of the project tasks related to the CoSQ Identify measurement requirements and resources 2. Development of a list of typical tasks related to the CoSQ Define data measures 3. Categorization of the tasks related to the CoSQ Select and plan measures 4. Development and application of weighting rules Define measurement criteria 5. Determination ti of the confidence of the weighting rules Define data collection and analysis (Laporte et al. 2012) 29

30 Stage 1. Identification of Project Tasks Rl Related to the CoSQ Software engineering process is composed of 4 types Life Cycle, Process, - Sub-process, - Activity. Type Process Element Life Cycle Primary Life Cycle Process Process Development Sub-process Code and Debug Activity Unit Tests Representation of BSEP elements WBS Element Task Name Effort [hours] Code document Monitoring-Unit Testing 91.1 Sample registration of a task in the Project accounting system (Laporte et al. 2012) 30

31 Stage 2. Development of a list of tasks related to the Cost of Software Quality A list of tasks related to the CoSQ is developed for this project. Primary Life Cycle Processes Supply Development System Requirements Analysis System Architectural Design Sft Software Requirements tanalysis Software Architectural Design Software Detailed Design Software Coding and Testing Software Integration Software Validation Testing Subset of CoSQ-related Tasks Organizational Life Cycle Processes Management Estimate the Project Plan the Project Manage Risks Track kthe Project Infrastructure Plan Infrastructure Activities Establish the Project Infrastructure Establish the Global Infrastructure Maintain the Infrastructure (Laporte et al. 2012) 31

32 Stage 3. Categorization of tasks related to the CoSQ All 1,121 tasks were sorted as follows: Implementation (I), Evaluation (E), Prevention (P), Rework (R) (internal and external anomalies). Task Identification WBS Element Task Name I E P R 2410 Code Trace Requirements - - x - Example of classification of the Requirements Traceability Task (Laporte et al. 2012) 32

33 Stage 4. Development of weighting rules Many tasks belong to more than one CoSQ category e.g. The Test and coding task overlaps the Evaluation and Implementation CoSQ categories. Twenty-seven (27) weighting rules have been defined. Rule Number Name of Rule Typical Tasks Implementation Evaluation Prevention Rework 3 Process Audit Software Project & Process Audit 7 Review ew Design review ew 100% 12 Problem Correction and Coding 17 Training 26 Follow-up & Validation Corrections, Debugging and Final Coding Training of new resources 30% 70% 100% 100% Follow-up & Validation (all 85% 15% releases) (Laporte et al. 2012) 33

34 Stage 5. Determining the Confidence of Wihi Weighting Rules What is the level of confidence of the CoSQ measures? Another component was added to the rules "H" for high precision, "M" for medium precision, and "L" for low precision. Confidence with the Weighting Rules Percentage (%) H High 88% M Medium 11% L Low 0,2% Total number of activities 1,121 (Laporte et al. 2012) 34

35 Cost of Software Quality Measurement Results Perform Evaluation Prevention Rework CoSQ Hours 59,231 18,551 2,025 8,791 28,824 Percentage 67 % 17 % 4% 11 % 32 % Total effort (i.e. cost) of the project: 88,598 hours Number of software tasks: 1,121 tasks (Laporte et al. 2012) 35

36 Cost of Software Quality at Raytheon compared to Bombardier Transportation Start of intiative 70 CMM level 1 CMM level 3 Percentage of total projec ct cost Rework Appraisal TCoSQ Cost of Conformance Prevention Total Cost of Software Quality (40%) 32 % Percentage of Evaluation ( 20%) 17 % Percentage of Rework ( 13%) 11 % Percentage of Prevention ( 7%) 4 % Rework Year Adapted from (Haley 1996) 36

37 Cost of Quality and Maturity Levels A study of the Software Engineering Institute (SEI) showed that rework varies between 15% and 25% of the cost of developing a CMM Maturity Level of 3. CMM Maturity Level Percentage of frework 1 50 % 2 25 % to 50 % 3 15 % to 25 % 4 5 % to 15 % 5 5 % Relationship between CMM Maturity Levels and percentage of rework 11 % (Gibson et al. 2006) 37

38 Recommandations 1. Continue data collection and CoSQ Measurement Continue the data collection using procedures and methods (for example use of tools, database structures) within the SDG. 2. Control activities related to the correction of problems Most defects are found mainly during the execution of software requirements, software design and coding tasks Reduce the defects by increasing prevention effort e.g. Execute more peer reviews (walk-through, inspection) 3. Present the CoSQ measurement results to Management To enable them to develop better budgets for business process improvement. Establish a scoreboard to display the CoSQ and the results of process improvement activities to illustrate the relationships between them. (Laporte et al. 2012) 38

39 Recommandations 4. Measure the CoSQ at other Bombardier Transportation sites To identify opportunities for reducing costs, To provide a basis to budget development and quality activities, i i To use the results of CoSQ measures to improve processes. Project WBS Inputs CoSQ Measurement Process Typical CoSQ list of tasks Weighting rules Tasks 1. Identify CoSQ tasks, using Project Plan and WBS 2. Record CoSQ effort by category 3. Apply weighting grules 4. Compute the CoSQ 5. Develop findings and recommendations 6. Review findings and recommendations with management 7. Produce final CoSQ Project Report 8. Archive Report Outputs CoSQ Report Findings Recommendations Entry Criteria Project Plan Approved Measures Effort (staff hours) Exit Criteria CoSQ Project Report Approved (Laporte et al. 2012) 39

40 Ingeniería para la Industria Content t Introduction Business Rationale for Software Quality Assurance and the Measurement of the Cost of Quality Undergraduate/Graduate Software Quality Assurance Courses and the Measurement of the Cost of Quality Measurement of the Cost of Software Quality A Case Study Conclusion 40

41 Conclusion The concept of Cost of Software Quality is a powerful tool to help quantify the rework, indentify weaknesses and prevention activities, The Software Quality Assurance courses we teach help software engineers in understanding the value of the quality assurance practices learned in class and put into practice in the laboratories, The Case study at Bombardier Transportation demonstrated the business value of the measurement of the Cost of Software Quality. Best Approach = Better Faster Cheaper 41

42 Ingeniería para la Industria Thank you for your attention 42

43 Contact Information Claude Y Laporte Voice: [email protected] Web: Public site of WG 24 Free access to Deployment Packages, presentation material and articles: p p g 43

44 References Fanmuy, G., L Ingénierie et Management des Systèmes pour les PME/TPE et petits projets, Association Française d'ingénierie Système (AFIS)/International Council on Systems Engineering (INCOSE), May 24th, 2011, Paris, France. Gibson, D.L., D.R. Goldenson, and K. Kost, Performance Results of CMMI -Based Process Improvement. Technical Report, CMU/SEI TR-004, ESC-TR , 2006 Haley, T., Ireland, B., Wojtaszek, E., Nash, D., Dion, R., Raytheon Electronic Systems Experience in Software Process Improvement, Technical Report CMU/SEI-95-TR-017, Software Engineering Institute, November Iberle K., «But Will It Work For Me», Proceedings of the Pacific Northwest Software Quality Conference,,p p , Portland, United States 2002, Iberle K., «They Don t Care About Quality», Proceedings of STAR East, Orlando, United States, ISO/IEC JTC1/SC7 N3288, New Work Item Proposal Software Life Cycles for Very Small Enterprises, Mai ISO/IEC 12207:2008, Information technology Software life cycle processes, International Organization for Standardization/ International Electrotechnical Commission: Geneva, Switzerland. ISO/IEC Lifecycle Profiles for Very Small Entities (VSEs) Part 1: Overview. International Organization for Standardization/International Electrotechnical Commission: Geneva, Switzerland. ISO/IEC 15289: Systems and software engineering - Content of systems and software life cycle process information products (Documentation) Kabli, S., Conception, réalisation et mise a l essai de trousses de déploiement pour faciliter et accélérer l implémentation de la norme ISO/CEI par les très petites structures, ÉTS, Konrad, M., Overview of CMMI Model, Presentation to the Montréal SPIN, November 21, 2000, Montréal, Canada. Kroll, P.; Kruchten, P.; The Rational Unified Process Made Easy A Practionner s Guide to the RUP.; Addison-Wesley, 2003 Laporte, Claude Y., Berrhouma, Nabil, Doucet, Mikel, Palza-Vargas, Edgardo, "Measuring the Cost of Software Quality of a Large Software Project at Bombardier Transportation", Software Quality Professional Journal, ASQ, Vol. 14, number 3, June 2012, p Laporte, Claude, Y, Fanmuy, Gauthier, Ptack, Ken, The Development of Systems Engineering i International ti Standards d and Support Tools for Very Small Enterprises, 22nd Annual International Symposium of the International Council on Systems Engineering, Rome, July 9-12, Laporte, Claude Y., Berrhouma, Nabil, Doucet, Mikel, Palza-Vargas, Edgardo, "Measuring the Cost of Software Quality of a Large Software Project at Bombardier Transportation", Software Quality Professional Journal, ASQ, Vol. 14, number 3, June 2012, p Laporte, C.Y., Alexandre, S., O Connor, R., A Software Engineering Lifecycle Standard for Very Small Enterprises, in R.V. O Connor et al. (Eds.): EuroSPI 2008, CCIS 16, pp Long, L., The Critical Need for Software Engineering Education, Crosstalk - The Journal of Defense Software Engineering, January 2008, pp Reifer D., «Let the Numbers Do the Talking», Crosstalk The Journal of Defense Software Engineering, march Rogers, Everett M., Diffusion of Innovations, fifth edition, Free Press, New York,