Supporting the CMMI Metrics Framework thru Level 5. Márcio. Silveira. page 1

Transcription

1 September EDS-Electronic Electronic Data Systems do Brasil Ltda. Márcio Silveira page

2 Agenda Objective EDS Overall Process Improvement Strategy Measurement Elements of the CMMI Model M&A and High-Maturity Practices Processes and Procedures for M&A and High-Maturity Lessons Learned Q&A page 2

3 Objective This presentation shows the Software Engineering Framework that EDS utilizes at its Applications Development and Maintenance Centers in Brazil that supports the measurement requirements of the CMMI model thru level 5. During the presentation you will learn the basics and advanced measurement requirements from the CMMI model and see how EDS implemented the processes and procedures to support these requirements. CMMI is registered in the U.S. Patent and Trademark Office by Carnegie Mellon University. page 3

4 Corporate Objectives Business Drivers Technology Drivers PI Opportunities Voice of Customer Voice of Management EDS Overall Process Improvement Strategy EDGE - Enabling Delivery and Global Excellence Inputs/Drivers Strategic Business Plan (SBP) Strategies/Initiatives Measurement (Monitor & Control) Organization Improvement Strategy Governance Council (Board) page 4

5 Edge (Work-types, Work-types, Processes, Activities) page 5

6 Tailoring the... OSSP-Org. Plans Tailoring Criteria Requirements and Approach Documentation s Defined Process File Tailoring Execution Tailoring Tool Waivers Tailoring Decisions s Defined Process File (PDP) Starting Point Schedule Workbook Structure Amendment History page 6

7 M&A e High-Maturity (CMMI-Dev.2) LEVEL FOCUS NAME PROCESS AREAS 5 Continuously Improving OPTIMIZING Process Organizational Innovation & Deployment Causal Analysis and Resolution 4 Predictable Process QUANTIT. MANAGED Organizational Process Performance Quantitative Management 3 Standard and Consistent Process DEFINED Requirements Development Technical Solution Product Integration Verification Validation Integrated Management Risk Mgmt. Decision Analysis and Resolution Organizational Process Focus Organizational Process Definition Organizational Training 2 Disciplined Process MANAGED Requirements Mgmt. Planning Monitoring and Control Measurement and Analysis Process and Product Quality Assurance Configuration Mgmt. Supplier Agreement Mgmt. Ad Hoc PERFORMED (None) page 7

8 M&A Objectives The purpose of Measurement and Analysis is to develop and sustain a measurement capability that is used to support management information needs by: Specifying the objectives of measurement and analysis such that they are aligned with identified information needs and objectives Specifying the measures, analysis techniques, and mechanism for data collection, data storage, reporting, and feedback Implementing the collection, storage, analysis, and reporting of the data Providing objective results that can be used in making informed decisions, and taking appropriate corrective actions page 8

9 Supporting M&A Goal SPs Goal : Align Measurement and Analysis Activities SP. Establish Measurement Objectives SP. Establish Measurement Objectives Strategic Strategic Business Plan Business Plan Service Service Level Level Agreement Agreement Performance and Performance and Quality Quality Goals and Objectives Goals and Objectives SP.2 Specify Measures to Address Measurement Object. SP.2 Specify Measures to Address Measurement Object. SP.3 Specify data collection and storage procedures SP.3 Specify data collection and storage procedures SP.4 Specify analysis procedures SP.4 Specify analysis procedures Organizational Organizational Measurement Measurement Plans Plans Measurement Measurement Plans Plans Organization OPP/QPM OPP/QPM Organizational Organizational Quantitative Quantitative Management Management Approach Approach Quantitative Quantitative Management Plan Management Plan page 9

10 Supporting M&A Goal 2 SPs Goal 2: Provide Measurement Results Measurement Measurement Plans Plans Performance and Performance and Quality Quality Goals and Objectives Goals and Objectives Communication Communication Plan Plan SP 2. Collect Measurement Data SP 2. Collect Measurement Data SP 2.3 Store Data and Results SP 2.3 Store Data and Results SP 2.2 Analyze Measurement data SP 2.3 Store Data and Results SP 2.2 Analyze Measurement data SP 2.3 Store Data and Results SP 2.4 Communicate Results SP 2.4 Communicate Results Source Source of Data of Data Tracking Tracking Reports (Local Reports (Local Repository) Repository) Presage Presage (Enterprise (Enterprise Metrics Metrics Repository) Repository) Metrics Metrics Analysis Report Analysis Report PTR Consolidation Process Organizational Organizational Metrics Metrics Repository Repository Organizational Organizational Metrics Metrics Analysis Report Analysis Report Organization page 0

11 OPP & QPM Objectives The purpose of Organizational Process Performance (OPP) is to establish and maintain a quantitative understanding of the performance of the organization s set of standard processes in support of quality and process-performance objectives by: Selecting processes to be managed quantitatively Establishing process performance measurements and quality and process performance objectives Creating and maintaining Process performance baselines and models The purpose of Quantitative Management (QPM) is to quantitatively manage the project s defined process to achieve the project s established quality and process-performance objectives by: Establishing and maintaining the project s quality and process-performance objectives Identifying suitable sub-processes that compose the project s defined process based on historical stability and capability data found in process-performance baselines or models Selecting the sub-processes of the project s defined process to be statistically managed Monitoring the project to determine whether the project s objectives for quality and process performance are being satisfied, and identifying appropriate corrective action Selecting the measures and analytic techniques to be used in statistically managing the selected sub-processes Establishing and maintaining an understanding of the variation of the selected sub-processes using the selected measures and analytic techniques Monitoring the performance of the selected sub-processes to determine whether they are capable of satisfying their quality and process-performance objectives, and identifying corrective action Recording statistical and quality management data in the organization s measurement repository page

12 Supporting OPP Goal SPs Goal : Establish Performance Baselines and Models M&A - SP. Establish Measurement Objectives M&A - SP. Establish Measurement Objectives Strategic Strategic Business Plan Business Plan Service Service Level Level Agreement Agreement Performance and Performance and Quality Quality Goals and Objectives Goals and Objectives SP. Select Processes SP. Select Processes SP.2 Establish Process-Performance Measures SP.2 Establish Process-Performance Measures SP.3 Establish Quality and Process-Performance Object. SP.3 Establish Quality and Process-Performance Object. Organizational Organizational Quantitative Quantitative Management Management Approach Approach Quantitative Quantitative Management Plan Management Plan SP.4 Establish Process-Performance SP.4 Establish Process-Performance Baselines (PPBs) Baselines (PPBs) SP.5 Establish Process-Performance Models SP.5 Establish Process-Performance Models (PPMs) (PPMs) Organization Organizational Organizational Metrics Metrics Repository Repository PPBs PPBs Statistical & Statistical & Simulation Tools Simulation Tools PPMs PPMs page 2

13 Supporting QPM Goal SPs Goal : Quantitatively Manage the SP. Establish the s Objectives SP. Establish the s Objectives SP.2 Compose the Defined Process SP.2 Compose the Defined Process SP.3 Select the Sub-processes that w/be Statis. Managed SP.3 Select the Sub-processes that w/be Statis. Managed Organizational Organizational Quantitative Quantitative Quantitative Quantitative Management Management Management Plan Management Plan Approach Approach Performance and Performance and Quality Quality Goals and Objectives Goals and Objectives SP.4 Manage Performance (more in Goal 2) SP.4 Manage Performance (more in Goal 2) Tracking Tracking Reports (Local Reports (Local Repository) Repository) Metrics Metrics Analysis Report Analysis Report Quantitative Quantitative Management Plan Management Plan Defined Defined Process Process (PDP) (PDP) Organization page 3

14 Supporting QPM Goal 2 SPs Goal 2: Statistically Manage Sub-process Performance SP 2. Select Measures and Analytic Techniques SP 2. Select Measures and Analytic Techniques SP 2.2 Apply Statistical Methods to Understand Variation SP 2.2 Apply Statistical Methods to Understand Variation SP 2.3 Monitor Performance of the Selected Sub-processes SP 2.3 Monitor Performance of the Selected Sub-processes SP 2.4 Record Statistical Management Data SP 2.4 Record Statistical Management Data Tracking Tracking Reports (Local Reports (Local Repository) Repository) Metrics Metrics Analysis Report Analysis Report Storyboard Storyboard Quantitative Quantitative Management Plan Management Plan Statistical & Statistical & Simulation Tools Simulation Tools PPBs & PPMs PPBs & PPMs Organization page 4

15 Strategic Business Planning Key element Business Objectives + Market Drivers, Improvement Opportunities + Strategic Planning Customer s expectations Performance and Quality - Goals and Objectives = page 5

16 Performance and Quality Goals and Objectives Defines the process-performance and quality goals and objectives from the organization. It is a key output of the SBP activities. It defines: Suggested measurements and objectives at organizational level and project level. High-level measurement plan Reporting mechanism page 6

17 Organizational Measurement Plans (Pattern) Defines the process to collect, store, analyze and report the measurements collected by the organization and projects. Basically contains: Linkage between the measurements and the performance and quality goals objectives Detailed information about the metrics that will be collected, stored and analyzed. Analysis and reporting procedures Triggers Roles & responsibilities Organizational Metrics Repository information page 7

18 Measurement Plans It is an instantiation of the Organizational Measurement Plans (Pattern). s reuse the pattern and : Justify what doesn t apply to the project. Add project specific needs page 8

19 Organizational Quantitative Management Approach It addresses establishment of process-performance and quality objectives for the organization. These objectives will be used by projects to establish projectlevel process-performance and quality objectives, and determine the measurements that will be used to support these objectives. Basically contains : Quality goals and process-performance objectives Common Organizational Objectives Common Objectives ( Type) Organizational Baselines locations (PPBs) Organizational Performance Models (PPMs) Performance Quantitative Management General Procedures Organizational Procedures (PPBs, PPMs, Tools, etc.) Procedures page 9

20 Quantitative Management Plan It is an instantiation of the Organizational Quantitative Management Approach. s follow the organizational approach by : Selecting the common objectives that are aplicable to the project. Identifying project specifics objectives Identifying processes and sub-processes affected by the objectives. Creating a plan to collect, store, analyze and report on quantitative management data. (DMAIC) Identifying Baselines and Models that will be created and/or reused. Documenting all specific procedures that must be followed. page 20

21 SAP/MSPS/Presage Presage SAP Presage Tasks Assignments MSPS Effort Consolidation & Upload page 2

22 Tracking Reports (PTR) Local Repository page 22

23 Tracking Reports (PTR) Analysis Procedures page 23

24 Tracking Reports (PTR) Consolidation Tracking Reports - Effort Data Change Request data Size data Tracking Reports 2. Tracking Reports N Consolidation Process Organizational Organizational Metrics Metrics Repository Repository page 24

25 Within schedule Meet requirements Quality Prompt, clear and effective communications Open communication English proficiency Changes effectively managed. Issues/Risks effectively managed. Overall project management. Appropriate skills Adequate level of knowledge Organizational Metrics Analysis Report Some examples : Productivity: Hours per FP per platform KLOC per resource per month per platform Quality Assurance: Average number of non-conformances per audit Average days for non-conformances closing Defects: Defect density (#defects / size * 00) Defect containment (#defeitos detected in peer reviews/ados em revisões/#total defects) by deliverable/phase. Estimates: Jan 2.5 Avg Review Effort by WPR Completed (hrs) May Apr Mar Feb Avg Review Effort per Work Product Review Completed Org Avg Review Effort per Work Product Review Completed Linear (Avg Review Effort per Work Product Review Completed) Jun Effort variation by project/service Request by platform/language Duration variation by project/service Request by platform/language n=34 00% 90% 80% 70% 60% 0% Client Satisfaction Survey 2006 Summary OPP Baselines and Models 50% 40% 30% 20% Customer Satisfaction Index 0% Target page 25

26 Statistical & Simulation Tools Minitab/Crystal Ball I-M R Chart of Effort Tickets from _ X=2.4 Individual Value UCL= LB= Observation Moving Range UCL= 6.79 MR=2.08 LCL= Observation page 26

27 Name Description Main Characteristics Design Defect Density Testing Productivity Effort Variation Ticket Effort Resolution It shows the expected range of Defect Density (number of Defects by Pages of Documentation * 00) for the External Reviews. It shows the range of expected Appl.CR productivity when a Test Case is created. It shows the range of expected effort variation that would normally be achieved by a Application CR following the minor modification process with similar characteristics. It shows the behaviour of the ticket resolution process. -Design Phase -Health Care -Design Work Products -Minor Enhancement - Produce Phase -Health Care, Transportation -Test Case -Minor Enhancement -Main Phases (Design, Produce and Testing). -Health Care, Transportation - Primary Language (Cobol, Cool:Gen, Java, Shell Script) -Minor Enhancement - On-going support - Ticket category - Effort ticket resolution Process Performance Baselines (PPBs) page 27

28 Pilot and After Pilot %Reused Test Cases Regression 95%CI 95%PI S R-Sq 66.5% R-Sq(adj) 65.4% Fitted Line Plot - Metavance Defect Team - only SEs, all PCVs Coding = Code line Qty _avg Code line Qty _avg**2 40 Regression 95% C I 95% P I 30 S R-Sq 30.6% 20 R-Sq(adj) 24.6% Code line Qty _avg Apr/2007 to Apr/ Fitted Line Plot Defect Density = Business Skill Business Skill** Business Skill** Work Product Creator Business Skill 3.0 Fitted Line Plot Defect Density = Business Skill Business S kill** Business S kill** Work Product Creator Business Skill S R- Sq 70.6% R- Sq (ad j ) 66.2% 3.0 Regression 95% PI Reg ression 95% PI S R-Sq 70.6% R-Sq(adj) 66.2% Process Performance Models (PPMs) Life Cycle GAD QMS Predictability Productivity Quality Planning External Defect Density (Assignment) Standard Process Business Skill Business Skill Design Standard Process Business Skill External Defect Density Business Skill Cheklist Mentoring Fitted Line Plot Test Case Creation Productivity = %Reused Test Cases Test Case Creation Productivity Produce Standard Process Test Case Creation Reuse Standard Process Business Skill Testing Standard Process Business Skill Dev Co ding Ticket Standard Process Library Reuse Knowledge Repository Training Guideline Non Dev Ticket Standard Process Knowledge Base page 28

29 APPS Rio HM Approach Overview 28 / 0 Sep 2008 / EDS INTERNAL Average al Defects without Internal Review Max (UCL) Average Min (LCL) ABC Appl. CR ID 234 Phase Design Application Change System Area Billing Work Product Design document Defect Density = (WP Creator Business Skill) (WP Creator Business Skill)** (WP Creator Business Skill)**3 The Manager decides to perform an internal review considering the possibility of retain 74% of the forecasted defects Confidence Intervals for Forecasted Total of External Defects Forecasted Total of External Defects without Internal Review Forecasted Total of External Defects with Internal Review Work Product Externally Reviewed Work Product Externally Reviewed Defect Density = Business Skill Business Skill** Business Skill** Work Product Creator Business Skill Regression 95% PI S R-Sq 70.6% R-Sq(adj) 66.2% UCL=33.9 LCL=-8.8 UCL=32.5 LCL=0 27 / 0 Sep 2008 / EDS INTERNAL Storyboard DMAIC Initiatives Quality The Results External Defect Density External Defect Density Before Improv ement (A pr to Sep, 2007) Pilot (Sep to Nov, 2007) A fter Improv ement (F eb to A pr, 2008) UC L=29.3 UC L=9.2 X=25.9 X=8.6 UC L=7.4 _ X= I-MR Chart of External Defect Density 5 LC L= LC L= Work Product Reviewed LC L=-.0 Before Improv ement (A pr to Sep, 2007) P ilot (Sep to Nov, 2007) A fter Improv ement (F eb to A pr, 2008) UC L= MR=38.9 UC L=89.2 MR= Mean from 8.6 to 3,2 defects by 00PODs 88% UC L=7.4 MR=5.3 LC L=0 LC L=0 LC L= Work Product Reviewed DMAIC Initiatives Quality Client: Metavance is the main client of ASFOA Rio. It represents 4 managed projects with 50 resources (about 30% of the Centre). It was very important to consolidate the perception of this client from ASFOA Rio as a high quality centre. The Business Need External Defect Density External Defect Density Increase the quality of the final product; Reduce the bottleneck on the application US SME. I-MR Chart of External Defect Density Health Care industry, Design phase, Developer role (Period Considered: from April to September 2007) _ X=26. MR=40.6 APPS Rio HM Approach Overview 30 / 0 Sep 2008 / EDS INTERNAL APPS Rio HM Approach Overview 26 / 0 Sep 2008 / EDS INTERNAL DMAIC Initiatives Quality DMAIC Initiatives Quality The Simulation In order to evaluate the alternatives a simulation was performed, evaluating the impact on External Defect Density and Cost. Cristal Ball, a Monte Carlo simulator, was the tool used. The Main Contributor The analysis of defects raised at ASFOA Rio External Reviews showed that: Health Care industry detects 98.5% of defects; Design phase is the origin of 68.9% of defects; Developer role injects 88.0% of defects. Root Cause Analysis There was a correlation between External Defect Density and Work Product creator Business Skill. Current Mean Defect Density: 55.6 Option Mean Defect Density: 35.2 (-36%) Option 2 Mean Defect Density: 54.4 (-2,2%) Option 3 Mean Defect Density: 55.6 (0%) Current Mean Cost: US$ 243 Option Mean Cost: US$ 2.32 ( +7.8%) Option 2 Mean Cost: US$ 2.74 ( +.4%) Option 3 Mean Cost: US$ 2.84 ( +.9%) > External Defect Density Fitted Line Plot The Correlation Coefficient (R) is 0.8 and it explains 66.2% of population, what was a good correlation. The Defect Density increases when the Business Skill is lower than 2.0. APPS Rio HM Approach Overview DMAIC Initiatives Quality The Improvement A model was developed and deployed on Health Care projects. The Manager will use it to analyze the benefits of perform an internal review to retain defects. POD 50 WP Creator Business Skill.30 Internal Reviewer Business Skill 3.70 Organizational Process Performance Model Defect Density by Business Skill Data Regression Equation Forecasted Total of External Defects without Internal Review APPS Rio HM Approach Overview Forecasted Total of External Defects with Internal Review Analysis Result Forecasted Total of External Defects 29 / 0 Sep 2008 / EDS INTERNAL page 29

30 Lessons Learned. M&A/OPP/QPM will force you to create a structured process for Strategic Business Planning. If you don t have it will be very difficult to implement M&A/OPP/QPM practices 2. Have a long-term strategy to store measurement data, not only M&A but also SPC data. 3. Have an integrated set of tools supporting your Software Engineering projects, otherwise metrics collection will be a nightmare. 4. Reduce the effort, as much as you can, for metrics collection process, if possible, collect at the same time that the task is being performed. 5. Do specialize people on SPC, Six Sigma otherwise levels 4 and 5 will be a very hard journey. Consider to have statisticians around. 6. Excel will not be sufficient in many cases for QPM/OPP. 7. Do start with few performance indicators and then sophisticate and add additional ones, please remember, they must be aligned to companies s goals/objectives. 00 performance indicators will not allow you to take any conclusion. 8. Keep these indicators stable, as much as possible, so that you can analyze them periodically. 9. Do not individualize data at personal level, particularly on QPM/OPP analysis where sub-process analysis may lead you to see detailed data on people. page 30

31 Q&A? page 3

32 September eds.com page 32