EASPI EASPI. The Integrated CMMI-based Improvement Framework for Test and Evaluation. Jeffrey L. Dutton Principal Consultant

Transcription

1 The Integrated CMMI-based Improvement Framework for Test and Evaluation Jeffrey L. Dutton Principal Consultant Engineering and Services Performance Improvement LLC 22

2 Copyrights and Service Marks CMMI is registered in the U.S. Patent and Trademark Office by Carnegie Mellon University SCAMPI is a Service mark of Carnegie Mellon University All other materials 2011 (Distributed Unlimited)

3 Background and Experience Credentials - SCAMPI Lead Appraiser (Certified for Development and Services) - (Lean) Six Sigma Black Belt - Certified Scrum Master - Invited Member, CMMI-SVC Advisory Group - Visiting Scientist, Software Engineering Institute Education - B.S. Aerospace Engineering - M.S. Operations Research T&E Experience - Deputy Test Director, Cruise Missile IOT&E - HQ AFOTEC Operations Analyst for Space Transportation System - Satellite Operations Center OT&E

4 Perspective - Provider of Test Services

5 1 st Order T&E Issues and Challenges Integration of DT/OT Technological challenges in test environments, systems of systems testing, and net-centric system designs/operations Test range aging, consolidation, and obsolescence Increasing competition for T&E funding Aging T&E work force (disappearing expertise) How to provide credible, timely, high-value test results in this environment

6 2 nd Order T&E Challenges Change the fundamental context of the conversation Migrate from anecdotal proof of T&E value to consistently demonstrated value propositions Make T&E improvements systemic - Consider improving test technology, test process, and knowledge as a holistic set - Prioritize improvements against the expectation of customer-perceived value in a systematic and predictable way - Use root cause analysis to get at systemic problems - Ensure improvements are implemented consistently across the appropriate test domain - Ensure persistence of improvements across time

7 Meeting the Systemic Challenges (Three things we can do - my view) Value Proposition: - Think about T&E as a range of test services - Provide ever-increasing value propositions to the customer Performance Goals: - Increase test life cycle efficiency, velocity, and responsiveness - Decrease cost as an absolute - Learn to rapidly accommodate technological evolution In systems development (e.g. Model Drive Architecture) In system operations (e.g. net centric warfare) In test environments (e.g. virtual ranges) Adopt a T&E Improvement framework that supports systemic improvement

8 What Systemic Improvement Frameworks are Available? ISO * Lean Thinking Six Sigma Capability Maturity Model Integration (CMMI ) Theory of Constraints*

9 Lean Thinking What is? Focus on customer value Value stream mapping (workflows) Waste elimination Cadence and synchronization Rapid organizational learning Process doers are process owners Reliance on tacit knowledge and skilled team members Agile management Value proposition: High velocity Lean (smart) processes and process efficiency Builds mature teams quickly Rapid response to customer needs Downside: No improvement infrastructure Suffers from lack of consistency and persistence

10 Six Sigma What is? Statistical mechanisms for process control Process variability Central tendency Some mechanisms: Regression and correlation Tests of Hypothesis Analysis of variance Statistical process control Experimental design Process performance modeling and optimization Value proposition: Downside: High(er) cost Allows prediction of project performance Leading vs. lagging indicators High degree of process control (e.g. six sigma) Extensive timelines (improved by lean)

11 The CMMI Three models, or sets of domain-specific best practices - Development (systems, software, hardware) - Services - Acquisition Addresses management as well as mission functions Robust organizational framework for improvement - Improvement, quality assurance, metrics, decision analysis, configuration management, causal analysis, supplier mgt. Robust benchmarking mechanisms (SCAMPI appraisals) Well defined increments of process maturity and capability - Capability levels - Maturity levels Available SEI-authorized training

12 What should a T&E improvement framework do? Energize consistent focus on and understanding of customer needs and priorities Provide an organizational infrastructure for improvement Provide framework for strategic value analysis Ensure any/all T&E service performance improvement ideas, concepts, technologies, etc. are encouraged and fairly analyzed Enable consistent implementation across test programs and domains Support persistence of improvements across time Support high velocity of improvements when appropriate Ultimately enable development of mature test services Provide benchmarking mechanism

13 Comparison of Systemic Attributes (10 highest, 1 lowest) Framework Supports Strategic Analysis Focus on Customer Needs Holistic Improvement Infrastructure Scalable Invokes Consistency Invokes Persistence Improvement Velocity Benchmarking ISO Lean Thinking Six Sigma TOC CMMI

14 The CMMI as an Integrating Improvement Framework As it turns out - Lean and Six Sigma shortcomings are identical to those found in other industries.. Software development Systems development IT service management Engineering services Capability Maturity Model Integration (CMMI) - Provides a robust framework for performance improvement - Corrects Lean and Six Sigma deficiencies - Provides an extensible appraisal method Lesson Learned from the engineering and IT services communities Lean and Six Sigma efforts carried out in a CMMI framework result in a lean, flexible, cost efficient, strategically and tactically responsive improvement framework

15 How the CMMI works with Lean and Six Sigma CMMI practices for service development and operation - Provides framework for focus and development of test services - Sharpens focus on services CMMI practices for service management - Provides framework for mature management and leadership CMMI practices for performance improvement - Provides framework for mature, consistent improvement strategies - Provides framework for the organization for improvement CMMI support processes - Ensure the consistence implementation of processes and the persistence of results CMMI appraisal methods - Provide benchmarks of process maturity and potential for persistent performance improvements

16 Examples of Recent Lean/6 σ CMMI-Based Improvement Efforts Software development organization - Adopted CMMI for Development - Implemented Lean Software Development - Achieved 50% reduction in waste - Improved product throughput by 10 fold - Achieved performance improvement goals in 1/3 of expected time IT Services organization - Adopted CMMI for Services - Implemented Information Technology Infrastructure Library (ITIL) - Implemented Lean operations - Re-organized to focus on customer service - Now recognized for superior IT operations Software development organization - Adopted CMMI for development - Implemented Lean Software Development - Achieved 75% reduction in waste - Improved product throughput by 300%

17 The CMMI for Services Service Establishment and Delivery Service System Development Service System Transition Service Delivery Incident Resolution and Prevention Strategic Service Management Process Management Organizational Process Focus Organizational Process Definition Organizational Process Performance Organizational Process Management Organizational Training Project and Work Management Work Planning Work Monitoring and Control Integrated Work Management Quantitative Work Management Supplier Agreement Management Requirements Management Risk Management Capacity and Availability Management Service Continuity Support Configuration Management Process and Product Quality Assurance Measurement and Analysis Decision Analysis and Resolution Causal Analysis and Resolution

18 How The is the CMMI for for Test Services and Evaluation Structured? (mental model on (The how Process the CMMI-SVC Areas) can be applied) Service Tests Establishment Services Est. and Delivery Service Test System System Development Development Service Test System System Transition Transition Service Test Operations Delivery Incident Resolution and and Prevention Strategic Service Management Management of Test Services Test Process Management Organizational Process Focus Organizational Process Definition Organizational Process Performance Organizational Process Management Organizational Training Project Test Program and Work Management Work Test Planning Work Test Program Monitoring Monitoring and Control and Control Integrated Work Test Management Quantitative Work Test Management Supplier Agreement Management Test Requirements Management Management Risk Management Capacity and Availability Management Service Continuity Continuity of Test Services Test Support Configuration Management Process and Product Quality Assurance Measurement and Analysis Decision Analysis and Resolution Causal Analysis and Resolution

19 Advantages of CMM-Based Integrated Performance Improvement Framework for Test and Evaluation Improves test efficiency and velocity Motivates focus on Test Services across the organization Enables a high degree of consistency in test operations Enables improvements to persist over time (process stability) Improves communication among stakeholders Promotes management approaches that protect workers from chaos Promotes management that focuses on enabling multi-functional teams to provide Test Services Provides mature practices for identification and mitigation of Test Service risks Enables measurement and subsequent analysis of key parameters across the test service life cycle Enables persistent improvement in quality of test services and work products Enables management of test service configurations Enables the achievement of business objectives Provides synergistic framework for Lean and Six Sigma (and others)

20 Application of Lean/CMMI/ 6σ to T&E Organizations Government test organizations - Enables substantial increases in focus on customer needs across the test life cycle - Improves efficiency and focus on supporting field test organizations - Improves effectiveness in acquisition of test services from contractors - Increases velocity of test planning and delivery cycle Contractor test services - Enables articulation and definition of test services - Enables measurable improvement of test services - Enables process-capable performance measurement Test Teams - DT/OT - Gov/Contractor

21

22 For further information Contact - Jeffrey L. Dutton - jeff@easpi.com - Off: Cell Request a (free*) seminar or workshop on this topic. See. Or - (this briefing is available on line)

23 Next Month..