Risk Management Guide for DoD Acquisition

Transcription

1 Risk Management Guide for DoD Acquisition March 1998 DEPARTMENT OF DEFENSE DEFENSE ACQUISITION UNIVERSITY DEFENSE SYSTEMS MANAGEMENT COLLEGE PUBLISHED BY THE DEFENSE SYSTEMS MANAGEMENT COLLEGE PRESS FORT BELVOIR, VIRGINIA

2 Please comments on recommended changes to: ii

3 OFFICE OF THE UNDER SECRETARY OF DEFENSE 3000 DEFENSE PENTAGON WASHINGTON, DC RISK MANAGEMENT GUIDE Acquisition reform has changed the way the Department of Defense designs, develops, manufactures, and supports systems. Our technical, business, and management approach for acquiring and operating systems has, and continues to, evolve. For example, we no longer can rely on military specifications and standards to define and control how our developers design, build and support our new systems. Today we use commercial hardware and software, promote open systems architecture, and encourage streamlining processes, just to name a few of the initiatives that affect the way we do business. At the same time, the Of flee of the Secretary of Defense has reduced the level of oversight and review of programs and manufacturers plants. While the new acquisition model gives government Program Managers and their contractors broader control and more options than they have enjoyed in the past, it also exposes them to new risks. OSD recognizes that risk is inherent in any acquisition program and considers it essential that Program Managers take appropriate steps to manage and control risks. In late 1996, the Under Secretary of Defense, Acquisition and Technology [USD(A&T)] tasked the Director, Test, Systems Engineering, and Evaluation (DTSE&E) to review DoD risk management practices and techniques. In response, DTSE&E/Systems Engineering established a Risk Management Working Group that examined the Services, individual acquisition programs, and commercial industry s treatment of risk. The results of the study served as the basis for the risk management section (2.5.2) in the Defense Acquisition Deskbook. The study also identified the need to update existing risk training material to reflect the new way DoD conducts business. iii

4 This document is a product of a joint effort among the DTSE&E, the Defense Acquisition University, and the Defense Systems Management College (DSMC). It is Based on the material developed by the DoD Risk Management Working Group, included In the Defense Acquisition Deskbook. Mark D. Schaeffer Deputy Director, Test, Systems Engineering and Evaluation/ Systems Engineering Thomas M. Crean President Defense Acquisition University Lenn Vincent RADM, SC, USN Commandant Defense Systems Management College iv

5 PREFACE In December 1995, the Under Secretary of Defense, Acquisition and Technology [USD(A&T)], issued a memorandum entitled Reducing Life Cycle Costs for New and Fielded Systems, in which he established the policy and strategy to develop and field affordable weapon systems that are responsive to user s needs. One of the foundations of the strategy is the concept of Cost as An Independent Variable (CAIV), the Department of Defense (DoD) equivalent of commercial best practices. The CAIV concept recognizes that there are risks to be taken and risks to be avoided. When risks are taken, we will put in place appropriate risk management and contingency plans. Other initiatives, such as acquisition streamlining and revision of the DoD 5000 series documents, were ongoing when the USD(A&T) memorandum was published; each affected program risk. Also at this time, the DoD Inspector General was writing a critical report of the Department s management of risk; the report recommended measures to control risk of acquisition programs. Figure P-1 shows some of the initiatives that impact risk management. Figure P-1. DoD Renewed Emphasis on Risk Management With these initiatives as the basis, the USD(A&T) tasked the Director, Test, Systems Engineering, and Evaluation (DTSE&E) to (1) review DoD risk management practices and techniques, (2) determine whether new approaches were needed to improve risk management, and (3) report the results to USD(A&T). In response, DTSE&E established a Risk Management Working Group composed of members of the OSD staff, representatives of the Services, and members of other DoD agencies v

6 involved in systems acquisition. This group reviewed pertinent DoD directives (DoDD) and regulations, examined how the Services managed risk, studied various examples of risk management by companies in commercial industry, and looked at DoD training and education activity in risk management. The Working Group coordinated with other related efforts in DoD. For example, the Joint Aeronautical Commanders Group Risk Guide was a valuable source of information. The workshops for the CAIV Flagship programs provided current, real-world examples of Program Mangers implementing the CAIV initiative and risk management. Membership of the Working Group included a representative from USD(A&T) Acquisition Program Integration/Program Management (API/PM) who kept members informed on the status of the Integrated Program Management Initiative. Other sources of information were the Software Engineering Institute Risk Initiative, the Open Systems Initiative, and Safety and Cost Estimating communities. DTSE&E summarized the findings of the investigation and presented the results to the USD(A&T) in July The findings and recommendations of the Working Group are summarized below. vi

7 DTSE&E briefed the results to the Defense Manufacturing Council, an advisory body to the USD(A&T), which directed that the recommendations be incorporated in the Defense Acquisition Deskbook. Following that guidance, DTSE&E wrote the risk management portions of the Deskbook. The Risk Deskbook write-up forms the basis for this guide. The goal of the Risk Management Guide is to provide acquisition professionals and program management offices with a reference for dealing with system acquisition risks. It has been designed as an aid in classroom instruction and as a reference for practical applications. This guide reflects the efforts of many people. Mr. Mark Schaeffer, Deputy Director, Systems Engineering, DTSE&E, who chaired the Risk Management Working Group and Mr. Mike Zsak, from the DTSE&E, Systems Engineering Support Office, were the driving force behind the risk management initiative. Paul McMahon and Bill Bahnmaier from the DSMC faculty and DSMC Press guided the composition of the Guide. Special recognition goes to the Institute for Defense Analyses team composed of Louis Simpleman, Ken Evans, Jim Lloyd, and Gerald Pike, who compiled the data and wrote major portions of the text. vii

8 viii

9 CONTENTS Chapter 1 INTRODUCTION Purpose and Scope Organization of the Guide Approach to Risk Management DoD Risk Management Policies and Procedures... 2 Chapter 2 RISK AND RISK MANAGEMENT Introduction Overview Risk Management Structure and Definition Risk Discussion Characteristics of Acquisition Risk Program Products, Processes, Risk Areas, and Risk Events Risk Planning Purpose of Risk Plans Risk Planning Process Risk Assessment Purpose of Risk Assessments Risk Assessment Process Timing of Risk Assessments Conducting Risk Assessments Risk Dating Risk Handling Purpose of Risk Handling Risk Handling Process Risk Monitoring Risk Documentation Chapter 3 RISK MANAGEMENT AND DOD ACQUISITION PROCESS Introduction Overview DoD Acquisition Process Characteristics of the Acquisition Process Integrated Product and Process Development (IPPD) Continuous Risk Management Program Stability Reduction of Life-Cycle Costs Event-Oriented Management Modeling and Simulation Risk Management Activities during Acquisition Phases Phase Subsequent Phases ix

10 3.6 Risk Management and Milestone Decisions Risk Management and the Acquisition Strategy Risk Management and CAIV Chapter 4 RISK MANAGEMENT AND PROGRAM MANAGEMENT Introduction Overview Program Manager and Risk Management Risk Management is a Program Management Tool Risk Management is a Formal Process Risk Management is Forward-Looking Risk Management is Integral to Integrated Product and Process Development (IPPD) Risk Management Organization in the PMO Risk Management Organizational Structure Risk Management Responsibilities Contractor Risk Management Contractor View of Risk Government/Contractor Relationship Risk Management and the Contractual Process Risk Management: Pre-Contract Award Early Industry Involvement: Industrial Capabilities Review Developing the Request for Proposal The Offeror s Proposal Basis for Selection Source Selection Risk Management: Post-Contract Award Risk Management Reporting and Information System Risk Management Training Chapter 5 RISK MANAGEMENT TECHNIQUES Introduction Overview Risk Planning Techniques Description Procedures Risk Assessment Techniques Product (WBS) Risk Assessment Process (D0D M) Risk Assessment Program Documentation Evaluation Risk Identification Threat and Requirements Risk Assessment Cost Risk Assessment Quantified Schedule Risk Assessment Expert Interview Analogy Comparison/Lessons-Learned Studies x

11 5.5 Risk Prioritization Description Procedures Risk-Handling Techniques General Risk Avoidance Risk Transfer Risk Control Risk Assumption Risk Monitoring General Earned Value Management Technical Performance Measurement Integrated Planning and Scheduling Watchlist Reports Management Indicator Systems Risk Management Information Systems and Documentation Risk Management Reports Software Risk Management Methodologies Software Risk Evaluation (SRE) Boehm s Software Risk Management Best Practices Initiative Risk Management Method APPENDIX A DOD RISK MANAGEMENT POLICIES AND PROCEDURES... A-1 1. DoD Defense Acquisition, March A-1 2. DoD Regulation R. Mandatory Procedures for Major Defense Acquisition Programs (MDAPs) and Major Automated Information System (MAIS) Acquisition Programs, March 15, A-2 3. DoD Directive (DoDD) , OSD Cost Analysis Improvement Group (CAIG), November 24, A-6 4. DoD M Cost Analysis Guidance and Procedures, December A-6 APPENDIX B GENERIC RISK MANAGEMENT PLAN... B-1 Sample Risk Management Plan... B-1 Preface... B-1 Sample Format for Risk Management Plan... B-2 Sample Risk Management Plan for the XYZ Program... B Introduction... B Risk Management... B Risk Planning... B Risk Management Information System and Documentation... B-21 Annex A Critical Program Attributes... B-23 xi

12 Annex B Program Risk Reduction Schedule... B-24 Annex C Program Metric Examples... B-25 Annex D Management Information System and Documentation... B-27 APPENDIX C GLOSSSARY... C-1 APPENDIX D BIBLIOGRAPHY... D-1 FIGURES 2-1 Risk Management Structure Transition from Development to Production Critical Process Areas and Templates A Risk Management Plan Format Risk Assessment Example of a WBS Dependent Evaluation Structure Decentralized Risk Management Organization Risk Planning Technique Inputs and Output Sample Format for Risk Management Plan Product (WBS) Risk Assessment Techniques Inputs and Outputs Process (DoD M) Risk Assessment Technique Inputs and Outputs Plan Evaluation Technique Inputs and Outputs Phase 0 Correlation of Selected Documents (Example) Threat and Requirement Risk Assessment Technique Inputs and Outputs Cost Risk Assessment Top-Level Diagram Schedule Risk Assessment Technique Inputs and Outputs Expert Interview Technique Inputs and Outputs Analogy Comparison/Lessons-Learned Studies Top-Level Diagram Risk Prioritization Technique Inputs and Outputs Risk Aggregation Techniques Inputs and Outputs Sample of a List of Prioritized Risks Example Showing Detailed List of Top-Level Risk Information Example of More Complex Combination of Risk Level and Scheduled Tasks Conceptual Risk Management and Reporting System Appendix B 2-1 Risk Management and the Acquisition Process... B-27 B-1 Conceptual Risk Management and Reporting System... B-27 xii

13 TABLES 2-1 Risk Assessment Approaches Likelihood Criteria (Example) Consequences Criteria (Example) Overall Risk Rating Criteria (Example) Risk Ratings (Example) Overall Risk Rating (Example) Notional Description of Risk Management Responsibilities Significant Risks by Critical Risk Areas Risk Management Reference Documents Critical Risk Areas and Example Elements Examples of Demonstration Events Watchlist Examples Examples of Product-Related Metrics Examples of Product Metrics Examples of Cost and Schedule Metrics Data Base Management Systems Elements Software Risk Management Steps Top 10 Software Risks Best Practices Initiative Risk Management Method xiii

14 xiv

15 1 INTRODUCTION Risk has always been a concern in the acquisition of DoD systems. The acquisition process itself is designed, to a large degree, to allow risks to be controlled from conception to delivery of a system. Unfortunately, in the past, some Program Managers and decision makers have viewed risk as something to be avoided. Any program that had risk was subject to intense review and oversight. This attitude has changed. DoD managers recognize that risk is inherent in any program and that it is necessary to analyze future program events to identify potential risks and take measures to handle them. Risk management is concerned with the outcome of future events, whose exact outcome is unknown, and with how to deal with these uncertainties, i.e., a range of possible outcomes. In general, outcomes are categorized as favorable or unfavorable, and risk management is the art and science of planning, assessing, and handling future events to ensure favorable outcomes. The alternative to risk management is crisis management, a resource-intensive process that is normally constrained by a restricted set of available options. 1.1 PURPOSE AND SCOPE This Risk Management Guide is designed to provide acquisition professionals and program management offices with a reference book for dealing with system acquisition risks. It is intended to be useful as an aid in classroom instruction and as a reference book for practical applications. Most of the material in this guide is derived from the Defense Acquisition Deskbook. Readers should refer to Paragraph of the Deskbook for any new information. 1.2 ORGANIZATION OF THE GUIDE The Risk Management Guide discusses risk and risk management, defines terms, and introduces basic risk management concepts (Chapter 2). Chapter 3 examines risk management concepts relative to the DoD acquisition process. It illustrates how risk management is an integral part of program management, describes interaction with other acquisition processes, and identifies and discusses the various types of acquisition risks. Chapter 4 discusses the implementation of a risk management program from the perspective of a program management office (PMO). This chapter focuses on practical application issues such as risk management program design options, PMO risk management organizations, and criteria for a risk-management information system (MIS). Chapter 5, the final chapter, describes a number of techniques that address the aspects of risk management, i.e., planning, identifying, analyzing, handling, and monitoring. This Guide is a source of background information and provides a starting point for a 1

16 risk management program. None of the material is mandatory, Program Managers should tailor the approaches and techniques to fit their programs. The Risk Management Guide also contains appendices that are intended to serve as reference material and provide backup detail for some of the concepts presented in the main portion of the Guide. 1.3 APPROACH TO RISK MANAGEMENT Based on the DoD model contained in the Defense Acquisition Deskbook (DAD) (described in Chapter 2), this Guide emphasizes a risk management approach that is disciplined, forward looking, and continuous. In 1986, the Government Accounting Office (GAO), as part of an evaluation of DoD policies and procedures for technical risk assessments, developed a set of criteria as an approach to good risk assessments. These criteria, with slight modification, apply to all aspects of risk management and are encompassed in the Guide s approach. They are: (1) Planned Procedures. Risk management is planned and systematic. (2) Prospective Assessment. Potential future problems are considered, not just current problems. (3) Attention to Technical Risk. There is explicit attention to technical risk. (4) Documentation. All aspects of the risk management program are recorded and data maintained. (5) Continual Process. Risk assessments are made throughout the acquisition process; handling activities are continually evaluated and changed if necessary; and critical risk areas are always monitored. 1.4 DOD RISK MANAGEMENT POLICIES AND PROCEDURES DoD policies and procedures that address risk management for acquisition programs are contained in four key DoD documents. DoDD contains the policy on risk management and is amplified further by the information in DoD R. The latter document integrates risk management into the acquisition process, describes the relationship between risk and various acquisition functions, and establishes some reporting requirements. DoDD and DoD M address risk and cost analysis guidance as they apply to the Office of the Secretary of Defense. Appendix A is an extract of existing risk management policies and procedures from all of these documents. The DoD 5000 series contains strong statements on risk management but requires elaboration to help the Program Manager (PM) establish an effective risk management program. The information furnished in the Risk Management section of the Deskbook supports and expands the contents of the DoD 5000 series. The DoD risk management policies and procedures provide the basis for this Guide, which complements the Deskbook by elaborating on risk management concepts and by providing greater detail for applying techniques. 2

17 2 RISK AND RISK MANAGEMENT 2.1 INTRODUCTION This Chapter introduces the concepts of risk and risk management by explaining the DoD risk-related definitions and by identifying the characteristics of acquisition risks. It also presents and discusses a structured concept for risk management and its five subordinate processes. 2.2 OVERVIEW The DoD risk management concept is based on the principles that risk management must be forward-looking, structured, informative, and continuous. The key to successful risk management is early planning and aggressive execution. Good planning enables an organized, comprehensive, and iterative approach for identifying and assessing the risk and handling options necessary to refine a program acquisition strategy. To support these efforts, assessments should be performed as early as possible in the life cycle to ensure that critical technical, schedule, and cost risks are addressed with mitigation actions incorporated into program planning and budget projections. PMs should update program risk assessments and tailor their management strategies accordingly. Early information gives them data that helps when writing a Request for Proposal and assists in Source Selection planning. As a program progresses, new information improves insight into risk areas, thereby allowing the development of effective handling strategies. The net result promotes executable programs. Effective risk management requires involvement of the entire program team and also requires help from outside experts knowledgeable in critical risk areas (e.g., threat, technology, design, manufacturing, logistics, schedule, and cost). In addition, the risk management process should cover hardware, software, the human element, and integration issues. Outside experts may include representatives from the user, laboratories, contract management, test, logistics, and sustainment communities, and industry. Users, essential participants in program trade analyses, should be part of the assessment process so that an acceptable balance among performance, cost, schedule, and risk can be reached. A close relationship between the Government and industry, and later with the selected contractor(s), promotes an understanding of program risks and assists in developing and executing the management efforts. Successful risk management programs generally have the following characteristics: Feasible, stable, and well-understood user requirements and threat; A close relationship with user, industry, and other appropriate participants; A planned risk management process, integral to the acquisition process; 3

18 An acquisition strategy consistent with risk level and risk-handling strategies; Continual reassessment of program and associated risks; A defined set of success criteria for all performance, schedule, and cost elements, e.g., Acquisition Program Baseline (APB) thresholds; Metrics to monitor effectiveness of risk-handling strategies; Effective Test and Evaluation Program; Formal documentation. PMs should follow the guidelines below to ensure that a management program possesses the above characteristics. Assess program risks, using a structured process, and develop strategies to manage these risks throughout each acquisition phase. Identify early and intensively manage those design parameters that critically affect capability, readiness, or cost. Use technology demonstrations/modeling/simulation and aggressive proto-typing to reduce risks. Use test and evaluation as a means of quantifying the results of the risk-mitigation process. Include industry and user participation in risk management. Use Developmental Test and Evaluation (DT&E) and early operational assessments when appropriate. Establish a series of risk assessment reviews to evaluate the effectiveness of risk handling against clearly defined success criteria. Establish the means and format to communicate risk information and to train participants in risk management. Prepare an assessment training package for members of the program office and others, as needed. Acquire approval of accepted risks at the appropriate decision level. In general, management of software risk is the same as management of other types of risk and techniques that apply to hardware programs are equally applicable to software intensive programs. However, some characteristics of software make this type of risk management different, primarily because it is difficult to: Identify software risk. Estimate the time and resources required to develop new software, resulting in potential risks in schedule and cost. Test software completely because of the number of paths that can be followed in the logic of the software. Develop new programs because of the rapid changes in information technology and an ever-increasing demand for quality software personnel. 2.3 RISK MANAGEMENT STRUCTURE AND DEFINITIONS Although each risk management strategy depends upon the nature of the system being developed, research reveals that good strategies contain the same basic processes and structure shown in Figure 2-1. The application of these processes vary with acquisition phases and the degree of system definition; all should be integrated into the program management function. The elements of the structure are discussed in the following paragraphs of this Chapter; however, in order to form a basis for discussion, the Defense Acquisition Deskbook definitions for the processes and elements of risk management include: 4

19 Figure 2-1. Risk Management Structure Risk is a measure of the potential inability to achieve overall program objectives within defined cost, schedule, and technical constraints and has two components: (1) the probability of failing to achieve a particular outcome and (2) the consequences of failing to achieve that outcome. Risk management is the act or practice of dealing with risk. It includes planning for risk, assessing risk areas, developing risk-handling options, monitoring risks to determine how risks have changed, and documenting the overall risk management program. Risk planning is the process of developing and documenting an organized, comprehensive, and interactive strategy and methods for identifying and tracking risk areas, developing risk-mitigation plans, performing continuous risk assessments to determine how risks have changed, and assigning adequate resources. Risk events, things that could go wrong for a program or system, are elements of an acquisition program that should be assessed to determine the level of risk. The events should be defined to a level that an individual can comprehend the potential impact and its causes. For example, a potential risk event for a turbine engine could be turbine blade vibration. There could be a series of potential risk events that should be selected, examined, and assessed by subject matter experts. The relationship between the two components of risk, probability and consequence, is complex. To avoid obscuring the results of an assessment, the risk associated with an event should be characterized in terms of its two components. There is still a need for backup documentation containing the supporting data and assessment rationale. Risk assessment is the process of identifying and analyzing program areas and critical technical process risks to increase the likelihood of meeting performance, schedule, and cost objectives. Risk identification is the process of examining the program areas and each critical technical process to identify and document the associated risk. Risk analysis is the process of examining each identified risk area or process to refine the description of the risk, isolating the cause, and determining the effects. Risk priority is defined in terms of its probability of occurrences, its consequences, and its relationship to other risk areas or processes. Risk handling is the process that identifies, evaluates, selects, and implements op- 5

20 tions in order to set risk at acceptable levels given program constraints and objectives. This includes the specifics on what should be done, when it should be accomplished, who is responsible, and associated cost. The most appropriate strategy is selected from these handling options. For purposes of the Guide, risk handling is an all-encompassing term whereas risk mitigation and risk control are subsets of risk handling. Risk monitoring is the process that systematically tracks and evaluates the performance of risk-handling actions against established metrics throughout the acquisition process and develops further riskhandling options, as appropriate. Risk documentation is recording, maintaining, and reporting assessments, handling analysis and plans, and monitoring results. It includes all plans, reports for the Program Manager and decision authorities, and reporting forms that may be internal to the program management office. 2.4 RISK DISCUSSION Implicit in the definition of risk is the concept that risks are future events and that there is uncertainty associated with the program if these events occur. Therefore, there is a need to determine, as much as possible, the probability of a risk event occurring and to estimate the impact (consequences) if it occurs. The combination of these two factors determines severity. For example, an event with a low probability of occurring, yet with severe consequences, may be a candidate for handling. Conversely, an event with a high probability of happening, but the consequences of which do not affect a program, may be acceptable and require no handling. To reduce uncertainty and apply the definition of risk to acquisition programs, PMs must be familiar with the types of acquisition risks, understand risk terminology, and know how to measure risk. These topics are addressed in the next several sections Characteristics of Acquisition Risk Acquisition programs tend to have numerous, often interrelated, risks. They are not always obvious; relationships may be obscure; and they may exist at all program levels throughout the life of a program. Risks are in the PMO (program plans, etc.); in support provided by other Government agencies; in threat assessment; and in prime contractor processes, engineering and manufacturing processes, and technology. The interrelationship among risk events may cause an increase in one because of the occurrence of another. For example, a slip in schedule for an early test event may adversely impact subsequent tests, assuming a fixed period of test time is available. Another important risk characteristic is the time period before a risk future event occurs; because time is critical in determining riskhandling options. If an event is imminent, the PMO must resort to crisis management. An event that is far enough in the future to allow management actions may be controllable. The goal is to avoid the need to revert to problem solving by managing risk. An event s probability of occurrence and consequences may change as the development process proceeds and information becomes available. Therefore, throughout the development phase, PMOs should reevaluate known risks on a periodic basis and examine the program for new risks Program Products, Processes, Risk Areas, and Risk Events Program risk includes all risk events and their relationships to each other. It is a toplevel assessment of impact to the program 6

21 when all risk events at the lower levels of the program are considered. Program risk may be a roll-up of all low-level events; however, most likely, it is a subjective evaluation of the known risks by the PMO, based on the judgment and experience of experts. Identifying program risk is worthwhile because it forces the PMO to consider relationships among all risks and may identify potential areas of concern that would have otherwise been overlooked. One of the greatest strengths of a formal, continuous risk management process is the proactive quest to identify risk events for handling and the reduction of uncertainty that results from handling actions. A program office has continuous demands on its time and resources. It is, at best, difficult, and probably impossible, to assess every potential area and process. To manage risk, the PMOs should focus on the critical areas that could affect the outcome of their programs. Work Breakdown Structure (WBS) product and process elements and industrial engineering and manufacturing processes contain most of the significant risk events. Risk events are determined by examining each WBS element and process in terms of sources or areas of risk. Following are some typical risk areas: Threat. The sensitivity of the program to uncertainty in the threat description, the degree to which the system design would have to change if the threat s parameters change, or the vulnerability of the program to foreign intelligence collection efforts (sensitivity to threat countermeasure). Requirements. The sensitivity of the program to uncertainty in the system description and requirements except for those caused by threat uncertainty. Design. The ability of the system configuration to achieve the program s engineering objectives based on the available technology, design tools, design maturity, etc. Test and Evaluation. The adequacy and capability of the test and evaluation program to assess attainment of significant performance specifications and determine whether the systems are operationally effective and suitable. Modeling and Simulation (M&S). The adequacy and capability of M&S to support all phases of a program using verified, valid, and accredited M&S. Technology. The degree to which the technology proposed for the program has been demonstrated as capable of meeting all of the program s objectives. Logistics. The ability of the system configuration to achieve the program s logistics objectives based on the system design, maintenance concept, support system design, and availability of support resources. Production/Facilities. The ability of the system configuration to achieve the program s production objectives based on the system design, manufacturing processes chosen, and availability of manufacturing resources. Concurrency. The sensitivity of the program to uncertainty resulting from the combining or overlapping of life-cycle phases or activities. Capability of Developer. The ability of the developer to design, develop, and manufacture the system. The contractor should have the experience, resources, and knowledge to produce the system. Cost/Funding. The ability of the system to achieve the program s life-cycle support objectives. This includes the effects of budget and affordability decisions and the effects of inherent errors in the cost estimating technique(s) used (given that the technical requirements were properly defined). 7

22 Management. The degree in which program plans and strategies exist and are realistic and consistent. The Government s acquisition team should be qualified and sufficiently staffed to manage the program. Schedule. The adequacy of the time allocated for performing the defined developmental tasks. This factor includes the effects of programmatic schedule decisions, the inherent errors in the schedule estimating technique used, and external physical constraints. Critical risk processes are the developer s engineering and manufacturing processes which, historically, have caused the most difficulty during the development phases of acquisition programs. These processes are design, test, production, facilities, logistics, and management. Three of these processes are included in the critical risk areas and are addressed separately to emphasize that they focus on processes. DoD M, Transition from Development to Production, describes them using templates. See Figure 2-2 for an example of the template for product development. The templates are the result of a Defense Science Board task force, composed of Government and industry experts, who identified engineering processes and control methods to minimize risk in both Government and industry. The task force defined these critical events in terms of the templates, which are briefly discussed later. The figure also shows funding as a process that, unlike others, is a Government process. Additional areas, such as manpower, environmental impact, systems safety and health, and systems engineering, that are analyzed during program plan development provide indicators for additional risk. The PMO should consider these areas for early assessment since failure to do so could cause dire consequences in the program s latter phases. In addition, PMs should address the uncertainty associated with security an area sometimes overlooked by developers but addressed in the Acquisition System Protection (ASP) section of the Deskbook and Air Force Pamphlet ASPWG PH-1, Acquisition System Protection Program Work Book, September 1994, and in the Deskbook. Defense Systems Management College, Program Management Teaching Note, Acquisition Program Protection, 12 January However, in addition to the guidance given there, PMs must recognize that, in the past, classified programs have experienced difficulty in access, facilities, clearances, and visitor control. Failure to manage these aspects of a classified program could affect schedules. 2.5 RISK PLANNING Purpose of Risk Plans Risk planning is the detailed formulation of a program of action for the management of risk. It is the process to: Develop and document an organized, comprehensive, and interactive risk management strategy. Determine the methods to be used to execute a PM s risk management strategy. Plan for adequate resources. Risk planning is iterative and includes the activities to assess, control, monitor, and document the risk associated with a program. The result is the Risk Management Plan (RMP) Risk Planning Process The PMO should periodically review the plan and revise it, if necessary. Some events such as: (1) a change in acquisition strategy (2) preparation for a major decision point, (3) technical audits and reviews, (4) an update of other program plans, and (5) prepa- 8

23 Figure 2-2. Transition from Development to Production Critical Process Areas and Templates ration for a Program Objective Memorandum (POM) submission may drive the need to update an existing plan. Planning begins by developing and documenting a risk management strategy. Early efforts establish the purpose and objective, assign responsibilities for specific areas, identify additional technical expertise needed, describe the assessment process and areas to consider, delineate procedures for consideration of handling options, define a rating scheme, dictate the reporting and documentation needs, and establish report requirements and monitoring metrics. This planning should also address evaluation of the capabilities of potential sources as well as early industry involvement and program. The PM s strategy to manage risk provides the program team with direction and basis for planning. Initially formalized during a program s Concept Exploration Phase and updated for each subsequent program phase, the strategy should be reflected in the program s acquisition strategy, which with requirement and threat documents, known risks, and system and program characteristics are sources of information for PMO use to devise a strategy and begin developing a risk management plan. Since the program s risks are affected by 9

24 the Government and contractor team s ability to develop and manufacture the system, industry can provide valuable insight into this area of consideration. The plan is the roadmap that tells the Government and contractor team how to get from where the program is today to where the PM wants it to be in the future. The key to writing a good plan is to provide the necessary information so the program team knows the objectives, goals, and the PMO s risk management process. Since it is a map, it may be specific in some areas, such as the assignment of responsibilities for Government and contractor participants and definitions, and general in other areas to allow users to choose the most efficient way to proceed. For example, a description of techniques that suggests several methods for evaluators to use to assess risk is appropriate, since every technique has advantages and disadvantages depending on the situation. Appendix B contains an example of a risk plan and a summary of the format is shown in Figure 2-3. In a decentralized PMO risk management organization, the program s risk management coordinator may be responsible for risk management planning. See Sections 4.4, Risk Management Organizations, and 5.3, Risk Planning Techniques. 2.6 RISK ASSESSMENT Purpose of Risk Assessments The primary objective of assessments is to identify and analyze program risks so that the most critical among them may be controlled. Assessments are factors that managers should consider in setting technical, schedule, and cost objectives because they provide an indication of the likelihood of achieving the desired outcomes Risk Assessment Process Risk assessment is the problem definition stage of management that identifies, analyzes, and quantifies program events in terms of probability and consequences. The results form the basis for most risk management actions. It is probably the most difficult and time-consuming part of the management process. There are no quick answers or shortcuts. Tools are avail- Introduction Program Summary Definitions Risk Management Strategy and Approach Organization Risk Management Process and Procedures Risk Planning Risk Assessment Risk Handling Risk Monitoring Risk Management Information System, Documentation and Reports Figure 2-3. A Risk Management Plan Format 10

25 able to assist evaluators in assessing risk, but none are totally suitable for any program and may be misleading if the user does not understand how to apply them or interpret the results. Despite its complexity, risk assessment is one of the most important phases of the risk process because the caliber and quality of assessments determine the effectiveness of a management program. The components of assessment, identification analysis and prioritization, are performed sequentially with identification being the first step. Risk identification begins by compiling the program s risk events. PMOs should examine and identify program events by reducing them to a level of detail that permits an evaluator to understand the significance of any risk and identify its causes, i.e., risk drivers. This is a practical way of addressing the large and diverse number of potential risks that often occur in acquisition programs. For example, a WBS level 4 or 5 element may be made up of several risk events associated with a specification or function, e.g., failure to meet turbine blade vibration requirements for an engine turbine design. Risk events are best identified by examining each WBS product and process element in terms of the sources or areas of risk, as previously described in Paragraph Risks are those events that evaluators, (after examining scenarios, WBS, or processes), determine would adversely affect the program. Evaluators may initially rank events by probability of occurrence based on a judgment of consequence before beginning analysis to focus on those most critical. Risk analysis is a technical and systematic process to examine identified risks, isolate causes, determine the relationship to other risks, and express the impact in terms of probability and consequences. In practice, the distinction between risk identification and risk analysis is often blurred because there is some risk analysis that occurs during the identification process. For example, if, in the process of interviewing an expert, a risk is identified, it is logical to pursue information on the probability of it occurring, the consequences, the time associated with the risk (i.e., when it might occur), and possible ways of dealing with it. The latter actions are part of risk analysis and risk handling, but often begin during risk identification. Prioritization is the ranking of risk events to determine the order of importance. It serves as the basis for risk-handling actions. Integrated Product Teams (IPTs) typically perform risk assessments in a decentralized risk management organization as described in paragraph 4.4. If necessary, the team may be augmented by people from other program areas or outside experts. Paragraph 5.4, Risk Assessment Techniques, elaborates on this for each of the described assessment techniques Timing of Risk Assessments The assessment process begins during the last half of Phase 0, Concept Exploration, and continues throughout the subsequent phases. The PMO should continually reassess the program at increasing levels of detail as the program progresses through the acquisition phases and more information becomes available. There are, however, times when events may require new assessments, i.e., a major change in the acquisition strategy. Paragraph lists other events that could cause risk assessments to be performed. 11

26 2.6.4 Conducting Risk Assessments There is no standard approach to assessing risk because methods vary according to the technique employed, the phase of the program, and the nature of the program itself; however, some top-level actions are typically common to all methods. They are grouped in Figure 2-4 into pre-risk assessment activities, risk identification activities, and risk analysis activities Pre-Risk Assessment Activities. The risk management plan may describe the actions that compose this activity. Typically, a Figure 2-4. Risk Assessment 12

27 program-level IPT may conduct a quick-look assessment of the program to identify the need for technical experts (who are not part of the team) and to examine areas that appear most likely to contain risk. The program s risk coordinator, or an outside expert, may train the IPTs, focusing on the program s risk strategy, definitions, suggested techniques, documentation, and reporting requirements. Paragraph 4.9, Risk Management Training, provides some suggestions for training Risk Identification Activity. To identify risk events, IPTs should break down program elements to a level where they, or subject matter experts, can perform valid assessments. The information necessary to do this varies according to the phase of the program. During the early phases, requirement, threat documents, and acquisition plans may be the only program-specific data available. They should be analyzed to identify events that may have adverse consequences. A useful initial identification exercise is to perform a mission profile for the system as suggested in DoD M, Transition from Development to Production. Using this methodology, the developer creates a functional and environmental profile for the system and examines the low-level requirements that the system must meet to satisfy its mission requirements. The IPTs may then study these requirements to determine which are critical. For example, in an aircraft profile, it may be apparent that high speed is critical. If the speed requirement is close to that achieved by existing aircraft, this may not be a concern. However, if the speed is greater than that achieved by today s aircraft, it may be a critical risk area. Since aircraft speed depends, among other things, on weight and engine thrust, it would be desirable to enlist the help of a materials expert to address weight and an engine expert to assess engine-associated risk. Another method of decomposition is to create a WBS as early as possible in a program. Figure 2-5 is a simple example of a decomposition based on the WBS for an aircraft. The figure shows an important requirement of the decomposition process, the establishment of goals, e.g., don t exceed the weight budget or objective. Further it is beneficial at this time to estimate a cost distribution for each risk event. Figure 2-5. Example of a WBS Dependent Evaluation Structure 13

28 During decomposition, risks are identified from experience, brainstorming, lessons learned from similar programs, and guidance contained in the risk management plan. The examination of each event is an exploratory exercise to identify the critical risks. The investigation may show that risks are inter-related. For example, the weight of an aircraft affects its speed, but also impacts the payload, range, and fuel requirements. These have design and logistics consequences and may even affect the number of aircraft that must be procured to meet objectives. Critical risks need to be documented as specified in the risk management plan and may include the scenario that causes the risk, planned management controls and actions, etc. It may also contain an initial assessment of the consequences to focus the risk assessment effort Risk Analysis Activity. Analysis begins with a detailed study of the critical risks that have been identified. The objective is to gather enough information about the risks to judge the probability of occurrence and the impact on performance, cost, and schedule if the risk occurs. Impact assessments are normally subjective and based on detailed information that may come from: Table 2-1. Risk Assessment Approaches Comparisons with similar systems, Relevant lessons-learned studies, Experience, Results from tests and prototype development, Data from engineering or other models, Specialist and expert judgments, Analysis of plans and related documents, Modeling and simulation, Sensitivity analysis of alternatives. Depending on the particular technique and the risk being analyzed, some supporting analysis may be necessary, i.e., analysis of contractor processes, such as design, engineering, fault tree analysis, engineering models, simulation, etc. Analyses provide the basis for subjective assessments. A critical aspect of risk analysis is data collection. Two primary sources of data are interviews of subject matter experts and analogy comparisons with similar systems. Paragraph 5.4 contains a technique for collecting both types of data for use in support of the techniques listed in Table 2-1. Periodically, sets of risks need to be prioritized in preparation for risk handling, and 14

29 aggregated to support program management reviews. Paragraph 5.5, Risk Prioritization, describes methods for accomplishing this Risk Assessment by Risk Category. Each risk category, e.g., technical, cost, and schedule, includes a core set of assessment tasks and is related to the other two categories. This relationship requires supportive analysis among areas to ensure the integration of the assessment process. For example, a technical assessment probably should include a cost and schedule analysis in determining the technical risk impact. The results of the assessments, normally conducted by IPTs follow: Technical Assessment Provides technical foundation, Identifies and describes program risks, i.e., threat, technology, design, manufacturing, etc., Prioritizes risks with relative or quantified weight for program impact, Analyzes risks and relates them to other internal and external risks, Quantifies associated program activities with both time duration and resources, Quantifies inputs for schedule assessment and cost estimate, Documents technical basis and risk definition for the risk assessment. Schedule Assessment Evaluates baseline schedule inputs, Incorporates technical assessment inputs to program schedule model, Evaluates impacts to program schedule based on technical team assessment, Performs schedule analysis on program integrated master schedule, Quantifies schedule excursions reflecting effects of cost risks, including resource constraints, Provides Government schedule assessment for cost analysis and fiscal year planning, Reflects technical foundation, activity definition, and inputs from technical and cost areas, Documents schedule basis and risk impacts for the risk assessment. Cost Estimate and Assessment Builds on technical and schedule assessment results, Translates technical and schedule risks into cost, Derives cost estimate by integrating technical assessment and schedule risk impacts to resources, Establishes budgetary requirements consistent with fiscal year planning, Determines if the phasing of funds supports technical and acquisition approach, Provides program cost excursions from: - Near term budget execution impacts, - External budget changes and constraints. Documents cost basis and risk impacts Risk Rating Ratings are an indication of the potential impact of risks on a program; they are a measure of the likelihood of an event occurring and the consequences of the event. They are often expressed as high, moderate, and low. A group of experts, who are familiar with each risk area (e.g., design, logistics, produc- 15