Top Systems Engineering Issues In US Defense Industry

Transcription

1 National Defense Industrial Association Systems Engineering Division Task Group Report Top Systems Engineering Issues In US Defense Industry September 2010 Final-v11-9/21/2010 Background The NDIA Systems Engineering Division decided in early December 2009 to update the Top 5 Issues in Systems Engineering (SE) report that was issued initially in 2003 and updated in This was coordinated with the Director, Systems Engineering, Office of the Under Secretary of Defense, Acquisition Technology & Logistics, who serves as the primary OSD interface to the NDIA Systems Engineering Division. The issues related to our defense industry are complex, affecting both the industry participants as well as the government participants. A Task Group was formed, inputs were solicited in advance, and a reconciliation meeting with about 18 members was held on March 17 th and 18th, Although scores of separate issues were identified, the group found that the bulk of these actually fell into five major issue categories. The detailed results, including the status of the previous 2006 SE issues, are described below. Status of Activities against the Top Systems Engineering Issues for 2006 # 2006 Issue 2010 Status 1 Key systems engineering practices known to be effective are not consistently applied across all phases of the program life cycle. 2 Insufficient systems engineering is applied early in the program life cycle, compromising the foundation for initial requirements and architecture development. Institutionalization of practices has shown value when adopted but adoption tends to be spotty Determination of proficiency in applying practices appears to be problematic Improving by necessity in complex systems Policy updates (5000.2, competitive prototyping and earlier decisions) imply SE engagement, but are not explicit (subsequent revisions may have already addressed this or are in work) 1

2 3 Requirements are not always well-managed, including the effective translation from capability statements into executable requirements to achieve successful acquisition programs. 4 The quantity and quality of systems engineering expertise is insufficient to meet the demands of the government and the defense industry. 5 Collaborative environments, including SE tools, are inadequate to effectively execute SE at the joint capability, system of systems (SoS), and system levels. WSARA requirements for development planning are believe to be an improvement Variability in approaches to requirements definition, validation and consolidation continue Resource issues persist, in government and industry. Shortages: leadership, domain, architects, systems engineers Initiatives: acquisition workforce, STEM, crosstraining Value of having experience to enhance educated workforce is better understood State of the practice techniques not widely utilized Multiple tools are available but little guidance on preferences exists. Emphasis on SoS seems to have diminished Identification of the Top Systems Engineering Issues for 2010 The Systems Engineering Division reviewed the top Systems Engineering Issues from the previous two versions (2003 and 2006) and identified additional issues during the workshop. The issues were discussed, reviewed and assigned to affinity groups and then prioritized into the top 5 issues by voting and consensus. For each of the five issues, additional details and explanations were developed, and recommendations were gathered to address their correction or to drive improvement. Special attention was given to making the recommendations actionable and specific. One issue from 2003 and 2006 related to the workforce remains almost unchanged from the previous lists but the recommended actions are believed to be more focused and actionable in this version. One other issue from 2006 related to engineering practices remains but has been made more specific, hopefully enabling more precise and focused action. Actions assigned to the NDIA are intended to be executed by the Systems Engineering Division. Where possible, the NDIA SED will solicit the collaboration of other industry groups (e.g., INCOSE, IEEE, AIA, and AIAA) to address the actions, in order to try to have a greater influence across the SE community. The top 5 Systems Engineering issues as of 2010 were deemed to be the following. Please note that the issues are not listed in any priority. 1. Increasingly urgent demands of the warfighter are requiring effective capabilities to be fielded more rapidly than the conventional acquisition processes and development methodologies allow. 2. The quantity and quality of Systems Engineering expertise is insufficient to meet the demands of the government and defense industry 3. Systems engineering practices known to be effective are not consistently applied or properly resourced to enable early system definition 4. Technical decision makers do not have the right information & insight at the right time to support informed & proactive decision making or may not act on all the technical information available to ensure effective & efficient program planning, management & execution. 2

3 5. Lack of technical authority can impact the integrity of developed system and result in cost/schedule/system performance impacts as the technical solution is iterated and reworked in later stages of the development. Top 5 SE Issues and Discussion Points Issue 1: Increasingly urgent demands of the warfighter require effective capabilities be fielded more rapidly than the conventional acquisition processes and development methodologies allow. The following main points provide amplification of this issue: A quick response capability is needed to satisfy urgent mission operational needs, but may be encumbered by traditional full-up processes dictated by the Defense Acquisition System. In a rapid response environment, a strong SE capability along with appropriate Subject Matter Experts is crucial to make effective trades on technical approaches and processes according to the urgency of the need, characteristics of the effort, and to facilitate an explicit determination and assessment of the impact of risk (e.g., technical, sustainment, production, interoperability) and the level of risk acceptable to support a decision to field the capability. Design, development and deployment decisions (such as operational use of prototypes) have far-reaching impacts. The entire program life cycle impacts (e.g., planning, development, logistics, and sustainment) must be understood when making SE tradeoffs. Rapid response to prioritized needs depends heavily on a tightly integrated and capable team which includes engagement from needed functional stakeholders. Critical success factors for responding rapidly to urgent needs (e.g., stable architecture, leveraging existing capability, clear and stable description of the capability gap, domain expertise, reliable technology, highly capable team) are generally understood, but are not yet well-documented and used in decisionmaking for when rapid response may be feasible and appropriate. Issue 1 Recommendation: NDIA Systems Engineering Division to convene a joint government/industry study, with cross-functional representation including T&E, to develop risk-driven guidance on tailoring SE processes and activities and DoD acquisition requirements from the Defense Acquisition System to achieve rapid acquisition and deployment. Details of this recommendation include the following points to be addressed: 3

4 Industry experience in accelerated acquisition for different program types and constraints can be leveraged to provide input on characterizing the tradeoffs, critical success factors, and measures of effectiveness that may be applicable to specific program situations and associated potential life cycle impacts. There are inevitable SE tradeoffs required to balance all constraints (cost, schedule, technical maturity, process rigor, and sustainment) to make informed program decisions for a rapid response to urgent mission needs. Successful material solutions are a function of well documented architectural and design baselines of existing systems and components allowing effective reuse, integration, and verification in the extension of system functionality. Key government stakeholders, such as the Rapid Fielding Office and Defense Portfolio Managers, should be engaged in developing these recommendations and ensuring their relevance and viability in a rapid acquisition environment. The make-up of the team and organizational approach need to be optimized for effective execution. Recommendations related to Issue 5 should be reviewed when considering this point. Issue 2: The quantity and quality of systems engineering expertise is insufficient to meet the demands of the government and defense industry Discussion Points: The following main points, many of which carry over from 2006, provide amplification on this issue: Experienced, trained systems engineering personnel are in short supply, in both industry and government. This situation is exacerbated in government by the migration of systems engineers in mid-career due to inequities in compensation and incentives vs. industry. Program management and systems engineer frequent turnover and fewer opportunities for On the Job Training (OJT) are impediments to success. Inconsistent methods in industry and government to ensure the quality and competency of systems engineering personnel. Limited human resources with needed technical skills are available to fill an increasing demand for systems engineers. Inconsistent definition of SE roles and responsibilities results in inefficient use of government and industry resources. Interfaces between systems engineering and other disciplines are often not adequately addressed by the inexperienced technical workforce. Limited numbers of experienced engineers limit the opportunities for mentoring and developing the next generation of system engineers. Recommendations: 4

5 1) Develop systems engineering expertise through role definition, selection, training, career incentives, and broadening systems thinking into other disciplines. Detailed actions for this recommendation should include: a. NDIA to develop and publish specific system engineering role definition for the defense industry consistent with the SEBOK and competency models. b. Continue to work with major universities and professional societies to add an introductory course in SE in all undergraduate engineering and technical management degree programs (task for NDIA SED Education and Training (E&T) Committee). c. Work with major universities and recommend SE curricula to improve consistency across university programs and help achieve standardization of skill sets for graduates. (Task for NDIA SED E&T Committee) d. Work with DAU to provide SE as a mandatory skill set requirement for promotion into government program management, and include requirement for industry program management qualifications to include SE. e. Develop a reference SE Competency model that is useful across DoD and its contracting community. Work this in conjunction with INCOSE to establish a single model, if possible. f. Work with INCOSE to develop domain-specific extensions for SE certification that are related to the needs of the defense industry. Promote SE certification with the acquisition extension, a relevant domain extension, or both. 2) Establish systems engineering job codes or other indicators within government service to emphasize systems engineering in the recruiting of new hires. This recommendation should include the following action: a) Establish a working group under NDIA SED E&T Committee to evaluate approaches to require systems thinking training for other disciplines and program management to get the benefit of a systems perspective across domains and expand the systems engineering candidate pool 3) OSD should establish policy/guidance for services to include systems engineering expertise in RFP/proposal efforts and program start-up teams with reach-back capability. Consideration should be given to integrating this initiative with the OSD transition team concept. Discrete budgets should be allowed for the SE effort and fenced from cuts without a thorough review of the implications. a. Recommend that Industry emulate this approach where not mandated for ACAT 1 programs. 4) NDIA SED Education and Training Committee should investigate and recommend specific methods and techniques to accelerate the training and development of Systems Engineers to include purposeful OJT rotation programs to complement course work. 5

6 5) Provide contract incentives to increase the development of systems engineers on programs. Issue 3: Systems engineering practices known to be effective are not consistently applied or properly resourced to enable early system definition Discussion Points: The following main points provide amplification on this issue: Without proper and comprehensive SE, PMs and other decision makers (i.e., Milestone Decision Authorities (MDAs)) do not know what information they lack in terms of technical issues and risks PMs and MDAs do not always have the information they need to make informed decisions Programs do not always start with good requirements/capabilities Programs do not always capture evolving requirements adequately and determine the impacts of the evolving requirements changes Programs do not always present information to decision makers in a manner that conveys fully potential impacts the program under review will have upon other programs and their capabilities. (e.g., Systemsof- Systems environment) There are no consistently applied methods for presenting key SE information to decision makers. This constraint is compounded by a high turnover rate of PMs and those in the acquisition decision chain. It is difficult to use currently available standard SE tools early in life cycle (MDD/MS-A timeframe). In addition, many of the tools are not readily available and the engineers have not been trained in their use. How to perform disciplined SE is known, while its consistent application is not. Price to Win contract strategies force tradeoffs in the resources allocated to SE process during proposal and execution. It is unclear if the government at all levels of the acquisition process can detect that these tradeoffs have been made and understand the near and long term program and technical management considerations of these tradeoffs. The same is true for prime contractors and their subcontractors. While substantial improvements have been made to emphasize systems engineering, many decision makers are still somewhat unaware of the value of SE, and activities which cannot be readily measured are often not performed. Performing SE activities prior to Milestone A does not guarantee that a program will succeed, but not performing early SE almost always guarantees it will fail1. NOTE: The recent emphasis on performing SE is 1 Pre Milestone A and Early Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Acquisition. Available online: 6

7 clearly noted, although the effect may not show up immediately and continued emphasis is strongly recommended to help assure success If you don t understand the fundamentals, checklists will not help. Experience is critical putting more emphasis on OJT rotation as part of systems engineering training and development. Affordability and associated trades are not always performed. In many cases there are disincentives to perform the full life cycle affordability trades. Affordability strategy should be mandatory for all programs and included in the government s acquisition strategy. Schedules tend to drive what should be event-driven reviews. Fiscal year pressures enhanced by the DoD requirement to move through acquisition phases and product the first article tend to emphasize reviews driven by schedule instead of having met predefined entry criteria. Issue 3 Recommendations: Identify key SE information that decision makers need and develop consistent methods for presenting that information. Establish a working group to determine why proven tools and techniques are not consistently applied. To create a receptive audience to gain full value from these recommendations to be effective, implementation of Issue 2 recommendations are encouraged. Establish a working group to develop consistent methods for presenting key SE information to decision makers, such as program managers (PMs) and Milestone Decision Authorities (MDAs). Examples include developing SE WBS guidance that enables a consistent SE methodology with standard definitions of tasks and deliverables starting prior to Milestone A and continuing through program execution. Develop a recommended template for presenting key SE information, including activities, value/expected results, risk of not performing the activities, and future consequences. The goal is to ensure that programs start with good requirements/capabilities and capture evolving requirements and interfaces. Make the SE discipline a required process in the transition from S&T to early concept development. Issue 4: Decision makers do not have the right information at the right time to support informed and proactive decision making that ensures effective and efficient program planning, management and execution. Discussion Points: Predictive measures 7

8 We seek to be able to identify & use key leading indicators to predict technical success based on program-unique success factors. Currently, most programs lack consistent application of value-added measures that provide predictive insight and the ability to effectively package and communicate the impact to decision makers -- across the life cycle based on information needs of decision makers and that change behaviors. Scientific based We seek to be able to collect and analyze industry quantitative and qualitative information on SE methods, practices, and solution approaches to characterize them for effective and efficient usage under defined conditions. Decisions are often not based upon engineering practice, methods, and approaches. Decision makers often lack the confidence in the practices and in the people applying them. Roadmap to address systemic issues We seek to be able to identify and address systemic program technical issues in order to elevate the priority of attention given to trying to find possible solutions. This should be able to be applied within or across organizational entities. Need data from items 1 & 2 in order to accomplish this. We need to identify the program-specific information needs and cause and effect factors, measure them, and makes sense of the patterns in order to inform inprocess decisions. Issue 4 Recommendations: (1) NDIA will create a roadmap that programs can use to facilitate the definition and implementation of a technical measurement process to support engineering management decisions. The roadmap should be coordinated with INCOSE Measurement Working Group and PSM. The roadmap would provide suggestions for packaging and communicating measurement results into information products that decision makers can use The roadmap would provide information on combinations of measures that often go hand-in-hand in providing insight (focusing on leading and predictive insight) i.e., that give the decision maker a richer set of related/correlated information The roadmap would provide reference points or vectors to other existing sources of detailed measurement information, acting to some extent as a guide to the Measurement Body of Knowledge. It is not the intent to recreate or duplicate other existing sources of measurement 8

9 information (e.g. PSM Guidebook, SE Leading Indicators Guide, Technical Measurement Guide, etc). Rather the intent is to develop a short (5-10 pages), process-focused how-to guide that steps the practitioner through the fundamentals of selecting the appropriate value-added measures for the program situation/environment and using those measures to gain insight, leveraging the consistent approach in the existing references. (2) NDIA should continue a research agenda to investigate causes and effects for program systems engineering success and characterization of SE methods, practices, and solution approaches. Investigate the identification of context factors and how they dynamically interact. Include NAVAIR. The intent is to identify and define the measures that when analyzed in combination, provide insight (particularly leading insight) into program execution. The conditions, context, contingencies, circumstances and/or acquisition lifecycle timing may all be important elements for consideration. It will be essential to identify and define the program context factors (i.e., implementation conditions) that are likely to make a difference for effective/efficient execution of the SE methods and practices. These will be the subject of investigation to determine which context factors are relevant in SE planning. Because of the broad range of this issue and the need to consider Program Management (PM) in the solution, NDIA SED should coordinate with the NDIA Program Management Systems Committee and consider a workshop along the lines of what was done for the top SE and SW top issues several years ago. (3) NDIA should continue to work with other parties to explore & potentially create a data collection and analysis capability in order to develop technical management benchmarks. Need to create the value proposition that justifies the considerable effort likely required (e.g., OSD/Systems Engineering root cause analysis data base, USC as part of the Systems Engineering Research Center, SEI - Software Engineering Institute, PSM - Practical Software and Systems Measurement). The intent is to identify collaborative partners to: o Identify and consistently define a set of data that can provide a wide range of insight o Pool measurement data and contextual information together that would allow for benchmarking trends to be developed. The output of this effort would be used in coordination with recommendations (1) and (2), specifically, and the acquisition community (DoD and Industry) in general. Issue 5: 9

10 Lack of technical authority can impact the integrity of developed systems and result in cost/schedule/system performance impacts as the technical solution is iterated and reworked in later stages of the development. The following discussion points provide amplification on this issue: System engineering fundamentals and program tradeoffs are not properly implemented in programs. Critical technical issues are not identified, raised and resolved in a timely manner. This can result in excessive rework when the technical shortcoming has to be resolved in later acquisition phases when it is more difficult and costly to address. One of the many reasons for prolonged development with later significant rework is attributed to the fact that Engineering is often not empowered to do the proper System Engineering the first time. The well established methods for proper System Engineering are often not performed, even in an abbreviated way. Thus, at some point after the system is initially field tested, problems arise in the operational environments that require significant engineering efforts that should have been accomplished in development. This problem stems from early contract and/or program decisions to eliminate steps in the engineering process. When these decisions are made, the engineers have no recourse to challenge these decisions. In many cases, the technical baseline definition of a system and management of that baseline is not established before field testing. This creates ambiguity in the delivered product and makes it difficult to repeat the manufacturing of the system. This will then require subsequent unplanned additional engineering activity. A technical authority empowered to drive and then approve/control the baseline would help to ameliorate this circumstance. Technical reviews are not rigorous, comprehensive disciplined, and do not employ a sound methodology. This is one root cause for critical issues not being identified and raised early in the development life cycle. Often there is no senior technical authority (or a specified chain of technical authority) to respond to the issues addressed above when they do get identified. When a decision is made that compromises the engineering activities and thus may compromise the integrity of the delivered system, there is often no independent mechanism in place to elevate the issue for resolution because a chain of technical authority has not been established. This situation results in insufficient checks and balances on the Program Office in decisions it makes on technical matters. In some cases, there may be an identified Technical Authority and processes within the program office technical staff and the government or contractor engineering organization that is empowered to drive the technical aspects of the program. In those cases, technical decision-making capability can be enhanced with the use of independent experts to broaden the technical and domain expertise in the program office. In most cases after examining current methods and approaches to implementing technical authority on program, it does not appear that there are sufficient technical authority mechanisms in place on AF, ARMY, NAVY and NASA 10

11 program current methods. Where these mechanisms may exist, they are not empowered to drive the technical aspects of the program. The issue of technical authority on programs and its empowerment to drive all aspects of engineering and technical management is a systemic issue that extends beyond the government and has driven contractor behaviors as well. Recommendations: Strengthen the technical integrity of the delivered systems by implementing the following three actions: 1) Technical Oversight and Issue Resolution: (Industry and Government) Every program needs an organizational capacity for technical oversight vested with the skills, ability, knowledge, responsibility, and authority to make programmatic decisions supporting the technical integrity of the system. An independent path or chain of command for technical authority must be established to raise significant issues, and to balance program authority. Add a requirement to the SEP guidance that the SEP must explicitly identify the technical authority (TA) for the program: 1. the TA role, authority, responsibility, and the performance reporting chain 2. TA conduct and chairing of technical reviews 3. the mechanism for elevating technical issues beyond the program office for resolution. Examine whether OSD can establish a policy that all services must enact a technical authority process and capacity to resolve issues at a higher level than the Program Manager for serious issues. 2) Technical Reviews: (Industry and Government) There is a need to establish event-based technical reviews with enforced entry criteria and go deeper into technical details. The government PMO (and contractor s programs) must staff these reviews with experienced engineers who can operate somewhat independently of the Program Office and ensure that appropriate technical authority is being applied to the program. The matrix organization (people not directly doing the work) must support peer reviews. NDIA review and update guidance, as needed, for entry/exit criteria and application for consistent rigor in technical reviews. Consistency will need to be flowed through development contracts. 3) Parallel Career Paths: (Government) Create a career path for senior technical people with broad expertise and wide experience who can see strategic viewpoints that leads to seniority and authority equivalent to the Program Manager and PEO. In practice these engineers are by definition (rules) always subordinated to the Program Manager; but there needs 11

12 to be a technical authority chain of command to balance program authority, specifically the Chief Systems Engineer on the program. The Chief Engineer is presumed to possess the adequate skill set, the experience of a systems thinker / systems integrator, and is recognized as the technical authority for the program. NDIA SE Division discuss with NDIA PM Systems Committee and OSD to determine if there are actions that can be taken through NDIA to support this For additional information, contact the NDIA Systems Engineering Division: Bob Rassa, Chair, Raytheon Co; RCRassa@Raytheon.com Hal Wilson, Vice-Chair, Northrop Grumman Corp; hal.wilson@ngc.com Geoff Draper, 2 nd Vice-Chair, Harris Corp; gdraper@harris.com Sam Campagna, Committee Executive, NDIA; scampagna@ndia.org 12

13 Appendix A Task Group Participant Organizations Alion Science & Technology American Rheinmetall Munitions The Boeing Company Booz Allen Hamilton Harris Corporation Lockheed Martin Corporation NASA HQ, Engineering Northrop Grumman Corporation OUSD(AT&L)DDR&E/SE Raytheon Company Rockwell Collins Stevens Institute of Technology Software Engineering Institute SynGenics Corporation US Army HQ DA - CIA US Army RDECOM-ARDEC USAF - Air Force Office of Scientific Research US Navy SPAWAR Systems Center - Pacific Vitech Corporation 13