Systems Engineering Assumptions and Comparison Tests

Transcription

1 Systems Engineering Assumptions and Comparison Tests i

2 ii

3 Systems Engineering Assumptions and Comparison Tests Tan Li Graduate Report September 2008 Faculty of Civil Engineering and Geosciences in cooperation with DHV B.V. Graduate student Name: Tan Li Address: Lisztstraat BN Delft Tel: Student number: Graduate Committee Prof.dr.ir. H.A.J. de Ridder (DCP) Dr.ir. G.A. van Nederveen (DCP) Dr.ir. M.G.C.E. Reniers (DHV) Ing. J. Hekker (DHV) Ir. G. Arends (GEO) Alle rechten voorbehouden. Niets uit deze uitgave mag worden vermenigvuldigd en/of openbaar gemaakt door middel van druk, fotokopie, microfilm of op welke andere wijze dan ook, zonder voorafgaande toestemming van DHV B.V. iii

4 iv

5 Abstract Systems Engineering becomes more and more practical these days. It is original from Aerospace Engineering, and it spreads to other domains rapidly. Nowadays, the application of Systems Engineering concept in Civil Engineering is a hotpot calling for more attention and deeper research. Starting with the thinking of several assumptions abstracted from practical work, I choose the most attractable one to my further study. That is the core part of Systems Engineering-Validation and Verification Processes. I also finish some research on general application condition of Systems Engineering and the circumstance on Design & Construction Processes, which will be the premise of speaking about Validation and Verification Processes. Based on the knowledge gained from literature study (Theory Part) and the information collected from Interview Stage (Practical Part), I have some comparison upon the most interesting issues. Then I formulize an online questionnaire to test people s attitudes and further indicate my hypotheses. With the reactions got from this questionnaire, I would like to survey three areas. First, what do most people think about the critical issues in the use of Systems Engineering? Second, which different opinions are shown by standing on different points of views? Third, is there still enough space to place more Systems Engineering concept in practical Civil Engineering work? I create a rich database to record all these people s ideas. Then by using statistic analysis, I provide scientific results on each statement and make proper analysis in both vertical (standing on each group of statements, comparing different opinions that different roles shown) and horizontal (standing on each role, comparing its opinion upon several related statements) directions. My conclusion hits the most important parts of Systems Engineering in Civil Engineering domain nowadays. And we could learn from my research that, Systems Engineering is not only a simple traditional engineering, but also has more extension contents to benefit all people involved with Civil Engineering projects. As most of people s expectations, Systems Engineering will catch more emphasis and be applied widely in future Civil Engineering domain. v

6 Keywords -Systems Engineering -Design &Construction Processes -Validation &Verification Processes -Questionnaire Survey -Statistic analysis vi

7 Contents Contents Chapter 1: Introduction General information Questions and Goal of study Assumption 1-About proper requirements in Requirements stage Assumption 2-About freedom and criteria for trade-offs in Options & Alternatives stage Assumption 3-About validation and verification Limitations Methodology Find out initial assumptions based on practice Literature study and hypotheses giving Collect information and formulize questionnaires survey Conclusions, thesis writing...8 Chapter 2: Theory Study General Introduction of Systems Engineering Definition of System Definitions of Systems Engineering Origins of Systems Engineering Use of Systems Engineering Systems Engineering Process State the problem Investigate alternatives Model the system Integrate system components Launch the system Assess Performance Re-evaluation Verification & Validation Verification Process Validation Process Intermediate Result-Additional Questions About Systems Engineering About Verification and Validation...25 Chapter 3: Practical Research Interview Scheme Interview Report Outline for Interviews Report for Interview with Martijn Ottenhoff Report for Interview with Marco Mijnders Report for Interview with Rene Kuiper...33 I

8 Contents Report for Interview with Toine Vos TIPs for Interviews Intermediate Result-Additional Questions Systems Engineering in different roles within Civil Engineering domain General process and checking work in design...38 Chapter 4 Questionnaires Survey Comparison of Theory study and Practical research About Systems Engineering About the working process and checking works Interesting Issues for Questionnaires The relationship of Clients, Designers and Contractors Design and Construction Process Verification & Validation Process Online Questionnaires...48 Chapter 5: Introduction General information Use Systems Engineering in daily work The using condition of Systems Engineering in practical work Who can benefit most from the use of Systems Engineering The collaboration condition within projects Design and Construction Process Daily Process Critical Issue in design process Contract and organization situation The members of design group The flow of design & construction process Dealing with possible change Validation & Verification Process General knowledge of Validation & Verification (WHAT) The necessity of Validation and Verification (WHY) The responsibility of Validation & Verification (WHO) The occasion to use Validation & Verification (WHEN) How to use Validation & Verification (HOW) Create new concept in Civil Engineering domain Chapter 6 Conclusion and Suggestion Conclusion General Systems Engineering Design and construction processes The Validation and Verification processes Suggestion References Appendix II

9 Introduction Chapter 1: Introduction 1.1 General information Systems Engineering becomes more and more practical these days. Original from Aerospace Engineering, it spreads to other domains rapidly. The application of Systems Engineering concept within Civil Engineering is a hotpot calling for more attention and deeper research. I have a background of Structure Engineering for my bachelor degree, but I chose Design & Construction Processes for the master study with the intention of being more comprehensive. Based on some basic knowledge I have learned in Process Management specification, I would like to do more work in the application of Systems Engineering within Civil Engineering domain. My department in DHV is doing project management in Environment and Transportation, dealing with infrastructure projects. I will focus on this field, hoping to contribute to improve the development of Systems Engineering in infrastructure projects. 1.2 Questions and Goal of study In the beginning of this section, I would like to show a life cycle description of a project procedure. We have a chart of famous Vee-Model to process the procedure of Systems Engineering as below (see Figure1-1). It illustrates the Systems Engineering activities during the life cycle stages. In this model, time and system maturity proceed from left to right. The core of it depicts the evolving baseline from stakeholders requirements agreement to identification of a system concept to definition of systems components that will comprise the final product. 1

10 Introduction Figure 1-1 Vee-Model [1] After being familiar with the life cycle of a system, it is necessary to have some suitable descriptions of the applying of Systems Engineering in Civil Engineering specially. The life cycle process in Civil Engineering project is listing below. Requirements Options Alternatives Design Construction Operation Demolition Figure 1-2 Life Cycle Process in Civil Engineering Project The engineering department of DHV takes part in the first four stages. They have three main issues to deal with as listing below: General quality improvement (Quality Management) Consultancy by using Systems Engineering (as a product) within clients organizations Main driver: Within engineering process for construction companies. By enabling systems engineers to coordinate the interactions between engineering specialists, systems stakeholders and operators, and construction, DHV domain also pays 2

11 Introduction attention to the addressing of conformance with the expectations and legislated requirements of society. As regards to the study of my master thesis, I would like to put more emphasis on requirements within DHV and check the validation process of the design work to optimize the application of Systems Engineering within Civil Engineering domain. We have some assumptions in different stages of the life cycle of the process of a project. I will generally explain them in the following paragraphs. However, thinking about these Systems Engineering assumptions and comparing with the practical issues, the third assumption becomes more attractable. I will choose this to investigate for my thesis. Then I will show more explanation on Assumption 3. And as this assumption has some relation to the first assumption about the requirements. It is also necessary to look into this field as well sometimes Assumption 1-About proper requirements in Requirements stage Problems definition The definition of the requirements is the basis of creating an effective product. Relevant departments of consultant companies communicate with the clients and possibly with stakeholders, input data for the project, such as the description of users needs or services that the project will provide, cost, schedules, constraints, terms and conditions of the agreement, and industry specifications and standards. However, in the real working process, the stakeholder Requirements are always incomplete, unstructured and indistinctly. Incomplete: There is always some information of the users requirements missing, which needs to be added afterwards. And it s also common to forget some implementation constraints for design and also function during the analysis of requirements. Unstructured: The initial information got is not in the right form to be analyzed, which creates difficulties for transforming the stakeholders requirement-driven view of desired services into a technical view of a required product. The integrity and traceability of system requirements is not sufficient. Indistinctly: Unjustified constraints and unrealistic or competing objectives exist during the analysis phase, which leads to the unclear of the final requirements given. Sometimes, source and rationale for some requirements missed. And it s hard to avoid deriving requirements that are not consistent with constraints. Assumption given Stakeholders always fail by nature to give a satisfied description of what they want. So the problem will be the awareness within DHV departments on the crippled set of requirements, which leads to lots of inconvenience in the following stages. 3

12 Introduction Questions asked How to initiate an effective communication with clients to improve understanding set expectations? How to receive the correct requirements from stakeholders in such way that can be used in contract? How to check whether I have received effective requirements correctly? And by using which kinds of tools? Goal Create some methods to solve the problems listed above; preferably draw a legible and structured procedure for intended audience to follow with. Make a detailed box of requirement which is starting with the consideration of: 1. Different types of clients 2. Different location(language boundary for communication, different norms) 3. Minimum requirements from stakeholders Assumption 2-About freedom and criteria for trade-offs in Options & Alternatives stage Problems definition During the procedure of describing functional and performance requirements, services and attributes, timeline requirements, data flow requirements, DHV design has merely freedom as it should be in Options stage. There are some detail specifications to limit design brainstorm. And in alternative stage, when faced with many options, it is always hard to decide which one is most suitable because of the incredible criteria. Assumption given We need: expectations of clients been clarified, design freedom been addressed and the agreement of criteria had been made. Questions asked How to establish right boundary conditions? How much details we should go into during design phase? How to establish where there is design freedom? Is it possible to think of some criteria even from the beginning of design phase? What are the most useful criteria to find out the most enforceable alternative? Goal Make analysis of design freedom by sufficient communication with the client to avoid unnecessary work, and then save energy in Options stage. Seek for the possibility to provide a complete database including of all the limitations during design. 4

13 Introduction Assumption 3-About validation and verification Background Theory Requirements Evolution Stakeholder Requirements Technical Requirements Figure 1-3 Comparison of Requirements [2] FROM: Stakeholder Requirements -Generally in the language of users, customers or specialists -Often non-technical and imprecise -Sometimes is a surrogate for the real need or expectation TO: Technical Requirements -Adequate for design, build, or text of a product or process -Needs to be in the language of the engineer Verification-to check against intended use to build it right The purpose of the Verification Process is to confirm that all requirements are fulfilled by the system elements and system-of-interest. It confirms that all elements of the system-of-interest perform their intended functions and meet the Performance Requirements (Technical Requirements) allocated to them. The outputs of this process are documentation of the verification results, a record of any corrective actions recommended, Design Feedback/Corrective Actions taken, and evidence that the system element or system satisfies the requirements, or not. Validation-to check against intended use to build the right thing The purpose of the Validation Process is to confirm that the realized system complies with the Stakeholder Requirements. System validation is subject to approval by the project authority and key stakeholders. It performs a comparative assessment as a means to determine if Stakeholders Requirements and defined measures of effectiveness have been correctly translated into technical design specifications and measures of performance. The system design team uses the results of this process to forecast success in meeting the expectations of users and the acquirer, as well as to offer feedback to identify and correct performance deficiencies before implementation. We could compare the Verification and Validation Process as below, then from which we will get a general impression of the differences. 5

14 Introduction Inputs Outputs Two Types Verification Validation -Architectural design requirements -Integrated System Released for Validation -Supplied system elements -Validation Criteria for stakeholders -Integration Plan requirements -Verifiable system -Validation Procedures -Results of integration testing -Reported results of validation activities -Problem resolution records and corrective actions taken -Product & Process Qualification -Requirements Validation -Full compliance to specification -Check for traceability -Product prequalification may be - Have we missed any requirements? necessary when Product is - Do we have extraneous redesigned requirements? -Process prequalification may be cnecessary when Process is restarted -Product Acceptance -Product Validation -Full compliance to key criteria -Check if meeting needs and -Done on every unit or on sample expectations of stakeholders is -Can be done prior to shipment or einstallation Table 1-1 Comparison of Verification and Validation [3] Existing Situation Validation Right Things Right Right Things Wrong Wrong Things Right Wrong Things Wrong Verification Figure 1-4 working Smart of Validation and Verification Currently, most consultant companies have put a huge effort to the verification process. They keep on doing the things right, according to the Technical Requirements. There are quite too many regulations needed to check every step further. However, sometimes the Stakeholders Requirements were not translated into Technical Requirements quite well. Then if keep relying on the Technical Requirements all the time, it may results in some useless products which are unnecessary for the users. This leads to time and energy wasted in unneeded work. So, more attention is calling for to make the Validation Process in the central domain and keep the design work in a correct direction all the time. 6

15 Introduction Assumption given -Requirements are not to be verified since solution is prescribed by client. -Many derived requirements to make them verifiable; beyond the level of detailing for making the product. Questions asked What are people s attitudes towards Validation and Verification? Are these kinds of checking work applying in practical work nowadays? Do working people really know the differences between Validation and Verification processes? And based on their different role they are playing, do they have different opinions on some range and responsibility problems? Goal Find out people s attitudes towards checking works. Provide the data supporting for comparison of Validation and Verification. Based on theory and reactions people gave, seek for deeper reasons for problems existing in Civil Engineering domain. Feedback from Stakeholder Requirements should be made suitable and intelligence to support the further step of design. 1.3 Limitations There are some boundary conditions to limit a project. A Civil project could never be ideal and perfect. It can be better than the last one, but never becomes the best one. New technology and new concept emerges with time flies. And design and construction procedure will try to catch up these steps all the way. At the same time, my theory research is limited to a countable literatures and publications. The practical learning is only within Dutch Civil Engineering domain. And the later questionnaires survey provides 72 reactions out of all involved Civil Engineering people. 1.4 Methodology Find out initial assumptions based on practice First of all, as I will improve the reflection of today s study theory on practice in Systems Engineering concept, I need initiate basic knowledge about the principle of Systems Engineering by reading relevant study materials. Looking into the formulated text by heart to make the definition understood thoroughly is necessary. Then, conduct the most interesting parts and try to think about difficulties and risks if further research is carried out 7

16 Introduction there. Discuss with supervisors to learn the current situation of the application of Systems Engineering in real work within Civil Engineering domain. As mentioned above, Systems Engineering was original in Aerospace field but just started in Civil Engineering field nowadays. So the first task of the thesis should be learning to know how it works daily. Find out where there are some problems to be improved and figure them out sharply. Describe some answerable questions to deal with in further research of the thesis. Then, establish main objective and critical methods with effective review to guide myself in a direct path Literature study and hypotheses giving Process a deep research about relevant literatures to see how other scholars have tried to answer the questions that I mentioned in proposal. Compare them and structure the most useful parts into clear text. Draw conclusions by systematic analysis. Initial adaptable hypotheses and create suitable models to satisfy the required questions. Hypotheses are preferred to be given to solve those problems based on the literature study. Interview people by using pertinence questionnaires or face to face contact to gather the reflections whether the hypotheses are true or not. For example in CPM (Contract & Project Management) department of DHV, people are dealing with several of projects and facing with plenty of difficulties. They have experiences and would have some opinions in Systems Engineering as it is going more and more common practice nowadays Collect information and formulize questionnaires survey Collect the feedbacks from literature study and practical research part. Then make analysis of them and find out the most interesting topics. Usually, it could be imagined that the information collected from in-depth interview is huge and mass. So organizing them into a clear explanation is required Conclusions, thesis writing Analyze on each statement testing in questionnaires. Draw conclusions to answer the questions mentioned at the beginning stage, optimize them with detailed consideration and particular characterization. Then finalize the master thesis. 8

17 Theory Study Chapter 2: Theory Study 2.1 General Introduction of Systems Engineering Definition of System A system is a whole that cannot be divided into independent parts without losing its essential characteristics as a whole. It follows from this definition that, a system s essential defining properties are the product of the interactions of its parts, not the actions of the parts considered separately. Therefore, when a system is taken apart, or its parts are considered independently of each other, the system loses its essential properties. Furthermore, when the performance of each part taken separately is improved, the performance of the system as a whole may not be, and usually isn t. Definition of a System: A system is a set of interrelated components which interact with one another in an organized fashion toward a common purpose Definitions of Systems Engineering A system is an integrated composite of people, products, and processes that provide a capability to satisfy a stated need or objective. Systems Engineering is a profession, a process, and a perspective as illustrated by these three representative definitions. [4] Systems Engineering is a discipline that concentrates on the design and application of the whole (system) as distinct from the parts. It involves looking at a problem in its entirety, taking into account all the facets and all the variables and relating the social to the technical aspect. (Ramo1) Systems Engineering is an iterative process of top-down synthesis, development, and operation of a real-world system that satisfies, in a near optimal manner, the full range of requirements for the system. (Eisner2) Systems Engineering is an interdisciplinary approach and means to enable the realization of successful systems. (INCOSE3) Certain keywords emerge from this sampling- interdisciplinary, iterative, socio-technical, and wholeness. In a general principle, Systems Engineering is an interdisciplinary engineering 9

18 Theory Study management process that evolves and verifies an integrated, life-cycle balanced set of system solutions that satisfy customer expectations and meet public acceptability Origins of Systems Engineering The modern origins of Systems Engineering can be traced to the 1930 s followed quickly by other programs and supporters. Table below offers a thumbnail of some important highlights in the origins and history of the application of Systems Engineering (Table 1) Rocket locomotive; progenitor of main-line railway motive power 1937 British multi-disciplinary team to analyze the air defense system Bell Labs supported NIKE development SAGE Air Defense System defined and managed by MIT 1956 Invention of systems analysis by RAND Corporation 1962 Publication of A Methodology for Systems Engineering 1969 Jay Forrester (Modeling Urban Systems at MIT) 1994 Perry Memorandum urges military contractors to adopt commercial practices such as IEEE P Release of ISO/IEC Table 2-1: Important Dates in the Origins of Systems Engineering as a Discipline [5] With the introduction of the international standard ISO/IEC in 2002, the discipline of Systems Engineering was formally recognized as a preferred mechanism to establish agreement for the creation of products and services to be traded between two enterprises the acquirer and supplier Use of Systems Engineering As can be readily inferred from the nature of the earliest projects, the Systems Engineering discipline emerged as an effective way to manage complexity and change. Both complexity and change have escalated in our products, services, and society. Reducing the risk associated with new systems or modification to complex systems continues to be a primary goal of the systems engineer. 10

19 Theory Study Figure 2-1: Process of Systems Engineering [1] Before talking about the Systems Engineering domain, we have to consider about the Systems Engineering tools first. Systems Engineering tools are strategies, procedures, and techniques that aid in performing Systems Engineering on a project or product. The figure above demonstrates the whole progress of Systems Engineering (Figure 2-1). Based on this procedure, we know that Systems Engineering applies to the full life cycle of systems, including conception, development, production, utilization, support and retirement of systems, whether performed internally or externally to an organization. There is a wide variety of systems in terms of their purpose, domain of application, complexity, size, novelty, adaptability, quantities, locations, life spans and evolution. Systems Engineering comprises the life cycle of any man-made system. Many related fields may be considered tightly coupled to Systems Engineering. These areas have contributed to the development of Systems Engineering as a distinct entity. For example, Control engineering, Industrial engineering, Operations research, Performance engineering and so on. There are not so much mentioned in Civil Engineering area so far. So an application Systems Engineering method within civil engineering becomes the research objective. We need to transform the existing method within other domains to provide sufficient evidence in order to make Systems Engineering applicable in Civil Engineering. 11

20 Theory Study 2.2 Systems Engineering Process Systems Engineering is an interdisciplinary process that ensures that the customer's needs are satisfied throughout a system's entire life cycle. [6] This process is comprised of the following seven tasks. [7] 1. State the problem. Stating the problem is the most important Systems Engineering task. It entails identifying customers, understanding customer needs, establishing the need for change, discovering requirements and defining system functions. 2. Investigate alternatives. Alternatives are investigated and evaluated based on performance, cost and risk. 3. Model the system. Running models clarifies requirements, reveals bottlenecks and fragmented activities, reduces cost and exposes duplication of efforts. 4. Integrate. Integration means designing interfaces and bringing system elements together so they work as a whole. This requires extensive communication and coordination. 5. Launch the system. Launching the system means running the system and producing outputs -- making the system do what it was intended to do. 6. Assess performance. Performance is assessed using evaluation criteria, technical performance measures and measures -- measurement is the key. If you cannot measure it, you cannot control it. If you cannot control it, you cannot improve it. 7. Re-evaluation. Re-evaluation should be a continual and iterative process with many parallel loops. This process can be summarized with the acronym SIMILAR. [8] Figure 2-2 The Systems Engineering Process The purpose of Systems Engineering is to produce systems that satisfy the customers' needs, increase the probability of system success, reduce risk and reduce total-life-cycle cost. 12

21 Theory Study The system life cycle The system life cycle has seven phases: (1) discovering system requirements, (2) investigating alternatives, (3) full-scale engineering design, (4) implementation, (5) integration and test, (6) operation, maintenance and evaluation and (7) retirement, disposal and replacement. However, the system life cycle is different for different industries, products and customers State the problem The problem statement starts with a description of the top-level function that the system must perform or the deficiency that must be ameliorated. It includes system requirements stated in terms of what must be done, not how to do it. It might be composed in words or as a model. The problem statement is one of Systems Engineering's most important products. An elegant solution to the wrong problem is less than worthless. Understand customer needs Customers seldom know what they want or need. Systems Engineers must enter the customer's environment and find out how the customer will use the system. We must exceed, not merely meet, customer expectations. Discover system requirements There are two types of system requirements: mandatory and tradeoff. [9] Mandatory requirements insure that the system satisfies the customer's operational need. Mandatory requirements (1) specify the necessary and sufficient conditions that a minimal system must have in order to be acceptable (They are usually written with the words shall or must.), (2) must be passed or failed, there is no middle ground (i.e. They do not use scoring functions.), and (3) must not be susceptible to trade-offs between requirements. Mandatory requirements state the minimal requirements necessary to satisfy the customer's need. After understanding the mandatory requirements, Systems Engineers propose alternative candidate designs, all of which satisfy the mandatory requirements. Then the tradeoff requirements are evaluated to determine the preferred designs. Tradeoff requirements (1) should state conditions that would make the customer happier (They are often written with the word should.), (2) should use scoring functions [10] [11]to evaluate the criteria, and (3) should be evaluated with multi-criterion decision aiding techniques [12][13] because there will be trade-offs between these requirements. Sometimes there is a relationship between mandatory and tradeoff requirements, e.g. a mandatory requirement might be a lower threshold value for a tradeoff requirement. The words optimize, maximize, minimize and simultaneous should not be used in stating 13

22 Theory Study requirements. Quality function deployment (QFD) is useful in identifying system requirements [14][15]. Validate requirements Validating requirements means ensuring that (1) the recommended solution satisfies the actual needs of the customer, (2) the description of the requirements is consist and complete, (3) a system model can satisfy the requirements and (4) a real-world solution can be tested to prove that it satisfies the requirements. If Systems Engineering discovers that the customer has requested a perpetual-motion machine, the project should be stopped. Requirements are often validated by reference to an existing system that meets most of the requirements Investigate alternatives Alternative designs are evaluated based on performance, cost, schedule and risk criteria. No design is likely to be best on all criteria, so multicriteria decision aiding techniques should be used to reveal the preferred alternatives. For important decisions formal tradeoff studies should be performed. A tradeoff study is not something that you do once at the beginning of a project. Throughout a project you are continually making tradeoffs: creating team communication methods, selecting components, choosing implementation techniques, designing test programs, or maintaining schedule. Many of these tradeoffs should be formally documented. These are the components of a tradeoff study: Problem statement, Evaluation criteria, Weights of importance, Alternative solutions, Evaluation data, Scoring functions, Scores, Combining functions, Preferred alternatives, and Sensitivity analysis. [16] Performance and cost criteria show how well the system satisfies its requirements. Evaluation criteria are often called Measures of Effectiveness. Such measurements are made throughout the evolution of the system: based first on estimates by the design engineers, then on models, simulations, prototypes and finally on the real system. Evaluation criteria are used to help select amongst alternative designs and they are used to quantify system requirements. Technical performance measures (TPM's) are used to track the progress of design and manufacturing. They are measurements that are made during the design and manufacturing process to evaluate the likelihood of satisfying the system requirements. Measures are often related to the process, not the product. [17] Therefore, they do not always relate to specific system requirements. Rather some measures relate to the company's mission statement and subsequent goals. A useful measure is the percentage of requirements that have changed after the System Requirements Review. 14

23 Theory Study Model the system Models will be developed for most alternative designs. The model for the preferred alternative will be expanded and used to help manage the system throughout its entire life cycle. Many types of system models are used, such as physical analogs, analytic equations, state machines, block diagrams, functional flow diagrams, object-oriented models, computer simulations and mental models. Systems will usually be more successful if the models have observable states. [18] Systems Engineering is responsible for creating a product and also a process for producing it. So, models should be constructed for both the product and the process. Process models allow us, for example, to study scheduling changes, and perform sensitivity analyses to show the effects of delaying or accelerating certain subprojects. Running the process models reveals bottlenecks and fragmented activities, reduces cost and exposes duplication of effort. Product models help explain the system. These models are also used in tradeoff studies and risk management. Design the system The overall system must be partitioned into subsystems; subsystems must be partitioned into assemblies, etc. Reusability should be considered in creating subsystems. For new designs, subsystems should be created so that they can be reused in future products. For redesign, subsystems should be created to maximize the use of existing, particularly commercially available, products. Create sequence diagrams Arguably the first step in designing a system should be creating sequence diagrams. This means collecting typical sequences of events that the proposed system will go through. Sequences can be descriptions of historical behavior or can be based on mental models for future behavior Define system architecture Defining the system architecture means choosing the high level switches that will determine the components and subsystems of the system. Rechtin and Maier (1997) state that System Architecting is done in the first two phases of the system life cycle: discovering system requirements and concept exploration. [19] Functional decomposition 15

24 Theory Study Systems engineers do functional decomposition on new systems (1) to map functions to physical components, thereby ensuring that each function has an acknowledged owner, (2) to map functions to system requirements, and (3) to ensure that all necessary tasks are listed and that no unnecessary tasks are requested. This list becomes the basis for the work breakdown structure. When analyzing (or re-engineering) an existing system, Systems Engineers do functional analysis to see what the system does in order to improve its performance (often called value engineering), and they also do functional decomposition to see what the system is supposed to do. Sensitivity analyses In a sensitivity analysis each parameter or requirement is varied and the effects on the outputs are observed. Sensitivity analyses can be used to point out the requirements and parameters that have the biggest effects on cost, schedule and performance. They are used to help allocate resources. [20] Assess and manage risk There are two types of risk: risk of project failure (due to cost overruns, time overruns or failure to meet performance specifications) and risk of harm (usually called personnel safety). A failure modes and effects analysis and risk mitigation must be performed. [21] Project risk can be reduced by supervising quality and timely delivery of purchased items. [22] Reliability analysis Major failure modes must be analyzed for probability of occurrence and severity of occurrence. [23] Integrate system components Integration means bringing things together so they work as a whole. System integration means bringing subsystems together to produce the desired result and ensure that the subsystems will interact to satisfy the customer's needs. Design and manage interfaces Interfaces between subsystems and interfaces between the main system and the external world must be designed. Subsystems should be defined along natural boundaries and should be defined to minimize the amount of information to be exchanged between the 16

25 Theory Study subsystems. Well-designed subsystems send finished products to other subsystems. Feedback loops around individual subsystems are easier to manage than feedback loops around interconnected subsystems Launch the system Launching the system means doing what the system was intended to do, e.g. running the system and producing outputs. [8] Design engineers produce designs for the product and the process to make it. Configuration management Configuration management (also called modification management) ensures that any changes in requirements, design or implementation are controlled, carefully identified, and accurately recorded. All stakeholders should have an opportunity to comment on proposed changes. Decisions to adopt a change must be captured in a baseline database. This baseline is a time frozen design containing requirements for functions, performance, interfaces, verification, testing, cost, etc. Baselines can only be changed at specified points in the life cycle. All concerned parties must be notified of changes to ensure that they are all working on the same design. Project management Project management is the planning, organizing, directing, and controlling of company resources to meet specific goals and objectives within time, within cost and at the desired performance level [22][23]. Project management creates the work breakdown structure, which is a product-oriented tree of hardware, software, data, facilities and services. It displays and defines the products to be developed and relates the elements of work to be accomplished to each other and to the end product. It provides structure for guiding team assignments and cost and tracking control. Documentation All of these Systems Engineering activities must be documented in a common repository, often called the Engineering Notebook. The stored information should be location, platform, and display independent: which means any person on any computer using any tool should be able to operate on the fundamental data. Assumptions, results of tradeoff studies and the reasons for making critical decisions should be recorded. These documents should be alive and growing. Lead teams Complex systems cannot be designed by one person. Consequently engineers work on 17

26 Theory Study Integrated Product Development Teams (IPDTs). These teams are interdisciplinary with members from Business, Engineering, Manufacturing, Testing, etc. IPDTs are often led by Systems Engineers, either formally or informally. [25] Assess Performance During the operation and maintenance phase of the system life cycle the performance of the system must be measured. Initially these measurements will be used to verify that the system is in compliance with its requirements. Later they will be used to detect deterioration and initiate maintenance. Prescribe tests Early in the system life cycle Systems Engineering should describe the tests that will be used to prove compliance of the final system with its requirements. However, most testing should be performed by built-in self-test equipment. Conduct reviews Systems Engineering should ensure that the appropriate reviews are conducted and documented. The exact reviews that are appropriate depends on the size, complexity and customer of the project. The following set is common: Mission Concept Review, System Requirements Review (SRR), System Definition Review, Preliminary Design Review (PDR), Critical Design Review (CDR), Production Readiness Review (PRR), and System Test. Full-scale engineering design begins after the Preliminary Design Review. Manufacturing begins after the Critical Design Review. [26][9] Verify requirements Systems Engineering must prove that the final system satisfies each system requirement. Requirements may be verified by inspection, analysis, demonstration, test, logical argument, modeling or simulation. Total system test The system that is finally built must be tested to see (1) that it satisfies the mandatory requirements, and (2) how well it satisfies the tradeoff requirements Re-evaluation Re-evaluation is arguably the most important task of Systems Engineering. For centuries engineers have used feedback to control systems and improve performance. It is one of 18

27 Theory Study the most fundamental engineering tools. Re-evaluation means observing outputs and using this information to modify the system inputs, the product or the process. Re-evaluation should be a continual process with many parallel loops. 19

28 Theory Study 2.3 Verification & Validation Verification Process General presentation The Verification process confirms that Design Synthesis has resulted in a physical architecture that satisfies the system requirements. Verification represents the intersection of Systems Engineering and test and evaluation. The objectives of the Verification process include using established criteria to conduct verification of the physical architecture from the lowest level up to the total system to ensure that cost, schedule, and performance requirements are satisfied with acceptable levels of risk. Further objectives include generating data (to confirm that system, subsystem, and lower level items meet their specification requirements) and validating technologies that will be used in system design solutions. The focus and nature of verification activities change as designs progress from concept to detailed designs to physical products. Earlier design stages: focus on proof of concept for system, subsystem and component levels Later stages: focus on the system meets the customer requirements. Figure 2-3 SE Vee-model [27] 20

29 Theory Study The "Vee" model is a way of representing the Systems Engineering design and development process. The left side of the "Vee" depicts the decomposition and definition of the system requirements and specifications at the beginning of the system's life cycle. These specifications are handed off to the design engineers in the middle of the life cycle, at the base of the "Vee." After the individual physical components are developed, responsibility passes back to the systems engineer who integrates and tests the system -- the right side of the "Vee". So, time proceeds from left to right in the "Vee" model. Design is a top-down process while the verification activity is a bottom-up process. Components will be fabricated and tested prior to the subsystems. Subsystems will be fabricated and tested prior to the completed system Verification Process Activities.[28] a) Define the strategy for verifying the systems throughout the life cycle. b) Define a verification plan based on system requirements. c) Identify and communicate potential constraints on design decisions. d) Ensure that the enabling system for verification is available and associated facilities, equipment and operators are prepared to conduct the verification. e) Conduct verification to demonstrate compliance to the specified design requirements. f) Make available verification data on the system. g) Analyze, record and report verification, discrepancy and corrective action information Text & Evaluation.(SYSTEMS ENGINEERING FUNDAMENTALS) Test & Evaluation policies and procedures directly support the system engineering process of Verification. Testing is the means by which objective judgments are made regarding the extent to which the system meets, exceeds, or fails to meet stated objectives. The purpose of evaluation is to review, analyze, and assess data obtained from testing and other means to aid in making systematic decisions. The purpose of Test & Evaluation is to verify technical performance, operational effectiveness, operational suitability; and it provides essential information in support of decision making. The Program office plans and manages the test effort to ensure testing is timely, efficient, comprehensive and complete- and that test results are converted into system improvements. Test planning will determine the effectiveness of the verification process. Careful attention to test planning can reduce program risk. The key test planning document is the Test and Evaluation Master Plan. 21

30 Theory Study Test & Evaluation Master Plan (TEMP) The Test & Evaluation Master Plan is a mandatory document prepared by the program office. The TEMP is a valuable Verification tool that provides an excellent template for technology, system, and major subsystem-level Verification planning. The TEMP includes a reaffirmation of the user requirements, and to an extent, an interpretation of what those requirements mean in various operational scenarios. The required TEMP format: Part 1: System Introduction. Part 2: Integrated Test Program Summary. Part 3: Developmental Test & Evaluation Outline. Part 4: Operational Test & Evaluation Outline. Part 5: Test & Evaluation Resource Summary The compare of Developmental Test & Evaluation and Operational Test & Evaluation Developmental Test & Evaluation/Operational Test & Evaluation The Developmental Test & Evaluation (DT&E) verifies that the design solution meets the system technical requirements and the system is prepared for successful OT&E. Focus on verifying system technical requirements Program office responsibility that is used to develop the design Operational Test & Evaluation (OT&E) programs are structured to determine the operational effectiveness and suitability of a system under realistic conditions, and to determine if the minimum acceptable operational performance requirements as specified in the ORD and reflected by the Key performance parameters have been satisfied. Focus on verifying operational requirements. Independent evaluation of design maturity that is used to determine if the program should proceed to full rate production. Developmental Tests Operational Tests Controlled by Program manager Controlled by independent agency One-on-one tests Many-on-many tests Controlled environment Realistic/tactical environment with Contractor environment operational scenario Trained, experienced operators No system contractor involvement Precise performance objectives and User troops recently trained threshold measurements Performance measures of Test to specification operational effectiveness and Developmental, engineering, or suitability production representative test Test to operational requirements article Production representative test article Table 2-2 Comparison of developmental tests and operational tests 22

31 Theory Study Validation Process The purpose of the Validation Process is to provide objective evidence that the services provided by a system when in use comply with stakeholders requirements Validation Process Activities. [28] a) Define the strategy for validating the services in the operational environment and achieving stakeholder satisfaction. b) Prepare a validation plan. c) Ensure that any operators, enabling system for validation and associated facilities are ready in order to conduct validation. d) Conduct validation to demonstrate conformance of services to stakeholder requirements. e) Make available validation data on the system according to legal, regulatory or product sector requirements. f) As appropriate to agreement terms or organizational objectives, conduct validation to isolate that part of the system giving rise to a non-conformance. g) Analyze record and report validation data according to criteria defined in the validation strategy Common Approaches and Tips. [5] Validation methods during the concept phase include developing assessment scenarios exercising all system modes, and demonstrating system-level performance over the entire operating regime. The system design team uses the results of this activity to forecast success in meeting the expectations of users and the acquirer, as well as to provide feedback to identify and correct performance deficiencies before implementation. 23