Dependable (Safe/Reliable) Systems. ARO Reliability Workshop Software Intensive Systems

Transcription

1 Dependable (Safe/Reliable) Systems Composing, Analyzing and Validating s to Assess / Develop / Validate Methods and Supporting Tools for the Creation of Dependable Systems ARO Reliability Workshop Intensive Systems May Frederick T. Sheldon, Ph.D. Oak Ridge National Laboratory Applied Engineering Research Laboratory Engineering for Dependable Systems Research

2 What is a software process? Activities things that are done Products inputs and outputs Sequencing relationship among the activities and products How does sequencing work? Previous Phase Verification Check Present Phase Compliance with previous phase requirements Conformance to the standards of this phase Adequate basis for subsequent phases Next Phase 2

3 Verification versus Validation Verification determines if the products of a given phase of the SLC fulfill the requirements established during the previous phase Proof of transformation conformance Did we build the product right? Validation checks if the program, as implemented, meets the expectations of the customer to ensure compliance with software requirements Did we build the right product? 3

4 Verification & Validation in Context Definition System Definition What the System is supposed to do? System Development Generation Development What the software is supposed to do? Verification Coding & Component Testing Is the doing what it is supposed to do? Integration & SW System Testing Validation Validation Testing System Validation Hardware Integration Is the System doing what it is supposed to do? System Validation Testing 4

5 Right process for the product to ensure and no silver bullet! High quality software with competitive cost and cycle time... Quality [Defects] Cycle Time Cost we must shrink the triangle! 5

6 Quality Attribute(s) Versus Cost Relation Moore s law of Engineering Cost Product Attr ibute(s) (e.g., Reliability, Safety) 6

7 SLC Expenditure Profile Maintenance Cost Maintenance Need to validate effective SE methods and tools Validation Specification and Implementation Without Formal Specification Validation and Implementation Specification With Formal Specification 7

8 Generic Process Models The waterfall model Separate and distinct phases of specification and development Evolutionary development Specification and development are interleaved (e.g., Extreme Prgming) Formal transformation Mathematical system model formally transformed to an implementation Reuse-based development The system is assembled from existing components Hybrid Methodologies Prototyping for high-risk specifications (e.g., Spiral Model) A heavyweight methodology has many rules, practices, and documents. A lightweight methodology few rules / practices easy to follow. 8

9 Evolutionary development Concurrent Activities Specification Initial Version Outline Description Development Intermediate versions Validation Final Version 9

10 Transformational development Formal Transformations Formal transformations T1 T2 T3 T4 Formal specification R1 R2 R3 Executable program P1 P2 P3 P4 Proofs of transformation correctness Proof of Transformation Correctness 10

11 Spiral Model Boehm Determine Objectives, Alternatives, and Constraints Cumulative Cost Progress Through Steps Evaluate Alternatives; Identify and Resolve s Commitment Review Partition Plan Next Phases and Life Cycle Plan Development Plan Integration and Test Plan 1 Concept of Operation Validation Validation and Verification Implementation 2 3 Acceptance Test Product Integration and Test Operational Simulations, Models, Benchmarks Unit Test Detailed Code Develop, Verify Next Level Product 11

12 Prototyping Environment Plan Life-Cycle Plan Development Plan Concept of Operation 1 Validation 2 Requirement 3 Determine Objective, Alternatives, Constraints Cumulative Cost Progress Through Steps Evaluate Alternatives Identify, Resolve s Testbed/Fielded s Review Commitment Partition Plan Life-Cycle Plan Development Plan Integration and Test Plan Concept of Operation 1 Validation Validation & Verification 2 Requirement Product 3 Operational Simulations, Models, and Benchmarks Benchmarks Plan Next Phase Detailed Unit Test Code Integration and Test Operational System Implementation Acceptance Test 12

13 System How does a process HWCI work? Hardware System SRR SDR SSR Functional Baseline Allocated Baseline Hardware (HWCI) Development Preliminary Preliminary PDR PDR Detailed - May be multiple reviews and may be integrated with hardware reviews Detailed CDR CDR (CSCI) Development Coding and CSU Testing Fabrication CSC Integration and Testing TRR Testing PCA CSCI Testing System Integration And Testing PCA FCA FCA FQR Testing And Evaluation Reviews Product Baseline Production and Deployment SRR - System Review SDR - System Review SSR - Specification Review PDR - Preliminary Review CDR - Critical Review TRR - Test Readiness Review FCA - Functional Configuration Audit PCA - Physical Configuration Audit FQR - Formal Qualification Review 13

14 CMM Level Characteristics Key Challenges Result 5 Optimizing Focus on process improvement Data gathering is automated and used to identify weakest process elements Numerical evidence used to justify application of technology to critical tasks Rigorous defect-cause analysis and detect prevention Focus on process optimization to reduce errors Still human-intensive process Maintain organization at optimizing level Productivity & quality 4 Managed (Quantitative) Measured process: estimates/actuals, error-cause analysis Minimum set of quality and productivity measurements established Process database established & resources to analyze & maintain its data Focus on technology management and insertion Changing technology Problem analysis Problem prevention 3 Defined (Quantitative) Process defined and institutionalized Engineering Process Group established to direct improvements Focus on design skills, design tracking Focus on various types of traceability Process measurement Process analysis Quantitative quality plans 2 Repeatable (Intuitive) Process dependent on individuals Established basic project controls with strength in doing similar work Process faces major risk when presented with new challenges Lacks orderly framework for improvement Focus on collecting various types of "trend" data Focus on management and tight project control Training Technical practices (reviews, testing) Process focus (standards, process groups) 1 Initial (Ad hoc/chaotic process) No formal procedures, cost estimates, project plans No management mechanism to ensure procedures are followed, tools not well integrated, and change control is lax Senior management does not understand key issues Project management Project planning Configuration management quality assurance 14

15 Key Issues in Reliability Provide readier access to formal methods for developers of critical systems by further integration of informal and formal methods. Develop better methods for analysis of product families and safe reuse of Commercial-Off-The- Shelf software. Improve the testing and evaluation of critical systems through the use of requirements-based testing, evaluation from multiple sources, model consistency, and virtual environments. 15

16 Key Issues in Reliability Ensure SW technology transfer from the lab to practice with case studies to facilitate earlier adoption of potentially cost savings / safety ensuring advances. Advance the use of runtime monitoring to detect faults and recover to a safe state, as well as to profile system usage to enhance reliability/safety analyses. Promote collaboration with related fields in order to exploit advances in areas such as security and survivability, software architecture, theoretical computer science, human factors engineering, and software engineering education. 16

17 Research Summary and Plans Experience with numerous formalisms/tools and will continue to Assess, develop and validate methods and supporting tools for the creation of dependable systems Develop an integrated framework for composing, analyzing and validating system and software models Extend to security related verification and validation 17

18 Contact Information Frederick T. Sheldon, Ph.D. Engineering for Dependable for Systems Oak Ridge National Laboratory Phone: Fax: URL: 18