The Software Development Life Cycle: An Overview. Last Time. Session 8: Security and Evaluation. Information Systems Security Engineering

Transcription

1 The Software Development Life Cycle: An Overview Presented by Maxwell Drew and Dan Kaiser Southwest State University Computer Science Program Last Time Brief review of the testing process Dynamic Testing Methods Static Testing Methods Deployment in MSF Deployment in RUP Session 8: and Evaluation General s Engineering Concepts s Engineering Process Relation of ISSE Process to other Processes Product, Process & Resource Evaluation Course Evaluations s Engineering General s Engineering Concepts s Engineering Process Relation of ISSE Process to other Processes s Engineering Process Discover Needs Discover Needs Define Design Assess Effectiveness Mission/Business Description Policy Consideration Mission Needs Statement (MNS) Concept of Operations (CONOPS) Users / Users Representatives Implement 1

2 Define Functionality Objectives - MoE Context/Environment Requirements - RTM Functional Analysis Define Functional Allocation - CM Preliminary Design Baseline Configuration Detailed Design - CI Implement Procurement Build Test Assess Effectiveness Interoperability Availability Training Human/Machine Interface Cost ISSE Activities Describing information protection needs Generating information protection based on needs early in the systems engineering process Satisfying the at an acceptable level of information protection risk Building a functional information protection architecture based on Allocating information protection functions to a physical and logical architecture Designing the system to implement the information protection architecture Balancing information protection risk management and other ISSE considerations within the overall system context of cost, schedule, and operational suitability and effectiveness ISSE Activities - Continued Participating in trade-off studies with other information protection and system engineering disciplines Integrating the ISSE process with the systems engineering and acquisition processes Testing the system to verify information protection design and validate information protection Supporting the customers after deployment and tailoring the overall process to their needs 2

3 Discover Protection Needs Mission Layered Requirements Hierarchy MISSION/BUSINESS Threat Analysis Policy ARCHITECTURE More More Specific Specific Functions Protection Policy DESIGN More More Abstract Abstract Requirements Protection Requirements Components IMPLEMENTATION Specifications Figure 3-2 Impact of Mission, Threats, and Policies on Protection Requirements Mission Protection Needs What kind of information records are being viewed, updated, deleted, initiated, or processed (classified, financial, proprietary, personal private, etc.)? Who or what is authorized to view, update, delete, initiate, or process information records? How do authorized users use the information to perform their duties? What tools (paper, hardware, software, firmware, and procedures) are authorized users using to perform their duties? How important is it to know with certainty that a particular individual sent or received a message or file? Threats to Management Types of Legitimate users and uses of information Threat agent considerations - Capability - Intent - Willingness - Motivation - Damage to mission Protection Policy Considerations Why protection is needed What protection is needed How protection is achieved not considered at this stage Protection Policy Issues The resources/assets the organization has determined are critical or need protection The roles and responsibilities of individuals that will need to interface with those assets (as part of their operational mission needs definition) The appropriate way (authorizations) authorized individuals may use those assets (security ). 3

4 Define Protection Protection Objectives MoE Context/Environment Protection Requirements RTM Functional Analysis Protection Objectives Should Explain: The mission objectives supported by information protection objective The mission-related threat driving the information protection objective The consequences of not implementing the objective protection guidance or policy supporting the objective Design Protection Functional Allocation Preliminary Protection Design Detailed Protection Design Preliminary Protection Design Activities Reviewing and refining Discover Needs and Define activities' work products, especially definition of the CI-level and interface specifications Surveying existing solutions for a match to CI-level Examining rationales for proposed PDR-level (of abstraction) solutions Verification that CI specifications meet higher-level information protection Supporting the certification and accreditation processes Supporting information protection operations development and life-cycle management decisions Participating in the system engineering process Detailed Protection Design Activities Reviewing and refining previous Preliminary Design work products Supporting system- and CI-level design by providing input on feasible information protection solutions and/or review of detailed design materials Examining technical rationales for CDR-level solutions Supporting, generating, and verifying information protection test and evaluation and procedures Tracking and applying information protection assurance mechanisms Verifying CI designs meet higher level information protection Completing most inputs to the life-cycle security support approach, including providing information protection inputs to training and emergency training materials Reviewing and updating information protection risk and threat projections as well as any changes to the set Supporting the certification and accreditation processes Participating in the system engineering process Implement Protection Procurement Build Test 4

5 Implement Protection General Activities Updates to the system information protection threat assessment, as projected, to the system's operational existence Verification of system information protection and constraints against implemented information protection solutions, and associated system verification and validation mechanisms and findings Tracking of, or participation in, application of information protection assurance mechanisms related to system implementation and testing practices Implement Protection General Activities (cont.) Further inputs to and review of evolving system operational procedure and life-cycle support plans, including, for example, Communication (COMSEC) key distribution or releasability control issues within logistics support and information protection relevant elements within system operational and maintenance training materials A formal information protection assessment in preparation for the Verification Review Inputs to Certification and Accreditation (C&A) process activities as required Participation in the collective, multidisciplinary examination of all system issues Build Protection Physical Integrity. Have the components that are used in the production been properly safeguarded against tampering? Personnel Integrity. Are the people assigned to construct or assemble the system knowledgeable in proper assembly procedures, and are they cleared to the proper level necessary to ensure system trustworthiness? Test Protection Activities Reviewing and refining Design Protection work products Verifying system- and CI-level information protection and constraints against implemented solutions and associated system verification and validation mechanisms and findings Tracking and applying information protection assurance mechanisms related to system implementation and testing practices Providing inputs to and review of the evolving life-cycle security support plans, including logistics, maintenance, and training Continuing risk management activities Supporting the certification and accreditation processes Participating in the systems engineering process Assess Effectiveness Interoperability. Does the system protect information correctly across external interfaces? Availability. Is the system available to users to protect information and information assets? Training. What degree of instruction is required for users to be qualified to operate and maintain the information protection system? Human/Machine Interface. Does the human/machine interface contribute to users making mistakes or compromising information protection mechanisms? Cost. Is it financially feasible to construct and/or maintain the information protection system? Relation to Other Processes Acquisition Process Risk Management Process DITSCAP Common Criteria International Standard 5

6 ISSE and Acquisition Process Flows Risk Management Process Requirements Identification Phase User Program Needs Concept Exploration Phase Identify New Concepts Engineering & Implementation Translate Into Design Production & Operational Support Build Production Understand Mission Objectives Understand Protection Needs (Services) Req. Req. Test Concepts Formalize Concepts Specify Components Purchase Components Formal Testing Support Over Lifecycle Acquisition Engineering (ISSE) Process Assess Effectiveness Implement Decision Actions Risk Management Cycle Characterize Risk Posture Discover Protection Needs Define Protection Design Protection Implement Protection Decide What Will Be Done Characterize What Can Be Done Users / Users Representatives Figure 3-5 Risk Management Process Risk Decision Flow Risk Plane Risk Decision Flow Y - CONSEQUENCE Improvements Countermeasure Identification & Characterization Mission Critical Parameter Trade-Off High A-1 Risk Analysis Compare and Contrast Available Attacks Develop Theory of Adversarial Behavior Develop Theory of Mission Impact Compare and Contrast Various Courses of Action Decide on Courses of Action Med. A-3 A-2 Low Vulnerability and Attack Identification & Characterization Threat Identification & Characterization Mission Impact Identification & Characterization Low Med. X - LIKELIHOOD OF SUCCESS High Figure 3-7 Risk Plane Foundation Research and Incidence Analysis Document Mission Need Phase 1 Definition Concepts & Relationships Registration Negotiation Agreement SSAA Phase 2 Verification Life Cycle Activity (1 to n) Owners impose value wish to minimize Development Activity Correct SSAA Certification Analysis Acceptable Reanalyze Phase 3 Validation Certification Evaluation of Certify The Integrated Ready For Certification Develop Recommendation DITSCAP Flow Countermeasures that may be reduced by may be aware of Threat Agents that exploit Vulnerabilities leading to to reduce that may possess Risk Phase 4 Post Accreditation Accreditation Granted give rise to Threats that increase to to Assets SSAA Operation Change Requested wish to abuse and/or may damage Figure 3-9 Concepts and Relationships Compliance Validation Required in the Common Criteria 6

7 TOE physical Assets requiring protection protection TOE purpose purpose Evaluation Concepts & Relationships Establish Establish security Environment material (PP/ST) Assurance Techniques Evaluation Assumptions Assumptions Threats Threats Organizational security problems security problems produce Gives evidence of CC CC catalog Establish security objectives objectives objectives Establish Requirements Objectives material (PP/ST) Protection Profile require Owners Assurance giving Confidence Functional Assurance Requirements for for the the Requirements material (PP/ST) that Countermeasures Establish TOE TOE summary summary specifications minimize Risk TOE summary specification Specification material (PP/ST) to Assets Use of Evaluation Results PPs Evaluated Products Catalog Catalog (optional) (optional) Catalog Evaluated product Product Questions? Develop & evaluate TOE TOE Evaluation results (alternatives) Accredit system Accredited system accreditation criteria Figure 3-12 Uses of Evaluation Results Evaluation General Techniques Evaluating the Product Evaluating the Process Evaluating Resources Categories of Evaluation Feature analysis: rate and rank attributes Survey: document relationships Case study sample from variables Formal experiment sample over variables 7

8 Example Feature Analysis Feature Tool 1: t-oo-l Tool 2: ObjecTool Tool 3: Importance EasyDesign Good user interface Object-oriented design Consistency checking Use cases Runs on Unix Score Case Study Types Sister projects: each is typical and has similar values for the independent variables Baseline: compare single project to organizational norm Random selection: partition single project into parts Table Design tool ratings Formal Experiment Controls variables Uses methods to reduce bias and eliminate confounding factors Often replicated Instances are representative: sample over the variables (whereas case study samples from the variables) Evaluation Steps Setting the hypothesis the tentative supposition that we think explains the behavior we want to explore Maintaining control over variables decide what effects our hypothesis Making investigation meaningful determine the degree to which results can be generalized Pitfall Common Evaluation Pitfalls Description 1. Confounding Another factor is causing the effect. 2. Cause or effect? The factor could be a result, not a cause, of the treatment. 3. Chance There is always a small possibility that your result happened by chance. 4. Homogeneity You can find no link because all subjects had the same level of the factor. 5. Misclassification You can find no link because you cannot accurately classify each subject s level of the factor. 6. Bias Selection procedures or administration of the study inadvertently bias the result. 7. Too short The short-term effects are different from the long-term ones. 8. Wrong amount The factor would have had an effect, but not in the amount used in the study. 9. Wrong situation The factor has the desired effect, but not in the situation studied. Assessment vs. Prediction An assessment system examines an existing entity by characterizing it numerically Prediction system predicts characteristic of a future entity; involves a model with associated prediction procedures deterministic prediction (we always get the same output for an input) stochastic prediction (output varies probabilistically) Table Common pitfalls in evaluation. Adapted with permission from (Liebman 1994) 8

9 Product Quality Models Boehm s Model Boehm s Model ISO 9126 Model ISO 9126 Model Targeting Item Target Malpractice level Fault removal efficiency > 95% < 70% Original fault density < 4 per function point > 7 per function point Slip or cost overrun in excess 0% > 10% of risk reserve Total creep < 1% per month average > 50% (function points or equivalent) Total program documentation < 3 pages per function point > 6 pages per function point Staff turnover 1 to 3% per year > 5% per year Table Quantitative targets for managing US defense projects. (NetFocus 1995) Software Reuse Producer reuse: creating components for someone else to use Consumer reuse: using components developed for some other product Black-box reuse: using component without modification Clear- or white-box reuse: modifying component before reusing it Process Evaluation Postmortem Analysis a post-implementation assessment of all aspects of the project Process Maturity Models development has built in feedback and control mechanisms to spur improvement 9

10 Postmortem Analysis Design and promulgate a project survey to collect relevant data. Collect objective project information. Conduct a debriefing meeting. Conduct a project history day. Publish the results by focusing on lessons learned. Table When post-implementation evaluation is done. Time period Percentage of respondents (of 92 organizations) Just before delivery 27.8% At delivery 4.20% One month after delivery 22.20% Two months after delivery 6.90% Three months after delivery 18.10% Four months after delivery 1.40% Five months after delivery 1.40% Six months after delivery 13.90% Twelve months after delivery 4.20% Table Required questions for level 1 of process maturity model. Capability Maturity Model (CMM) Question number Question Does the Software Quality Assurance function have a management reporting channel separate from the software development project management? Is there a software configuration control function for each project that involves software development? Is a formal process used in the management review of each software development prior to making contractual commitments? Is a formal procedure used to make estimates of software size? Is a formal procedure used to produce software development schedules? Are formal procedures applied to estimating software development cost? Are profiles of software size maintained for each software configuration item over time? Are statistics on software code and test errors gathered? Does senior management have a mechanism for the regular review of the status of software development projects? Do software development first-line managers sign off on their schedule and cost estimates? Is a mechanism used for controlling changes to the software? Is a mechanism used for controlling changes to the code? CMM level Initial Repeatable Defined Managed Optimizing Table Key process areas in the CMM (Paulk et. al. 1993) Key process areas none Requirements management Software project planning Software project tracking and oversight Software subcontract management Software quality assurance Software configuration management Organization process focus Organization process definition Training program Integrated software management Software product engineering Inter-group coordination Peer reviews Quantitative process management Software quality management Fault prevention Technology change management Process change management Evaluating Resources People Maturity Model goal is to improve workforce 10

11 Table People capability maturity model. (Curtis, Hefley and Miller 1995) Level Focus Key practices 5: optimizing Continuous knowledge and skills improvement Continuous workforce innovation Coaching Personal competency development 4: managed Effectiveness measured and managed, high performance teams developed 3: defined Competency-based workforce practices 2: repeatable Management takes responsibility for managing its people 1: initial Organizational performance alignment Organizational competency management Team-based practices Team-building Mentoring Participatory culture Competency-based practices Career development Competency development Workforce planning Knowledge and skills analysis Compensation Training Performance management Staffing Communication Work Questions? Course Evaluations 11