Proposed Cybersecurity T&E Process

Proposed Cybersecurity T&E Process M r P e t e C h r i s t e n s e n Te s t a n d E v a l u a t i o n P o r t f o l i o M a n a g e r T h e M I T R E C o r p o r a t i o n 1 5 N o v e m b e r 2 0 1 3 W i t h g u i d a n c e a n d s u p p o r t f r o m D r D a v e B e l l, M s S u s a n M a y, M s J e a n P e t t y D A S D D T & E : D r S t e v e n J. H u t c h i s o n, M r To m S i m m s, M r Te r r y M u r p h y O S D D O T & E : M r D a v e A l a n d. A n d m a n y o t h e r s Public Release Pending 2013 The MITRE Corporation. All rights reserved.

2 What, Why and How? What do we want to accomplish? Provide an overview of Proposed OSD Cybersecurity T&E Process Gather your ideas and feedback Why is this important? Threats in Cyberspace are exploiting vulnerabilities at alarming rates DOD policies and procedures are changing to help DOD mitigate risks Government, FFRDC, SETA and Industry partners must collaborate to deliver operationally effective systems How will we do it? 1. Present proposed Cybersecurity T&E Process 2. Gather your feedback as we go and Have fun as we do it! Cyber Goths

Bottom Line Up Front 3 Cyberspace is an ambiguous term Liberally applied prefix to anything! Cyber Space links Social, Information and Physical Networks Massive Attack Surface exposes Information to Threats! Cyber Threats exploit vulnerabilities Threats exercise a Kill Chain DOD SE and T&E Communities must collaborate Or Mission Critical Information will remain vulnerable Systems Acquisition and Test focus must shift Assure the Mission Cyber T&E must assess ability to execute Missions Understand Threats, Evaluate Attack Surface and Kill Chain to close vulnerabilities US DOD T&E Community is working the issue DT&E and DOT&E collaborated on methodology Working Policy, Infrastructure and Workforce

4 Cybersecurity and DOD DoD missions increasingly depend upon complex, interconnected IT environments. These environments are inherently vulnerable, providing opportunities for adversaries to negatively impact DoD missions. A comprehensive T&E program is required to address cybersecurity, starting early in the acquisition lifecycle, to provide early discovery and allow correction of developmental and operational issues in order to support the warfighter. DOD Information Network Graphics Source: WIKIPEDIA Commons

5 Approved US Govt. Cybersecurity Definition Cybersecurity The prevention of damage to, protection of, and restoration of computers, electronic communications systems, electronic communications services, wire communication, and electronic communication. This includes information contained therein, to ensure its availability, integrity, authentication, confidentiality, and nonrepudiation. Defined in National Security Presidential Directive-54/Homeland Security Presidential Directive-23

Working Definition: Attack Surface 6 Information Domain Information Network Image Source: Josh O Sullivan : MITRE Corp. Cyberspace Exposes Information and Data via Interconnected Social, Physical and Information Networks Aug 2011: Comprehensive Experimental Analyses of Automotive Attack Surfaces Source: : University of California, San Diego, University of Washington Attack Surface: A system s exposure to reachable and exploitable cyber vulnerabilities within the system boundaries Source: SANS Attack Surface Problem: http://www.sans.edu/research/security-laboratory/article/did-attack-surface

Working Definition: Cybersecurity Kill Chain and Cyber Attack Lifecycle 7 Cybersecurity Kill Chain: A sequence of actions performed by an adversary to execute cyber attacks with specific objectives, such as data theft. MITRE: Cyber Attack Lifecycle Cyber Attack Lifecycle: A framework to understand and anticipate the moves of cyber adversaries at each stage of an attack. Source: Mandiant APT 1 Attack Cycle

Working Definition: Cybersecurity T&E 8 Cybersecurity T&E: Examination of security measures to reduce the attack surface and mitigate kill chain effects in order to evaluate system resilience in response to threat representative cyber attacks. Cybersecurity T&E is not executed in a vacuum! In collaboration with Users, PM, Systems Engineers, Security Controls Assessors Beginning prior to MS A and in conjunction with existing Systems Engineering and Systems Security Engineering Processes In an incremental and iterative manner prior to Identify and verify baseline security requirements Mitigate exposed vulnerabilities and Assess a system s resilience to execute Critical Operational Missions in response to threat representative cyber attacks including the ability to restore normal operations. Graphic Source: WIKIPEDIA Commons

Challenges with DOD Cybersecurity, Capabilities Development, Systems Acquisition and Test 9 Word Cloud: Google Search for Cyber Security 6,970,000 results (0.31 seconds) Word Cloud: Google Search for DOD 5000 8,620,000 results (0.27 seconds)

Integrated Cybersecurity T&E Includes SE/SSE and CIO Disciplines and Artifacts 10 Materiel Solution Analysis (MSA) RMF Security Plan Early Security Testing (test data) SE Cybersecurity Requirements Validation RMF Security Assessment Plan & TEMP RMF Security Assessment Report/POA&M (attack surface analysis) DT Blue Team Testing T&E Step 5 Operational Cyber Vulnerability Evaluation DT Red Team Kill Chain Testing

Cybersecurity T&E: Planned and Executed in Collaboration: Acquisition, CIO, SE and T&E Aligned 11 Cybersecurity Engineering and Test must be integrated and iterative throughout the acquisition lifecycle Includes all communities Systems Engineering, IT, Security Control Assessor, etc. Integrated Product Teams must align artifacts activities, within acquisition milestones and events.

Cybersecurity T&E Process 12 Step 1 Understand Cybersecurity Requirements Beginning at Pre-MS A or Pre-EMD, with update at Milestone C: Understand Cybersecurity requirements and develop an approach for cybersecurity T&E. Requirements may be specified or implied. Step 2 Characterize the Cyber Attack Surface Beginning at Pre-EMD: Characterize the attack surface; in the integrated environment, determine possible threat vectors. Step 3 Understand the Cybersecurity Kill Chain Post CDR: Analyze and evaluate potential vulnerabilities to determine measures to improve resilience. Step 4 Cybersecurity DT&E Step 5 Operational Cyber Vulnerability Evaluation Step 6 Cyber Operational Resiliency Evaluation Prior to MS C: Cybersecurity DT&E event in a realistic mission environment, with use of cyber range, CNDSP, representative users and Cybersecurity threat representation. Realistic developmental Cybersecurity DT&E event In conjunction with MS C, operational test and evaluation event to assess residual vulnerabilities and risk. Director OT&E must approve entry to Step 6, based on resolution of vulnerabilities. Operational Test Post-MS C operational test and evaluation event to assess operational capabilities to fight through Cyber Attacks Realistic Cyber Threat Test Note: Steps may be iterative to resolve exposed vulnerabilities

Example Cybersecurity T&E Example Mapped to the Acquisition Lifecycle 13 Process fits but is not limited to the DoDI 5000.02 milestones Steps are mapped to both milestones and design review steps Programs have latitude on timing of Step activities Process shifts discovery earlier within the acquisition life cycle and builds in fix-it intervals The findings in any one step may require revisiting a prior step

Simple Example: Comprehensive Experimental Analyses of Automotive Attack Surfaces 14 Modern automobiles are pervasively computerized Engine, Transmission, Body, Airbag, Antilock Brakes, HVAC, Keyless Entry Control, etc. Attack Surface extensive Telematics: Blue Tooth, Cellular, Wi-Fi, Keyless Entry Attack Surface is easily exploited OBD Diagnostics CD players Bluetooth Cellular radio/ Wi-Fi Allow long distance vehicle control, location tracking, incabin audio exfiltration Source: : University of California, San Diego: Stephen Checkoway, Damon McCoy, Brian Kantor, Danny Anderson, Hovav Shacham, and Stefan Savage University of Washington: Karl Koscher, Alexei Czeskis, Franziska Roesner, and Tadayoshi Kohno

Step 1 - Understand Cybersecurity Requirements 15 Identify cybersecurity requirements for Cybersecurity T&E - Review all available program resources - Capabilities Document, Architectures, RFP, System Specification, Program Protection Plan, - Identify critical operational missions and associated information systems - Identify critical mission dependencies on hardware/software components that may be susceptible to cybersecurity intrusions - Identify critical data exchanges and interfaces (include non-programmatic systems if applicable) - Identify additional implied (derived) and essential requirements Identify cyber threat environment to be emulated in test Identify MAC/CL or RMF security categorization Identify cybersecurity test organization(s), including - DIACAP/RMF security controls assessor - Blue Team - Red Team Identify Cybersecurity T&E resources - Cyber range resources(e.g., National Cyber Range (NCR), DoD IA Range, Joint Information Operations Range (JIOR)) (See Backups for more detail) - M&S or tools for cybersecurity Integrate cybersecurity into T&E events and/or Plan for dedicated cybersecurity test events as appropriate and if possible Plan may need to be revised and updated as understanding of Attack Surfaces and Vulnerabilities is refined

16 Cyber Security Requirements Specified requirements Requirements clearly identified in program documentatio ICDs/CDDs, CONOPs, Product Specifications and PPP DoD regulations, such a DoDI 8500.02 (DIACAP). DIACAP IA Controls (IACs) are identified as requirements based on a system s MAC/CL designation Implied requirements Implied requirements are translated into technical requirements that enable the capabilities defined in CONOPS and other operational documentation AKA as Derived Requirements Requirements driven by operational capabilities Requirements driven by acquisition approach and/or technology choices Use of COTS/GOTS and free open source software (FOSS). Implied tasks include additional tasks the developer must accomplish to operate securely Includes the Cyber Threat environment Objective of Step 2 Characterize the cyber attack surface to identify the additional implied cybersecurity requirements. Essential requirements Essential tasks are those that must be achieved to provide sufficient resilience to support mission accomplishment in the presence of cyber attack Objective of Step 3 Analysis of potential kill chain activities to identify essential cybersecurity requirements necessary to improve resilience of the operational system to cyber attack.

17 Cybersecurity Testing Resources Security Controls Assessors (SCAs) Assesses compliance to IA controls Executes the Security Assessment Plan (SAP) Linked to the Certification and Accreditation of the system Based on Security Technical Implementation Guides (STIGs) or similar documentation Can be determined by multiple methods: hands-on testing, interviewing key personal, or examination of relevant artifacts Includes a review of operational and management security controls Conducted with full knowledge and assistance of systems administrators, owner and developer No harm to systems Vulnerability Assessment (Blue) Comprehensive Identifies any/all known vulnerabilities present in systems Reveals systemic weaknesses in security program Focuses on adequacy & implementation of technical security controls and attributes Full knowledge and cooperation of systems administrators Multiple methods used: hands-on testing, interviewing key personal, or examination of relevant artifacts No harm to systems Feedback to developers and system administrators for system remediation and mitigation Penetration Testing (Red) Graduation exercise Exploit one or more known or suspected weaknesses Focus attention on specific problem or attack vector Both internal and external threats Develop an understanding of the inherent weaknesses of a technology Model actions of a defined internal or external hostile entity Conducted covertly with minimal staff knowledge May harm systems and components and require clean up

Example Step 1: Understanding Cybersecurity Requirements/Develop T&E Approach 18 Urban Assault Vehicle Example Requirements Resources 1. System Threat Assessments 2. Capabilities Documents 3. Information Support Plan 4. Program Protection Plan 5. Mission Assurance Category: III 6. Confidentiality Level: Classified 7. Contract Specs Cybersecurity T&E Approach 1. Early T&E involvement 2. Requirements Analysis 3. Design Reviews 4. Contactor SIL Testing 5. Blue Team DT&E Cyber Range 6. Red Team DT&E Event 7. Red Team OT&E in Field DODAF Architecture Products

Step 2 - Characterize the Cyber Attack Surface 19 Characterize the attack surface to identify additional implied cybersecurity requirements and possible threat vectors In the integrated environment, determine possible threat vectors - Utilize cybersecurity SMEs to assist in analyzing the attack surface to determine likely avenues of cyber attack - Examine PPP, System Design, system architecture products (e.g., SV-1. SV-6 viewpoints) to identify interfacing systems, services, and data exchanges that may expose the system to potential threat exploits - Examine system CONOPS to understand roles and responsibilities of a system operators, administrators, and the computer Network Defense Service Provider (CNDSP) - Identify host environment provisions for system protection, monitoring, access control, system updates, etc. - Evaluate early DIACAP/RMF and other security test artifacts

Example Step 2: Characterize the Attack Surface 20 Vehicle Attack Surface 1. Vehicle to Vehicle Comms 2. Telematics 3. Keyless Entry 4. OBD II 5. Radio 6. Anti Theft Urban Assault Vehicle Attack Surface T&E Assess 1. Evaluate Contractor/SIL Security Artifacts 2. Baseline Cybersecurity posture 3. Approach to close/mitigate vulnerabilities 4. Likelihood of attack? 5. What happens if/when exploited?

Step 3 Understand the Cybersecurity Kill Chain 21 Analyze and evaluate potential vulnerabilities to determine measures to improve resilience (cyber range or lab) - Develop initial concept for cyber security testing activities at the component and subsystem level Identify test opportunities to conduct cybersecurity testing in a system of systems context (such as JITC interoperability testing) Identify and integrate DIACAP/RMF security controls assessment activities into unit testing. Functional testing, etc. Evaluate early DIACAP/RMF artifacts - Perform a vulnerability assessment using a Blue Team, to determine likely avenues of cyber attack and the most likely threat exploits Include or emulate the CNDSP Enumerate discovered vulnerabilities and supply to contractor for remediation - Analyze the kill chain to determine how the system would respond in the contested cyber domain

Prototype Cybersecurity Kill Chain Test Overview 22 Blue Team/Red Team Portrays APT Recon SUT and CNDSP Portray Test Items Detect Weaponize Deliver Exploit Control Execute Maintain APT Objectives Exfiltrate data Violate data availability Corrupt data integrity APT attempts multiple attacks while adjusting for success or failure Data Collection Attacker actions Defender detections Defender actions Mission activity Source: Institute for Defense Analysis (IDA), February 2013 Defenders attempt to analyze attacks and determine courses of action Deny Disrupt Degrade Deceive Destroy Recover Defender Objectives Protect Against Intrusions Detect Intrusions React to Intrusions Mitigate Intrusions Determine Responses Restore After intrusion

23 Example Step 3: Kill Chain Analysis Vehicle Attack Surface 1. Deny Vehicle/Vehicle Comms 2. Intercept Telematics 3. Clone Keyless Entry 4. Corrupt OBD II 5. Monitor Radio 6. Disable Anti Theft Urban Assault Vehicle Attack Surface T&E Activities 1. Verify/Exercise Critical Mission Threads 2. Exploit Kill Chain 3. Use Blue Team to Assess Vulnerabilities Vehicle SV-6 Systems Data Exchange Requirements

Step 4 - Cybersecurity T&E 24 Evaluate system-of-systems cybersecurity in a mission context, using realistic threat exploitation techniques Conduct Red Team assessment to identify remaining vulnerabilities - Red Team emulates the threat adversary TTPs - Red Team attempts to exploit the attack surface and execute cyber kill chain activities - Include or emulate the CNDSP - Include typical users if available Identify exploitable threat vectors and vulnerabilities Analyze results to determine impact to mission Collaborate with PM and SE to recommend corrective actions to improve resilience - May include non-materiel solutions such as TTP and recommendations to the CNDSP Cyber resiliency is the ability of a nation, organization, or mission or business process (and supporting systems) to anticipate, withstand, recover from, and evolve to improve capabilities in the face of, adverse conditions, stresses, or attacks on the supporting cyber resources it needs to function.

25 Threat-Based Testing Guided by a validated cyber threat assessment STAR Service/Component Capstone Threat Assessment Focus testing on exploits and TTPs consistent with the threat portrayal Cyber Contested Environment portrayed by Vulnerability Assessment Teams (Blue and Red) How are mission functions impacted by threat adversary? Graphic Sources: WIKIPEDIA Commons

Example Step 4: Full Up DT&E Red Team Event Urban Assault Vehicle Autobahn Mission Exercise Critical Missions 1. Tx/RX Vehicle/Vehicle Comms 2. Cellular Phone Calls 3. Use Keyless Entry 4. Upload/Download OBD II Data 5. Tune Radio 6. Anti Theft 26 T&E Activities 1. Establish Representative Cyber Environment with Threats and Users 2. Conduct Red Team Assessment 3. Understand Mission Impacts 4. Evaluate Test Data 5. Produce DT&E Assessment

27 Step 5 Operational Cyber Vulnerability Evaluation Step 5 - An operational cyber vulnerability assessment to determine readiness for operational evaluation Purpose OTA or a Blue Team will conduct an overt, cooperative, and comprehensive vulnerability assessment in an operational environment Evaluate Configuration management, patch management, network access controls, and system cybersecurity tools Leverage production-representative DT&E data to the maximum extent possible Provide vulnerability evaluation results and recommendations to materiel developers, as appropriate for remediation Vulnerability results should not be provided to Red Teams performing Step 6 Correcting all vulnerabilities found during this step will be the entrance criteria for Step 6 (Note) Vulnerabilities identified in Step 5 may require re-testing Preparation for IOT&E This step may also make use of available developmental test events and data as appropriate.

Example Step 5: Operational Cyber Vulnerability Evaluation 28 Exercise Critical Missions 1. Tx/RX Vehicle/Vehicle Comms 2. Cellular Phone Calls 3. Use Keyless Entry 4. Upload/Download OBD II Data 5. Tune Radio 6. Anti Theft T&E Activities 1. Establish Representative Cyber Environment with Threats and Users 2. Conduct Blue Team Assessment 3. Evaluate Test Data 4. Determine readiness for OT&E

29 Step 6 - Cyber Operational Resiliency Evaluation Step 6 - A full-up operational test of the system-of-systems in a representative operational and threat environment Purpose Conduct an independent and comprehensive evaluation of protect, detect, react, restore capabilities, to include exploitation potential, and mission impact. Some system information and network information may be provided to the Red Team to facilitate the cybersecurity evaluation Red Team should not have access to the detailed Step 5 vulnerability evaluation Discover: How well do the system s cybersecurity capabilities protect key/critical information and data? Does the system s ability to detect penetration and penetration attempts support the rapid identification of hostile cyber activity? Does the system support rapid reaction and mitigation of penetration/exploitation? Does the system support reconfiguration and restoration of critical services, data, and functions? Systems with High/Medium risk to CIA for system information; COOP and contingency plans must be evaluated

Example Step 6: Penetration Testing with Representative Threat 30 Urban Assault Vehicle Autobahn Mission Exercise Critical Missions 1. Tx/RX Vehicle/Vehicle Comms 2. Cellular Phone Calls 3. Use Keyless Entry 4. Upload/Download OBD II Data 5. Tune Radio 6. Anti Theft T&E Activities 1. Establish Representative Cyber Environment with Threats and Users 2. Conduct Red Team Assessment 3. Understand Mission Impacts 4. Evaluate Test Data 5. Produce OT&E Assessment

Cybersecurity T&E Key Take Aways! 31 Cybersecurity T&E Process activities begin pre-milestone A and continue throughout the Acquisition Lifecycle Collaborative process helps translate cybersecurity requirements, host environment, threats, etc. into testing activities Cybersecurity T&E process requires the development and testing of mission-driven cybersecurity requirements Requires systems engineering, systems security engineering and T&E expertise. Test and Evaluation Master Plan (TEMP) must detail How testing will provide the information needed to assess cybersecurity and Inform Systems engineering, Risk Management and Acquisition Decisions. Test activities must integrate RMF security controls assessments and Tests of commonly exploited and emerging vulnerabilities early Cybersecurity DT&E is expected to identify issues related to resilience of military capabilities before MS C Early developmental T&E provides data and feedback to the PM and SE Teams Informs requirements, facilitates change to minimize impact on cost, schedule, and performance Cybersecurity OT&E is expected to ensure that the system under test can Withstand realistic threat representative cyber-attacks and Return to normal operations in the event of a cyber-attack. Cybersecurity T&E Process represents a shift left Because it requires early T&E involvement.

Closing 32 Next time someone says Cyber.. Stop and ask them what they really mean Cyber Goths Attack Surface in Cyberspace is massive How big is your attack surface? Is your information protected? Cyber Threats exploit vulnerabilities Cyber Kill Chain must be understood and disrupted! Current US DOD processes are being changed Systems Engineering, Systems Security Engineering, Developmental and Operational Test Communities are collaborating Cyber Security must protect Mission Critical Information Information is the What Mission Assurance is the Why! T&E seeks to ID Attack Surface and Disrupt Kill Chain! Close High Risk Vulnerabilities Early US DOD T&E Community: Actively working the problem Methodology and Policy in work to shift discovery to left Cyber T&E Infrastructure and Workforce will enable and execute

Questions, Comments, Recommendations? Pete Christensen T&E Portfolio Manager OSD Portfolio pchris@mitre.org T h e i n t e r n e t i s o n e g i g a n t i c w e l l - s t o c k e d f r i d g e r e a d y f o r r a i d i n g ; f o r s o m e s t r a n g e r e a s o n, p e o p l e g o u p t h e r e a n d j u s t g i v e s t u f f a w a y. M e g a ' Z i n e s, M a c w o r l d ( 1995) Public Release Pending 2013 The MITRE Corporation. All rights reserved.