Testing & Training Cross Talk Forum: Scoping the Cyber T&E Challenge

Transcription

1 Testing & Training Cross Talk Forum: Scoping the Cyber T&E Challenge Ryan Norman Test Resource Management Center 4800 Mark Center Drive Suite 07J22 Alexandria, VA (571)

2 Cyberspace T&E Problem Cyberspace vulnerabilities leave our warfighters vulnerable Warfighter s mission is at risk to cyberspace attacks A cyberspace attack could render systems inoperable and prohibit warfighters from completing their missions Cyberspace vulnerabilities in DoD warfighting systems and information systems are our biggest concern How can the T&E community help? Cyberspace vulnerabilities need to be tested and resolved during system development, prior to operational testing Existing DoD laboratories and ranges the testbeds for DoD systems are essential to test cyberspace defense TRMC Immediate Focus: Computer Network Defense (CND) of Weapon and IT Acquisition Systems 1

3 Specialty Engineering Development/Design Engineering Decision Authority Input Requirements Definitions (JCIDS) Cyberspace T&E within the Acquisition Process Operational Need Measures of Effectiveness/Suitability Output to Next Phase of Development or Lifecycle Support Transition Identify Cyberspace Measures Decomposition & Definition Cyberspace Vulnerability Assessments Component Level Testing to the chipset level Requirements Analysis Understand System of Logical Systems Dependencies Analysis Design Solution Detailed Design System Measures of Performance Allocated Functions & Performance Requirements Component/Interface Definition Element Design Criteria Validation Ensure System of Systems Mitigation of Verification Cyberspace Threats Integration Confirmation Audits OT&E Validation against actual Cyber Threats Realization & Assessment DT&E Verification in Realistic Environment Integrated Component Testing Implementation Tech Data & Training Pubs Component Characterization Hardware Fabrication Software Creation/Coding V Chart Source: DAU Acquisition Community Connection 2

4 Cyberspace T&E Strategy Overview Test & Evaluation that accurately and affordably measures cyberspace effectiveness and vulnerabilities of warfighting systems and DoD information systems, to verify the warfighter s capability to achieve mission success while operating in cyberspace Cyberspace Attack Effects and Impacts: Unauthorized Access Unauthorized Use Disruption of Ops Loss of Control Data Corruption Data Fabrication Target Spoofing Cyberspace Test Tools Systems Under Test Cyberspace T&E Vision Federated Cyberspace T&E Capability Cyberspace Threat Representations Cyberspace Test Instrumentation An integrated T&E enterprise capable of creating a realistic cyberspace test environment at all required security levels, collecting performance & vulnerability test data, and assessing effects Four Major Thrusts 1.Cyberspace T&E Process Additional activities to test cyberspace during the acquisition process 2.Cyberspace T&E Methodology Test approach to adequately assess cyberspace capabilities/limitations 3.Cyberspace T&E Workforce T&E training to enable T&E professionals to conduct future cyberspace T&E 4.Cyberspace T&E Infrastructure Existing DoD Labs, Ranges, & Networks Industry & Academia Accessible Common Framework for: Cyberspace Environment Tools Cyberspace Test Instrumentation 3

5 Some Tenets of a Successful Cyberspace T&E Strategy 1. Mission Focus 2. Enterprise Perspective Beyond DoD 3. Tiered Management Structure & Accountability 4. Leveraging Existing Processes & Capabilities 5. Utilize Agile Concepts Cyberspace T&E Vision: T&E that accurately and affordably measures cyberspace effectiveness and vulnerabilities of warfighting systems and DoD information systems, to verify the warfighter s capability to achieve mission success while operating in cyberspace 4

6 Cyberspace T&E Strategy: Infrastructure Enterprise that supports DoD cyberspace testing needs Automated IA Test Tools Analysis Tools Test Control Sim/Stim Instrumentation Infrastructure Core Traits Managed Common Framework: Ensure interoperability of developed infrastructure Robust & Contained Environment Instantiation: Enterprise components that address cyberspace T&E needs Non-obtrusive instrumentation: Data capture of cyberspace measures Assessment Teams Readily Available: Infrastructure that spans C/S/A s and interfaces with other government, industry, and academia asappropriate Enterprise Knowledge Management: Infrastructure must enable collaboration, learning, and sharing data Net Ops Red Teams Credentialed Centers of Excellence : Managed & implemented process for documenting quality of cyberspace facilities and capabilities 5

7 The TRMC Blueprint : Putting Test Capabilities on the DoD Map Quadrennial Defense Review Strategic Planning Guidance Service T&E/S&T Working Groups DoD Strategic Plan for T&E Resources Service T&E Needs and Solutions Process TRMC Joint Investment Programs Risk mitigation needs Technology shortfalls Requirements Risk mitigation solutions Capabilities Advanced development (6.3 Funding) (6.4 Funding) (6.5 Funding) Service Improvement & Modernization/ Programs Acquisition Programs / Advanced Concept Technology Demonstrations T&E Multi-Service / Agency Capabilities DoD Corporate Distributed Test Capability 6

8 JMETC, JIOR, and You We must all work together to maximize our ability to perform cyberspace test and training The TRMC Joint Mission Environment Test Capability (JMETC) now has responsibility for Cyber T&E infrastructure Will leverage existing TRMC processes to ensure the best bang for the buck Investments can be beneficial to test & training communities Coordination has begun with Joint Staff J7 Joint and Coalition Warfare JMETC and JIOR have been in coordination for several years Working towards established processes for persistent connectivity in support of emerging test & training requirements Obviously, there are many policy / IA impediments Progress being made slowly but surely 7

9 Summary Cyberspace vulnerabilities leave our warfighters vulnerable Cyberspace T&E must become as agile as the threat Cyberspace T&E Vision Thrust Areas: 1. Process 2. Methodology 3. Workforce 4. Infrastructure Starting in FY13, the Joint Mission Environment Test Capability is resourced to provide Cyberspace Test Infrastructure Testing and Training collaboration essential for mutual success of both missions 8

10 Questions? Ryan Norman Test Resource Management Center 4800 Mark Center Drive Suite 07J22 Alexandria, VA (571)

11 Backup Slides Not Intended for Presentation

12 Computer Network Operations Defend Cyberspace Defensive Cyberspace Operations (DCO) Actions taken to protect, monitor, analyze, detect and respond to unauthorized activity within Department of Defense information systems and computer networks. CND employs IA capabilities to respond to unauthorized activity within DoD information systems and computer networks in response to a CND alert or threat information Note: CND also employs intelligence, counterintelligence, law enforcement, and other military capabilities to defend DoD information and computer networks. Own Cyberspace Offensive Cyberspace Operations (OCO) Actions taken through the use of computer networks to disrupt, deny, degrade, or destroy information resident in computers and computer networks, or the computers and networks themselves. Computer Network Exploitation (CNE) Enabling operations and intelligence collection capabilities conducted through the use of computer networks to gather data from target or adversary automated information systems or networks. Our T&E Infrastructure must support both Defending & Owning Cyberspace. Existing laboratories and ranges are essential to test CND. 11

13 Development Progression Development Stage DT&E Functions DT&E Cyberspace Functions Requirements Engage early Identify Cyber Defense Measures to be Evaluate testability tested Design Component Test Integration System Test Interoperability Cyberspace T&E Strategy: T&E Process DT&E Activities for Cyberspace Assist in engineering design Verify component specification compliance and functionality Verify system specification compliance Assess human-system integration Progress to plan for Reliability Growth Characterize system functionality Stress the system within an intended mission environment Provide assessments of capabilities and limitations Verify KPPs, COIs, CTPs, and KSAs Assess progress towards meeting COIs Assess system safety Support joint interoperability process Assess compatibility with legacy systems Assurance, Certification, and Accreditation Assess vulnerability, susceptibility, and threat Test individual hardware / software components (SW/HW Assurance) Ensure SUT instrumented to Chipset level Test integrated components in integration lab SUT w/digital mission traffic & scenario simulation Penetration tests Penetration effects tests Test & Evaluate standards conformance Perform live interoperability tests Information Assurance Activities Cyberspace Testing Look beyond compliance to performance Certify system readiness for OT&E Test in realistic cyber environments - Benign - Threat Test in COCOM mission scenarios 12

14 Mechanism(s) for sharing best practices for cyberspace T&E throughout the community Define test procedures to measure and assess cyberspace capabilities and limitations Design Reference Missions (DRM) Threat Reference Designs (TRD) Example IA Attributes: Availability, Integrity, Authentication, Confidentiality, Nonrepudiation CND Metric Types: Cyberspace T&E Strategy: Methodology Test Procedures & Best Practices for Cyberspace Protection: Available, timely, & reliable access to data and information Detection: Measured time to determine threat has penetrated defenses Reaction: Measured Response from Human Factors, SUT, and Network Warfighting Environment Test Environment Restoration: Measured time and degree to which operations are returned to normal 13

15 Cyberspace T&E Strategy: Workforce Cyberspace Curriculum for T&E Professionals Cyberspace activities/operations require an additional skill set for the T&E Workforce (current focus is on Weapon Systems Acquisition) Additional Cyberspace T&E skill set needs to include knowledge of: Performing T&E Cyberspace functions. Computer Network Operations (CNO) Technology trends Potential Cyber Threats (Communications, Software and Hardware) Cyberspace T&E infrastructure, Environment Tools, Test Instrumentation, and Test Control Tools T&E Cyberspace functions include: Identifying Cyber Defense Measures that need to be tested Assess vulnerability, susceptibility, and threat Planning and executing of Penetration Tests on Systems and the evaluation of results Planning, Executing, and Evaluating System-of-Systems Test events in realistic cyberspace environments (both benign and threat) Strategy: Leverage ongoing competency assessment of the DoD T&E Workforce to identifying gaps, create Learning Objectives, and develop training material on Cyberspace T&E Encourage the use of external Cyberspace training opportunities 14

16 Cyberspace T&E Strategy Proposed Implementation Plan Define Traits of Major Thrusts Develop Cyberspace T&E Reqmts. Begin AoA s and CBA s Develop Cyberspace T&E Roadmap Increment #1 FY12+ Validate Progress through Test Pilots Increment #0 Cyberspace T&E Vision & Major Thrusts : Provides unified end state for cyberspace DT&E and test infrastructure. Continue to refine and socialize with C/S/As Define Traits of Major Thrusts : Functional decomposition of qualities necessary to achieve cyberspace T&E vision Develop Cyberspace T&E Requirements: TCRD-like document that describes process, methodology, workforce, and infrastructure requirements that must be met to achieve cyberspace T&E vision Begin Analysis of Alternatives (AoA) and Capabilities Based Assessments (CBA): Identify existing leverage opportunities that could help and risk items that could hinder achieving cyber T&E vision Develop Cyberspace T&E Roadmap: Implementation plan to realize defined requirements Validate Progress through Test Pilots: Ensures requirements are satisfied and matured as needed 15

17 InterTEC Cyber Event (ICE) 2011 Test Objectives 1. Conduct a series of tests that measure progress towards stakeholder goals: Verify Interoperability: Establish baseline operational mission performance in a benign environment conducted as part of InterTEC Systems Acceptance Testing (SAT) to leverage existing, proven methodology Conduct Cyber Defense Test: Conduct representative cyberspace attacks to evaluate mission performance in a threat-enriched operational environment 2. Gauge progress towards cyberspace T&E vision in context of cyberspace T&E major thrust traits: Assess Infrastructure connectivity, tools, and processes for repeatability and gaps Assess validity of utilized cyberspace T&E methodologies and measures JCAS Operational Use Case provides realistic context for achieving 2011 ICE test objectives 16

18 Key ICE 2011 Deliverables Three ICE 2011 Systems Under Test 1. [THRUST AREA: Methodology] Initial Methodologies for conducting cyberspace T&E 2. [THRUST AREA: Methodology] Preliminary set of CND DT&E metrics and measures 3. [THRUST AREA: Infrastructure] Gap analysis for a more robust red and blue cyberspace T&E infrastructure 17