This Conference brought to you by www.ttcus.com Linkedin/Group: Technology Training Corporation @Techtrain Technology Training Corporation www.ttcus.com
U.S. Army Intelligence and Security Command Army Intelligence and Big Data 29 Sept 2015 INSCOM the Army s Force for Dominant Intelligence
Why? Big Data All Sensors need Processing, Exploitation and Dissemination (PED) 3
Why? New Army Operating Concept 4
Unified Cloud Data (UCD) Unified Cloud Data (UCD): A Joint, Interagency Effort Aimed at Defining the Service Technical Approach to the ICITE Framework and Data While Solving the Global Processing, Exploitation & Dissemination (PED) Migration Challenges UCD: Our Big Data Reference Implementation of a Unified Data Layer All Data Work, Regardless of Type of Intelligence and Security Level All Analytics + Enrichment Processes Run against UCD: Write Once, Use Often All Indexes Make UCD Data Discoverable to Analysts Security Is Baked In Security Markings Are Integrated at the Source Level, Event / Document / Entity level, Down to Individual Attributes for an Entity UCD Pilot Army s Instantiation, Led by INSCOM, Assessed by Soldiers in Live Environment To Inform Army Programs of Record Army Mission Command and Army Cyber Pilots show Benefit of Unified Data for Operations-Intel Convergence for the Army Operational Concept (AOC) 5
Extending Cloud Advanced Analytics 6
Open Source Information Alerts Analysts Use Open Information Pilot New Analytics Train Analysts on Sources and Capabilities Sources Change; Terms Evolve Stay Engaged: Frequent Changes Share Information, including Concepts of Operation Put into Context t of What s Known Mature Techniques to Verify and Understand: Who, What, Why?? Certify for Policy Compliance Reinforce Mission and Legal Authorities Do the Right Analysis, the Right Way Leverage the Enterprise: Analytic Capabilities Data Approach: Pay for Data Once, Use Many Ways Enterprise Impact: Data Retention, Storage, Correlation, Cyber Security Big Data: Useful for Tipping i + Cueing, But Has Risks 7
The Enterprise is the Foundation Foundation Layer Backbone 8
Unified Cloud Data (UCD) Partners & Pilots 9
Unified Cloud Data (UCD) Partners & Pilots Demonstrated Value: + Soldiers used UCD + Live Data + Saved Mission Command 7+ months to pilot Ops/Intel Convergence, DTRA to pilot Constellation on UCD baseline + Remote Mgt + Puppet deployment lets 1 SysAdmin manage multipleucd sites + Piloted AWS GovCloud for Integration: Saved $ 10
Implementing Unified Cloud Data (UCD) 11
Good Feedback from Functional Assessment Assessed UCD + Value of Big Data Analytics to Inform Future Requirements Good Initial Feedback: Users Want More Access + More Data Improved Analyst Usability: Ease of Use: Easy Multi-INT exploitation of Unified Data with Widgets + Workflows Rapid Mastery: After 3 days training, Soldiers could use UCD for mission threads Speed of Analysis: Soldiers used UCD to do Country Study in 30 min (1/3 time) Fast Data Access: Facial Recognition in seconds against 100Ks of records Operations-Intelligence Convergence: Improved Situational Awareness: Blue Force + Red Data in Common Operating Picture Pre-Deployment Checks: Soldiers easily checked New Area: Know What s Known Enterprise Efficiencies and Security: Built-In Support: Self-Configurable Dashboards + Workflows Soldiers can share Info Sharing: Built-in Reports/Report Creation (no support needed), Coalition Info Sharing Enterprise Operations: Remote Admin by Fewer System Admins support multiple sites Improved Security: Cell-level Security, Thin Client/PKI, Separation of Roles 12
UCD Support of Intel Functions UCD handles many kinds of data Sensors Data from Many Sources/Types Images Audio Video Messages Public Info Mission Command Etc. Separate Data from Analytics Security: Provenance Security Labels Metadata Tagging Extract Entities + Geo/Temporal Attributes Metrics more Velocity + Data Ingestion Content Real-Time Cell-Level Security Analytics Update Indexes Data Access: Match User Roles/Authorizations against Data Security Analyst s Conclusions Enrich Data User Authorizations Community Partners Correlate All Data Context-Based Data Navigation + Beneficial to All Domains: + Operations-Intel Convergence + Medical Support + Logistics Support Map Reduce Analytics Enable Data Sharing Analysts Enrich Correlated Data: + Know What s Known Now + Helps Analysts Connect the Dots Supports Big Data Analytics and Multi-Discipline Fusion for AOC environments: A2AD, Megacities, Ad Hoc Response 13
UCD Support of Intel Functions UCD handles many kinds of data Separating Data from Apps lets Analytics Use Same Data: Read Once, See Data Many Ways
UCD Support of Intel Functions UCD handles many kinds of data Separating Data from Apps lets Analytics Use Same Data: Read Once, See Data Many Ways Counter-Insider Threat Security checks
UCD Support of Intel Functions UCD handles many kinds of data Separating Data from Apps lets Analytics Use Same Data: Read Once, See Data Many Ways Counter-Insider Threat Security checks UCD implements full CRUD functionality: Analysts can Create, Read, Update, Delete
UCD Support of Intel Functions UCD handles many kinds of data Separating Data from Apps lets Analytics Use Same Data: Read Once, See Data Many Ways Counter-Insider Threat Security checks UCD Unifies Data UCD implements full CRUD functionality: Analysts can Create, Read, Update, Delete
UCD Support of Intel Functions Separating Data from UCD handles many kinds of data Apps lets Analytics Use Same D Data: U S t Read Once, See Data Many Ways UCD Unifies Data Fine-Grained Data Security Markings are stored in Accumulo Counter-Insider Threat Security checks UCD implements full CRUD functionality: Analysts can Create, Read, Update, Delete
UCD Support of Intel Functions Separating Data from UCD handles many kinds of data Apps lets Analytics Use Same D Data: U S t Read Once, See Data Many Ways UCD Unifies Data Fine-Grained Fi G i d Data D t Security Markings are stored in Accumulo Fast GPU-based Geospatial and Temporal indexing Counter-Insider Threat Security checks UCD implements full CRUD functionality: Analysts can Create, Read, Update, Delete
UCD Support of Intel Functions Separating Data from UCD handles many kinds of data Apps lets Analytics Use Same D Data: U S t Read Once, See Data Many Ways Counter-Insider Threat Security checks UCD implements full CRUD functionality: Analysts can Create, Read, Update, Delete UCD Unifies Data Integrated g Capabilities p such as Facial Recognition Fine-Grained Fi G i d Data D Security Markings are stored in Accumulo Fast GPU-based Geospatial and Temporal indexing
UCD Support of Intel Functions Separating Data from UCD handles many kinds of data Apps lets Analytics Use Same D Data: U S t Read Once, See Data Many Ways Counter-Insider Threat Security checks UCD implements full CRUD functionality: Analysts can Create, Read, Update, Delete UCD Unifies Data Integrated g Capabilities p such as Facial Recognition Fine-Grained Data Security Markings are stored in Accumulo Can Make Workflows for Analyst Tasks + for MapReduce Jobs Fast GPU-based Geospatial and Temporal indexing
Constraints We Must Handle Conform to the Enterprise Expect Change New Kinds of Data: OSINT/Social Media New User Needs: Heat Map New Capabilities from Partners: WAMI Track Extraction from NGA New Technology Components: GPU Processors for Geospatial Data Reduce Costs Software Licenses Physical Footprint IT Support Leverage Open Source / Other Software Increase Security Provenance: Track Every Interaction PKI and Thin Client Bastion Node Construct Remote Administration: Separate System Administration from the Data Exploit Exponential Increase in Data 22
UCD Lessons Learned Protect the Data: Build on a secure Cloud architecture with cell/object-level security, and extend security down to the weapon system to be able to Counter Insider Threat (with PKI, provenance etc.). "Big Data Strategy with Unified Data: Capture and triage vast, increasing amounts of data of all types, from many sources, with automated "Big Data" analytics. Analytics from different providers should leverage the same correlated data: write once, use many ways. Enterprise-Scale Remote Administration: Leverage automated build, delivery, install, configuration management, system administration, and monitoring to make operational use simple and scalable to the Enterprise level: automate routine tasks so humans focus on problems. Leverage Prior Capabilities with Enterprise Platform: Break apart legacy stovepipe capabilities from battlefield-proven Quick Reaction Capabilities (QRCs); re-host unique components on IC ITE security architecture to ensure needed capabilities endure for the future. Transform Capabilities Acquisition: To benefit from the innovation & speed of new capabilities, at long-term greatly reduced cost, incentivize organizations and individuals to use Open Source software, Agile business models of vendors & gov t organizations. Encourage and Reward Partnering: Strong INSCOM + Mission Command partnership integrated INSCOM UCD software onto MC tactical cloud servers: at NIE 15.1, 1 showed Intel/Ops Convergence, situational awareness, DIL operations, Bde TOC server consolidation Future Ready for Capabilities Integration: Future analytic capabilities that exploit Unified Data can be rapidly integrated and (done right) can inherit security. Need New Categories of Innovative Capability: Cyber Security and Info Assurance require new Enterprise analytics, to understand threats, attacks, system health but we also need Innovative Capabilities + Processes to demonstrate Info Assurance (IA) and Capabilities Security. 23
U.S. Army Intelligence and Security Command Questions INSCOM the Army s Force for Dominant Intelligence 24