Cyber Security With Big Data

Transcription

1 Cyber Security With Big Data Fast. Complete. Cost-Effec1ve. Harry J Foxwell, PhD Principal Consultant Oracle Public Sector Oct 2015

2 Safe Harbor Statement The following is intended to outline our general product direcoon. It is intended for informaoon purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or funcoonality, and should not be relied upon in making purchasing decisions. The development, release, and Oming of any features or funcoonality described for Oracle s products remains at the sole discreoon of Oracle. Oracle ConfidenOal Internal/Restricted/Highly Restricted 2

3 Pressures on Tradi1onal Tools Mobile Users Growing Data Volume & Variety Demand for PredicOve AnalyOcs Copyright 2014 Oracle and/or its affiliates. All rights reserved.

4 Big Data Issues Ingest Rates (Velocity) Simultaneous ingest and query cause conflicts and delays in the system Data Reten1on (Volume) Insufficient periods of Ome for idenoficaoon of trends in data Requirement is for mulo-year storage and retenoon = mulo-pb Poor Query Performance Hours-Days for results, reducing/limiong number of queries that can be executed Challenge for the analyst to visualize the problem and refine query dynamically Queries performed on separate compuong pladorms due to poor performance Need hierarchy of systems to conduct queries (?) 10/8/15 HHB Systems LLC 4

5 SIEM Deficiencies Ingest Rates Unable to capture all events in a medium-sized network Systems struggle to execute queries while collecong data Data Reten1on Log file sizes are increasing dramaocally Need data over long Omeframes to find the Advanced Persistent Threat Poor Query Performance Hours to days for queries to execute Analysts not able to conceptualize results and re-query Results Missed events & trends More analysts required, but focused on basic event ID, instead of cyber intel Excessive system architecture costs

6 SIEM Deficiencies, Example Use Case Ingest Rates Can only capture <4,500 Events Per Second (EPS) and require 70,000 EPS Data RetenOon Only retain 14 days of full data (limited data for 30 days) and require 5 years Poor Query Performance Analysts logger searches take up to 10 hours for a 7 day query Limited Scalability Unable to accommodate all potenoal users of the system Scaling up Exis1ng Logger Architecture Would Cost $22.5M!

7 Resul1ng Effects PotenOally missed events or longer remedia1on 1mes due to delays Some incidents are missed due to lack of storage capacity Takes longer to iden1fy a trend due to storage requirements Less produc1ve staff and more analysts required More personnel focused on lower levels (event idenoficaoon and validaoon) rather than on cyber intelligence Slow system performance Costly HW and SW soluoons 10/8/15 7

8 Today s Data Architectures User Tools OLTP Systems Data Warehouse BI Tools and Dashboards Custom/Advanced Analy1cs

9 Acquiring and Using More & Varied Data Data Sources Big Data Ecosystem User Tools High Volume Distributed File System NoSQL DB Data Warehouse BI Tools and Dashboards Custom/Advanced Analy1cs

10 Discovering Valuable Data Anywhere Data Sources Knowledge Discovery Engine Big Data Ecosystem User Tools Informa1on Discovery High Volume Distributed File System NoSQL DB Data Warehouse BI Tools and Dashboards Custom/Advanced Analy1cs

11 Oracle s Big Data Strategy There are a lot of commonly held beliefs about big data that need to be challenged, with the first being that you simply adopt Hadoop and are good to go. The problem is that Hadoop is a technology, and big data isn't about technology. Big data is about business needs. In reality, big data should include Hadoop and rela1onal [databases] and any other technology that is suitable for the task at hand. Ken Rudin, Head of AnalyOcs, Facebook Big Data = Hadoop + NoSQL + Rela1onal Oracle ConfidenOal Internal 11

12 2015 O Reilly Data Science Survey Four tools SQL, Excel, R, and Python remain at the top for the third year in a row Spark (and Scala) use has grown R is now used by more data professionals who otherwise tend to use commercial tools hqp://duu86o6n09pv.cloudfront.net/reports/2015-data-science-salary-survey.pdf Oracle ConfidenOal Internal/Restricted/Highly Restricted 12

13 Big Data Analy1cs Challenge Separate data access interfaces 13

14 Oracle Big Data Management Preserving investment with transparent Big Data access NoSQL 14

15 Oracle Big Data SQL Massively Parallel SQL Query across Oracle, Hadoop and NoSQL SQL SQL Offload Query to Data Nodes data subset Small data subset quickly returned Offload Query to Exadata Storage Servers Hadoop & NoSQL Oracle Database 12c 15

16 Oracle s Open Source Support Where & When it Makes Sense

17 Big Data Appliance X4-2 Sun Oracle X4-2L Servers with per server: 2 * 8 Core Intel Xeon E5 Processors 64 GB Memory 48TB Disk space (864 TB of storage in a Full Rack) Integrated Software (3.0): Oracle Linux Oracle Java VM Cloudera Distribution of Apache Hadoop (CDH) 5.0 Cloudera Manager 5.0 and Options Apache Spark Oracle R Distribution Oracle NoSQL Database

18 Cyber POC Architecture Port Mirror Security Analyst Protected Network 1. Copy of all data packets 2. Log Files Oracle Big Data and Rela1onal Data Stores Predic1ve Aqack RecogniOon, Session and File AnalyOcs

19 Enterprise Data Architecture Data Sources Knowledge Discovery Engine Big Data Ecosystem User Tools Informa1on Discovery High Volume Distributed File System NoSQL DB Rela1onal Data Machine Learning Algorithms BI Tools and Dashboards Custom/Advanced Analy1cs Real-Time Recommenda1ons

20 Oracle Big Data Management System DATA RESERVOIR DATA WAREHOUSE ANALYTICS Cloudera Hadoop Oracle Big Data SQL Oracle NoSQL Oracle R Advanced Analy1cs for Hadoop Oracle R Distribu1on Big Data Appliance Oracle Event Processing Big Data Discovery Oracle Big Data Connectors Oracle Data Integrator Oracle Big Data SQL Oracle Database In-Memory, Oracle Mul1-tenant Industry Models Oracle Industry Oracle Models Advanced Analy1cs Oracle Advanced Oracle Spa1al Analy1cs & Graph Oracle Spa1al & Graph Oracle GoldenGate Exadata Oracle Real Time Decisions Oracle Business Analy1cs Enterprise Business Intel Self-Service Discovery Data Mashup Endeca Info Discovery Exaly1cs OBIEE SOURCES

21 Example: Packet Analy1cs On Big Data Appliance Tested several different cluster sizes Processed data sets from TB Completed full analysis of 100TB of data in 16 hours Real-Time Analysis of a 14Gbs data stream

22 Example: ArcSight on Eng System Results Achieved 70,000 EPS during POC OpOmized later for over 115,000 EPS or 25x more data Exadata compression produced 14x storage improvement Queries returned 27x-6,500x faster Logger query that previously took 9 days is now back in <2 mins ~90% Cost Reduc1on (Saving over $20M)

23 Reduce the Incident Handling Life Cycle Enhance prepara1on and data availability to analysts Enable faster detec1on due to enhanced event correlaoon and processing Facilitates collabora1on and analyoc support for containment Enables enhanced valida1on of architecture to idenofy other similar aqack locaoons Captures and retains data for corporate history and community collaboraoon reducing post incident acovity resources and Omelines 10/8/15 Page 23

24 How Oracle Engineered Solu1ons Support Incident Response Accelerates mission 1melines, reducing threat 1melines Enables precision responses Enables a new class of real 1me strategies and analy1cs Speeds dispatch to mgmt, enables analy1cs and automa1on Enables more dynamic organiza1onal management and response Integrate over longer period to catch low and slow aoacks & track campaigns Roadmap for maturing the incident response capability- provides scalability and flexibility for work flow and collabora1on How the program fits into the overall organizaoon enables integra1on SOC/CERT/ Focused Intelligence 10/8/15 Page 24

25 Enables focus on Cyber Intelligence The enabling plarorm of advanced automa1on and cyber analy1cs 10/8/15 25

26 Q&A

27 Oracle ConfidenOal Internal/Restricted/Highly Restricted 27

28 Net Collector Filtering, Aggrega1ng, Reconstruc1ng Network Packets Integrated low-cost probes Up to 40Gbps at each probe Embedded real-ome NoSQL data store Intrusion Detec1on System & Deep Packet InspecOon capability ExtracOon of metadata, content and staosocs Smart data transportaoon

29 Net Aggregator Managing and Analyzing the Data Mul1ple data models for different types of analysis Unified system metadata Real-Ome and batch processing IntegraOon with External Repositories and AnalyOcs via open APIs

30 Analysis Desktop Rich visualiza1on and Informa1on discovery Network (graph) & map view Guided navigaoon Data & text analyocs Interac1ve dashboards ReporOng and publishing Session ReconstrucOon 3rd Party Analy1cs Copyright 2014 Oracle and/or its affiliates. All rights reserved.

31 Endeca InformaOon Discovery Chaos Becoming Order Chaos Becoming Order Data that was: Previously disjointed Difficult to access Hard to understand Is now: Presented in COPs Single sign on Visually driven Discovery learning Copyright 2014 Oracle and/or its affiliates. All rights reserved.

32 Oracle Big Data Discovery Changing the Game for Agile Government Innova1on on Big Data Profile Find Understand Transform Discover Predict Collaborate Easily add data and see it automaocally and cononuously cataloged, enriched and related Use familiar guided search across massive amounts of diverse data Know what s important from diagnosoc analysis of millions of data characterisocs Powerful tools to quickly clean up and wrangle dirty data so it s ready to go Uncover valuable new insights Use new insights to define and refine predicove models Publish, share and evolve as you learn more 32

33 Breakthrough InnovaOon: Oracle Big Data Discovery The Visual Face of Hadoop Radically simplify data prep and Analysis process 33

34 Extending Data Management Big Data Management System Rela1onal Run Agency Integrate exisong systems Support mission-criocal tasks Protect exisong expenditures Insure skills relevance NoSQL Scale Agency Meet mobile challenges Accelerate developer agility Scale-out economically Serve data faster Hadoop Change Agency Disrupt fraudsters Improve supply chains Leverage new paradigms Exploit new analyses Oracle ConfidenOal Internal/Restricted/Highly Restricted 34

35 Overcoming Barriers to AdopOon of New Technologies INTEGRATION SKILLS SECURITY SQL Engineered Systems SQL on All Data Database Security on All Data ConfidenOal 35

36 Engineered for Big Data Management at Scale Engineered System for Hadoop & NoSQL Security - AuthenOcaOon - AudiOng - EncrypOon High Availability Easy Upgrades Rapid Provisioning Big Data Appliance Engineered Data Connectors Exadata Engineered System for Oracle Database Intelligent Storage Smart Scan Storage Indexing Advanced Compression Easy Upgrades Easy ConsolidaOon Integrated Enterprise Management

37 Govern & Secure All your Data Hadoop, NoSQL & Rela1onal ExisOng Capability AuthenOcaOon through Kerberos AuthorizaOon through Apache Sentry AudiOng through Oracle Audit Vault EncrypOon for Data-at-Rest Network EncrypOon Big Data SQL adds Advanced Security on Hadoop & NoSQL Masking and RedacOon Virtual Private Database Fine-grain Access Control Oracle ConfidenOal Internal 37

38 Enterprise Cyber Data Architecture Common Opera1ng Picture Big Data Connectors Aoack Discovery Unstructured Data Rela1onal Data Security Admin Data Packets HDFS Storage NoSQL DB S O A SIEM Interface Log File Data (RDBMS) Internal Network Devices/Souware