Fighting Cyber Fraud with Hadoop. Niel Dunnage Senior Solutions Architect

Transcription

1 Fighting Cyber Fraud with Hadoop Niel Dunnage Senior Solutions Architect 1

2 Summary Big Data is an increasingly powerful enterprise asset with many potential user cases in this case we ll explore the relationship between big data and cyber security Cloudera, Inc. All rights reserved.

3 Quick facts Founded 2008, by former employees of Employees Over 750 Global 24x7 Support Professional Services Follow-the-sun capability; Pro-active & Predictive Support Programs Dedicated Support Engineers; Support Centers in NA, Europe & Asia World class services delivery teams worldwide Mission Critical Thousands of enterprise customers rely on Cloudera: 50% of the Fortune 50; 65% of the Fortune 500, Top Defense & Intelligence Agencies The Largest Ecosystem Cloudera University Open Source Leaders Over 1200 Members of our Partner Program, ClouderaConnect Over 40,000 people trained around the world Cloudera employees are founders of most of the Apache Hadoop ecosystem projects, and leading contributors to all of them 3

4 Leading the way in data management powered by Hadoop CLOUDERA FOUNDED BY MIKE OLSON AMR AWADALLAH & JEFF HAMMERBACHER CLOUDERA RELEASES CDH THE FIRST COMMERCIAL APACHE HADOOP DISTRIBUTION CLOUDERA REACHES 100 PRODUCTION CUSTOMERS CLOUDERA ENTERPRISE 4 THE STANDARD FOR HADOOP IN THE ENTERPRISE CLOUDERA IMPALA CLOUDERA NAVIGATOR CLOUDERA SEARCH THE ENTERPRISE DATA HUB LAUNCHED CDH Cloudera Manager CLOUDERA ENTERPRISE 4 ASK BIGGER QUESTIONS ENTERPRISE DATA HUB HADOOP CREATOR DOUG CUTTING JOINS CLOUDERA CLOUDERA MANAGER: FIRST MANAGEMENT APPLICATION FOR HADOOP CLOUDERA UNIVERSITY EXPANDS TO 140 COUNTRIES CLOUDERA CONNECT REACHES 300 PARTNERS TOM REILLY JOINS AS CEO OVER 800 PARTNERS IN CLOUDERA CONNECT 4

5 Agenda Data: - The new oil How to scale out with unreliable workers How enterprises pool and share storage and computation resources Enterprise data governance and growing up Hadoop security Deploying machine learning at scale Empowering creative data science Cloudera, Inc. All rights reserved.

6 This Morning Cloudera, Inc. All rights reserved.

7 Cyber Security:- Data is a valuable commodity DDOS Data Exfiltration Confidential customer records Transaction data Reputation attack False flag Fake data Insider Threat Operations designed to deceive in such a way that the operations appear as though they are being carried out by entities, groups or nations other than those who actually planned and executed them hacker has released a database dump from a US fantasy football website ( claiming that it was timed to coincide with the NFL has continued to support OpSaudi, claiming further attacks on websites connected to Saudi Aramco. Anonymous Italy and Operation Green Rights (OpGR) have released the contents of an account connected to an Italian steel producer, in connection to accusations of pollution against the company Cloudera, Inc. All rights reserved.

8 Cloudera s Approach to Hadoop Security Comprehensive Standards-based Authentication Centralized, Granular Authorization Native Data Protection End-to-End Data Audit and Lineage Compliance-Ready Meet compliance requirements HIPAA, PCI-DSS, Encryption and key management Transparent Security at the core Minimal performance impact Compatible with new components Insight with compliance Cloudera, Inc. All rights reserved.

9 Enterprise Data Hub Users Cases OSINT Analysis Fraud Detection Log Processing Performance Management Risk Manageme nt Innovation and Advantage Ask bigger questions in the pursuit of discovering something incredible Operational Efficiency Perform existing workloads faster, cheaper, better ETL Acceleration Active Archive EDW Optimization Deep Exploratory BI Historical Compliance Cloudera, Inc. All Rights Reserved.

10 Our Design Strategy The Enterprise Data Hub A fully integrated Hadoop ecosystem One pool of data One metadata model One security framework One set of system resources Metadata, Navigator Select CPU_Met from application WHERE (USAGE > 1000) LEFT OUTER JOIN ON application_id where application_type IS Non_Critical Batch Processing Spark, MAPREDUCE, HIVE & PIG Interactive SQL CLOUDERA IMPALA HDFS TEXT, RCFILE, PARQUET, AVRO, ETC. Interactive Search CLOUDERA SEARCH Engines Machine Learning Spark Mlib,MAHOUT, Oryx Resource Management YARN Storage Integration graph.vertices.filter{case(id, _) => id== }.collect Math & Statistics SAS, R Hbase/ Accumulo RECORDS REST (Webhdfs), File (Fuse) Flume, Sqoop Stream Processing Spark streaming Security, Navigator, Sentry Cloudera, Inc. All rights reserved.

11 Offence:- Fraud Detection Fully Automated at scale User Cases Distributed parallel execution with chained joins Historical processing at scale Machine Learning, malware/anomaly detection, spam filters etc Combined real time and batch predictors 12 12

12 Big Data Economics Ask bigger questions Agile (2 week cycle) Linear scaling Robust and economic crypto security Creative fail fast innovation Powers productivity insights Increasing infrastructure ROI Increasing business ROI Defeating fraudulent activity Evaluating risk Innovate Predict Ingest Discover Cloudera, Inc. All Rights Reserved.

13 Data Ingest NRT Ingest Flume Optimized to flow real time event data into the Hadoop cluster Spark Streaming for near real time micro batch aggregations Twitter streaming Kafka Log API Bulk Load Sqoop for structured Fuse file system access API Web / Hue Data Enrichment Flume interceptors Kite Morplines module Configuration based interceptors that can enrich data. For example extracting facets, entity extraction applying regulatory tags collect Client Client Client Client enrich Agent Agent Agent buffer store Cloudera, Inc. All rights reserved.

14 Near Real time Access to threats View the geographic distribution of Slowloris DDOS taken from Apache web server logs Help isolate unpatched servers Identify source of attacks LogUtils.createStream(...).filter(_.getText.contains( 408 Error")).countByWindow(Seconds(10)) stream.join(historiccounts).filter { case (word, (curcount, oldcount)) => curcount > oldcount } Cloudera, Inc. All rights reserved.

15 Machine Learning Real-time large-scale machine learning predictive analytics infrastructure build on Hadoop Collaborative filtering and recommendation Classification and regression, Clustering 16 16

16 VARs and Monte Carlo Simulations Under reasonable circumstances, how much can you expect to lose? Monte Carlo simulation, involves posing thousands or millions of random market scenarios and observing how they tend to affect a portfolio of financial instruments VAR based on TimePeriod, Portfolio and Confidence level This technique is easily parallelizable and as such is a great fit for Hadoop and Spark in particular Until recently required complex MPI C++ code Easily implemented in Hadoop and feasible across hierarchies of financial instruments (P&L Accounts) Backtest to validate the VAR Curation of Market Factors is important Can shape portfolio investments for instruments that trial as loss making Cloudera, Inc. All rights reserved.

17 Applying VAR Techniques to Cyber Threat Monitoring with Hadoop Historical event data processing at scale Hadoop as a service shared with financial governance applications Treat s spent on vendors and software like instruments and portfolios? Anomaly detection of network traffic by learning what is normal Siloed applications have previously made it hard to have a tangible value of finanicial risk. Risk calculations tend towards the subjective ie low (FIS APT) high (insider threat) Cloudera, Inc. All rights reserved.

18 Internal Threat Dashboard Overall Risk Assessment: Risk Per Category: Online Banking Access: Public Records: Financial transaction rate: Online Activity: Social Media Activity: Regular purchases Foreign Travel: Ranked List of High Risk Personnel: Name Risk Score Kim Burgess 94 Guy Hughes 93 Jeff Maclaen 87 Ed Snowden 86 Mary Smith 82 Open Cases: Customers with Risk Scores that Recently Changed Name Old Score New Score John Smith Rob Jones Jim Fisher Henry Johnson Sue Leefield Name Risk Score Customers Dodgy Ecomm.biz 94 John Smith, Rob Jones. Brentford Shopping Centre 93 Jim Fisher, Henry Johnson 19

19 20 Analytics

20 Defense: - Security Features Hadoop Security: - Kerberos simplified deployment with Cloudera Manager Sentry: - provides unified authorization with a single policy for Hive, Impala and Search HDFS Extended ACL s and HBase cell level access control Navigator encrypt and key trustee deliver compliant data security Via Gazzang acquisition Navigator provides data management layer including audit, access control reviews, data classification and discovery, and lineage Cloudera, Inc. All rights reserved.

21 Kerberos Security Perimeter Security Guarding Technical Network access to the cluster itself Concepts: Authentication isolation Kerberos Kerberos: A computer network authentication protocol that works on basis of tickets to allow nodes to prove identity to each other in a secure manner using encryption extensively Messages are exchanged between: Client Server Kerberos Key Distribution Center (KDC). Note this is not part of Hadoop, but most Linux Distros come with MIT Kerberos KDC. Passwords are not sent across network, Instead passwords are used to compute encryption keys Authentication status is cached (don t need to send credentials with each request) Timestamps are essential to Kerberos (make sure system clocks are synchronized!) Cloudera, Inc. All rights reserved.

22 Apache Sentry Access Security Access Defining what users and applications can do with data Technical Concepts: Authorization Permissions Sentry Sentry provides unified authorization across multiple access paths A single authorization policy will be enforced for Impala, Hive and Search Role based access at Server, Database, Table or View granularity Multi-tenant: Separate policies for each database / schema Cloudera, Inc. All rights reserved.

23 Cloudera Navigator Visibility Visibility Reporting on where data came from and how it s being used Technical Concepts: Auditing Lineage Cloudera Navigator Auditing and Access Management View, granting and revoke permissions across the Hadoop stack Identify access to a data asset around the time of security breach Generate alert when a restricted data asset is accessed Lineage Given a data set, trace back to the original source Understand the downstream impact of purging/modifying a data set Metadata Tagging and Discovery Search through metadata to find data sets of interest Given a data set, view schema, metadata and policies Lifecycle Management Automate periodic ingestion of data Compress/encrypt a data set at rest Purge a dataset/replicate data set to a remote site Cloudera, Inc. All rights reserved.

24 Cloudera, Inc. All rights reserved.

25 Encryption at rest Navigator Encrypt and Key Trustee Encrypt any File, Directory AES-256 Encryption Unique Access controls Process Based, NOT users / groups 100% Transparent Separation of Duties Key Management AES encryption keys stored on separate Key Trustee server Key manager breach, data is safe Data Server breach, data is safe Process Based ACL s Linux File, Directory AES-256 Encryption Linux Server / VM Encrypt client GPG Linux Server / VM Key Trustee Server 26 Gazzang gazzang.com/products/cloudencrypt-for-aws