Who Am I? Mark Cusack Chief Architect 9 years@rainstor Founding developer Ex UK Ministry of Defence Research InfoSec projects

Transcription

1 1

2 Who Am I? Mark Cusack Chief Architect 9 years@rainstor Founding developer Ex UK Ministry of Defence Research InfoSec projects 2

3 RainStor: a SQL Database on Hadoop SCALE (MPP, Shared everything) LOAD COMPRESS ANALYZE SQL92 +Pig +Hive +MapReduce Billions of Records/Day 20 40X (95%+) Rapid Response (Secs) Batch (Mins Hrs) Secure & Comply 3

4 Compliance vs Security Industry/Government Compliance Regulations Privacy Integrity Trust SEC, Sarbanes Oxley, Dodd Frank Reform, EU Data Retention, PCI, HIPAA, Defend information from unauthorized access, use, modification, destruction, recording, Compliance Joint Requirements Security 4

5 Security & Compliance Risks 5

6 Hadoop & Big Data SECURITY? Social Media Logs Clickstreams Credit Card Trade Personal Information LOW VALUE DATA HIGH VALUE DATA Recommendation Engines Data Sandboxing Log Processing Audits Regulatory Reporting (Eg. SEC, SOX) Lawful Intercept 6

7 What s Missing in Hadoop? How secure is my data within Hadoop? How do I prevent malicious users from accessing my Big Data infra? How do I ensure data compliance with regulatory bodies? Security and Compliance 7

8 Is a Compliant Application Secure? Not always! Compliance assessments: black and white Security assessments: shades of grey Compliance standards based on InfoSec Take a pragmatic approach with Hadoop Can Hadoop meet compliance requirements? Consider Hadoop and PCI DSS 8

9 Key PCI DSS Requirements Payment Card Industry Data Security Standard Protect sensitive data Minimize data retention period Control access to data business need to know Audit access to data Manage and protect user credentials LOW VALUE DATA Same requirements in other compliance standards 9

10 Protecting Sensitive Data on Hadoop Encryption of data at rest HDFS should be considered untrusted HDFS has no built in file level encryption No key management scheme in Hadoop Encryption of data on the move Many protocols in Hadoop: TCP/IP, HTTP, RPC No encryption at the flick of a switch in Hadoop 10

11 Data Retention Periods on Hadoop File level data disposition No policies for deciding when to delete files Deletion from HDFS is manual process No support for physically shredding files Record level data disposition No way to selectively delete individual records E.g. Banking requirement to delete all data belonging to a single customer 11

12 Control Access to Data on Hadoop File level access control HDFS file based permissions and ownership Difficult to integrate with the enterprise Record level access control GRANT/REVOKE role based access control vs metastore permissions in Hive are confusing and difficult to configure Data masking Need to prevent some users from seeing full data E.g. CC number: XXXX XXXXXX 1234 Hadoop/Hive has no mechanism for masking data 12

13 Audit Access to Data on Hadoop Log access to sensitive data Record who, what, when Need to track data access & attempts to change logging levels and access control mechanisms Hadoop only records filesystem requests Data tamper proofing Integrity at the HDFS block level Does not cover attempt to change data values 13

14 Protect User Credentials on Hadoop User authentication Good support through Kerberos integration in Hadoop Kerberos may not be suitable for every enterprise Kerberos configuration in Hadoop is complex Protection of credentials on the move and at rest Can t pass credentials securely to MapReduce tasks Job configuration persisted to disk unencrypted 14

15 How is the Hadoop Community Addressing the Gaps? Distro vendors Intel: Secure Hadoop Cloudera: Sentry & Navigator Hortonworks: Knox & Falcon Third party vendors Voltage, Protegrity RainStor 15

16 The RainStor + Hadoop Stack Top BI Tools of Stack & Dashboards BI/Visualization API Java Pig Hive Programming/API (incl JDBC/ODBC) MapReduce/YARN Real time SQL (Optimizer/Filter) Computation Security & Compliance Security Hadoop Files Sequence, ORC, RainStor Files (Compression) File Format Hadoop Distributed File System File System Apache RainStor Vendor Specific 16

17 RainStor s Patented Compression AA $ $ DELL Value and Pattern De duplication Compress by 40x then encrypt (all in memory) Lower encryption and decryption overhead 0004 Example of Stock Trades 17

18 RainStor Security Capabilities & Features Secure Large Volumes of Structured Data on Hadoop Privacy Data Encryption/Key Management Data Masking SQL92 Views Tamper proofing Audit Trail Record level Delete Data Disposition Integrity Trust Kerberos Authentication Role based Authorization LDAP / Active Directory Linux PAM Support 18

19 Recommendations Address encryption and key management Add detailed auditing and tamper proofing Add fine grained access control and masking Add data retention rules Simplify configuration for security & compliance Use RainStor! We provide these features out of the box 19

20 Thank You 20