How to Hadoop Without the Worry: Protecting Big Data at Scale

Transcription

1 How to Hadoop Without the Worry: Protecting Big Data at Scale SESSION ID: CDS-W06 Davi Ottenheimer Senior Director of Trust EMC

2 Big Data Trust. Redefined Transparency Relevance Resilience 2

3 3

4 4

5 5

6 Bluetooth 6

7 2012 RSAC Breach Data Lessons 1854 GHOST MAP CHOLERA VORONOI NETWORK BREACHES Source Observation Dr. John Snow

8 Yet Google Wrong 3 rd Year in a Row Algorithmic accountability one of the biggest problems of our time system consistently overestimated flu-related visits over the past 3 years......especially inaccurate around the peak of flu season when such data is most useful

9 Therefore A Build-Your Own Market Infrastructure NoSQL DB NewSQL DB MPP DB Graph DB Crowdsourcing Hadoop on Premise Cloud Clustering Monitoring Data Transformation Storage AppDev Security Analytic Platform BI Platform Machine Learning Location, Ppl, Events Search Crowdsourcing Statistical Computing Analytics Business Analytics Data Science Unstructured Data Data Viz Social Analytics SMB Log Analytics Applications Advertising Finance Government Health Publishing Utilities HR Legal Education Security Marketing Science OSS Framework Query Access Workflow Real-Time Stats ML Deployment Search Data Sources Markets / Warehouses User Services Devices/Things Research 9

10 BYO Scale-Out Compute Environment Client Client MapReduce HDFS Masters Job Name Node Name Node (checkpoint) secondary Slaves 10

11 BYO Scale-Out Compute Environment Sync Sqoop ODBC REST Script Pig SQL Hive Tez HCatalog NoSQL Hbase Accumulo Stream Storm Distributed OS (MR -> YARN) Search Solr ML Mahout Ops Ambari Zookeeper Oozie Flume Security Knox Rhino Sentry XASec GRC Falcon Distributed Store (HDFS) 11

12 BYO Scale-Out Compute Environment Authentication Authorization Audit Kerberos + Tokens Some ACLs Scattered Logs Security Knox Rhino Sentry XASec Confidentiality??? Hadoop Distribution #1 Access, Data, Perimeter, Visibility Hadoop Distribution #2 Policy, Audit, Access, Encryption 12

13 BYO Scale-Out Compute Environment Authentication Daemons run as single user (hadoop) user process hdfs mapred group hadoop namenode, datanode, secondary namenode jobtracker, tasktracker, child tasks users hdfs, mapred sudo -u hdfs hadoop fs -rmr / 13

14 BYO Scale-Out Compute Realities 1. Data Shared 2. Networks Open 3. Nodes Distributed 4. Web Services Open 5. Access Controls Open 6. Clients Unauthenticated 14

15 BYO Scale-Out Compute Risk Symptoms Kerberos (Randomness, Scalability) Job Ticket / Service Delegation Data Node Authority (non-acl) API Lack of Multi-Tenancy Awareness Local Disk Map Output Access via HTTP Service 15

16 Scale-Out Integrity Concerns Tweet Errors Balance Sheet Errors 16

17 Scale-Out Availability Concerns 1 Admin : 30,000+ Nodes switch switch switch switch switch job tracker name node name node client A A1 B3 A2 A3 secondary B1 B2 client B Rack 1 Rack 2 Rack 3 Rack 4 Rack n 1/2 PETABYTE 17

18 Phase One Scale-Out Query Analysis Management Computation Inventory Storage Data + Compute Node Data + Compute Node Data + Compute Node Checkpoint Data + Compute Node Data + Compute Node Data + Compute Node 18

19 Phase Two Resilience Query Analysis Management Computation Inventory Storage Data Compute Node Compute Node Compute Node Compute Node Compute Node Compute Node I I I I Storage 19

20 Phase Three Classification Query Analysis Management Computation Inventory Storage Work ~ Data I Work b Home b Work a Home a I I I Storage Bob Alice 20

21 Phase Four Least Authority Query Analysis Management Computation Inventory Storage S1341dfeqaas2ia1kjg3af Work b Home b Work ~ Work a Home a I I I I Data Storage Bob Alice 21

22 Phase Three Classification Accumulo Cell-Level Access Dist Key/Values NoSQL (HBase) Performance Evolution 2006 Google BigTable 2008 NSA 2011 accumulo.apache.org // specify which visibilities we are allowed to see Authorizations auths = new Authorizations("public"); Scanner scan = conn.createscanner("table", auths); scan.setrange(new Range( user100", user200")); scan.fetchfamily("attributes"); for(entry<key,value> entry : scan) { String row = entry.getkey().getrow(); Value value = entry.getvalue(); } 22

23 Protecting Big Data at Scale CIFS HTTP Gig-e 10 Gig-e Protocols NFS FTP HDFS for Hadoop REST for Object Single FS / Volume Client/App Layer Network Layer Operating Environment Intra-Communication 23

24 Protecting Big Data at Scale Multi-Tenancy Aware Full-ACL File Systems Kerberos Authentication High-Resilience Architecture Name Node Continuous Availability Data Protection (BC/DR, Snapshots, etc.) SEC 17a-4 Compliant WORM 24

25 Big Data Trust. Redefined Transparency Relevance Resilience 25

26 Thank