Hadoop Meets Exadata. Presented by: Kerry Osborne. DW Global Leaders Program Decemeber, 2012

Transcription

1 Hi Hadoop Meets Exadata Presented by: Kerry Osborne DW Global Leaders Program Decemeber, 2012

2 whoami Never Worked for Oracle Worked with Oracle DB Since 1982 (V2) Working with Exadata since early 2010 Work for Enkitec ( (Enkitec owns an Exadata Half Rack V2/X2) (Enkitec owns an Oracle Big Data Appliance) Exadata Book (recently translated to Chinese) Hadoop Aficionado Blog: kerryosborne.oracle-guy.com 3

3 What s the Point? Data Volumes are Increasing Rapidly Cost of Processing / Storing is High Scalability is Big Concern And 4

4 Hadoop Is A Virus * Stolen from Orbitz 5

5 Google Trends 6

6 Google Trends 7

7 Google Trends 8

8 Disjointed Presentation??? Architectures Integration Approaches Oracle Products Exadoop Case Study 9

9 Traditional RDBMS Architecture w o r k DB Server Compute Plumbing Storage Storage 10

10 Traditional Oracle Architecture RAC w o r k workers Cache (SGA) dbwr lgwr etc Block Mapper (ASM) Storage 11

11 HDFS/Hadoop Architecture HA? w o r k Name Node Job Tracker datanode tasktracker datanode tasktracker workers workers Storage Storage 12

12 HDFS/Hadoop Architecture HA? w o r k Job Tracker Block Mapper (namenode) datanode tasktracker datanode tasktracker workers workers Storage Storage 13

13 Exadata Architecture RAC w o r k Block Mapper (ASM) workers Cache Storage Node Storage Node workers workers Storage Storage 14

14 HDFS/Hadoop Architecture HA? w o r k Job Tracker Block Mapper (namenode) datanode tasktracker datanode tasktracker workers workers Storage Storage 15

15 Oracle + Hadoop Integration 16

16 Obligatory Marketing Slide 17

17 Oracle Big Data Appliance Prebuilt Hadoop Stack in a Rack Engineered System Open Source Software Includes Cloudera Distribution 18

18 Oracle Big Data Appliance 19

19 BDA Software 20

20 Top Secret Feature of BDA 21

21 Integration Options Many Ways to Skin the Cat Fuse Sqoop Oracle Big Data Connectors 22

22 Fuse External Tables 23

23 Sqoop (SQL-to-Hadoop) Graduated from Incubator Status in March 2012 Slower (no direct path?) Quest has a plug-in (oraoop) Bi-Directional 24

24 Oracle Big Data Connectors Oracle Loader for Hadoop - OLH Oracle Direct Connector for HDFS - ODCH Oracle R Connector for Hadoop ORHC Oracle Data Integrator Application Adapter for Hadoop Note: All Connectors are One Way 25

25 Oracle Data Integrator Application Adapter for Hadoop ODIAAH? 26

26 Oracle R Connector for Hadoop (ORHC) Provides ability to pull data from Oracle RDBMS Provides ability to pull data from HDFS Provides access to local file system Not really a loader tool Most useful for analysts 27

27 Oracle Loader for Hadoop (OLH) Implemented as a MapReduce job (oraloader.jar) Saves CPU on DB Server Can convert to Oracle datatypes Can partition data and optionally sort it Online direct into Oracle tables Can load into Oracle via JDBC or OCI Direct Path Offline generate preprocessed files in HDFS (DP format) 28

28 Oracle Direct Connector for HDFS (ODCH) Uses External Tables Fastest - 12T per hour Can load DP files preprocessed by OLH Allows Oracle SQL to query HDFS data Doesn t require loading into Oracle Downside uses DB CPU s 29

29 Exadoop * Mad Scientist Project 30

30 Exadoop Unusual Situation! Exadata Half Rack with 4 Spare Storage Servers Company Playing with Big Data Technology Exadata Cells Very Similar to BDA Servers 4 Cells Mini BDA! (happy face) 31

31 Exadoop Layout Exa Half Rack Exadoop - 4 Compute Nodes - Exa Compute Nodes X X X X X X X X X X X X - 7 Storage Nodes (252TB) Big Fat Pipe - Exa Storage Nodes (108TB raw) - Hadoop Cluster (144TB raw) 32

32 Exadoop Applications Telecom Company Call Detail Records Dumped by Switches Loaded into HDFS via Flume 33

33 Exadoop Proposed Architecture SIP Server - Exa Compute Apex App Flume Agent Packet Sniffer - Exa Storage - Hadoop Cluster HDFS CDR Error Codes Java App Hbase 34

34 Exadoop Applications 35

35 Wrap Up Is Hadoop the right tool for the job? Maybe All the Cool Kids Are Doing It! 36

36 Questions? Contact Information : Kerry Osborne kerry.osborne@enkitec.com kerryosborne.oracle-guy.com 37