Copyright 2014 Oracle and/or its affiliates. All rights reserved.

Transcription

1

2 Naval Postgraduate School Report on the Oracle Cloudera Solution for Big Data Tom Plunkett Lead Author, Oracle Big Data Handbook Arijit Das Research Associate Professor, US Navy Postgraduate School Copyright 2014 Oracle and/or its/ or affiliates. All rights Copyright 2014 Oracle and its affiliates. All reserved. rights reserved. Oracle Confidential Internal

3 Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle s products remains at the sole discretion of Oracle. Oracle Confidential Internal/Restricted/Highly Restricted 3

4 Oracle Big Data Handbook "In this book, some of Oracle's best engineers and architects explain how you can make use of big data. They'll tell you how you can integrate your existing Oracle solutions with big data systems, using each where appropriate and moving data between them as needed." Doug Cutting, co-creator

5 Big Data in Healthcare Finding relationships between genes and cancer types genes and five major cancer types 20 million medical publication abstracts Genes from 60Million patients mirna for a simulated 900Million population

6 Big Data is about asking questions that we couldn t in the past because we either didn t have the means to collect the data or couldn t analyze or interpret it if we did have the data. And then getting answers to those questions. At least 100x faster. Dr. Jack Collins Director, Advanced Biomedical Computing Center Frederick National Laboratory for Cancer Research Frederick National Laboratory for Cancer Research üwinner, 2012 Government Big Data Solutions Award üfinalist, 2013 Excellence.Gov Awards ülaureate, 2013 Computerworld Honors Program

7 Briefing Agenda Big Data s Potential & Myths Oracle s Big Data Strategy & Innovations The Naval Postgraduate School Use Case Oracle Confidential Internal 7

8 Big Data s Potential & Myths Public Sector Use Cases Oracle Confidential Internal 8

9 Governments Face Increasing Data Demands It used to be more with less now it is efficiency and effectiveness Everyone is struggling with what they don t know Agencies need to empower users to discover patterns in the data Agencies need to bring data together from the silos Use technology to unlock the value of unstructured data 9

10 Cyber Security Data Warehouse Leaving Attackers No Place to Hide M ultiple data models for different types of analysis Unified system metadata Real- time and batch processing Open APIs for Integration with 3rd party data and analytics Interactive Dashboards Guided Attack Discovery Oracle Confidential Internal 10

11 Myth or Fact? Common Wisdom: Reality: 1.Big Data = Hadoop 2. 3.Hadoop is mainly for Unstructured Data 4. 5.Hadoop will replace the Data Warehouse 6. 7.Hadoop is a Data Integration Solution 1. Big Data = Hadoop + NoSQL + Relational Hadoop is for both Structured, Semi- structured and Unstructured Data Hadoop is a supplement to the Data Warehouse Hadoop is not a Data Integration Solution (but data integration is critical to Big Data!) 8. Oracle Company Confidential 11

12 Fact: Big Data = Hadoop + NoSQL + Relational There are a lot of commonly held beliefs about big data that need to be challenged, with the first being that you simply adopt Hadoop and are good to go. The problem is that Hadoop is a technology, and big data isn't about technology. Big data is about business needs. I n reality, big data should include Hadoop and relational [databases] and any other technology that is suitable for Ken the task at hand. Rudin, Head of Analytics, Facebook Oracle Confidential Internal 12

13 Fact: Hadoop is Mostly Used w/structured Data üvast majority of actual Hadoop projects are utilizing data sourced from structured data repositories Oracle Company Confidential 13

14 Fact: Hadoop is Not a DW Replacement üfor the past 3 years, fewer and fewer survey respondents indicate an intent to try Hadoop for DW replacements üno major analyst suggests that Hadoop is capable of Source: Gartner Survey for Data Warehousing Magic Quadrant Oracle Company Confidential 14

15 Oracle s Big Data Strategy and Innovations Oracle Confidential Internal 15

16 Extending Data Management BigData Management System Hadoop Change Government Disrupt fraudsters Improve supply chains Leverage new paradigms Exploit new analyses Relational NoSQL Scale Government Meet mobile challenges Accelerate developer agility Scale-out economically Serve data faster Run Government Integrate existing systems Support mission-critical tasks Protect existing expenditures Oracle Confidential Internal/Restricted/Highly Restricted 16

17 Oracle Big Data Management System DATA RESERVOIR Cloudera Hadoop Oracle Big Data SQL Oracle NoSQL DATA WAREHOUSE Oracle Oracle Database Database In- Memory, Oracle Big Data Oracle MultiIndustry - tenant Models Connectors Oracle Industry OracleModels Advanced Analytics Oracle Advanced Oracle Spatial & Analytics Graph Oracle Data Integrator Oracle Spatial & Graph Oracle Big Data SQL Exadata Oracle Oracle Real GoldenGate Time Decisions SOURCES Oracle R Advanced Analytics for Hadoop Big Data Appliance Oracle R Oracle Event Oracle Distribution GoldenGate ANALYTICS Oracle Business Analytics Enterprise BI Self- Service Discovery Data Mashup X Exalytics Endeca Info Discovery OBIEE

18 Overcoming Barriers to Adoption of New Technologies INT EGR ATI SKI LLS SEC URI TY SQ L Eng ine ere SQL on All Dat aba se Confidential 18

19 Engineered for Big Data Management at Scale Engineered System for Hadoop & NoSQL Security - Authentication - Auditing - Encryption High Availability Easy Upgrades Rapid Provisioning Engineered System for Oracle Database Intelligent Storage Engineer ed Data Connecto rs Big Data Exadata Appliance Integrated Enterprise Management Smart Scan Storage Indexing Advanced Compression Easy Upgrades Easy Consolidation

20 Big Data Analytics Challenge Separate data access interfaces 20

21 Oracle Big Data Management Preserving investment with transparent Big Data access NoSQL 21

22 Oracle Big Data SQL One Query Spanning Oracle Database, Hadoop & NoSQL Query Data in RDBMS, Hadoop & NoSQL Fast Massive Parallelism Oracle SQL Oracle NoSQL DB HDFS Data Node Oracle Database Storage Server Oracle NoSQL DB HDFS Data Node Oracle Database Storage Server Storage Indexes Filtered Locally Minimized Data Movement

23 Govern & Secure All your Data Hadoop, NoSQL & Relational Existing Capability Authentication through Kerberos Authorization through Apache Sentry Auditing through Oracle Audit Vault Encryption for Data-at-Rest Network Encryption Big Data SQL adds Advanced Security on Hadoop & NoSQL Oracle Confidential Internal 23

24 Oracle Big Data Discovery Changing the Game for Agile Government I nnovation on Big Data Profile Find Easily add data and see it automatically and continuously cataloged, enriched and related Use familiar guided search across massive amounts of diverse data Understa nd Transfor m Discove r Know what s important from diagnostic analysis of millions of data characteristics Powerful tools to quickly clean up and wrangle dirty data so it s ready to go Uncover valuable new insights 20% Predict Use new insights to define and refine predictive models Collabora te Publish, share and evolve as you learn more 80% Oracle Confidential Internal 24

25 Breakthrough Innovation: Oracle Big Data Discovery The Visual Face of Hadoop Radically simplify data prep and Analysis process 25

26 US Navy Big Data Use Case Arijit Das Research Associate Professor Oracle Confidential Internal 26

27 Naval Postgraduate School Overview (NPS) Graduate study in science, math, engineering, business & policy. Many from the US astronaut community hold a degree from NPS. Run by the Navy, and open to US and coalition partners. Students are all DoD affiliated (active duty, civilians, & contractors) Ongoing applied research projects. Vendor collaboration through support contracts, including Oracle. 27

28 NPS Research group (Computer Scientists): Our group does applied research in Database related topics. Consists of students, interns, & civilian employees. Leverage service contracts to get answers to the difficult questions. Vendor training classes to enhance in-house knowledge. Currently using Oracle Database and Middleware products. 28

29 NPS Research Study: Project funded by Naval entities (Navy + Marine Corps). One year effort and has been extended. Student thesis work. Focus on Oracle Big Data tools for the USMC Data warehouse. Work is ongoing and the results will be handed over to the sponsors. 29

30 Research Study Question 1 (HDFS + Data warehouse): USMC Contractors created a backup of the DW and shipped it to NPS. Schema was loaded into an Oracle 11G DB at NPS. SQL queries were run using SQL Developer. Data from Relational DB tables were exported to JSON format. JSON data was loaded into HDFS (Hadoop Distributed File System). Java Map Reduce equivalent of the SQL queries were written and tested. 30

31 Research Study Question 2 (Data movement): Looked at Oracle Big Data Connectors. Configuration files were set up. Run MapReduce jobs were in HDFS. Oracle Loader was used to load data from HDFS into DB tables. Sqoop (open source loader) testing is also planned. 31

32 Oracle Loader for Hadoop Text Avro Parquet Hive Log files JSON Parallel load, optimized for Hadoop Automatic load balancing Convert to Oracle format on Hadoop Save database CPU Load specific Hive partitions Sequence files Compressed files And more

33 Research Study Question 3 (SQL on HDFS data files): Not everybody is comfortable with writing Java Map Reduce. SQL is easier to work with. Generate the CSV data for all the DB tables. Move all the data to HDFS. Install the SQL connector for HDFS. Run SQL in the DB against HDFS (data files). OSCH OSCH OSCH OSCH 33

34 Oracle SQL Connector for HDFS OSCH OSCH External Table OSCH OSCH Text Hive Compressed files create table customer_address ( ca_customer_id number(10,0), ca_street_number char(10), ca_state char(2), ca_zip char(10)) organization external ( TYPE ORACLE_LOADER DEFAULT DIRECTORY DEFAULT_DIR ACCESS PARAMETERS ( ) PREPROCESSOR HDFS_BIN_PATH:hdfs_stream ) LOCATION ( addr1, addr2, addr3 )) Parallel query and load Load into database or query in place Access text or Hive over text Access compressed data Access specific Hive partitions Kerberos authentication

35 Research Study Question 4 (HDFS Hardware Cluster): Looked at both Apache & Cloudera Hadoop. Current rack has nodes (Dell Hardware/Redhat Linux 64 bit OS) Name Node server (most powerful) also has the Oracle 11G DB. Rack organization and power distribution a major consideration. All 8 nodes have 2 network interfaces (Ethernet 1 GB / Infiniband 10 GB). Intra Node communication via Infiniband switches (10 GB/s 35

36 Research Study Question 5 (Oracle Big Data Appliance): Use Oracle data masking tools to create a morphed version of the data. Install the Appliance VM at NPS. Test connectors, Java MapReduce, SQL. Plan on loading data to a appliance (needs to be coordinated with Oracle). Testing and benchmarking. 36

37 Contributors: Nicholas Bitto (Java MapReduce). Greg Belli (Oracle Big Data Connectors). Erik Lowney (Big Data Cluster). USMC sponsor. USMC contractors. 37

38 Q&A For More Information Contact: ORACLE1 or oracle.com Oracle's best engineers explain how you can make use of big data. how you can integrate your existing solutions with big data systems, using each where appropriate." Doug Cutting, co-creator of Apache Hadoop

39 Oracle Confidential Internal 39

40