Oracle Big Data SQL. Architectural Deep Dive. Dan McClary, Ph.D. Big Data Product Management Oracle

Transcription

1

2 Oracle Big Data SQL Architectural Deep Dive Dan McClary, Ph.D. Big Data Product Management Oracle Copyright 2014, Oracle and/or its affiliates. All rights reserved.

3 Safe Harbor Statement The following is intended to outline our general product direcmon. It is intended for informamon purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or funcmonality, and should not be relied upon in making purchasing decisions. The development, release, and Mming of any features or funcmonality described for Oracle s products remains at the sole discremon of Oracle. Oracle ConfidenMal Internal/Restricted/Highly Restricted 3

4 Agenda The Data AnalyMcs Challenge Why Unified Query MaWers SQL on Hadoop and More: Unifying Metadata Query Franchising: Smart Scan for Hadoop Oracle ConfidenMal Internal/Restricted/Highly Restricted 4

5 Data AnalyMcs Challenge (2012) Separate data access interfaces 5

6 Tables on NoSQL SQL on Hadoop 6

7 Data AnalyMcs Challenge: 2013 No comprehensive SQL interface across Oracle, Hadoop and NoSQL 7

8 Oracle Big Data Management System Unified SQL access to all enterprise data NoSQL 8

9 How to Get the Most From Polyglot Persistence Rapid Development Simple SemanDcs AcDve Archive Data ExploraDon Scale- out AnalyDcs Business- cridcal processes SensiDve Data ExisDng ApplicaDons

10 SQL on Hadoop and More: Unifying Metadata

11 Why Unify Metadata? CREATE TABLE customers! SELECT name FROM customers! sales Catalog customers Query CREATE across TABLE sources sales! à Integrate new metadata SELECT No changes customers.name, for users and sales.amount! applicamons Seamlessly handle schema- on- read Exploit remote data distribumon HolisMcally opmmize queries SALES CUSTOMERS

12 How Data is Stored in HDFS Example: 1TB File {"custid": ,"movieid":null,"genreid":null,"mme":" :00:00:07","recommended":null,"acMvity":8} {"custid": ,"movieid":1948,"genreid":9,"mme":" :00:00:22","recommended":"N","acMvity":7} {"custid": ,"movieid":null,"genreid":null,"mme":" :00:00:26","recommended":null,"acMvity":9} Block B1 {"custid": ,"movieid":11547,"genreid":44,"mme":" :00:00:32","recommended":"Y","acMvity":7} {"custid": ,"movieid":11547,"genreid":44,"mme":" :00:00:42","recommended":"Y","acMvity":6} {"custid": ,"movieid":null,"genreid":null,"mme":" :00:00:43","recommended":null,"acMvity":8} {"custid": ,"movieid":null,"genreid":null,"mme":" :00:00:50","recommended":null,"acMvity":9} {"custid": ,"movieid":608,"genreid":6,"mme":" :00:01:03","recommended":"N","acMvity":7} Block B2 {"custid": ,"movieid":null,"genreid":null,"mme":" :00:01:07","recommended":null,"acMvity":9} {"custid": ,"movieid":27205,"genreid":9,"mme":" :00:01:18","recommended":"Y","acMvity":7} {"custid": ,"movieid":1124,"genreid":9,"mme":" :00:01:26","recommended":"Y","acMvity":7} {"custid": ,"movieid":16309,"genreid":9,"mme":" :00:01:35","recommended":"N","acMvity":7} {"custid": ,"movieid":11547,"genreid":44,"mme":" :00:01:39","recommended":"Y","acMvity":7}} Block B3 {"custid": ,"movieid":424,"genreid":1,"mme":" :00:05:02","recommended":"Y","acMvity":4} 1 block = 256 MB Example File = 4096 blocks InputSplits = 4096 PotenMal scan parallelism Oracle ConfidenMal Internal/Restricted/Highly Restricted 12

13 How MapReduce and Hive Read Data Consumer Scan and row creamon needs to be able to work on any data format Create ROWS & COLUMNS SCAN Data Node disk Data definimons and column deserializamons are needed to provide a table RecordReader => Scans data (keys and values) InputFormat => Defines parallelism SerDe => Makes columns Metastore => Maps DDL to Java access classes 13

14 SQL- on- Hadoop Engines Share Metadata, not MapReduce Hive Metastore Oracle Big Data SQL SparkSQL Hive Impala Hive Metastore Table DefiniMons: movieapp_log_json Tweets avro_log Metastore maps DDL to Java access classes Oracle ConfidenMal Internal/Restricted/Highly Restricted 14

15 Extend Oracle External Tables CREATE TABLE movielog (! click VARCHAR2(4000))! ORGANIZATION EXTERNAL (! TYPE ORACLE_HIVE! DEFAULT DIRECTORY DEFAULT_DIR! ACCESS PARAMETERS! (! com.oracle.bigdata.tablename logs! com.oracle.bigdata.cluster mycluster! ))! REJECT LIMIT UNLIMITED;! New types of external tables ORACLE_HIVE (inherit metadata) ORACLE_HDFS (specify metadata) Access parameters for Big Data Hadoop cluster Remote Hive database/table DBMS_HADOOP Package for automamc import 15

16

17 A Smarter External Table Oracle Table You define: Table name Oracle types Any Degree of Parallelism HDFS Data You get: AutomaMc discovery of Hive table metadata AutomaMc translamon from Hadoop types AutomaMc conversion from any InputFormat Fan- out Parallelism across the Hadoop cluster 17

18 StorageHandlers: Extensibility Beyond HDFS Oracle Big Data SQL StorageHandlers are a metadata bridge. Hive Metastore

19

20 Query Franchising: Smart Scan for Hadoop

21 Language- level FederaMon Fails Been there, done that. Hadoop Part with sites as! (! select s.custid as cust_id, listagg(s.site, ',')! within group (order by s.custid) site_list! from shortcodes s! group by custid! ),! select c.first_name, c.last_name, c.age, c.state_province, s.site_list! from customers c, sites s;!! Database Part

22 Language- level FederaMon Fails Been there, done that. select s.custid as cust_id,! listagg(s.site, ',')! within group (order by! s.custid) site_list! from shortcodes s! group by custid!! Operators exist in both places? Is their behavior consistent? How do you negomate resources?

23 Query Franchising dispatch of query processing to self- similar compute agents on disparate systems without loss of opera9onal fidelity

24 Query Franchising: Uniform Behavior, Disparate LocaMon Top- level plan created HolisMc plan plan for all work Distribute to franchises based by locamon 2 2 Franchisees carry out local work Franchises secure and umlize resources All franchises speak the internal language 2 3 Global operamons opmmized Adapts to local variamon Nothing lost in translamon

25 What Can Big Data Learn from Exadata? Intelligent Storage Maximizes Performance SELECT name, SUM(purchase)! FROM customers! GROUP BY name;! 1 Oracle SQL query issued Plan constructed Query executed Oracle Exadata Storage Server 2 Smart Scan Works on Storage Filter out unneeded rows Project only queried columns Score data models Bloom filters to speed up joins Oracle Exadata Storage Server CUSTOMERS

26 Big Data SQL: A New Hadoop Processing Engine MapReduce and Hive Processing Layer Spark Impala Search Big Data SQL Resource Management (YARN, cgroups) Storage Layer Filesystem (HDFS) NoSQL Databases (Oracle NoSQL DB, Hbase) Oracle ConfidenMal Internal/Restricted/Highly Restricted 26

27 Smart Scan for Hadoop: OpMmizing Performance Big Data SQL Server Smart Scan External Table Services Data Node Oracle on top Apply filter predicates Project columns Parse semi- structured data Hadoop on the boxom Work close to the data Schema- on- read with Hadoop classes TransformaMon into Oracle data stream Disk Oracle ConfidenMal Internal/Restricted/Highly Restricted 27

28 Mapping Hadoop to Oracle Parallel Query and Hadoop HDFS NameNode Hive Metastore B B B 1 InputSplits 2 PX Determine Hadoop Parallelism Determine schema- for- read Determine InputSplits Arrange splits for best performance Map to Oracle Parallelism Map splits to granules Assign granules to PX Servers PX Servers Route Work Offload work to Big Data SQL Servers Aggregate Join Apply PL/SQL

29 Big Data SQL 1.1 Introduces Enhanced Parallelism Big Data SQL 1.0 Hadoop DoP linked to RDBMS DoP Lead to many idle PQ processes Required explicit declaramon Big Data SQL 1.1 Unlink Hadoop and RDBMS DoP AutomaMc max Hadoop parallelism Even on serial tables An average of 40% faster Even at equivalent DoP

30 Big Data SQL Dataflow Big Data SQL Agent Smart Scan External Table Services SerDe RecordReader Data Node Disks Read data from HDFS Data Node Direct- path reads C- based readers when possible Use namve Hadoop classes otherwise Translate bytes to Oracle Apply Smart Scan to Oracle bytes Apply filters Project Columns Parse JSON/XML Score models

31

32 But How Does Security Work? DBMS_REDACT.ADD_POLICY(! object_schema => 'MCLICK',! SELECT object_name * FROM => my_bigdata_table! 'TWEET_V',! WHERE column_name SALES_REP_ID => 'USERNAME',! = SYS_CONTEXT('USERENV','SESSION_USER'); policy_name => 'tweet_redaction',!! function_type => DBMS_REDACT.PARTIAL,! function_parameters => 'VVVVVVVVVVVVVVVVVVVVVVVVV,*,3,25',! expression => '1=1'! );! B B B *** Filter on SESSION_USER Database security for query access Virtual Private Databases RedacMon Audit Vault and Database Firewall Hadoop security for Hadoop jobs Kerberos AuthenMcaMon Apache Sentry (RBAC) Audit Vault System- specific encrypmon Database tablespace encrypmon BDA On- disk EncrypMon

33 SQL, Everywhere Futures

34 More Lessons from Exadata Move less data à Go faster 1 2 Storage Indexes Skip reads on irrelevant data Big Hadoop Blocks ~ Big Speed Up Caching Cache frequently accessed columns HDFS Caching

35 Try it out: Oracle Big Data Lite hxp:// appliance/oracle- bigdatalite html

36

37

38 Fan- Out Parallelism Approach #1 Schema- on- Write, Small blocks RDBMS PQ PQ Host 1 Host 2 Host 3

39 Fan- Out Parallelism Approach #2 Schema- on- Read, Large Blocks