Big Data Management System Solution Overview

Transcription

2 Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle s products remains at the sole discretion of Oracle. Copyright 2014 Oracle and/or its affiliates. All rights reserved. 2

3 Program Agenda Oracle positioning Technical proposition Storage Treatment Data Management Visualization Active domains Copyright 2014 Oracle and/or its affiliates. All rights reserved. Oracle Confidential Internal/Restricted/Highly Restricted 3

4 Enterprise Big Data Analytics Architecture Enabling you to Create Value from Data Discovery + Business Analytics BIG DATA ANALYTICS BIG DATA APPLICATIONS Mobile + Web + On-device CREATE VALUE FROM DATA Data Reservoir + Data Warehouse BIG DATA MANAGEMENT BIG DATA INTEGRATION Streaming + Batch Copyright 2014 Oracle and/or its affiliates. All rights reserved. Oracle Confidential #BeyondBigData 4

6 Getting Started with Big Data Build Foundation Enrich Transform ETL Offload ISV Platform Integration Enhance Existing Data Warehouse and BI Key Business Initiatives Copyright 2014 Oracle and/or its affiliates. All rights reserved.

7 Driving Business Value from Technology Innovation Use the Right Tool for the Job and benefit from the Power of AND Hadoop NoSQL Relational Change the Business Disrupt competitors Disintermediate supply chains Leverage new paradigms Exploit new analyses Scale the Business Serve data faster Meet mobile challenges Scale-out economically Run the Business Integrate existing systems Support mission-critical tasks Protect existing expenditures Ensure skills relevance Copyright 2014 Oracle and/or its affiliates. All rights reserved. 7

8 SOURC ES Oracle Big Data Management System ORACLE BIG DATA SQL DATA RESERVOIR DATA WAREHOUSE Cloudera Hadoop Oracle NoSQL Oracle R Advanced Analytics for Hadoop Oracle R Distribution Oracle Big Data Connectors Oracle Data Integrator Oracle Oracle Database Database In-Memory, Oracle Industry Multi-tenant Models Oracle Industry Models Oracle Advanced Oracle Analytics Advanced Analytics Oracle Spatial & Graph Oracle Spatial & Graph Big Data Appliance Exadata Oracle Event Processing Apache Flume Oracle GoldenGate Oracle Data Integrator Oracle GoldenGate Oracle Event Processing Copyright 2014 Oracle and/or its affiliates. All rights reserved.

10 Recap: Big Data Appliance Overview Big Data Appliance X4-2 Sun Oracle X4-2L Servers with per server: 2 * 8 Core Intel Xeon E5 Processors 64 GB Memory 48TB Disk space Integrated Software: Oracle Linux, Oracle Java VM Oracle Big Data SQL* Cloudera Distribution of Apache Hadoop EDH Edition Cloudera Manager Oracle R Distribution Oracle NoSQL Database * Oracle Big Data SQL is separately licensed Copyright 2014 Oracle and/or its affiliates. All rights reserved. 10

11 Recap: Standard and Modular Starter Rack is a fully cabled and configured for growth with 6 servers In-Rack Expansion delivers 6 server modular expansion block Full Rack delivers optimal blend of capacity and expansion options Grow by adding rack up to 18 racks without additional switches Copyright 2014 Oracle and/or its affiliates. All rights reserved. 11

12 Oracle Big Data SQL A New Architecture Powerful, high-performance SQL on Hadoop Full Oracle SQL capabilities on Hadoop SQL query processing local to Hadoop nodes Simple data integration of Hadoop and Oracle Database Single SQL point-of-entry to access all data Scalable joins between Hadoop and RDBMS data Optimized hardware High-speed Infiniband network between Hadoop and Exadata Copyright 2014 Oracle and/or its affiliates. All rights reserved. Oracle Confidential Internal/Restricted/Highly Restricted 12

14 Intelligent Query Optimization One Query Spanning Oracle Database, Hadoop & NoSQL Query Data in RDBMS, Hadoop & NoSQL Oracle SQL Fast Massive Parallelism Filtered Locally Minimized Data Movement Oracle NoSQL DB BDS Server Oracle NoSQL DB BDS Server HDFS Data Node BDS Server HDFS Data Node BDS Server Oracle Database Storage Server Oracle Database Storage Server Copyright 2014 Oracle and/or its affiliates. All rights reserved.

15 Data Stored in Hadoop Example: 1TB File Hadoop/NoSQL Ecosystem {"custid": ,"movieid":null,"genreid":null,"time":" :00:00:07","recommended":null,"activity":8} {"custid": ,"movieid":1948,"genreid":9,"time":" :00:00:22","recommended":"n","activity":7} {"custid": ,"movieid":null,"genreid":null,"time":" :00:00:26","recommended":null,"activity":9} Block B1 {"custid": ,"movieid":11547,"genreid":44,"time":" :00:00:32","recommended":"y","activity":7} {"custid": ,"movieid":11547,"genreid":44,"time":" :00:00:42","recommended":"y","activity":6} {"custid": ,"movieid":null,"genreid":null,"time":" :00:00:43","recommended":null,"activity":8} {"custid": ,"movieid":null,"genreid":null,"time":" :00:00:50","recommended":null,"activity":9} {"custid": ,"movieid":608,"genreid":6,"time":" :00:01:03","recommended":"n","activity":7} Block B2 {"custid": ,"movieid":null,"genreid":null,"time":" :00:01:07","recommended":null,"activity":9} {"custid": ,"movieid":27205,"genreid":9,"time":" :00:01:18","recommended":"y","activity":7} {"custid": ,"movieid":1124,"genreid":9,"time":" :00:01:26","recommended":"y","activity":7} {"custid": ,"movieid":16309,"genreid":9,"time":" :00:01:35","recommended":"n","activity":7} {"custid": ,"movieid":11547,"genreid":44,"time":" :00:01:39","recommended":"y","activity":7}} Block B3 {"custid": ,"movieid":424,"genreid":1,"time":" :00:05:02","recommended":"y","activity":4} 1 block = 256 MB Example File = 40 blocks InputSplits = 40 Potential parallelism on the BDA Copyright 2014 Oracle and/or its affiliates. All rights reserved. Oracle Confidential Internal/Restricted/Highly Restricted 15

16 Enhance Oracle External Table Performance CREATE TABLE movieapp_log_json (click VARCHAR2(4000)) ORGANIZATION EXTERNAL (TYPE ORACLE_HIVE DEFAULT DIRECTORY DEFAULT_DIR ) PARALLEL 20 REJECT LIMIT UNLIMITED; Previously external tables were file-centric 1 file == 1 unit of parallelism Enhanced external tables understand parallelism Automatically map external units of parallelism to Oracle Granules 1 Input Split == 1 Oracle Granule Copyright 2014 Oracle and/or its affiliates. All rights reserved. 16

17 Query Execution on Hadoop select last_name, state, movie, genre from movielog m, customer c where genre= comedy and c.custid = m.custid 1 Query compilation determines: Data locations Data structure Parallelism 3 2 Parallel reads using Big Data SQL Server: Parallel unit: PQ Slaves & InputSplits Filter rows and project columns HDFS NameNode 1 Hive Metastore HDFS Data Node BDS Server 2 HDFS Data Node BDS Server 3 Process filtered result Move relevant data to database Join with database tables Apply database security policies Copyright 2014 Oracle and/or its affiliates. All rights reserved.

18 Big Data SQL Server Minimizes Data Movement Big Data SQL Server Hadoop Smart Scan 1. Apply filter predicates 2. Apply column projections 3. Apply row-level functions JSON Parsing External Table Services 1. Read using Hadoop Classes 2. Convert to Oracle Data Stream Data Node Work close to the data Scans and serializations from Hadoop classes Transformation into Oracle data stream Smart Scan: Emit only relevant data Apply filter predicates Include complex predicates, e.g. JSON_EXISTS Bloom filters for faster joins Score Data Mining models Project columns Include projections from nested structures Disk Copyright 2014 Oracle and/or its affiliates. All rights reserved. Oracle Confidential Internal/Restricted/Highly Restricted 18

21 Oracle Communications Data Model Reference Architecture Oracle Comms Apps (BSS/OSS) Oracle Comms Ntwk Products (Tekelec & Acme) ETL/ELT Adapters Real-Time Adapters Big Data Platform (Hadoop/NoSQL) B Customer Experience Operations Other Oracle Apps (CRM, ERP, etc.) Third Party Relational Data Warehouse (OCDM) Monetization Third Party Sources Data Sources Adapters Feedback Loop Data Management Analytic Apps To Other Apps Copyright 2014 Oracle and/or its affiliates. All rights reserved.

22 Data Stored in Hadoop Example: Files with JSON data Hadoop/NoSQL Ecosystem {"custid": ,"movieid":null,"genreid":null,"time":" :00:00:07","recommended":null,"activity":8} {"custid": ,"movieid":1948,"genreid":9,"time":" :00:00:22","recommended":"n","activity":7} {"custid": ,"movieid":null,"genreid":null,"time":" :00:00:26","recommended":null,"activity":9} {"custid": ,"movieid":11547,"genreid":44,"time":" :00:00:32","recommended":"y","activity":7} {"custid": ,"movieid":11547,"genreid":44,"time":" :00:00:42","recommended":"y","activity":6} {"custid": ,"movieid":null,"genreid":null,"time":" :00:00:43","recommended":null,"activity":8} {"custid": ,"movieid":null,"genreid":null,"time":" :00:00:50","recommended":null,"activity":9} {"custid": ,"movieid":608,"genreid":6,"time":" :00:01:03","recommended":"n","activity":7} {"custid": ,"movieid":null,"genreid":null,"time":" :00:01:07","recommended":null,"activity":9} {"custid": ,"movieid":27205,"genreid":9,"time":" :00:01:18","recommended":"y","activity":7} {"custid": ,"movieid":1124,"genreid":9,"time":" :00:01:26","recommended":"y","activity":7} {"custid": ,"movieid":16309,"genreid":9,"time":" :00:01:35","recommended":"n","activity":7} {"custid": ,"movieid":11547,"genreid":44,"time":" :00:01:39","recommended":"y","activity":7}} {"custid": ,"movieid":424,"genreid":1,"time":" :00:05:02","recommended":"y","activity":4} Copyright 2014 Oracle and/or its affiliates. All rights reserved. Oracle Confidential Internal/Restricted/Highly Restricted 22

23 Hive Provides SQL-like interface to data stored in HDFS Allows applications to process data stored in any format Tables capture metadata required to locate and parse data SQL query generates a MapReduce job to process the data Big Data SQL uses Hive metadata to simplify administration, but it s not required. Big Data SQL does not use Hive for query execution! Copyright 2014 Oracle and/or its affiliates. All rights reserved. Oracle Confidential Internal/Restricted/Highly Restricted 23

24 Schema on Read: MapReduce and Hive Simple Case: Single Column HiveQL: CREATE EXTERNAL TABLE movieapp_log_json ( click STRING ) ROW FORMAT DELIMITED LINES TERMINATED BY '\n' STORED AS TEXTFILE LOCATION '/user/oracle/applog'; > select * from movieapp_log_json {"custid": ,"movieid":null,"genreid":null,"time":" :00:00:07", {"custid": ,"movieid":1948,"genreid":9,"time":" :00:00:22", {"custid": ,"movieid":null,"genreid":null,"time":" :00:00:26", {"custid": ,"movieid":11547,"genreid":44,"time":" :00:00:32, {"custid": ,"movieid":11547,"genreid":44,"time":" :00:00:42", {"custid": ,"movieid":null,"genreid":null,"time":" :00:00:43", {"custid": ,"movieid":null,"genreid":null,"time":" :00:00:50", {"custid": ,"movieid":608,"genreid":6,"time":" :00:01:03, Classic DDL Data source metadata Copyright 2014 Oracle and/or its affiliates. All rights reserved. Oracle Confidential Internal/Restricted/Highly Restricted 24

25 Schema on Read: MapReduce and Hive Same Source with Columns Derived Using SerDe HiveQL: CREATE EXTERNAL TABLE movielog_cols ( custid int, movieid int, activity int, ) ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.jsonserde' STORED AS INPUTFORMAT 'org.apache.hadoop.mapred.textinputformat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.hiveignorekeytextoutputformat' LOCATION '/user/oracle/applog_json'; > select * from movielog_cols Classic DDL Data source metadata Copyright 2014 Oracle and/or its affiliates. All rights reserved. Oracle Confidential Internal/Restricted/Highly Restricted 25

26 Hive Metastore SQL Execution Engines Share Metadata Oracle Big Data SQL Shark Hive Impala Hive Metastore Table Definitions: movieapp_log_json movielog movieapp_log_avro Metastore maps DDL to Java access classes Copyright 2014 Oracle and/or its affiliates. All rights reserved. Oracle Confidential Internal/Restricted/Highly Restricted 26

27 Unify Metadata: Publish Hive Metadata to Oracle Catalog Hive Metastore CREATE TABLE movieapp_log_json (click VARCHAR2(4000)) ORGANIZATION EXTERNAL (TYPE ORACLE_HIVE DEFAULT DIRECTORY DEFAULT_DIR ) REJECT LIMIT UNLIMITED; Oracle Catalog Hive metadata External Table External Table Big Data Appliance + Hadoop/NoSQL Exadata + Oracle Database Copyright 2014 Oracle and/or its affiliates. All rights reserved. 27

28 Automation: Oracle Data Modeler Import Hive definitions into model Automatically generate Oracle DDL for imported tables Copyright 2014 Oracle and/or its affiliates. All rights reserved. Oracle Confidential Internal/Restricted/Highly Restricted 28

29 New Data Sources for Oracle External Tables CREATE TABLE movielog (click VARCHAR2(4000)) ORGANIZATION EXTERNAL ( TYPE ORACLE_HIVE DEFAULT DIRECTORY Dir1 ACCESS PARAMETERS ( com.oracle.bigdata.tablename logs com.oracle.bigdata.cluster mycluster) ) REJECT LIMIT UNLIMITED New set of properties ORACLE_HIVE and ORACLE_HDFS access drivers Identify a Hadoop cluster, data source, column mapping, error handling, overflow handling, logging New table metadata passed from Oracle DDL to Hadoop readers at query execution Architected for extensibility StorageHandler capability enables future support for other data sources Examples: MongoDB, Hbase, Oracle NoSQL DB Copyright 2014 Oracle and/or its affiliates. All rights reserved. 29

30 Use Rich Oracle SQL Dialect Over All Data Snapshot of Oracle SQL Analytic Functions Ranking functions rank, dense_rank, cume_dist, percent_rank, ntile Window Aggregate functions (moving and cumulative) Avg, sum, min, max, count, variance, stddev, first_value, last_value LAG/LEAD functions Direct inter-row reference using offsets Reporting Aggregate functions Sum, avg, min, max, variance, stddev, count, ratio_to_report Statistical Aggregates Correlation, linear regression family, covariance Linear regression Fitting of an ordinary-least-squares regression line to a set of number pairs. Frequently combined with the COVAR_POP, COVAR_SAMP, and CORR functions Descriptive Statistics DBMS_STAT_FUNCS: summarizes numerical columns of a table and returns count, min, max, range, mean, stats_mode, variance, standard deviation, median, quantile values, +/- n sigma values, top/bottom 5 values Correlations Pearson s correlation coefficients, Spearman's and Kendall's (both nonparametric). Cross Tabs Enhanced with % statistics: chi squared, phi coefficient, Cramer's V, contingency coefficient, Cohen's kappa Hypothesis Testing Student t-test, F-test, Binomial test, Wilcoxon Signed Ranks test, Chi-square, Mann Whitney test, Kolmogorov-Smirnov test, One-way ANOVA Distribution Fitting Kolmogorov-Smirnov Test, Anderson-Darling Test, Chi-Squared Test, Normal, Uniform, Weibull, Exponential Copyright 2014 Oracle and/or its affiliates. All rights reserved.

31 } next = linenext.getquantity(); if (!q.isempty() && (prev.isempty() (eq(q, prev) && gt(q, next)))) { state = "S"; return state; } if (gt(q, prev) && gt(q, next)) { state = "T"; return state; Pattern } Matching With Oracle SQL Snapshot of Oracle SQL Analytic Functions if (lt(q, prev) && lt(q, next)) { state = "B"; return state; } if (!q.isempty() && (next.isempty() (gt(q, prev) && eq(q, next)))) { state = "E"; return state; Simplified, } sophisticated, standards based syntax if (q.isempty() eq(q, prev)) { state = "F"; return state; } Finding Patterns in Stock Market Data - Double Bottom (W) Ticker 10:00 10:05 10:10 10:15 10:20 10:25 } return state; private boolean eq(string a, String b) { if (a.isempty() b.isempty()) { return false; } return a.equals(b); } private boolean gt(string a, String b) { if (a.isempty() b.isempty()) { return false; } return Double.parseDouble(a) > Double.parseDouble(b); } private boolean lt(string a, String b) { if (a.isempty() b.isempty()) { return false; } return Double.parseDouble(a) < Double.parseDouble(b); } public String getstate() { return this.state; } } BagFactory bagfactory = public Tuple exec(tuple input) throws IOException { SELECT first_x, last_z FROM ticker MATCH_RECOGNIZE ( PARTITION BY name ORDER BY time MEASURES FIRST(x.time) AS first_x, LAST(z.time) AS last_z ONE ROW PER MATCH PATTERN (X+ Y+ W+ Z+) DEFINE X AS (price < PREV(price)), Y AS (price > PREV(price)), W AS (price < PREV(price)), Z AS (price > PREV(price) AND z.time - FIRST(x.time) <= 7 )) long c = 0; String line = ""; String pbkey = ""; V0Line nextline; V0Line thisline; V0Line processline; V0Line evalline = null; V0Line prevline; boolean nomorevalues = false; String matchlist = ""; ArrayList<V0Line> linefifo = new ArrayList<V0Line>(); boolean finished = false; 250+ Lines of Java UDF 12 Lines of SQL 31 DataBag output = bagfactory.newdefaultbag(); if (input == null) { return null; } if (input.size() == 0) { return null; } Object o = input.get(0); if (o == null) { return null; } Copyright 2014, Oracle and/or its affiliates. All rights reserved. //Object o = input.get(0); if (!(o instanceof DataBag)) { int errcode = 2114; 20x less code Copyright 2014 Oracle and/or its affiliates. All rights reserved.

32 Govern All Data Oracle Big Data Appliance Oracle Database 12c SQL Store JSON data unconverted in Hadoop Store business-critical data in Oracle Data analyzed via SQL Apply advanced security on Hadoop Masking/Redaction Virtual Private Database Fine-grained Access Control DBMS_REDACT.ADD_POLICY( object_schema => 'hr', object_name => 'employee', column_name => 'social_sec_num', policy_name => 'redact_ssn', function_type => DBMS_REDACT.FULL, expression => '1=1' ); Copyright 2014 Oracle and/or its affiliates. All rights reserved. 32

36 Advanced Query & Analysis Full Power of SQL and Advanced Analytics Leverages All Your Data Relational, Hadoop and NoSQL Secure Unified Governance on All Data Fastest Performance Utilize SQL Processing Across the Platform Transparent to Applications No Changes to Application Code Big Data Management System Copyright 2014 Oracle and/or its affiliates. All rights reserved.

38 Why Is Big Data Important? Value Creation HEALTH CARE PUBLIC SECTOR MANUFACTURING COMMUNICATIONS RETAIL Reduce Prescription Fraud Improve Student Outcomes Accelerate Test Cycles to Reduce Backlog Offering New Services based on Location Data Better Predict Product Success In a big data world, a competitor that fails to sufficiently develop its capabilities will be left behind. McKinsey Global Institute Copyright 2014 Oracle and/or its affiliates. All rights reserved.

39