SQL - The Goto Language for Big Data Analy&cs!

Transcription

1 SQL - The Goto Language for Big Data Analy&cs! Analy&cal SQL SQL Made Great Hermann Bär, hermann.baer@oracle.com Product Management Data Warehousing 1

2 Who Am I Not?.. But Who SHOULD be Here.. Keith Laker keith.laker@oracle.com 19 years with Oracle Worked in consulsng, global support, onsite support and product mgnt. Part of Data Warehouse product management team Product Manager for analyscal SQL Based in Manchester, UK oracle-big-data blog twixer feed

3 Safe Harbor Statement The following is intended to outline our general product direc&on. It is intended for informa&on purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or funcsonality, and should not be relied upon in making purchasing decisions. The development, release, and Sming of any features or funcsonality described for Oracle s products remains at the sole discreson of Oracle. 3

4 Dear Reader Due to legal constraints we are not allowed to provide any material about Oracle s excisng upcoming new technology. We sincerely apologize for this. However, there is no reason to wait and not to use the current available version, Oracle Database 12c Release 12.1 up to its full potensal. We hope you will find this material useful. Please visit us also on OTN for further informason. Your Oracle team.

5 Agenda Some things you will be able to do today and SQL and Big Data CalculaSng approximate answers SQL made for analysis Summary some things you will be able to do tomorrow... All new features marked as. 5

6 SQL For Big Data

7 Requires Conceptual Common View of AnalyScal Big Data Architecture Language = SQL Data Streams ExecuSon InnovaSon AcSonable AcSonable AcSonable Events Insights InformaSon SQL Event Engine Data Data Factory Enterprise ReporSng Reservoir InformaSon Store Structured Enterprise Data Other Data Events & Data Discovery Lab Discovery Output 7

8 Approximate result-sets When good enough is in fact good enough

9 Exploring Today s Big Data Lakes Key business challenges Many queries rely on counts and/or stassscal calculasons NDVs, Pareto s 80:20 rule, idensfying outliers etc. Exact processing of large data sets is resource intensive Exploratory queries don t require completely accurate result Trending analysis, social analysis, sessionizason analyscs Oracle s solu&ons Provide approximate result capabilises in SQL Key objec&ves Return approximate results faster, minimal deviason from actual Use fewer resources, allowing more queries to run 9

10 Genng Approximate Uniqueness Counts Answer How many type quessons How many unique sessions today How many unique customers logged on How many unique events occurred COUNT (DISTINCT...) Returns the exact number of rows that contain dissnct values of specified expression Can be resource intensive because requires sorsng significantly faster solu&on is APPROX_COUNT_DISTINCT (expr) Processes large amounts of data significantly faster Uses HyperLogLog algorithm Negligible deviason from exact result Ignores rows containing null values Supports any scalar data type Does not support BFILE, BLOB, CLOB, LONG, LONG RAW, or NCLOB 10

11 Accuracy and Performance Results for accuracy Real world customer workload Accuracy that is typically 97% with Performance Results Real world customer workload 5-50x improvement 95% confidence Notes: This approach does not use sampling, it uses a hashbased approach Ignores rows that contain a null value for specified expression Supports any scalar data type other than BFILE, BLOB, CLOB, LONG, LONG RAW, or NCLOB 11

12 Count DisSnct Processing

13 Count DisSnct Processing COUNT(DISTINCT ) processing: SORT operasons 8GB of memory (PGA) 164GB of temp 13

14 Approximate Query Processing

15 Approximate Query Processing With approx query processing: No sort Only 540MB PGA Zero temp 1. 50x Faster 2. 15X Less Memory 3. No temp 15

16 Approximate Count DisSnct Comparison of COUNT(DISTINCT) with APPROX_COUNT_DISTINCT Independent exemplary performance and accuracy analysis Data courtesy of ChrisSan Antognini High accuracy with superior performance, using bounded memory For complete test details, see hxp://antognini.ch/2014/10/the-approx_count_dissnct-funcson-a-test-case/

17 SQL is made for Analysis Simpler code, faster results Copyright 2015, Oracle and/or its affiliates. All rights reserved. 17

18 The On-Going EvoluSon of AnalyScal SQL Introduction of window functions Statistical functions SQL model clause Partition Outer Join Data mining Pattern matching Top N clause Approx. count distinct JSON support 8i 9i 10g 11g 12c Enhanced window functions (percentile, etc) Rollup, grouping sets, cube SQL Pivot Recursive WITH ListAgg Nth value window Slide - 18

19 Making Code Simpler: Pattern Matching with SQL Java vs. SQL: Searching for W Patterns in Stock Trade Data package pigstuff; import java.io.ioexception; import private java.util.arraylist; class V0Line { import java.util.iterator; String state = null; public import org.apache.pig.evalfunc; String[] setstate(v0line attributes; lineprev, V0Line linenext) { import private org.apache.pig.pigexception; String boolean prev eq(string = " ; a, String b) { import org.apache.pig.backend.executionengine.execexception; private String boolean next = gt(string ; a, String b) { import org.apache.pig.data.bagfactory; public V0Line(String[] atts) { import org.apache.pig.data.databag; public attributes Tuple exec(tuple = atts; input) throws IOException { import org.apache.pig.data.datatype; } import import org.apache.pig.data.tuplefactory; public public String[] Schema getattributes() outputschema(schema { input) { import org.apache.pig.impl.logicallayer.frontendexception; return attributes; Schema.FieldSchema linenumber = new import org.apache.pig.impl.logicallayer.schema.schema; } Schema.FieldSchema("linenumber", DataType.CHARARRAY); /** Schema.FieldSchema pbykey = new * public void setstate(string state) { Schema.FieldSchema("pbykey", DataType.CHARARRAY); nbayliss this.state Schema.FieldSchema = state; count = new Schema.FieldSchema("count", */ } DataType.LONG); } } Schema tupleschema = new Schema(); tupleschema.add(linenumber); tupleschema.add(pbykey); tupleschema.add(count); return new Schema(tupleSchema); 250+ Lines of Java and PIG Find a W-shape paxern in a Scker stream: Output the beginning and ending date of the paxern Calculate average price each the W-shape Find only paxerns that lasted less than a week 19

20 Making Code Simpler: Pattern Matching with SQL Java vs. SQL: Searching for W Patterns in Stock Trade Data package pigstuff; import java.io.ioexception; import private java.util.arraylist; class V0Line { import java.util.iterator; String state = null; public import org.apache.pig.evalfunc; String[] setstate(v0line attributes; lineprev, V0Line linenext) { import private org.apache.pig.pigexception; String boolean prev eq(string = " ; a, String b) { import org.apache.pig.backend.executionengine.execexception; private String boolean next = gt(string ; a, String b) { import org.apache.pig.data.bagfactory; public V0Line(String[] atts) { import org.apache.pig.data.databag; public attributes Tuple exec(tuple = atts; input) throws IOException { import org.apache.pig.data.datatype; } import import org.apache.pig.data.tuplefactory; public public String[] Schema getattributes() outputschema(schema { input) { import org.apache.pig.impl.logicallayer.frontendexception; return attributes; Schema.FieldSchema linenumber = new import org.apache.pig.impl.logicallayer.schema.schema; } Schema.FieldSchema("linenumber", DataType.CHARARRAY); /** Schema.FieldSchema pbykey = new * public void setstate(string state) { Schema.FieldSchema("pbykey", DataType.CHARARRAY); nbayliss this.state Schema.FieldSchema = state; count = new Schema.FieldSchema("count", */ } DataType.LONG); } } Schema tupleschema = new Schema(); tupleschema.add(linenumber); tupleschema.add(pbykey); tupleschema.add(count); return new Schema(tupleSchema); SELECT first_x, last_z FROM ticker MATCH_RECOGNIZE ( PARTITION BY name ORDER BY time MEASURES FIRST(x.time) AS first_x, LAST(z.time) AS last_z ONE ROW PER MATCH PATTERN (X+ Y+ W+ Z+) DEFINE X AS (price < PREV(price)), Y AS (price > PREV(price)), W AS (price < PREV(price)), Z AS (price > PREV(price) AND z.time - FIRST(x.time) <= 7 )) 250+ Lines of Java and PIG SQL - 20x less code, 5x faster 12 Lines of SQL 20

21 Data Streams ExecuSon InnovaSon AcSonable Events Event Engine Events & Data Data Reservoir AcSonable Insights Discovery Lab Data Factory Discovery Output Enterprise InformaSon Store AcSonable InformaSon ReporSng Structured Enterprise Data Other Data SQL is Made for AnalyScs SQL analy&cs for all your data delivers faster processing, richer analyscs over all your data relasonal, Hadoop, Hive, NoSQL, event streams etc Faster and Smarter SQL Faster and resource efficient and processing for counsng and stassscal driven queries Richer SQL New richer funcsons for deeper analysis of big data, IoT data sets MATCH_RECOGNIZE 21

22 Safe Harbor Statement The preceding is intended to outline our general product direcson. It is intended for informason purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or funcsonality, and should not be relied upon in making purchasing decisions. The development, release, and Sming of any features or funcsonality described for Oracle s products remains at the sole discreson of Oracle. 22

23 23