Netezza BigData. IBM Big Data. Information Management IBM Corporation

Transcription

1 Netezza BigData IBM Big Data Information Management

2 Big Data? RFID (1.3B in 2005) 4.6 billion camera phones world wide? TBs M 2014! 100. GPS

3 Big Data,,.! 3

4 IBM Big Data InfoSphere BigInsights Hadoop-based low latency analytics for variety and volume Hadoop InfoSphere Information Server High volume data integration and transformation Information Integration MPP Data Warehouse Stream Computing InfoSphere Streams Low Latency Analytics for streaming data IBM InfoSphere Warehouse Large volume structured data analytics 4 IBM Netezza High Capacity Appliance Queryable Archive Structured Data IBM Netezza 1000 BI+Ad Hoc Analytics on Structured Data IBM Smart Analytics System Operational Analytics on Structured Data IBM Informix Timeseries Time-structured analytics

5 IBM Flattens the Time to Value Big Data Curve 5

6 Forrester Big Data Wave 6

7 Information Management Big Data:! " " #,. $ Big Data IBM Big Data Solutions Client and Partner Solutions Big Data User Environment Developers End Users Admin. Traditional data sources (ERP, CRM, databases, etc.) Big Data Enterprise Engine Streaming analytics Internet-scale analytics Source data (Web, sensors, logs, media, etc. ) 7

8 % Netezza BigData 8

9 &"- - ' '( ' )' (, Twitter, Facebook), ". %! : *"$ "+.,. # ' : 9 '"( '( (. ") URL ',. " "(. : IBM InfoSphere BigInsights Enterprise Edition: Custom text annotators Flexible query support )$"

10 *'CDR!! 10 "

11 ! / GPS - ( ) ". ' $ % $ 0 " # ) 11

12 0 #$,!! 30-60!,! % $!!, ": 20%! - #"( 120 : 120"./ / 12

13 % - "(, Streams ), $ " ',

14 " IBM Netezza Netezza BigData 14

15 Netezza!* 1$ 0 15

16 ? &,! # '(, )( * &10 100% + '(, $ (SQL, ODBC, JDBC, OLE DB) -! " SAS, SPSS, R!./ 16

17 Information Management & + "& &% %,, * MPP

18 #$ select DISTRICT, PRODUCTGRP, sum(nrx) from MTHLY_RX_TERR_DATA where MONTH = ' ' and MARKET = and SPECIALTY = 'GASTRO' FPGA Core CPU Core MTHLY_RX_TERR_DATA ( ) ( 0 0, Complex 1 Joins, Aggs, etc. sum(nrx) select DISTRICT, PRODUCTGRP, sum(nrx) where MONTH = ' ' and MARKET = and SPECIALTY = 'GASTRO' 18

19 Advanced Analytics with IBM Netezza Analytic Tools Data Warehouse Analytics Grid Data SQL ETL Demand Forecasting ETL ETL SQL C/C++, Java, Python, Fortran, Fraud Detection SQL 19

20 Advanced Analytics with IBM Netezza Analytic Tools Demand Forecasting Fraud Detection 20

21 IBM Netezza In-Database Analytics nzpackages for R GUI R nzengine for R CRAN Analytics Hadoop nzengine for Hadoop Software Development Kit UDFs UDAPs nzplug-in for Eclipse nzmatrix nzadaptors for Java, C, C++, Python, Fortran In-Database Analytics Parallel Analytics Accelerator Netezza AMPP Platform 21

22 Netezza Spatial (5 "$') 5 "$ ' ( 41,000 ' 4,600,000 () ! "! " # $ # $% # &# '()*+, $'!-(.'!/'00(*1&%21 '*30*4 5% 6% # 782 $ 56 9: :!"##!$## 4! # %!!"##!!$## -;+$;< 56'*30*45= -> *?(', 0(*@', AB+0(*B(,/<#</;C92DE;B#%792F;B 22

23 " IBM BigInsights Netezza BigData 23

24 "IBM BigInsights IBM BigInsights "" "$ "+ Apache Hadoop 2 " MapReduce #BigInsights "$ Hadoop MapReduce 24

25 % MapReduce %Hadoop 0 $ %" / (, )$" "!"#$!"#$!#$!#$ %! & ' %! & ' ( ) ( ) ( )!* ( )!##$ ( )!* ( )!##$ +! *+!## +! *+!## *! *!##$ *! *!##$ *! #$ *! #$,,,,,, (%- (%- - -!#$!#$! &! & ' # ' # %.$ %.$! /#! /# %0*)!#$ %0*)!#$ *** *** Distribute map tasks to cluster Hadoop Data Nodes 1. Map Phase (break job into small parts) 2. Shuffle (transfer interim output for final processing) MapReduce Application Shuffle 3. Reduce Phase (boil all output down to a single result set) Result Set Return a single result set 25

26 InfoSphere BigInsights A Full Hadoop Stack User Interface Management Console Development Tooling (ODS) Analytics Visualization Application Zookeeper Pig Hive Jaql MapReduce AdaptiveMR Avro Analytics ML Analytics Text Analytics Oozie Lucene Storage HBase HDFS GPFS-SNC Data Sources/ Connectors Streams DB2 LUW Netezza R Data Stage DB2 z Teradata Flume Informix Oracle 26

27 - GPFS HDFS 0 GPFS # # 3 % )$", HDFS! NameNode %, POSIX-! Security, Backup, Replication MapReduce 4$ Limited 4$ ' # # 27 ' 4$ -

28 NetezzaBigInsights 5BigInsights/Netezza )".. BigInsights ). BigInsights BigInsights 28

29 "*?(/G' ',; Netezza BigData 29

30 Streams (" 2 (listen, verb), (radio, noun) ***New*** population R ( s t, a t )! 30

31 Telephony Architecture Real-time Summary Statistics Combined churn prediction Summary Statistic Extraction Joint Churn Prediction Data Preprocessing Churn and Value Prediction Model (social network analysis based) Graph Construction Graph Edges and Nodes &'()*+,)-+*. Churn Prediction Model (call pattern based) SNAzzy Model - Social Network Analysis - Customer value extraction Call Detail Records Preprocessed CDRs Predictive Churn Model - Complex Decision Tree - Calling patterns/ user contracts 31

32 IBM InfoSphere Streams v2.0 '% % + %! Front Office 3.0 Eclipse IDE Streams Live Graph Streams Debugger Clustered runtime for near-limitless capacity RHEL v5.3 x86 multicore hardware InfiniBand Database Mining Financial Standard Internet Big Data (HDFS) ***New*** Text ***New*** User-defined toolkits 32

33 /". Streams Composite POS_TxHandling POS_Transactions TCP- Operator Source 1 Sales Sales Operator 2 TaxableSales Operator 3 TaxesDue TCP- Sink TCP- Source Deliveries Operator Inventory Operator Reorders 4 5 TCP- Sink composite POS_TxHandling { graph stream< > POS_Transactions = TCPSource() { } stream< > Sales = Operator1(POS_Transactions) { } stream< > TaxableSales = Operator2(Sales) { } stream< > TaxesDue = Operator3(TaxableSales) { } () as Sink1 = TCPSink(TaxesDue) { } stream< > Deliveries = TCPSource() { } stream< > Inventory = Operator4(Sales;Deliveries) { } stream< > Reorders = Operator5(Inventory) { } () as Sink2 = TCPSink(Reorders) { } } 33

34 "+ InfoSphere Streams: Instance InfoSphere Streams #" Processing Element (PE) ', Streams Job / Streams - PEs Instance Job Node PE operator Node Stream 1 PE Stream 1 Stream 3 Stream 3 PE Stream 2 Stream 4 Stream 5 34

35 ( & Streams: #(') #" ( Src Src OP stream OP OP Sink Sink "job Jobs ( instance Instance ( " node 'node node node h/w node node node node node node Streams instance 35

36 ." Standard Toolkit Relational Operators Filter Functor Punctor Adapter Operators FileSource FileSink DirectoryScan TCPSource TCPSink MetricsSink Utility Operators Custom Beacon Throttle Delay Barrier Pair JavaOp Sort Join Aggregate UDPSource UDPSink Export Import Split DeDuplicate Union ThreadedSplit DynamicFilter Gate - Internet Toolkit InetSource HTTP FTP HTTPS FTPS RSS file Database Toolkit ODBCAppend ODBCSource DB2SplitDB ODBCEnrich SolidDBEnrich DB2PartitionedAppend : DB2 LUW, IDS, soliddb, Netezza, Oracle, SQL Server, MySQL Financial Toolkit Data Mining Toolkit Big Data toolkit Text Toolkit User-Defined Toolkits /$

37 % ( "+ *,, " % $ "+ - /" - MapReduce '-!10 '" " (-!10 + MapReduce, ) -!10 + "" Streams + MapReduce IBM BigData 37

38 -"%$! 38