Database Appliances, MapReduce, and NoSQL Compared to RDBMS

Transcription

1 Appliances, MapReduce, and NoSQL Compared to RDBMS Prepared by Philip Czachorowsi Noember 17,

2 Disclaimer All technical information is presented as is. Any mentions of products does not constitute an endorsement. 2

3 Appliances, MapReduce, and NoSQL Compared to RDBMS Architecture Oeriew - Shared data - Shared nothing Big Data Big Data Technologies - Appliances - Big Data File Management NoSQL Moement Conclusions 3

4 Shared Data Architecture DB2 for z/os All data shared Coupling facility supports distributed loc management and buffer pool coherency Up to 32 data sharing members CF LPAR SRB SRB LPAR SRB SRB SRB SRB LPAR SRB SRB DB2 SRB SRB DB2 SRB SRB DB2 z/os z/os z/os 4

5 Shared Nothing Architecture (MPP) DB2 for LUW No shared data Collocated table joins executed locally at each node Non-collocated table joins require moing data seeral strategies Node LUW DB2 Node LUW DB2 High Speed Interconnect Node DB2 LUW Memory Memory Memory Nodes 5

6 Big Data Big Data Definition: - data sets whose size is beyond the ability of commonly used software tools to capture, manage, and process the data within a tolerable elapsed time.: Wiipedia Big Data Growth Characteristics: - Volume: Petabytes of data - Variety: Semi and unstructured data (text, ideo, logs, sensor data, etc. - Velocity: Rate that data is being created Example sizes - Yahoo captures 450 TB eery day (Forrester Report) - Yahoo has 100,000 s on 40,000 commodity computers (Forrester Report) - Twitter processes 7 TB of data eery day (IBM presentation) - The total world s data will grow from 800,000 petabytes in 2000 to 32 zettabtyes by 2020 (IBM paper) Most data is semi-structured or unstructured and not being stored 6

7 Big Data Examples Web logs Video, photographs Documents Detail eent data, lie RFID and sensor networs Social networ data Call detail records Internet searching Scientific and medical data Financial eents 7

8 Appliances Used for analytics Shared nothing architecture for scalability Moes processing to the data Hardware and software pacaged together Low administratie cost minimal setup and tuning 8

9 Appliances - Netezza Deployment options - Standalone - DB2 Analytics Accelerator for z/os V2.1 Adanced functions - Analytics and statistics - Hadoop 9

10 Appliances - Netezza IBM Netezza 1000 Data Warehouse Appliance 8 cores-fpga per S-Blade 12 S-Blades per rac Up to 10 racs 960 cores Scales from 1 TB to 1.5 petabytes SMP Host SQL Compiler and Optimizer Execution Engine High Speed Interconnect S-Blade Core Memory Core Core S-Blade Core Memory Core Core S-Blade Core Memory Core Core FPGA FPGA FPGA FPGA FPGA FPGA FPGA FPGA FPGA 10 FPGA Decompress data Restrict Project

11 MapReduce and Hadoop MapReduce - Deeloped by Google in Software framewor Support distributed computing of large datasets Executes on large number so s MPP model computer grids Uses commodity hardware - Batch oriented - Uses GFS for the file system Hadoop - Open source ersion of MapReduce, under the Apache project - Deeloped at Yahoo - Uses HDFS for the file system - Numerous other components deeloped to extend functions 11

12 Hadoop Architecture jobtracer tastracer Map 1 Part 1 Part 2 Part 3 Part n Reduce 1 Output Data Split 1 Split 2 Split 3 Split n tastracer Map 2 Map 3 Map n Part 1 Part 2 Part 3 Part n Part 1 Part 2 Part 3 Part n Part 1 Part 2 Part 3 Part n Reduce 2 Reduce 3 Reduce n HDFS 12

13 Hadoop-Relational Comparison Criteria Relational Hadoop* Schema Well formed with data integrity Any arbitrary data, no integrity checing Programming Model SQL (Declaratie) Low leel language (Procedural) Processing model Real time Batch Data storage Physical tables with indexes Files as output Query Optimization Extensie Minimal, procedural steps Parallelism Pre-determined by optimizer Dynamic scheduling of wor Consistency ACID BASE Scalability MPP DBMS can theoretically scale to massie numbers Deployed clusters with 1,000 s s SQL fault tolerance SQL statement is one UOW Checpoints each function Hardware Enterprise Commodity 13 *Numerous open source add on s are aailable to enhance Hadoop functionality

14 Open Source Components Layered on Hadoop Coordination Serice Zooeeper Distributed Serice for Log Data Flume Data Mining Library Mahout SQL Lie Language Jaql - JSON (Jaascript Object Notation) Query Language SQL Language Implementation Hie (HieQL) Job Control Oozie (Serer based) Data Flow Pig (Pig Latin) Systems HBase Cassandra File System HDFS Common Serices Hadoop Common 14

15 InfoSphere BigInsights IBM s Hadoop Implementation Adds enterprise features, Implements more robust infrastructure and connectiity - GPFS instead of HFS - Zooeeper: Coordination serice for distributed applications - Pig: High leel language for executing data flows and for - BigIndex: large scale indexing - Hie: SQL lie language - HBase: Columnar database - Jaql: JSON (Jaascript Object Notation) Query Language - 15

16 Example BigInsights Integration Options Netezza - BigInsighs has a connector to exchange data with Netezza - Uses Jaql - Can use BigInsights to process massie amount of data - Load into Netezza for analysis DB2 for LUW - DB2 UDF - BidInsights Jaql and JDBC InfoSphere Streams 16

17 NoSQL NoSQL Not only SQL The original intention was to implement modern web-scale databases The name was first used in 2009 Major data store types - Key-alue Stores - Document Stored - Extensible Record Stores or Wide Column Stores 17

18 NoSQL - Characteristics Shared nothing scalability - Horizontal scaling to large numbers of serers - Replication and distribution data oer serers - Exploits commodity hardware Simplicity - Simple API - No data schema - Optimized for a specific use Weaer concurrency model - BASE (Basically Aailable, Soft state, Eentually consistent) instead of ACID (Atomicity, Consistency, Isolation, and Durability) - CAP Theorem (Consistency, Aailability, Partitioning) Only two possible Open Source No standard data model lie relational 18

19 Conclusions 19 Critical record eeping data will remain in RDBMS databases - Data integrity and consistency (ACID) - Performance - SQL Standard Hadoop opens up opportunities to access and analyze new data - Enables businesses to gain new insights - Semi-structured as well as some structured data - Hadoop acts as an ETL tool, feeding analytical tools - Will eole and mature, learning from RDBMS RDBMS and Hadoop will co-exist - Processing of semi-structured or massie data in Hadoop - Analytics in RDBMS and database appliances NoSQL not liely to be embraced by large enterprises - Not liely to be used for critical data - Lac of a standard language and schema is a big disadantage - RDBMS endors may learn from NOSQL Hadoop changing rapidly - Open source - Implementation of higher layers, lie Pig, on top of Hadoop to simply deelopment - Addition of RDBMS features into Hadoop One size DBMS will not fit all problems

20 References (Partial List) A Comparison Of Join Algorithms For Log Processing In Mapreduce, Spyros Blanas, Jignesh M. Patel, Vu Ercegoac, Jun Rao, Eugene J. Sheita, Yuanyuan Tian. SIGMOD '10 Proceedings Of The 2010 International Conference On Management Of Data, 2010 Cohadoop: Flexible Data Placement And Its Exploitation In Hadoop, Mohamed Y. Eltabah, Yuanyuan Tian, Fatma Özcan, Rainer Gemulla, Aljoscha Krette, John Mcpherson. Proceedings Of The VLDB Endowment. Volume 4 Issue 9, June 2011 Ricardo: Integrating R And Hadoop, Sudipto Das, Yannis Sismanis, Kein S. Beyer, Rainer Gemulla, Peter J. Haas, John Mcpherson. SIGMOD '10 Proceedings Of The 2010 International Conference On Management Of Data, 2010 Emerging Trends In The Enterprise Data Analytics: Connecting Hadoop And DB2 Warehouse, Fatma Özcan, Daid Hoa, Kein S. Beyer, Andrey Balmin, Chuan Jie Liu, Yu Li.. SIGMOD '11 Proceedings Of The 2011 International Conference On Management Of Data, 2011 Efficiently Support Mapreduce-Lie Computation Models Inside Parallel DBMS, Qiming Chen, Andy Therber, Meichun Hsu, Hans Zeller, Bin Zhang, Ren Wu. IDEAS '09 Proceedings Of The 2009 International Engineering & Applications Symposium, 2009 Comparison Of Map-Reduce And SQL On Large-Scale Data Processing. Jenq-Shiou Leu, Yun-Sun Yee, Wa-Lin Chen ISPA '10 Proceedings Of The International Symposium On Parallel And Distributed Processing With Applications 2010 Will Nosql s Lie Up To Their Promise? Neal Leaitt. Computer, Volume 43 Issue 2, February 2010 Scalable SQL And Nosql Data Stores, Ric Cattell. ACM SIGMOD Record, Volume 39 Issue 4, December 2010 A Comparison Of Approaches To Large-Scale Data Analysis, Andrew Palo, Eri Paulson, Alexander Rasin, Daniel J. Abadi, Daid J. Dewitt, Samuel Madden, Michael Stonebraer. SIGMOD '09 Proceedings Of The 35th SIGMOD International Conference On Management Of Data, 2009 Mapreduce And Parallel DBMSS: Friends Or Foes? Michael Stonebraer, Daniel Abadi, Daid J. Dewitt, Sam Madden, Eri Paulson, Andrew Palo, Alexander Rasin. Communications Of The ACM, Volume 53 Issue 1, January 2010 OLTP Through The Looing Glass, And What We Found There, Staros Harizopoulos, Daniel J. Abadi, Samuel Madden, Michael Stonebraer. SIGMOD '08 Proceedings Of The 2008 ACM SIGMOD International Conference On Management Of Data, 2008 Hadoop: The Definite Guide, Tom White, 2010 Mapreduce: Simplified Data Processing On :Large Clusters, Jeffrey Dean, Sanjay Ghemawat, Communications Of The ACM, January, Understanding Big Data: Analytics for Enterprise Class Hadoop and Streaming Data, Chris Eaton, Dir Droos, Tom Deutsch, George PLapis, Paul Ziopoulos,