Accelerating GeoSpatial Data Analytics With Pivotal Greenplum Database

Transcription

1 Copyright 2014 Pivotal Software, Inc. All rights reserved. 1 Accelerating GeoSpatial Data Analytics With Pivotal Greenplum Database Kuien Liu Pivotal, Inc. FOSS4G Seoul 2015

2 Warm-up: GeoSpatial on Hadoop vs. P-RDB Now GeoSpatial Query Dimension Translator e.g., GeoHash, Hilbert Filling Range Query NoSQL Data System e.g., Hbase, MangoDB We do NOT have to reinvent the wheel! GeoSpatial Query Master Standby GeoSpatial GeoSpatial Network Interconnect Distributed File System e.g., GFS, HDFS Segment GeoSpatial Segment GeoSpatial Segment GeoSpatial (a) GeoSpatial query on NoSQL (b) GeoSpatial query on MPP Copyright 2014 Pivotal Software, Inc. All rights reserved. 2

3 Agenda Greenplum Database Overview Architecture & Features GeoSpatial supporting Big Data Analytics based on GPDB Big Data Suite Open Source Plan Data Science Case Studies Q&A 3

4 Greenplum Database Fully ACID Relational Database built for Big Structured Data SQL Standard Compliant Cluster based system running on Commodity hardware on top of the Linux OS Available as an EMC appliance 10+ years of R&D investment Enterprise product with install base 4

5 Feature Set Greenplum DB SYSTEM ACCESS CLIENT ACCESS PSQL, ODBC, JDBC BULK LOAD/UNLOAD GPLoad, GPFdist, External Tables, GPHDFS ADMIN TOOLS GP Command Center, GP Perfmon, GP Support 3 rd PARTY TOOLS Compatible with Industry Standard BI & ETL Tools DATA PROCESSING SQL STANDARD COMPLIANCE MASSIVELY PARALLEL PROCESSING (MPP) IN-DATABASE PROGRAMMING LANGUAGES PL/pgSQL, PL/Python, PL/R, PL/Perl, PL/Java, PL/C IN-DATABASE ANALYTICS & EXTENSIONS MADLib, PostGIS, PGCrypto DATA STORAGE FULLY ACID COMPLIANT TRANSACTIONAL DATABASE POLYMORPHIC STORAGE HEAP, Append Only, Columnar, External, Compression MULTI-VERSION CONCURRENCY CONTROL (MVCC) INDEXES B-Tree, Bitmap, GiST 5

6 MPP Shared Nothing Architecture Flexible framework for processing large datasets Master Host and Standby Master Host Master coordinates work with Segment Hosts Segment Host with one or more s s process queries in parallel Segment Hosts have their own CPU, disk and memory (shared nothing) High speed interconnect for continuous pipelining of data processing node1 Segment Host node2 Interconnect Segment Host Segment Host Segment Instance Segment Instance Segment Instance Master Host node3 SQL Segment Host Standby Master noden Segment Host 6

7 Query Execution Example Master Slice 1 Motion Gather SELECT s.beer, s.price FROM Bars b, Sells s WHERE b.name = s.bar AND b.city = Seoul Bars is distributed randomly Sells is distributed by bar Slice 2 Motion Gather Project s.beer, s.price Slice 2 Motion Gather Project s.beer, s.price s Scan Sells Segment 1 HashJoin b.name = s.bar Motion Redist(b.name) s Scan Sells Segment n HashJoin b.name = s.bar Motion Redist(b.name) Motion Redist(b.name) Motion Redist(b.name) Filter b.city = 'San Francisco' Filter b.city = 'San Francisco' b Scan Bars Slice 3 b Scan Bars Slice 3 Segment 1 Segment n 7

8 Parallel Query Optimizer Cost based optimizer designed for big data Allows for rapid development iteration Based on Cascades Framework for Optimizers Breaking the optimizer into multiple components components can be replaced and configured separately Algebraic model Operators input properties determine output properties Separating representation from optimization Separate component integrated into GPDB provider stats card unittests provider memory cost md gpos memo search xforms 8

9 Loading: Massively-Parallel Ingest Extreme speed and, immediate usability from files, ETL & Hadoop Fast Parallel Load & Unload No Master Node bottleneck 10+ TB/Hour per Rack Linear scalability Low Latency Data immediately available No intermediate stores No data reorganization Load/Unload To & From: File Systems Any ETL Product Hadoop Distributions Master Servers Query planning & dispatch gnet Network Interconnect Segment Servers Query processing & data storage External Sources Loading, streaming, etc ETL SQL File Systems 9

10 Polymorphic Storage User Definable Storage Layout TABLE SALES Jun Jul Aug Sep Oct Nov Dec Year -1 Year -2 Row-oriented Column-oriented External HDFS Row oriented faster when returning all columns HEAP for many updates and deletes Append-Optimized for insert mostly Use bitmap indexes on column or row tables to accelerate scans. Use B-Tree indexes on column or row for drill through queries Columnar storage compresses better Optimized for retrieving a subset of the columns when querying Compression can be set differently per column: gzip (1-9), quicklz, delta, RLE Less accessed partitions on HDFS with external partitions to seamlessly query all data Text, CSV, Binary, Avro, Parquet format All major HDP Distros 10

11 Morgan Stanley deployed multiple GPDB clusters for big data use cases including Risk Management, Sales & Trading 11

12 Focus Areas Open Source Cloud Enablement Analytics Database Engine Backup And Disaster Recovery 12

13 Reference Papers MAD Skills: New Analysis Practices for Big Data (VLDB 2009) Dynamic prioritization of database queries (ICDE 2011) Online Expansion of Large-scale Data Warehouses (VLDB 2011) The MADlib analytics library: or MAD skills, the SQL (VLDB 2012) HAWQ: a massively parallel processing SQL engine in hadoop (SIGMOD 2014) Orca: A Modular Query Optimizer Architecture for Big Data (SIGMOD 2014) Optimizing queries over partitioned tables in MPP system (SIGMOD 2014) The data revolution: how companies are transforming with big data (SIGIR 2014) Unveiling clusters of events for alert and incident management in large-scale enterprise it (SIGKDD 2014) Patents 13

14 Agenda Greenplum Database Overview Architecture & Features GeoSpatial supporting Big Data Analytics on GPDB Big Data Suite Open Source Plan Data Science Case Studies Q&A 14

15 Pivotal Big Data Suite With it to build the foundational blocks of big data analytic. Pivotal Greenplum Database Pivotal GemFire Pivotal HD Pivotal HAWQ OSS Support for Spring XD OSS Support for Redis OSS Support for RabbitMQ Redis for Pivotal Cloud Foundry RabbitMQ for Pivotal Cloud Foundry Pivotal HD for Pivotal Cloud Foundry * Pivotal HAWQ For Pivotal Cloud Foundry * *(Requires Pivotal Cloud Foundry v1.3) 15

16 Greenplum Database: Analytical Maturity CLIENT ACCESS & TOOLS CLIENT ACCESS ODBC, JDBC, OLEDB, MapReduce, etc. 3 rd PARTY TOOLS BI Tools, ETL Tools Data Mining, etc ADMIN TOOLS Greenplum Command Center Greenplum Package Manager PRODUCT FEATURES LOADING & EXT. ACCESS Petabyte-Scale Loading Trickle Micro-Batching Anywhere Data Access STORAGE & DATA ACCESS Hybrid Storage & Execution (Row- & Column-Oriented) In-Database Compression Multi-Level Partitioning Indexes Btree, Bitmap, etc. External Table Support LANGUAGE SUPPORT Comprehensive SQL Native MapReduce SQL 2003 OLAP Extensions Programmable Analytics Analytics Extensions (GeoSpatial, PR/R, PL/Java, PL/Python, PL/Perl) GREENPLUM DB SERVICES CORE MPP ARCHITECTURE Multi-Level Fault Tolerance Online System Expansion Workload Management Shared-Nothing MPP Parallel Query Optimizer Polymorphic Data Storage Parallel Dataflow Engine gnet Software Interconnect Scatter/Gather Streaming Data Loading 16

17 Example: Data-driven, Empirical Methods for Problem Solving Data Science Statistics Machine Learning Artificial Intelligence Econometrics. For example: Linear regression linregr() in MADlib, lm() in R Logistic Regression logregr() in MADlib, glm() in R K-Means Clustering kmeanspp( ) in MADlib, kmeans() in R 17

18 Online 1500 Open Source Meetup, 12 Sept. 2015, Beijing China Copyright Pivotal. All rights reserved. 18

19 Agenda Greenplum Database Overview Architecture Features Big Data Analytics on Pivotal Big Data Suite Open Source Plan Data Science Case Studies Q&A 19

20 Predicting Delivery Duration of International Shipments Customer Major postal service provider Business Problem Addressing the predictability of international shipments The MPP architecture of GPDB enabled rapid discovery of trends, anomalies, and actionable insights from their data We demonstrated to the customer that their data had great predictive power Challenges Many incomplete data rows (information not always available when shipment is outside US) Large volume of data spread across several tables and identifying trends, gaps, anomalies, and insights was a challenge Solution Built a regression model to predict delivery duration of international items 20

21 Wafer Bin Map Image Analytics Customer A major semiconductor company Business Problem Given a set of wafer bin map images, identify random die failures, outlying wafer bin maps and clusters of wafer bin maps Solution Denoising techniques applied to wafer bin map images for noise reduction Features extracted from wafer bin map images. Feature space is reduced for clustering and outlier detection. Challenges Deriving features from the wafer bin map images and feature reduction for clustering, outlier detection etc. 21

22 Route Optimization Customer A major courier delivery services company Business Problem Optimizing routing decisions while meeting the demand and satisfying the many business constraints to guarantee feasibility and compliance. Challenges Routing problems are known to be NP-Hard Solution Avoided expensive data movements by addressing demand forecasting and route optimization in the database Built a fully parallelized approximation algorithm that featured a variation of Floyd Warshall all pairs shortest paths and neighborhood searches Achieved significant reduction in fuel consumption over a greedy initial feasible solution Size of the operation. Delivery of 3 million packages a day with the largest fleet in the US Existing solution takes weeks to roll out monthly routing plans 22

23 Questions? Kuien Liu 23

24 A NEW PLATFORM FOR A NEW ERA