Anwendungsfälle für den Aufbau von Data Warehouses mit Echtzeit- Analyse extremer Datenmengen

Transcription

1 Anwendungsfälle für den Aufbau von Data Warehouses mit Echtzeit- Analyse extremer Datenmengen Dieter Capek HP Big Data Platform Presales

2 Agenda Systeme mit extremen Datenmengen Motivation Beispiele HP Vertica Technologie für extreme Datenmengen Nutzenaussage Technologische Merkmale Echtzeit Analyse Daten laden, SQL Erweiterungen Distributed R, Hadoop Integration 2

3 Systeme mit extremen Datenmengen 3

4 Expansion of the digital universe ZB 1.2ZB 40% by ZB Machine-generated data is a key driver in the growth of the world s data which is projected to increase 15x by 2020 (representing 40% of the digital universe) U.S. 32% ZB Western Europe 19% rest of the world 32% India4 % ZB IDC projects that the digital universe will reach 40 ZB by 2020, an amount that exceeds previous forecasts by 5 ZBs. China 13% Current breakdown of the digital universe 22% by 2020 By 2020, China alone is expected to generate 22% of the world s data 4

5 5 Mit so vielen Daten..

6 .. leben wir in einer Informationswüste 6

7 Data accessibility today Typical compromises Infrastructure that becomes unaffordable at scale Lock box Lock out for safety Analytics power that is accessible to only the few A trade-off between quality of insight & the speed of decisions Data is often past its effective expiration date to add value 7

8 for Facebook s CIO, Tim Campos, to get on stage in Europe and declare that, "A partner like HP Vertica thinks like we do and is a key part of Facebook s big data capabilities, is one the best endorsements, err likes, that any modern IT infrastructure vendor could hope for. - Dana Gardner, Briefings Direct 8 Watch the video to see how Vertica empowers Facebook

9 Zynga: A Large Vertica Installation 1. They are analyzing 70 Billion rows per day in real time 2. Vertica is running 10x - 100x faster than Hadoop 3. They are realizing 9x better compression on Vertica than Hadoop 4. They are adding 8 TB of raw data to Vertica daily 5. They are successfully running Vertica on a 230 node cluster 6. They have 2 PB of source data compressed to 500 TB unique report runs per day 8. 10,000 total reports run per day 9

10 CUSTOMER FEEDBACK We are now able to take rapidfire decisions on our assortment and store performance. THANK YOU, TIMOTHY. Maria Kim Assistant Merchant, +13 Target Group We are now able to take rapid-fire decisions on our assortment and store performance, thanks to the new information system. Maria Kim Assistant Merchant, +13 Target Group Valeriya Gamal Head of Reporting and Analytics Gloria Jeans Company

11 HP Vertica Technologie für extreme Datenmengen 11

12 HP Vertica Kurzinformation HP Vertica ist eine analytische Datenbank-Plattform, die speziell für Echtzeit-Analysen von Big Data Fragestellungen entwickelt wurde. HP Vertica besteht aus einem massiv parallelen Datenbanksystem und erweiterbaren analytischen Funktionen. Das System ist optimiert für Echtzeitanalysen von großen Datenmengen bis in den Petabyte Bereich. Vertica Firmengeschichte C-Store Forschungsprojekt Vertica wurde 2005 von den Datenbankforschern Michael Stonebraker und Andrew Palmer gegründet HP hat Vertica in 2011 gekauft 12

13 HP Vertica Analytics platform 7.0 High-performance data analytics platform purpose-built for big data Blazing fast analytics Gain insight into your data in near-real time by running queries 50x-1,000x faster than legacy products Massive scalability Infinitely scale your solution by adding an unlimited number of industry-standard servers Open architecture Protect and embrace your investment in hardware and software with built-in support for Hadoop, R, and a range of ETL and BI tools Optimized data storage Store 10x-30x more data per server than row databases with patented columnar compression Load & analyze growing forms of semi-structured data Quickly and easily load, explore, analyze emerging and rapidly growing forms of semi-structured data. Easy Set-Up and Administration Get to market quickly with your analytics initiatives at low cost of administration and maintenance 13 Speed, scalability, and openness at lower TCO

14 Turing Award 2014 to Michael Stonebraker 14

15 HP Big Data Conference 2015 August Westin Waterfront Hotel, Boston Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. +40 in-depth technical sessions Real-world customer use cases Network with hundreds of peers Hands-on hackathon and more

16 High-Performance DBMS With A Unique Combination Of Innovations Leverages existing BI, ETL, Hadoop / MapReduce and OLTP investments 16 No disk I/O bottleneck; simultaneously load & query Column Orientation Native DB-aware clustering on lowcost x86 Linux nodes Standard SQL Interface MPP Massively Parallel Processing High Availability Advanced Compression Auto Database Design Built-in redundancy that also speeds up queries Automatic setup, optimization, and DB management Up to 90% space reduction using 15+ algorithms > 50x 200x faster performance > High scalability from TBs to PBs > Simple integration with existing ETL and BI solutions > Superior performance on off-the-shelf hardware > Ultimate deployment flexibility

17 Columnar Storage Vertica intelligently organizes data on disk for each column Only reads columns involved in the query NOT every column compared to Row Storage Updateable column store Ideal for LOAD/READ intensive workloads with dramatic disk I/O reduction SELECT avg(price) FROM tickstore WHERE symbol = AAPL and date = 5/06/09 ; 17 AAPL AAPL BBY BBY Column Store - Reads 3 columns AAPL NYASE NYAASE NYASE NGGYSE NYGGGSE /05/ Row Store - Reads all columns AAPL NYASE NYAASE NYASE NGGYSE NYGGGSE /06/09 BBY NYASE NYAASE NYASE NGGYSE NYGGGSE /05/09 BBY NYASE NYAASE NYASE NGGYSE NYGGGSE /06/09 5/05/09 5/06/09 5/05/09 5/06/09

18 HP Vertica Cluster: Massively Parallel Processing Public network Dedicated private data network: full duplex 10 Gigabit switch Node 1 2 Oct Core (G8) 128 GB RAM Node 2 2 Oct Core (G8) 128 GB RAM Node N 2 Oct Core (G8) 128 GB RAM 10 TB 10 TB 10 TB 18

19 What is Vertica good at? 1. Real time query and data loading 2. Large volumes of data: up to petabytes 3. Multi-concurrency : 10s, 100s of queries at the same time 4. Complex analytical queries 5. Linear performance using commodity Linux servers (HP DL380 servers) 19

20 Echtzeit Analyse 20

21 Wie schnell kann man Daten in ein Datenbanksystem laden? 21

22 Daten laden mit HP Vertica: Experimente bei Facebook 22

23 SQL Analytics + - Built for Big Data Features Time series gap filing and interpolation Event window functions and sessionization Social Graphing Pattern matching Event series join Statistical functions Geospatial functions Benefits High performance (keep data close to CPU) Low cost (industry standard building blocks) Ease of use (automated + available) Use Cases Tickstore data cleanups CDR/VOD data analysis Clickstream sessionization Data aggregation and compression Monte Carlo simulation Graph algorithms Sensor Data Process Control Time Series SmartGrid 23

24 SQL Beispiel: Mustersuche in Sensordaten select factory_id, count(sensor_id) from (.. ) as foo MATCH (PARTITION BY factory_id, sensor_id ORDER BY timest DEFINE Normal AS (sensor_val between 0.2 and 0.7), Incr AS (sensor_val > sensor_val_last * 3), High AS (sensor_val >= 0.9), Drop AS (sensor_val <= 0.1 and sensor_val_last >= 0.8) PATTERN P AS (Normal+ Incr+ High* Drop) ROWS MATCH FIRST EVENT) ) foo2..; 24

25 Coverage: HP Vertica Flex Zone A new approach to loading, managing, and exploring semi-structured data 25

26 Vertica Cluster User Defined Extensions in R What is R Open source language for statistical computing Wide range of packages available for advanced data mining and statistical analysis Advantages of UDx in R Vertica automatically parallelizes the execution of user defined R code Optimized data transfer between Vertica and R 26

27 HP Distributed R und HP Vertica für parallele verteile Algorithmen 27

28 Combining the Power of Vertica and Hadoop Vertica Designed for Performance Interactive Analytics A rich SQL ecosystem Both Purposebuilt Scalable Analytics Platforms Hadoop Designed for Faulttolerance Batch Analytics A rich Programming Model Integration via M/R, HDFS, HCatalog 28

29 Sehr gutes Zusammenwirken von HP Vertica und Hadoop HP Vertica arbeitet sehr gut mit Hadoop Umgebungen zusammen. Die Integration kann erfolgen durch Lesen von Hadoop HDFS Daten Map Reduce Anwendungen können lesend und schreibend auf HP Vertica zugreifen Durch die Integration mit dem HIVE Catalog sind aus Vertica alle Datenformate lesbar, die in HIVE verwendbar sind (z.b. Parquet Dateien) HP bietet mit HP Vertica for SQL on Hadoop eine Produktvariante an, die HP Vertica direkt auf dem Hadoop Cluster mit Hadoop Datenhaltung ausführt. 29

30 Vielen Dank! 30