Software EMEA Performance Tour Berlin, Germany June

Transcription

1 Software EMEA Performance Tour 2013 Berlin, Germany June

2 The Vertica Database simply fast Dieter Capek,

3 Setting the scene

4 Defining Big Data: HP view HP s definition: Big Data is a class of data challenges, due to increasing volume, velocity, variety, and complexity, that are beyond the capabilities of the traditional software, architecture, and processes to effectively manage and utilize. Gartner "Big Data" is a popular term generally used to acknowledge the exponential growth, availability and use of information in the data-rich landscape of the emerging information economy era. McKinsey Report Big Data refers to datasets whose size is beyond the ability of typical database software tools to capture, store, manage, and analyze. 4

5 The challenges of Big Data Volume, Variety, Velocity, Complexity 59% 70% 85% 2% Worldwide information volume is growing at a minimum rate of 59% annually 70% to 85% of data is complex mixed data types. of currently deployed data warehouses will not scale sufficiently to meet new information volume and complexity demands by 2014 of corporations can deliver the right information, at right time to support enterprise outcomes all of the time * Source: Gartner, Coleman Parkes, October

6 HP Vertica Makes Information Matter ELEVATOR PITCH Vertica is a real time analytics database platform purpose built for Big Data. It consists of a massively parallel database and an extensible analytics framework optimized for real time analysis of data scaling from gigabytes to petabytes. VALUE PROPOSITION Vertica is purpose built to achieve a single goal build the industry s fastest analytics platform for Big Data, 50X 1000X faster on average than legacy solutions and with a much lower TCO. 7

7 600+ Customers and Growing since

8 Zynga: A Large Vertica Installation 1. They are analyzing 70 Billion rows per day in real time 2. Vertica is running 10x - 100x faster than Hadoop 3. They are realizing 9x better compression on Vertica than Hadoop 4. They are adding 8 TB of raw data to Vertica daily 5. They are successfully running Vertica on a 230 node cluster 6. They have 2 PB of source data compressed to 500 TB unique report runs per day 8. 10,000 total reports run per day 9

9 What does it fit in?

10 Unstructured Connectors Structured A complete Big Data analytics platform HP offers an integrated environment unmatched in the industry Data Sources Data Processing Data Analytics User Interfaces HP AppSystem for Apache Hadoop HP Hadoop Reference Architectures Databases Warehouses ERP, CRM SQL-compliant analytics Vertica Business Users Log Files Machine Data HP Hadoop solutions Hadoop applications Social media Customer calls s Meaning-based analytics Autonomy Technology Support and Consulting Services 11

11 Vertica 13 13

12 Vertica Analytics System

13 High-Performance DBMS With A Unique Combination Of Innovations Leverages existing BI, ETL, Hadoop / MapReduce and OLTP investments 15 No disk I/O bottleneck; simultaneously load & query Column Orientation Native DB-aware clustering on lowcost x86 Linux nodes Standard SQL Interface MPP Massively Parallel Processing High Availability Advanced Compression Auto Database Design Built-in redundancy that also speeds up queries Automatic setup, optimization, and DB management Up to 90% space reduction using 15+ algorithms > 50x 200x faster performance > High scalability from TBs to PBs > Simple integration with existing ETL and BI solutions > Superior performance on off-the-shelf hardware > Ultimate deployment flexibility

14 Columnar Storage Vertica intelligently organizes data on disk for each column Only reads columns involved in the query NOT every column compared to Row Storage Updateable column store Ideal for LOAD/READ intensive workloads with dramatic disk I/O reduction SELECT avg(price) FROM tickstore WHERE symbol = AAPL and date = 5/06/09 ; 16 AAPL AAPL BBY BBY Column Store - Reads 3 columns AAPL NYASE NYAASE NYASE NGGYSE NYGGGSE /05/ Row Store - Reads all columns AAPL NYASE NYAASE NYASE NGGYSE NYGGGSE /06/09 BBY NYASE NYAASE NYASE NGGYSE NYGGGSE /05/09 BBY NYASE NYAASE NYASE NGGYSE NYGGGSE /06/09 5/05/09 5/06/09 5/05/09 5/06/09

15 SQL Analytics + - Built for Big Data Features Time series gap filing and interpolation Event window functions and sessionization Social Graphing Pattern matching Event series join Statistical functions Geospatial functions Benefits High performance (Keep Data close to CPU) Low cost (Industry Standard building blocks) Ease of use (Automated + Available) Use Cases Tickstore data cleanups CDR/VOD data analysis Clickstream sessionization Data aggregation and compression Monte Carlo simulation Graph algorithms Sensor Data Process Control Time Series SmartGrid 17

16 Automatic Design & Administration Vertica Database Designer recommend a optimised database design for best performance for the user query needs Minimize DBA effort for essential physical database design Ability to re-design the database incrementally to optimize for changing workloads overtime Run before or after a Vertica database has been deployed. Online execution. Highly responsive to changing workloads DBA Provides > Logical schema Create table > Sample set of Typical queries Sample data > Fault Tolerance Level k-safety Database Designer Generates > Physical schema, compression to: Make queries in sample set run fast Fit within trickle load requirements Ensure all SQL queries can be answered A B C B A C (A B C A) (B A C B A) 18

17 Advanced Compression Vertica replaces slower disk IO with fast CPU cycles through aggressive compression Uses properties of the data like sorting & cardinality Operates across large numbers of rows Can be operated upon without decoding first Implements late materialization Decoded intelligently but as late as possible 19

18 Massively Parallel Processing Public network: 1Gigabit Dedicated private data network: full duplex 10 Gigabit switch Node 1 2 Oct Core (G8) 96+ GB RAM Node 2 2 Oct Core (G8) 96+ GB RAM Node N 2 Oct Core (G8) 96+ GB RAM 4-8TB 4-8TB 4-8TB 20

19 HP Vertica Use Cases

20 Sensor Data Analytics Goal Reduce maintenance costs, improve customer satisfaction, and create value-added services Challenges Incredible volumes of incoming sensor data to manage in a scalable manner High software and hardware costs to manage all this data with homegrown solutions Demanding product uptime requirements for customers Benefits Reduce service costs by predicting failures and meet higher SLAs Achieve overall lower TCO by managing data on less hardware Case Study Deliver value-added services, such as Smart Metering energy consumption programs 22

21 Click Stream Analytics Goal Parse and analyze Web site usage to understand customer engagement and behavior Challenges Specialized and cost-prohibitive analytical appliances do not scale Open source solutions require significant expertise Benefits Understand user behaviors and funnel effectiveness Optimize user experience, retention, and conversion rates Predict user engagement patterns and serve high-value content Monetize clickstream data and improve customer loyalty 23

22 Analyzing Billions of Clicks Challenge at HP.com Millions of visitors generate billion clicks per month Must store 5 years worth of data to get full value of year-overyear clickstream analysis Queries took 48 hours after each day s transactions Extremely complex website many pages are generated dynamically creating complex clickstream trails HP Vertica Solution Queries run in hours or even minutes; 48x 100x faster Industry-standard SQL accelerated acceptance and proficiency Speed of HP Vertica allows iterative and recursive analysis for deeper dives HP can build functionality tailored to individual interactions based on nuanced understanding of user behavior at an individual level 24

23 Security Goal Prevent, detect, and respond to security threats to networks, organizations, governments Challenges Current security forensics technology finds <1% of all security breaches Resources required for continual monitoring of network flows Need to analyze all data and methods to stay ahead of the next security problem Benefits Provide comprehensive actionable information on network activity Detect cyber events and minimize potential damage Prevent security breaches before any consequences occur 25

24 Improving Cyber Security HP IT Challenge at HP IT Global network comprises 16,000 switches, 10,000 routers and connects 300,000 users from 600 sites plus uncountable remote users Generates 150,000 data flows per second of mostly benign network traffic but some malicious activity Must react quickly to malicious activities, 24x7, while providing real-time network anomaly protection and prevention HP Vertica/Lancope Solution Lancope embeds HP Vertica to serve as its analytics engine HP Vertica analyzes NetFlow, sflow, Jflow, IPFix, and netstream flows from devices deployed across HP s network Part of an integrated solutions support security framework that protects HP s network Analyze historic data to help HP gain insight into malware and the techniques hackers use to breach corporate defenses 26

25 Summary

26 What is Vertica good at? 1. Real Time Query and Data Loading 2. Large Volumes of data TB s Upwards 3. Multi-Concurrency Real time Queries 10s, 100s 4. Complex/Analytical Queries 5. Linear Performance using Commodity MPP Linux Platforms 29

27 Vertica Information Sources External Website: Dieter Capek, HP Software, Presales: Stephan Holler, HP Software, Sales: 30

28 Value Added Service VAS: Industry Standard (TSIA)! Software Hersteller/Integratoren bieten hauptsächlich Services in den Phasen von Einkauf bis Integration an Value Added Services fokusieren sich stark auf die Inbetrieb- und Nutzungsphase VAS: Serving the Consumption GAP Die HP SW Value Added Services bestehen aus einer optimalen Mischung aus Education und Support Services unterstützt durch neueste SW Technologie 31 Wir erwarten Sie auf dem Service Stand!

29 HP Software Support portfolio HP Software Support Enterprise Support Around the clock access to product version updates and fixes, reactive problem resolutions and access to self-solve knowledge Premier Support Proactively optimize your environment *NEW* Critical Support Support for complex mission critical environments Basic Standard Priority Advisory Business Critical Access to self-solve knowledge and product updates Reactive support local business hours Access to self-solve knowledge and product updates Reactive support around the clock Enhanced reactive support local business hours with access to a pool of advanced engineers Pre-requisite: Enterprise Basic or Enterprise Standard support Assigned Technical Proactive Account Manager providing strategic support and solutions guidance and proactive support for products Assigned Enterprise Services Manager providing a single point of contact to lead strategic reviews and efficiently co-ordinate all support services to optimize your environment Annual and quarterly reviews and migrations planning from your assigned Critical Support Business Manager. Provides a new level of protection for your environment with rapid diagnosis, response and resolution of issues 32

30 Thank you Vielen Dank!