Big Data & the LAMP Stack: How to Boost Performance

Transcription

1 Big Data & the LAMP Stack: How to Boost Performance Jeff Kibler, Infobright Community Manager Kevin Schroeder, Zend Technologies, Technology Evangelist

2 Agenda The Machine-Generated Data Problem Benchmark: Methodology & Results The Infobright Approach Zend Technology

3 The Machine-Generated Data Problem

4 The Machine-Generated Data Problem Machine-generated data is the future of data management. Curt Monash, DBMS2 Machine-generated/hybrid data Weblogs Computer, network events CDRs Financial trades Sensors, RFID etc Online game data Human-generated data - input from most conventional transactions Purchase/sale Inventory Manufacturing Employment status change Rate of Growth

5 Current Database Technology: Hitting the Wall Today s database technology requires huge effort and massive hardware How Analytic Performance Issues are Typically Addressed by Pace of Data Growth Tune or upgrade existing databases 66% 75% Upgrade server hardware/processors 54% 70% Upgrade/expand storage systems 33% 60% Archive older data on other systems 30% 44% Upgrade networking infrastructure Don't Know / Unsure 4% 7% 21% 32% High Growth Low Growth 0% 20% 40% 60% 80% 100% Source: KEEPING UP WITH EVER-EXPANDING ENTERPRISE DATA By Joseph McKendrick, Research Analyst; Unisphere Research October 2010

6 What if you could achieve much higher performance analytics with no database tuning and much less hardware?

7 Benchmark: Methodology & Results

8 Benchmark Objective Evaluate Analytic Performance of Infobright/Zend versus MySQL/PHP Parameters Evaluated Loading Time Compression Monitoring Ease Query Resolution Results Software Evaluated 1. Infobright Enterprise Edition Zend Server MySQL PHP 5

9 Configurations Hardware MySQL CentOS5 64bit w/ 16GB Ram and 16 cores Indexed with Star Schema PHP 5 Infobright EE Single, flat table; no indexing Zend Server 5.0.4

10 Machine-generated Data Used National Climatic Data Center (NCDC) Sensor Data Global Daily Weather Reports 1929 to 2011 Publicly available - ftp://ftp.ncdc.noaa.gov/pub/data/gsod/ World s Largest Archive of Climate Data

11 Data Load Time and Compression Bulk Loaders: LOAD DATA INFILE Load Time Raw Size Compression DB Size Infobright 35 Min GB 20.5 : GB MySQL 100 Min GB 0.92 : GB Infobright continues to excel as data size grows to the terabyte scale

12 Monitoring During Test, Extensive Use of: Query Execution Warnings - Severe Slow Request Execution Error/Server/Access Logs - Fatal PHP Errors (Development) Code Tracing I found ZendServer to be extremely easy to use and to configure. Its functionality allows for easy debugging and monitoring of some tough queries we threw at the db. Plus, Zend Framework helped me get the code up and running very quickly. Alba Rico, Benchmark Lead

13 Queries - Baseline Baseline: Nine Non-Analytic, Simplistic Queries MyISAM (MySQL) Baseline Queries Brighthouse (Infobright) 1 SELECT COUNT(*) FROM reports; SELECT COUNT(*) FROM reports; 2 SELECT station, COUNT(*) FROM reports GROUP BY station; SELECT Station, COUNT(*) FROM reports GROUP BY station; 3 SELECT COUNT(DISTINCT station) FROM stations; SELECT COUNT(DISTINCT station) FROM reports; 4 SELECT COUNT(DISTINCT fips_id) FROM stations; SELECT COUNT(DISTINCT fips_id) FROM reports; 5 SELECT COUNT(DISTINCT country_name) FROM countries; SELECT COUNT(DISTINCT country_name) FROM reports; 6 SELECT DISTINCT full_date FROM reports; SELECT DISTINCT full_date FROM reports; 7 SELECT DISTINCT full_date, COUNT(*) FROM reports GROUP BY full_date ORDER BY full_date DESC; SELECT DISTINCT full_date, COUNT(*) FROM reports GROUP BY full_date ORDER BY full_date DESC 8 SELECT MAX(max_temp) FROM reports; SELECT MAX(max_temp) FROM reports; 9 SELECT COUNT(tornado) FROM reports WHERE tornado = TRUE; SELECT COUNT(tornado) FROM reports WHERE tornado = TRUE;

14 Queries - Analytic Simple #1: Reports a descending list of snow depths from all stations who reported a snow depth amount on a given date. Medium #1: Reports a list of station names (and the station's country) with the average precipitation and total precipitation calculated from all days that fall within a given range of dates as well as having reported snow on the same day. Only stations located above the 5000ft elevation mark are considered in this query. Medium #2: Reports the maximum max temperature, maximum precipitation, maximum wind speed, maximum wind gust, maximum standard deviation of the maximum temperature, and the total number of tornadoes reported for the state of Illinois for a given date range. Complex #1:Reports the country or countries with the maximum standard deviation in temperature for counties between the and latitude between a given range of dates. Complex #2: Reports the station, country name, precipitation total, and a description of the precipitation report for all reports between the given date where the country name begins with United'. Complex #3: Reports the station, country name, precipitation total, and a description of the precipitation report for all reports between the given date where the country name contains island. Similar to Q5, except the pattern matching on the country name is done on the end of the string versus the front.

15 Time (seconds) Query Performance IB/Zend and MySQL/PHP Performance Base #1 Base #2 Base #3 Base #4 Base #5 Base #6 Base #7 Base #8 Base #9 Simple #1 Infobright/Zend MySQL/PHP Med. #1 Med. #2 Comp. #1 Comp. #2 Comp. #3 Baseline 3-5: MySQL/PHP pulling from dimension table; Infobright/Zend pulling from fact. Baseline speeds can be drastically improved with addition of simple dimension tables in Infobright DB.

16 Conclusions Infobright + Zend = Powerful LAMP Pair Expedite Development Capabilities Monitor LAMP Performance 20X Stronger Data Compression* 280% Faster Data Loads* Superior Analytic Query Performance * Results from this benchmark

17 The Infobright Approach

18 What Infobright Delivers High performance with much less work and lower cost Faster queries without the work Fast load / High compression Low cost Fast time to production No indexes No projections or cubes No data partitioning Faster ad-hoc analytics Multi-machine Distributed Load Processor 10:1 to 40:1+ compression Less storage and servers Low-cost subscriptions 90% less administration Download in minutes Minimal configuration Implement in days

19 Column vs. Row Orientation - Use Cases ID Job Dept City # # # # # # 1 Shipping Operations Toronto 2 Receiving Operations Toronto 3 Accounting Finance Boston Row Oriented works if All the columns are needed Transactional processing is required Column-Based Storage 1 Shipping Operations Toronto 2 Receiving Operations Toronto 3 Accounting Finance Boston Column Oriented works if Only relevant columns are needed Reports are aggregates (sum, count, average, etc.) Benefits Very efficient compression Faster results for analytical queries Reading column takes similar CPU resources as reading a row

20 Data Packs and Compression 64K 64K 64K 64K Patent-Pending Compression Algorithms Data Packs Each data pack contains 65,536 data values Compression is applied to each individual data pack The compression algorithm varies depending on data type and distribution Compression Results vary depending on the distribution of data among data packs A typical overall compression ratio seen in the field is 10:1 Some customers have seen results of 40:1 and higher For example, 1TB of raw data compressed 10 to 1 would only require 100GB of disk capacity

21 Intelligence Not Hardware Creates information (metadata) about the data upon load, automatically Stores it in the Knowledge Grid (KG) KG is loaded into memory Less than 1% of compressed data size Uses the metadata when processing a query to eliminate / reduce need to access data The less data that needs to be accessed, the faster the response Sub-second responses when answered by the KG Architecture Benefits No need to partition data, create/maintain indexes, projections or tune for performance Ad-hoc queries are as fast as static queries, so users have total flexibility

22 The Knowledge Grid Knowledge Grid applies to the whole table Knowledge Nodes built for each Data Pack Information about the data DP1 DP2 DP3 DP4 DP5 DP6 Column A Column B Global knowledge String and character data Numeric data Distributions Built during LOAD Dynamic knowledge Knowledge Nodes answer the query directly, or Identify only required Data Packs, minimizing decompression, and Predict required data in advance based on workload Built per query E.g. for aggregates, joins

23 Granular Engine 1. Query received 2. Engine iterates on Knowledge Grid 3. Each pass eliminates Data Packs 4. If any Data Packs are needed to resolve query, only those are decompressed Query Q: How are my sales doing this year? Knowledge Grid 1% Results Compressed Data

24 Infobright Customer Performance Statistics Fast query response with no tuning or indexes Analytic Queries Alternative 1 Month Report (15MM events) Alternative Oracle Query Set Alternative 2+ hours with MySQL <10 seconds 43 min with SQL Server 23 seconds 10 seconds 15 minutes seconds Alternative BI Report Alternative Data Load 7 hours in Informix 17 seconds 11 hours in MySQL ISAM 11 minutes

25 Infobright and MySQL Complementary Technologies Infobright is architected on MySQL, the world s most popular open source database Provides a simple scalability path for MySQL users and OEMs No new management interface to learn MySQL integration enables seamless connectivity to BI tools and MySQL drivers for ODBC, JDBC, C/C++,.NET, Perl, Python, PHP, Ruby, Tcl, etc.

26 Analytic Use Cases Online Analytics Vertical Log / Events OEM/ Embedded Marketing/ Advertising Mobile analytics Behavioral analysis Web analytics Telecom: Network performance / management CDR analytics Capital markets Enterprise network and systems analysis Security (SIEM) High performance queries/reporting within ISV / SaaS application Stock analytics

27 Zend Technology

28 The Zend Solution CLOUD SERVICES LAYER Cloud Services SimpleCloud Web Services FRAMEWORK SERVICES LAYER Eclipse-based Auth Security RIA MVC Mobile Data Access Zend Framework Integration Session Clustering RUNTIME SERVICES LAYER Application Monitoring Cluster Management Zend Server Integration Configuration Management Application Deployment Enterprise PHP OPERATING SYSTEM

29 Monitoring

30 Get Started Zend Technologies Zend.com Download Zend Server and Zend Studio Watch recorded webinars Get training Infobrightinfobright.org Infobright.org Download ICE (Infobright Community Edition) Join the forums and learn from the experts! Infobright.com Download a free trial of Infobright Enterprise Edition, IEE Questions Jeff.kibler@infobright.com Kevin@zend.com Social Media twitter.com/zend twitter.com/infobright