Analytic Platforms: Beyond the Traditional Data Warehouse

Transcription

1 Analytic Platforms: Beyond the Traditional Data Warehouse Merv Adrian, IT Market Strategy Presented to TDWI & NYTECH - NYC, December 2010

2 Topics Introduction Business Case Technologies Uses Cases and Case Studies Getting Started and Best Practices Conclusions 2

3 What is an Analytic Platform? From the 2010 Study by Merv Adrian and Colin White (available at ) Most of the 223 professionals surveyed agreed with our definition: An analytic platform is an integrated and complete solution for managing data and generating business analytics from that data, which offers price/performance and time to value superior to non-specialized offerings. This solution may be delivered as an appliance (software-only, packaged hardware and software, virtual image), and/or in a cloud-based SaaS form. 3

4 Many Organizations Are Already Using One 44% claim to be using one already Our results showed ADBMS usage mirrors overall market 25% say they have no plans to Nonetheless, general satisfaction with analytic projects is low only 21.4% 4

5 Just How Big is the Market? Data warehouse revenues in 2009 were $8.6B, according to 451 Group estimates Leaving Teradata and Sybase aside, we estimate ADBMS vendor revenues as $330M Collectively, these vendors have sold their products to perhaps 1100 firms This is substantial although still only a tiny fraction of the overall market It is profoundly disruptive look at the response from the majors: Oracle ships Exadata, SAP buys Sybase, IBM buys Netezza, Teradata buys Kickfire and Gridscale, Microsoft almost ships It s accelerating and more change can be expected 5

6 Topics Introduction Business Case Technologies Uses Cases and Case Studies Getting Started and Best Practices Conclusions 6

7 What Issues Drive Analytic Platform Purchasers? Complex analyses Performance On-demand capacity User growth Load times Hardware growth & costs Let s step back and look at the business drivers 7

8 It s All About Growth Data Drives 42.6 % of respondents are keeping 3+ years for analytical purposes New sources at huge volumes, in surprising industries, like utilities. Consider smart meters: Transactions 350B Transactions / Year Changing Workloads e.g., Real-time optimization of a Power Grid by reading meters every 15 minutes 120M 3.65B Meter Reads per Month Meter Reads per Day 8

9 Data Types Proliferate Outside Classic Sources Over half of our respondents are using data outside DBMSs A quarter are using unstructured data and web logs 9

10 It s All About Growth Usage Steers Usage is growing as more analysts, programmers and increasingly processes use data How many users do you need your analytic platform to support? Analytic complexity grows as data mining, predictive modeling and advanced statistics become the norm 10

11 This Growth Underlies the Issues We Heard Data growth drives: Performance challenges Load time challenges Hardware growth: storage, processors Usage growth steers: All of the above and need for more sophistication in analytic capability 11

12 Analytic Platforms Target Cost and Complexity Hardware, software, power Analytic platforms aim to reduce costs for all with costeffective hardware, efficient software strategies, data compression and more Deployment options target moving CapEx to OpEx People costs Business analysts get productive tools that leverage their skills Programmers get support for development and test scenarios, sandboxes for iterative trials IT gets operations and management simplicity, not more and different complex tasks 12

13 Topics Introduction Business Case Technologies Uses Cases and Case Studies Getting Started and Best Practices Conclusions 13

14 RDBMS versus ADBMS RDBMSs form the underpinnings of most IT applications It is becoming increasingly more difficult for classic RDBMSs to support the wide range of use cases and workloads that exist in organizations Analytic DBMSs (ADBMSs) target the analytic processing part of the application spectrum Most ADBMSs use the relational model but there is growing interest in NoSQL solutions Even within the ADBMS segment, the ability of products to support a specific workload varies Customers need to match workloads to product features a POC is required in most cases 14

15 Customer Technology Requirements: Survey Results 15

16 ADBMS Development Considerations IT developers need to consider: SQL functionality and restrictions Programming languages supported Quality of the relational optimizer Support for in-database processing Physical storage options and restrictions End-users are mainly concerned with the features and restrictions of the interactive analytic tools they wish to use Most of organizations interviewed were: Using extreme analytic processing Had business users who were using ad hoc and sophisticated SQL queries against large volumes of detailed data 16

17 In-Database Processing ADBMS customers interviewed for this report considered in-database processing a key feature for ADBMS selection Provides the ability to push processing into the DBMS: Boosts performance not all processing may be parallelized Speeds implementation Makes complex analyses possible for users who have expertise to use functions but not to code them Products vary in their support for in-database processing: Built-in functions (scalar, aggregate, string, statistical, etc.) User defined functions Stored procedures MapReduce Predictive models 17

18 ADBMS Storage Options - 1 Wide variety of options Partitioning, indexing, hashing Row-based, column-based, hybrid data storage models Data compression Caching and in-memory data Shared disk versus shared nothing architecture Cause considerable discussion and debate, e.g., pros and cons of columnar storage Implementation differences make comparisons difficult Can add complexity and increase administration Options used should ideally be automated by the ADBMS, but this is difficult to achieve 18

19 ADBMS Storage Options - 2 Storage options should be transparent to SQL users, i.e., users shouldn t be forced to code SQL to exploit an underlying physical storage option Customer interviews showed that some level of SQL tuning may still be required (syntax, indexes, aggregates) Another approach is to use a brute-force approach of using hardware to achieve performance this is often the method used by NoSQL solutions 19

20 NoSQL Solutions No DBMS can satisfy today s wide range of workloads Some organizations develop their own NoSQL solutions, e.g., Google: MapReduce + BigTable DBMS + Google File System NoSQL definition (nosql-database.org): Non-relational, distributed, open-source & horizontally scalable Original focus was modern web-scale databases NoSQL now translates mostly to Not only SQL Several types of NoSQL database systems: Key Value Store, e.g., Amazon Dynamo Document Store, e.g., CouchDB, MongoDB Wide Column Store, e.g., Google BigTable, Cassandra, HBase Graph Databases, e.g., Neo4j, InfoGrid Only 7% of survey respondents were using Hadoop/MapReduce 20

21 RDBMS versus NoSQL NoSQL debate is reminiscent of the object-relational database wars of the 1980s reasons are similar Programmers prefer programmatic approaches for accessing and manipulating data, e.g., MapReduce Non-programmers prefer declarative languages, e.g., SQL Some organizations are reinventing the wheel by trying to extend NoSQL software with RDBMS features Better to recognize that both technologies have their benefits and can coexist The inclusion of MapReduce in ADBMS products offers some of the best of both worlds MapReduce is particularly attractive for the batch processing/transformation of large files of textual data 21

22 Administration Considerations Good administration capabilities rated high in both our survey results (54% rated it as very important) and customer interviews Several customers said that simple administration was an important product selection criterion because they didn t want to employ an army of database administrators Easy administration was particularly important when designing databases and storage structures, and when adding new hardware Several customers noted that as workloads increase in volume and became more mixed in nature, ADBMS workload management capabilities become more important 22

23 Deployment Models The deployment options offered by analytic platform vendors vary Some vendors provide a complete H/W and S/W package, while others deliver an integrated software package Some vendors also offer virtual images that are especially useful during for building and testing prototype applications Direction of some analytic platform vendors is to provide cloud-based offerings for deployment in the either a public or private cloud Ideally, a vendor should offer a variety of different deployment options for its analytic platform 23

24 Topics Introduction Business Case Technologies Uses Cases and Case Studies Getting Started and Best Practices Conclusions 24

25 Analytic Platform Use Cases data sources Analytic platform 1) Enterprise data warehouse with dependent data marts data sources Analytic platform data sources Analytic platform 3) Filtering, staging & transformation of data 2) Independent analytic solution: EDW business area independent data mart extreme analytic platform 25

26 Customer Use Cases: Survey Results 26

27 Case Studies: Traditional Data Warehousing Zions Bancorporation (sponsored by EMC/Greenplum) A financial services organization consisting of 8 commercial banks with 500 full-service banking offices Current RDBMS was too expensive to support predicted growth of EDW and new analytic processing requirements Greenplum was the most cost-effective short- and long-term solution 27

28 Case Studies: Traditional Data Warehousing Zions Bancorporation (sponsored by BMC/Greenplum) A financial services organization consisting of 8 commercial banks with 500 full-service banking offices Current RDBMS was too expensive to support predicted growth of EDW and new analytic processing requirements Greenplum was the most cost-effective short- and long-term solution Hoover s Inc., a D&B Company (sponsored by Teradata) Hoover's Online features an information database covering more than 65 million corporations and 85 million people Needed an integrated view of its customers for supporting sales, marketing and support, but had no data warehouse Teradata 2550 DW Appliance and Relationship Manager was the best fit to match its needs and achieved a positive ROI in 4 months 28

29 Case Studies: Extreme Analytic Platform - 1 Bet365 (sponsored by Kognitio) One of the world s leading online gambling groups with over 4 million users in 200 countries Required a system that could absorb hundreds of thousands of records per hour without impacting the delivery of analytics Kognitio met performance goals, supported 24-by-7 operations, and enabled bet365 to scale the platform to handle expected growth with minimal impact to operations 29

30 Case Studies: Extreme Analytic Platform - 1 Bet365 (sponsored by Kognitio) One of the world s leading online gambling groups with over 4 million users in 200 countries Required a system that could absorb hundreds of thousands of records per hour without impacting the delivery of analytics Kognitio met performance goals, supported 24-by-7 operations, and enabled bet365 to scale the platform to handle expected growth with minimal impact to operations comscore (sponsored by Aster Data) Provides digital marketing intelligence to more than 1600 organizations in over 40 countries Wanted to introduce a new census-based service that could load and analyze over 18 billion new rows of data a day Aster met performance goals, supported standard SQL & MapReduce, and was easy to expand to meet data growth 30

31 Case Studies: Extreme Analytic Platform - 2 CoreLogic LoanPerformance (sponsored by Sybase) Leading provider of information and services on mortgage financing, servicing and securitization Needed to maintain and analyze information on over 100 million active and paid-off mortgages Sybase IQ met performance goals (achieved 8x performance improvement and 40% data compression), used commodity hardware, and supported MicroStrategy toolset 31

32 Case Studies: Extreme Analytic Platform - 2 CoreLogic LoanPerformance (sponsored by Sybase) Leading provider of information and services on mortgage financing, servicing and securitization Needed to maintain and analyze information on over 100 million active and paid-off mortgages Sybase IQ met performance goals (achieved 8x performance improvement and 40% data compression), used commodity hardware, and supported MicroStrategy toolset MediaMath (sponsored by Netezza) Leader in the multi-billion dollar display advertising business Needed to analyze upwards of 15 billion ad impressions a day and calculate the fair market value of more than 50,000 ads/sec Netezza met performance requirements, and offered the simplicity and easy administration MediaMath was looking for 32

33 Case Studies: Extreme Analytic Platform - 3 Large International Banking Group (sponsored by Paraccel) Offers a wide range of services to its over 40 million customers Trading desk needed an analytic solution that could handle the ad hoc analysis of billions of rows of detailed loan/bond data ParAccel met the project s performance goals and design flexibility. Month-end loading was reduced from days to 2 hours and a key query was reduced from 3-4 days to 7 minutes 33

34 Case Studies: Extreme Analytic Platform - 3 Large International Banking Group (sponsored by Paraccel) Offers a wide range of services to its over 40 million customers Trading desk needed an analytic solution that could handle the ad hoc analysis of billions of rows of detailed loan/bond data ParAccel met the project s performance goals and design flexibility. Month-end loading was reduced from days to 2 hours and a key query was reduced from 3-4 days to 7 minutes Zynga (sponsored by Vertica) Online social gaming business with over 65 million active users daily and over 235 million active users each month Needed to analyze game data in real-time, which required the loading/analyzing of tens of billions of rows of data per day Vertica met query and load performance requirements and reduce disk storage needs 34

35 Topics Introduction Business Case Technologies Uses Cases and Case Studies Getting Started and Best Practices Conclusions 35

36 Begin At the Beginning: What Do You Have? What data are you using or not using? How much? Is external data likely to be needed? Don t ask IT what it has, ask users what they want Who is using the data? Who should be? How will it get where you need it to be? What skills do you have? What skills do you need? Operations expertise/staff? Own, host, or cloud? Vendors can help with consulting and with training Pre-built function libraries, business models, dashboards Will you report, slice and dice, model or predict? Can you use what you have? Extend or improve it? Better hardware utilization, virtualization, supplementation Software site licenses, -as-a-service options 36

37 Next Up: What Do You Need? Derived from prior questions and How much data drives how much storage? Who and what skills drives how many users? Determine concurrency what will run at the same time? Don t forget other systems as users, maintenance, backup How recent must the data be? Real-time costs real money Lower cost storage for lower usage data Build scenarios: this is what that costs Be firm on your budget 37

38 Get to a Short List It s reductive they have it or they don t And can prove it, or can t It s financially driven it fits the budget or it doesn t You can only negotiate so much Do they work with your chosen tools, partners, languages? Some will don t waste your time Will they do a POC? You ll be surprised some won t 38

39 POC Practices Perfect Procurement Your data Harder than you think have it ready and in volume Your workload Throw you hardest known problem on the table AFTER they start. Don t just test a few queries one at a time simulate your tough case Your people Be wary of systems only a team of specialists can stand up and get running unless they include that and SLAs in the price Test for failure For example, pull out a blade what happens? 39

40 Topics Introduction Business Case Technologies Uses Cases and Case Studies Getting Started and Best Practices Conclusions 40

41 It s Real and It s Time! Our case studies showed customer satisfaction Analytic platform vendors are growing, iterating rapidly, and learning Start now: Take a hard look at your needs Your competitors already are 41

42 Thank You! Merv Adrian Founder, IT Market Strategy 42