Big Data reconsidered Separating Hype from Reality for Hosting and Cloud Providers

Transcription

1 Big Data reconsidered Separating Hype from Reality for Hosting and Cloud Providers Matthew Aslett Research Director, Data Management and Analytics, 451 Research

2 Matthew Aslett Research Director, Data Management and Analytics Responsible for data management and analytics research agenda Focus on operational and analytic databases, including NoSQL, NewSQL, and Hadoop With 451 Research since 2007

3 Agenda Big Data : reconsidered Definition Drivers Technologies Use-cases Opportunities for hosting and cloud providers Data and the cloud Use-cases Opportunities and challenges HCTS preview What to expect at HCTS 2013 Q&A

4 Big Data: reconsidered Big Data has the potential to revolutionize the IT industry by enabling new business insight based on previously-ignored and under-utilized data. Big Data is massively over-hyped The challenge for users, vendors, investors and service providers alike (not to mention analysts) is to negotiate towards a successful outcome while avoiding the pitfalls Image: sezzles on Flickr

5 To understand big data There is no such thing as big data Big data is primarily driven by economics, not data There is no single big data usecase You don t need a big data strategy Big data is not just for Web startups There is no single big data platform There is no such thing as big data as a service

6 There s no such thing as big data As in data that is big. It s a trend, not a class or type of data Big data is phrase applied to new approaches to storing, processing and analyzing data But Big data has no agreed definition

7 There s no such thing as big data As in data that is big. It s a trend, not a class or type of data Big data is phrase applied to new approaches to storing, processing and analyzing data But Big data had no agreed definition, until June 2013 Good layman s definition, but overly simplistic, and focuses solely on volume

8 The three (or more) Vs Big data is used to describe data in relation to its growing: Volume The volume of data is too large for traditional database software tools to cope with Velocity The data is being produced at a rate that is beyond the performance limits of traditional systems Variety The data lacks the structure to make it suitable for storage and analysis in traditional databases and data warehouses

9 The three (or more) Vs Big data is used to describe data in relation to its growing: Volume The volume of data is too large for traditional database software tools to cope with Velocity The data is being produced at a rate that is beyond the performance limits of traditional systems Variety The data lacks the structure to make it suitable for storage and analysis in traditional databases and data warehouses HOW MUCH? HOW FAST? HOW VARIED?

10 The three (or more) Vs Big data is used to describe data in relation to its growing: Volume The volume of data is too large for traditional database software tools to cope with Velocity The data is being produced at a rate that is beyond the performance limits of traditional systems Variety The data lacks the structure to make it suitable for storage and analysis in traditional databases and data warehouses HOW MUCH? HOW FAST? HOW VARIED? However, data has always been growing in terms of the 3Vs And data will always continue to grow in terms of the 3Vs

11 The three (or more) Vs Big data is used to describe data in relation to its growing: Volume The volume of data is too large for traditional database software tools to cope with Velocity The data is being produced at a rate that is beyond the performance limits of traditional systems Variety The data lacks the structure to make it suitable for storage and analysis in traditional databases and data warehouses HOW MUCH? HOW FAST? HOW VARIED? Plus: However, data has always been growing in terms of the 3Vs And data will always continue to grow in terms of the 3Vs Veracity Variability Validity Viability Value V V

12 The three (or more) Vs Big data is used to describe data in relation to its growing: Volume The volume of data is too large for traditional database software tools to cope with Velocity The data is being produced at a rate that is beyond the performance limits of traditional systems Variety The data lacks the structure to make it suitable for storage and analysis in traditional databases and data warehouses HOW MUCH? HOW FAST? HOW VARIED? Plus: Veracity Variability Validity Viability Value V V However, data has always been growing in terms of the 3Vs And data will always continue to grow in terms of the 3Vs We stand by our statement that Big data has no agreed definition

13 There s no such thing as big data As a result Big data is being (mis)applied to almost anything related to data, whether appropriate or not And Big data as a term is also being used to avoid explaining complex concepts As such, Big data will inevitably face a backlash due to failure to live up to vague and unrealistic expectations

14 Big data is primarily driven by economics, not data It is now more economically feasible to store and process many data sets that were previously ignored - using clusters of commodity servers and advanced data processing software.

15 Big data is primarily driven by economics, not data It is now more economically feasible to store and process many data sets that were previously ignored - using clusters of commodity servers and advanced data processing software. Moved from storing 1% of data for 60 days in $100,000/TB To 100% of data for a year in $900/TB

16 Big data is primarily driven by economics, not data Big data is what happened when the cost of keeping information became less than the cost of throwing it away. George Dyson It is now more economically feasible to store and process many data sets that were previously ignored - using clusters of commodity servers and advanced data processing software. Moved from storing 1% of data for 60 days in $100,000/TB To 100% of data for a year in $900/TB

17 Big data is primarily driven by economics, not data Big data is what happened when the cost of keeping information became less than the cost of throwing it away? The processing and analysis of very large data sets in their entirety Increased adoption of massively parallel processing approaches Storage and analysis of both structured and multi-structured data Integration of external (social) and corporate data for more complete perspective Schema-free and schema-on-read approaches to data storage/analysis Adoption of exploratory analytic approaches to identify new patterns in data Predictive analytics as a fundamental component of BI strategies Machine-learning algorithms automate the reflection of collective intelligence Increased adoption of in-memory databases for rapid data ingestion Stream processing of sensor and other machine-generated data/events Real-time analysis of data prior to storage within the data warehouse/hadoop Interactive, native, SQL-based analysis of data in Hadoop and HBase are what happened when the cost of keeping information became less than the cost of throwing it away

18 Big data is primarily driven by economics, not data Big data is what happened when the cost of keeping information became less than the cost of throwing it away? The processing and analysis of very large data sets in their entirety Increased adoption of massively parallel processing approaches Storage and analysis of both structured and multi-structured data Integration of external (social) and corporate data for more complete perspective Schema-free and schema-on-read approaches to data storage/analysis Adoption of exploratory analytic approaches to identify new patterns in data Predictive analytics as a fundamental component of BI strategies Machine-learning algorithms automate the reflection of collective intelligence Increased adoption of in-memory databases for rapid data ingestion Stream processing of sensor and other machine-generated data/events Real-time analysis of data prior to storage within the data warehouse/hadoop Interactive, native, SQL-based analysis of data in Hadoop and HBase are what what users are concentrating on Not any number of Vs and not a big data strategy

19 There is no single big data use case Big data is what happened when the cost of keeping information became less than the cost of throwing it away? But there are common Characteristics: Volume Velocity Variety Technologies: Hadoop In-memory data processing Stream processing NoSQL databases Machine learning Predictive analytics Object storage User requirements: Totality Exploration Frequency Dependency

20 Total Data approach Big Data: The growing volume, velocity and variety of data BIG DATA

21 Total Data approach Big Data Technologies: New technologies being adopted to store and process that data BIG DATA TECHNOLOGY

22 Total Data approach Processing and analysis of very large data sets in their entirety Massively parallel processing approaches Both structured and multi-structured data External (social) and corporate data TOTAL DATA Total Data: The user trends driving the adoption of Big Data Technologies

23 Total Data approach Processing and analysis of very large data sets in their entirety Massively parallel processing approaches Both structured and multi-structured data External (social) and corporate data Schema-free and schema-onread data storage/analysis Predictive analytics as a fundamental BI tool Reflection of collective intelligence Identification of new patterns in data TOTAL DATA Total Data: The user trends driving the adoption of Big Data Technologies

24 Total Data approach Processing and analysis of very large data sets in their entirety Massively parallel processing approaches Both structured and multi-structured data External (social) and corporate data Schema-free and schema-onread data storage/analysis Predictive analytics as a fundamental BI tool Reflection of collective intelligence Identification of new patterns in data TOTAL DATA Real-time analysis of data prior to storage In-memory databases for rapid data ingestion Native, SQL-based analysis of data in Hadoop and HBase Stream processing of sensor and machine-generated data Total Data: The user trends driving the adoption of Big Data Technologies

25 Total Data approach Processing and analysis of very large data sets in their entirety Massively parallel processing approaches Both structured and multi-structured data External (social) and corporate data Schema-free and schema-onread data storage/analysis Predictive analytics as a fundamental BI tool Reflection of collective intelligence Identification of new patterns in data TOTAL DATA Real-time analysis of data prior to storage In-memory databases for rapid data ingestion Native, SQL-based analysis of data in Hadoop and HBase Stream processing of sensor and machine-generated data Management alongside existing data technologies

26 You don t need a big data strategy If you tell a vendor that you re working on your Big Data strategy, you ve made them very happy. That s because you ve told them two things: that you have no idea what your own actual business requirements are, and you re ready to spend money. Justin Sheehy, Basho CTO

27 You don t need a big data strategy If you tell a vendor that you re working on your Big Data strategy, you ve made them very happy. That s because you ve told them two things: that you have no idea what your own actual business requirements are, and you re ready to spend money. Justin Sheehy, Basho CTO Google does not have a big data strategy Facebook does not have a big data strategy ebay does not have a big data strategy Yahoo does not have a big data strategy These are the companies that inspired big data but they were all simply solving their own business problems

28 You don t need a big data strategy If you tell a vendor that you re working on your Big Data strategy, you ve made them very happy. That s because you ve told them two things: that you have no idea what your own actual business requirements are, and you re ready to spend money. Justin Sheehy, Basho CTO The processing and analysis of very large data sets in their entirety Increased adoption of massively parallel processing approaches Storage and analysis of both structured and multi-structured data Integration of external (social) and corporate data for more complete perspective Schema-free and schema-on-read approaches to data storage/analysis Adoption of exploratory analytic approaches to identify new patterns in data Predictive analytics as a fundamental component of BI strategies Machine-learning algorithms automate the reflection of collective intelligence Increased adoption of in-memory databases for rapid data ingestion Stream processing of sensor and other machine-generated data/events Real-time analysis of data prior to storage within the data warehouse/hadoop Interactive, native, SQL-based analysis of data in Hadoop and HBase

29 Big data is not just for Web startups Web search Social media Mobile apps Gaming Advertising WEB STARTUPS

30 Big data is not just for Web startups Web search Social media Telecommunications Manufacturing Retail Media Mobile apps Gaming Advertising WEB STARTUPS Insurance Technology Healthcare Financial services ENTERPRISES (DBMS ADOPTERS)

31 Big data is not just for Web startups Seismology Web search Social media Intelligence Energy Academia Telecommunications Manufacturing Retail Media Climate Pharma Mobile apps Gaming Aerospace Bioinformatics Advertising PUBLIC SECTOR (HPC ADOPTERS) WEB STARTUPS Insurance Technology Healthcare Financial services ENTERPRISES (DBMS ADOPTERS)

32 Big data is not just for Web startups Seismology Web search Social media Intelligence Energy Academia Telecommunications Manufacturing Retail Media Climate Pharma Mobile apps Gaming Aerospace Bioinformatics Advertising PUBLIC SECTOR WEB STARTUPS HPC + DBMS HPC + RDBMS Insurance Technology Healthcare Financial services ENTERPRISES

33 Big data and the cloud Cloud computing is a logical endpoint for the combination of grid computing, HPC, virtualization and utility computing As such, cloud computing feels like a potential logical endpoint for big data

34 Big data and the cloud Cloud computing is a logical endpoint for the combination of grid computing, HPC, virtualization and utility computing As such, cloud computing feels like a potential logical endpoint for big data However, adoption of cloud databases largely driven by dev and test

35 Big data and the cloud Cloud computing is a logical endpoint for the combination of grid computing, HPC, virtualization and utility computing As such, cloud computing feels like a potential logical endpoint for big data Barriers to adoption Data security, privacy and management concerns Complexity of large-scale data migration and configuration Database software, and licensing, is not designed for the cloud Opportunities Complex to configure, deploy and manage, and skilled staff are at a premium Enterprises want to make the most of existing tools/skills Opportunity for service providers to mask complexity with managed services rather than self-service cloud

36 Big data and the cloud Potential for increased revenue for DBaaS providers For 451 Research clients: DBaaS poised to drive next-generation database growth Next-Generation Operational Databases: Free (registration required) presentation of key findings But there is no such thing as big data as a service

37 There is no single big data platform Whereas the traditional data management platform is relatively straightforward

38 There is no single big data platform Whereas the traditional data management platform is relatively straightforward A big data management platform is made up of a combination of multiple optional components and moving parts

39 There is no such thing as big data as a service But there are opportunities for hosting and cloud providers at key points in the stack Particularly through partnerships

40 There is no such thing as big data as a service Hadoop vendors: Cloudera Hortonworks MapR Pivotal Hadoop enablers: Infochimps ZettaSet Continuuity Qubole Hadoop-aaS providers: TreasureData VertiCloud Mortar Data

41 There is no such thing as big data as a service NoSQL vendors: Basho 10gen DataStax Couchbase Neo4j Objectivity Sqrrl Data NoSQL-aaS providers: Cloudant MongoHQ Garantia Data

42 There is no such thing as big data as a service NewSQL vendors: NuoDB ParElastic GenieDB Clustrix TransLattice ScaleBase ScaleArc Continuent DW-aaS providers: Kognitio 1010data BitYota TempoDB

43 451 Research's annual HCTS-North America is the premier forum for executives in the hosting, cloud computing, datacenter and Internet infrastructure sectors. Creating a Digital Infrastructure Playbook for Next- Generation Datacenters 24 th 9:00-9:30am Data in the Cloud: Best Practices 24 th 1:30-2:00pm Who is doing what with big data in the cloud (public and private) and how do they balance the two? How do they determine best execution venue decisions etc? And what are service providers doing to help? The Rise of the Machines 25 th 1:25-2:05pm The Cloud as a Platform for Big Data Analytics of System Traffic and Log Monitoring for Better Systems Management

44 HCTS-North America Special Offer for Today s Webinar Attendees! 451 Research's annual HCTS-North America is the premier forum for executives in the hosting, cloud computing, datacenter and Internet infrastructure sectors. $250 Discount for Today s Webinar Attendees.* Please use discount code: WEBINAR250 For more information, or to register, please visit: *Offer Expires on 8/19/2013

45 A recording of this webinar along with the slides will be posted on our website within 48 hours: 451research.com/webinars

46 Questions?