BIG DATA SURVEY 2014 SURVEY

Transcription

1 BIG DATA SURVEY 2014 SURVEY There has been a tremendous amount of hype around Big Data projects and applications in recent years, but relatively little quantifiable evidence proving what, if any, business value these initiatives are delivering to organizations. The database has risen to prominence as the single most important determining factor behind whether a Big Data project succeeds or fails. While organizations have a plentitude of offerings now available to them, ranging from traditional RDBMS vendors and cloud databases to NoSQL and NewSQL, it s a double-edged sword making the right choice has never been more difficult. This was conducted with the goal of identifying opinions, attitudes and trends on the use of database technology to create applications that leverage the massive amounts of data coming into organizations today. What it uncovered is a great divide: the ability to successfully capture and store huge amounts of data is not translating to improved bottom-line business results. Seventy-two percent (72%) of the respondents ed admitted that their organizations cannot access, let alone utilize the vast majority of the Big Data they collect. And the reason their data is going to waste database performance deficiencies. Although most organizations are running multiple databases, the vast majority were not authorized to select an optimized database for their new Big Data application. Instead, they were forced to apply short-term fixes to improve system performance of existing databases (see figures 5.A-5.B). These measures, including adding additional caching and stream processing, were still not enough for some, as one in five respondents with the requisite project knowledge (i.e., excluding those answering don t know ) indicated they had to abandon their Big Data projects altogether (see figure 6). As evidenced by this, organizations that cannot quickly access, analyze and decision on the majority of their data are missing golden opportunities to deliver a more personalized customer experience, drive revenue growth and create competitive advantage.

2 BIG DATA SURVEY 2014 page 2 Survey methodology and respondent profile Commissioned January, 2014, the VoltDB collected responses from 368 qualified IT professionals, including programmers, data analysts, database developers and administrators, and others. These respondents spanned a wide cross-section of industries, including Information Technology, Finance and Insurance, Transportation and Retail, in organizations ranging in size from small and mediumsized businesses to the largest enterprises. What is your PRIMARY role? Figure 1.A 4.9% 3% Programmer/Analyst/ Developer 7.9% Other IT Professional Development Manager 35% Other IT Manager Database Administrator 9.2% Database Administration Manager Non-IT Professional Non-IT Manager 11.4% 11.7% 16% Figure 1.B Programmer/Analyst/Developer 35.9% Other IT Professional 16.0% Development Manager 11.7% Other IT Manager 11.4% Database Administrator 9.2% Database Administration Manager 7.9% Non-IT Professional 4.9% Non-IT Manager 3.0%

3 BIG DATA SURVEY 2014 page 3 Please indicate the industry your company operates in: Information Technology 34.2% Finance & Insurance 13.9% Retail Trade 6.5% Professional, Scientific, Technical Services 6.0% Transportation, Warehousing 5.7% Health Care, Social Assistance 4.6% Arts, Entertainment & Recreation 3.8% Manufacturing 3.3% Educational Services 3.3% Utilities 2.7% Wholesale Trade 1.9% Real Estate, Rental & Leasing 1.6% Mining, Quarrying, Oil & Gas Extraction 1.4% Public Administration 1.1% Construction 0.8% Management of Companies & Enterprises 0.8% Agriculture, Forestry, Fishing & Hunting 0.3% Administrative and Support of Waste Management & Remediation Services Accommodation, Food Services 0.3% Other 7.6% 0.3% Figure 2 What was your company s revenue for 2013 (in USD)? More than $5 Billion 24% of respondents were 12.2% $1-4.9 Billion employed by organizations earning 11.4% > $500 million $ Million 4.6% $ Million 6.5% $ Million 9.0% $ Million 6.0% $ Million 45% of respondents were 12.0% employed by organizations earning $1-9.9 Million 9.0% < $100 million Less than $1 Million 16.8% Don t know 12.5% Figure 3

4 BIG DATA SURVEY 2014 page 4 Which type of database is your organization currently using? (select all that apply) Traditional SQL-based RDBMS 70.9% MySQL 48.4% NoSQL 37.8% Data Warehouse 32.9% Large data set processing in a distributed computing environment 27.2% Mainframe Hierarchical Database 17.9% NewSQL 17.4% Other 7.6% Figure 4.A Figure 4.B 7.6% Traditional SQL-based RDBMS 27.2% Data Warehouse Mainframe Hierarchical Database 70.9% NewSQL NoSQL MySQL Large data set processing in a distributed computing environment 48.4% Other 32.9% 37.8% 17.4% 17.9%

5 BIG DATA SURVEY 2014 page 5 Which of the following have you had to implement due to your database-of-record s performance shortcomings? (select all that apply) Database caches 50.7% Batch ETL 35.4% Additional database licenses and supporting hardware 35.1% Stream processing engine because the database isn t fast enough to process streaming data We have never encountered performance problems with our database-of-record 30.9% 15.6% Figure 5.A Figure 5.B 15.6% Database caches Batch ETL Additional database licenses and supporting hardware Stream processing engine because the database isn t fast enough to process streaming data We have never encountered performance problems with our database-of-record 30.9% 50.7% 35.1% 35.4% In the last 24 months, have you had to abandon any Big Data projects because your database hasn t been able to support your application requirements? No 67.9% Yes 16.8% Don t Know 15.2% Figure 6

6 BIG DATA SURVEY 2014 page 6 What are the primary reasons the project failed? (select all that apply) Could not meet application performance requirements 50.0% Application could not meet minimum response time requirements 46.8% Database licensing fees were too expensive 37.1% Inability to ensure transactional consistency 32.3% Inability to extract business value from data in the required timeframe 32.3% Could not ingest data at sufficient rate 25.8% Other 6.5% Figure 7.A Figure 7.B Could not meet application performance requirements Application could not meet minimum response time requirements Database licensing fees were too expensive 25.8% 6.5% 50.0% Inability to ensure transactional consistency 32.3% Inability to extract business value from data in the required timeframe Could not ingest data at sufficient rate Other 46.8% 32.3% 37.1%

7 BIG DATA SURVEY 2014 page 7 Unused Data Has Little or No Value. The majority of respondents indicated that their organizations can t utilize most of the data they store for Big Data applications (see figure 8), despite the fact that doing so could drive real bottom line business benefits (see figure 9). What percentage of the data coming into your organization today are you able to access/utilize in an actionable way? Less than 50% 72.5% More than 50% 27.5% Figure 8 If you were able to access, analyze, and decision on the majority of data coming into your organization, what would you do with the insight? (select all that apply) Deliver a more personalized customer experience 49.6% Drive revenue growth 47.6% Create competitive advantages (e.g., get to market faster, differentiate, adjust promotions "on a dime" to achieve better results, etc.) 47.6% Identify areas for improved productivity/operations 45.9% Identify new market opportunities 39.8% Help refine our go-to-market strategy 35.4% Determine overall strategic business direction and and tactics 24.8% Figure 9 In-memory databases will trump traditional. Most respondents recognized the advantages of in-memory database architecture (see figure 10) and predicted it will become mainstream in the near term (see figure 11). The drivers for adoption of in-memory architecture include the ability to more quickly develop insights into the business as well as analyze real-time data and support real-time decision making (see figure 12).

8 BIG DATA SURVEY 2014 page 8 Do you believe databases should move to in-memory to deliver better performance characteristics? Yes 89.4% No 10.6% Figure 10 ANALYTICS ARE MOVING IN-MEMORY WITH THE DATABASE Respondents also indicated they believe in-memory analytics will become mainstream in the near term in order to close the time to insight gap in Big Data projects. [see figure 10]. When do you believe in-memory databases will become mainstream? 0-5 years 61.6% 6-10 years 27.2% More than 10 years 7.8% Never 3.4% Figure 11 What do you believe to be the main value of in-memory analytics? Allows for faster business insight 29.0% Analyzes dynamic or real-time data 28.4% Supports real-time decision making 20.2% Allows us to simplify the stack as well as IT infrastructure and processes 6.8% Don t know 5.7% Cost savings/decreased total cost of ownership 4.8% No background in specialty databases required 4.0% Other 1.1% Figure 12

9 BIG DATA SURVEY 2014 page 9 Recommendations Organizations must accurately define project requirements and desired outcomes when developing a Big Data application. There are many new databases available today, but they solve different problems and need to be evaluated carefully to ensure they deliver the required performance characteristics. For example, graph databases are better suited for those situations where data is organized by relationships vs. by row or document, and specialized text search systems should be considered appropriate in situations requiring real-time search as users enter terms. The starting point to harnessing the power of Big Data is the ability to actually access and utilize data in an actionable way. Data is useless if it cannot be acted upon within a time frame that delivers business value. Extracting business value is most challenging with fast data that is, data that provides maximum business value the moment it arrives, but loses value with the passage of time. In these situations, massive amounts of business value can be lost in a matter of milliseconds. It is critical to match database performance to the nature of the data that is being ingested, stored and analyzed. In the case of fast data, in-memory databases and inmemory analytics are the only practical option. Anything else is incapable of acting on the data fast enough to deliver optimal business value. being much faster than writing to and reading from a file system. Additionally, in-memory databases are designed to eliminate multithreading and locking overhead, two of the main reasons for poor database performance. However, until and unless organizations employ a database architecture purpose-built for exceptional speed that also supports real-time analytics and instant decision making on the massive volumes of data streaming in at the moment it arrives Big Data s value will go to waste. Conclusion Big Data is being wasted at most organizations. The majority of the organizations ed cannot effectively leverage the vast majority of the Big Data they collect due to database performance shortcomings. Although they have the ability to successfully capture and store huge amounts of data, that ability alone is not translating to improved bottom line business benefits. The next phase of evolution for Big Data projects will focus on fast data, where the implementation of in-memory databases and analytics will provide the performance required to ingest, analyze and act on this data in real time, enabling organizations to capture maximum business value. In-memory database architecture combined with immediate, intelligent data processing is crucial to tapping Big Data value. Storing data entirely in main memory has the advantage of