2013 BIG DATA OPPORTUNITIES SURVEY

Transcription

1 2013 BIG DATA OPPORTUNITIES SURVEY By Joseph McKendrick, Research Analyst Produced by Unisphere Research, a Division of Information Today, Inc. May 2013 Sponsored by

2 2 TABLE OF CONTENTS Executive Summary Big Data Initiatives The Enterprise Big Data Landscape Big Data Opportunities and Challenges Demographics

3 3 EXECUTIVE SUMMARY Big data isn t just the exclusive property of big web properties such as Google, Facebook, ebay and Yahoo. While these companies blazed the path in harnessing the analytic power of big data, organizations of all different sizes and industry groups are now leveraging big data in many ways as well. That s the finding of a survey of 304 data managers and professionals, conducted by Unisphere Research, a division of Information Today Inc. The survey, fielded in April 2013, revealed a range of practical approaches that organizations of all types and sizes are adopting to manage and capitalize on the big data flowing through their enterprises. It s important to separate the big data hype from the onthe-ground reality for enterprises in their day-to-day businesses. For example, there s a perception that big data only surrenders its wealth when crunched on the same large-scale clustering technologies that are employed by the big web companies. However, many big data problems and opportunities are being addressed on existing, conventional technologies such as relational databases. Most companies are not concerned with indexing the world s web information or delivering real-time recommendations to millions of users. Rather, they are focused on better understanding how a particular product or service is being received, or who is more likely to buy a new offering. The bottom line is that big data is more than simply amassing large quantities of bits and bytes and attempting to slice and dice it enough to glean some insights. Rather, it s a process of understanding the value of information to the business, and working with the business to achieve growth. The ability to manage and harvest important business data from datasets large and small is a concern for executives across the board, whether they are running manufacturing companies or schools. Key findings from the survey include the following: For many organizations today, handling data is not business as usual. More than two-fifths report they have formal big data initiatives in progress, most of which focus on greater customer analysis. Between one-third and one-half of the organizations surveyed report high levels of volume, variety, velocity, and value in data the four Vs that define big data. Two-fifths now maintain data stores in the hundreds of terabytes and greater, and three-fourths face restive user bases that want more data to do their jobs. Most executives intuitively understand the advantages big data can bring to their operations, especially with predictive analytics and customer analytics. However, organizational and skills issues may slow down their progress. Respondents represent a wide range of job titles and, organizations. Leading titles mentioned include DBAs (17%), data architects (13%) and IT consultants (11%). Another 15% of respondents are IT managers and executives, while 8% are business unit executives. More than one-fourth come from organizations with more than 10,000 employees, and the same number represent small firms with 100 or fewer employees. Industries represented include IT and tech, government, financial services, business and consumer services, and education. (See Figures ) Many respondents may be enthusiastic about the potential big data offers, but understand that it needs to be represented to corporate leaders as providing clear business benefits. As one respondent put it: Management sees the opportunity associated with closer understanding of customers and micro segmentation. But we need to avoid the words big data, as people do not really understand what it means but do associate big data with big costs. Still, there are a number of survey respondents whose companies are already seeing direct benefits from their big data efforts. Our organization strives best when it maximizes the success of its customers and direct salesforce, says a respondent. Big data properly analyzed not only provides the information to make better and timely business decisions, it also helps to determine behaviors and success factors in our direct sellers.

4 4 BIG DATA INITIATIVES For many organizations today, handling data is not business as usual. More than two-fifths report they have formal big data initiatives in progress, most of which focus on greater customer analysis. More than two-fifths of respondents indicate that their organizations have some type of formal big data initiative underway. (See Figure 1.) This may include the formation of special cross-organizational teams or committees to examine ways to better exploit this resource, or programs within departments such as marketing designed to mine and generate insights on available data. Management is starting to recognize the importance of big data, but is just unsure how we can integrate it into our company, says one respondent. It s not surprising that many of the largest organizations in this survey those with 1,000 employees and up are engaged in big data initiatives. But there are many small firms also engaging big data as well. More than a third of the smallest companies or agencies in the survey, 37%, report big data efforts, along with 43% of those organizations with employees numbering in the hundreds. (See Figure 2.) Industry groups in the survey leading the way with big data initiatives include services and retail (61%), financial services and insurance (58%), and IT and tech (45%). (See Figure 3.) What is driving these big data initiatives? There are a myriad of business reasons, but most respondents indicate that it is a way to improve internal processes. For a majority of the respondents with such efforts underway, 59%, there is a need to improve existing processes. In addition, another 41% are concerned with the need to create new business processes/models. Another 39% are concerned with the need to reduce time to access data. (See Figure 4.) Big data initiatives also represent a range of business activity. A majority of respondents with big data initiatives, 55%, indicate that they are focusing on customer analysis and segmentation. Close to half, 46%, are concerned with historical/archived data analysis, while 39% are applying big data analytics to production systems log monitoring and analysis. (See Figure 5.) The data being employed in many big data efforts isn t necessarily exotic or hard to capture. Rather, most organizations with formal big data efforts underway are employing information that is already sitting on machines or in storage systems within their enterprises. The survey finds a majority of those respondents with big data initiatives, 51%, say they are working with existing production or transactional data. Another 49% are working with real-time data feeds, suggesting that this is an important early step for many organizations seeking to capitalize on big data. Another traditional source of data that may already be captured and managed within the walls of enterprises ERP and CRM data also are the basis of many big data efforts. (See Figure 6.) Two-fifths of organizations also rely on archived or historical data for initiatives such as data warehousing as the foundation of their big data efforts. The further we can see back in historical trends in relation to today, both in-house and across the sector, the better able we are to make product and marketing decisions that decrease operation cost and increase revenue, says one respondent. Figure 1: Specific Enterprise Big Data Initiatives Underway? Yes 43% No, it's business as usual within our data environment 42% Don t know/unsure 15%

5 5 Figure 2: Specific Enterprise Big Data Initiatives Underway By Company Size 1 to 100 employees 37% 101 to 1,000 employees 43% 1,001 to 10,000 employees 47% 10,000+ employees 53% Figure 3: Specific Enterprise Big Data Initiatives Underway By Industry Retail/services 61% Financial services/insurance 58% IT services/products 45% Manufacturing 29% Government/education 27%

6 6 Figure 4: Drivers of Enterprise Big Data Initiatives Need to improve existing processes 59% Need to create new business processes/ 41% models Need to reduce time to access data 39% Need more visibility into our customer base 37% Management needs to develop/open new 32% lines of business We need more visibility into our internal 32% operations Need to realign more with customers 27% Competitors are doing it 25% Looking to package and sell information 13% Other 6%

7 7 Figure 5: Types of Big Data Initiatives Customer analysis/segmentation 55% Historical/archived data analysis 46% Production systems log monitoring/analysis 39% Website monitoring/log analysis 36% IT systems log monitoring/analysis 37% Competitive/market analysis 36% Content management (photos, images, 32% documents) Social media analysis 25% Testing/analyzing new products/r&d 25% Other 5%

8 8 Figure 6: Types of Data Used in Big Data Initiatives Production or transactional data 51% Real-time data feeds 49% Textual data (documents) 41% ERP data 41% CRM data 40% Archived/historical data 39% Web logs 34% Social media data 32% Multimedia (imagery, audio) 29% Machine to machine data 22% Sensor data 20% Spatial data 16% Other 3%

9 9 THE ENTERPRISE BIG DATA LANDSCAPE Between one-third and one-half of the organizations surveyed report high levels of volume, variety, velocity, and value in data the four Vs that define big data. Two-fifths now maintain data stores in the hundreds of terabytes and greater, and threefourths face restive user bases that want more data to do their jobs. Big data is often defined by the four Vs volume, variety, velocity and value. On all four counts, the survey shows, big data is everywhere, within a range of organizations. Volume: There are considerable volumes of data moving through enterprises. One-sixth of the organizations surveyed, in fact (17%), report having more than a petabyte of data they are currently managing, taking into account all clones, snapshots, replicas and backups. Another 22% report maintaining their total data across their organizations ranges in the hundreds of terabytes. (See Figure 7.) Not surprisingly, most of the highvolume data is seen within larger organizations. Sixty-eight percent of companies or agencies with more than 10,000 employees have data across their organizations exceeding 100 TBs, versus only 8% of the smallest firms with less than 100 employees. (See Figure 8.) Industry groups leading the way in terms of data volume include the financial services and insurance sector, where twothirds report enterprise data exceeding 100 TBs. Manufacturers are also big consumers of big data, with 39% reporting data scaling into the hundreds of terabytes. Retail and service firms follow at 36%. (See Figure 9.) The volume shows no signs of letting up. How will the overall volume of data in respondents organizations change over the next three years? Just about every respondent, 96%, expects their data to increase in volume. More than one-fourth of respondents, 26%, say it will double. (See Figure 10.) Variety: The ability to extract insights from unstructured data which is the essence of big data represents opportunities for real business returns. What percentage of data in respondents enterprises is unstructured data (web logs, social media data, sensor data, documents, imagery, audio)? A large segment, 42%, estimate that a significant portion of their data stores 25% or more is already in the form of unstructured data. (See Figure 11.) Interestingly, unstructured data is as much a part of the picture for small firms as it is large, global organizations. (See Figure 12.) In addition, the prevalence of unstructured data cuts across most major industry groups in the survey. (See Figure 13.) An overwhelming majority, 87%, say the amount of unstructured data in respondents organizations will increase over the next three years. Close to a third, 31%, expect their unstructured data stores to increase by more than 50% in that time period. (See Figure 14.) Velocity: The ability to move data from its sources to decision makers and applications at real or right-time speeds (delivered within one hour of creation) is another demand of big data. This is a capability that still has not been put in place at a majority of enterprises. Only one-fourth of respondents say that a significant amount of the data they maintain (more than 25%) is delivered real time. (See Figure 15.) There is significantly more progress among organizations that have formal big data programs in place, the survey also reveals. Close to one-third of respondents with big data programs report that a significant portion of their data is delivered real-time, versus 21% of respondents with no formal efforts underway. (See Figure 16.) Small to mid-sized businesses tend to have larger proportions of real-time data, the survey finds. For example, one-third of organizations with between 100 and 1,000 employees have a significant share real-time data available to their enterprises, compared to only 15% of the largest companies. (See Figure 17.) By industry, the financial services sector leads the way, with one-third of organizations in this category offering a significant segment of its data in real-time mode. The tech sector follows at 23%, and government agencies with 22%. (See Figure 18.) Over the next three years, close to a third of respondents, 30%, expect significant growth in the amount of real-time data being made available, exceeding 50% of their current data stores. (See Figure 19.) A similar percentage expect to see end-user demand for real-time increase as well. One-third say there will be a significant increase in demand, exceeding 50% over current levels. (See Figure 20.) As one respondent put it: The current systems we have do not provide instant-access dashboarding to data to make critical changes on each business day before that business day is over. Value: Finally, what makes big data meaningful is its ability to deliver insights to decision makers that move organizations forward. Three-fourths of respondents say they see users demanding access to more data to better drive their decisions and insights. (See Figure 21.) Everyone wants access to the data now and they also need queries to be fast, given the mountains of data we are facing, says one respondent. In addition, respondents were asked to indicate the importance of big data to their businesses. A majority, 58%, report that big data is extremely to very important. (See Figure 22.) There is a strong business case to be made for harnessing the power of big

10 10 data. One respondent outlined his company's commitment: We believe that additional information, derivable from analysis of a single large set of related data, as compared to separate smaller sets with the same total amount of data, allows correlations to be found to operational issues, the respondent says. Quick resolution would benefit our organization to serve our customers better and increase our subscriber base. Respondents whose organizations are actively addressing big data opportunities through formal programs are almost three times as likely to recognize the value of the resource. Close to nine-tenths of respondents with big data efforts indicate big data is extremely or very important to their businesses. (See Figure 23.) Small to mid-size businesses place the greatest importance on big data, the survey finds. Sixty-five percent of organizations with between 100 and 1,000 employees say big data is extremely or very important to their business, versus 35% of the largest organizations in the survey. (See Figure 24.) Financial services and tech firms are the most likely to see the business value in big data close to two-thirds of respondents within each category say this information is important. (See Figure 25.) Figure 7: Total Amount of Data Managed < 5 Terabytes (TBs) 27% 5 to 50 TBs 24% 50 to 100 TBs 11% 100 to 500 TBs 11% 500 TBs to 1 Petabyte (PB) 11% 1 to 10 PBs 8% >10 PBs 9%

11 11 Figure 8: Big Data Sites Managing More Than 100 Terabytes By Company Size 1 to 100 employees 8% 101 to 1,000 employees 28% 1,001 to 10,000 employees 47% 10,000+ employees 68% Figure 9: Big Data Sites Managing More Than 100 Terabytes By Industry Financial services/insurance 66% Manufacturing 39% Retail/services 36% IT services/products 33% Government/education 31%

12 12 Figure 10: Changes in Enterprise Data Volume Over Next Three Years Increase by more than 100% 26% Increase 51% to 100% 22% Increase 26% to 50% 21% Increase 10% to 25% 22% Increase 1% to 10% 6% No change 3% Decrease 0.35% Figure 11: Percentage of Enterprise Data That Is Unstructured None at this time 5% <10% 25% Between 10% to 25% 29% Between 25% to 50% 27% >50% 15%

13 13 Figure 12: Unstructured Big Data Sites With More Than 25% of Data Unstructured By Company Size 1 to 100 employees 44% 101 to 1,000 employees 34% 1,001 to 10,000 employees 46% 10,000+ employees 44% Figure 13: Unstructured Big Data Sites With More Than 25% of Data Unstructured By Industry Financial services/insurance 48% Manufacturing 48% IT services/products 45% Government/education 45% Retail/services 43%

14 14 Figure 14: Changes in Enterprise Unstructured Data Volume Over Next Three Years Increase by more than 100% 15% Increase 51% to 100% 16% Increase 26% to 50% 20% Increase 10% to 25% 19% Increase 1% to 10% 11% No change 5% Decrease 2% Don t know/unsure 11% Figure 15: Percentage of Enterprise Data Delivered Real-Time (Defined as within one hour of creation) None at this time 10% <10% 35% Between 10% to 25% 30% Between 25% to 50% 9% >50% 16%

15 15 Figure 16: Big Data Programs Impact on Percentage of Enterprise Data Delivered Real-Time (Respondents reporting more than 25% of data delivered real-time) Formal big data program 31% No big data efforts 21% Figure 17: Real-Time Big Data Sites With More Than 25% of Data Real-Time By Company Size 1 to 100 employees 29% 101 to 1,000 employees 32% 1,001 to 10,000 employees 18% 10,000+ employees 45% Figure 18: Real-Time Big Data Sites With More Than 25% of Data Real-Time By Industry Financial services/insurance 33% IT services/products 23% Government/education 22% Retail/services 18% Manufacturing 14%

16 16 Figure 19: Changes in Enterprise Unstructured Data Volume Over Next Three Years (Defined as within one hour of creation) Increase by more than 100% 13% Increase 51% to 100% 17% Increase 26% to 50% 20% Increase 10% to 25% 26% Increase 1% to 10% 10% No change 4% Decrease 0% Don t know/unsure 10% Figure 20: Anticipated Changes in End-User Demand for Enterprise Real-Time Data Over Next Three Years (Defined as within one hour of creation) Increase by more than 100% 16% Increase 51% to 100% 17% Increase 26% to 50% 23% Increase 10% to 25% 20% Increase 1% to 10% 10% No change 6% Decrease 0% Don t know/unsure 8%

17 17 Figure 21: Users Demanding Access to More Data? Don t know/unsure 14% Yes 75% No 12% (Total does not equal 100% due to rounding.) Figure 22: Importance of Big Data to Business Not important today 14% Extremely important 23% Very important 35% Somewhat important 27% (Total does not equal 100% due to rounding.) Figure 23: Big Data Programs Impact on Big Data Importance to Business (Respondents indicating big data is extremely or very important) Formal big data program 87% No big data efforts 32%

18 18 Figure 24: Importance of Big Data to Business By Company Size (Respondents indicating big data is extremely or very important) 1 to 100 employees 50% 101 to 1,000 employees 65% 1,001 to 10,000 employees 43% 10,000+ employees 35% Figure 25: Importance of Big Data to Business By Industry (Respondents indicating big data is extremely or very important) Financial services/insurance 65% IT services/products 65% Retail/services 57% Government/education 54% Manufacturing 43%

19 19 BIG DATA OPPORTUNITIES AND CHALLENGES Most executives intuitively understand the advantages big data can bring to their operations, especially with predictive analytics and customer analytics. However, organizational and skills issues may slow down their progress. For many organizations, big data offers a range of opportunities to improve and increase business. The most popular application cited is the ability to perform predictive analytics, cited by 62% of respondents. Another 53% say big data offers ways to better understand their customers. More than twofifths are linking their big data efforts directly to business growth, employing big data tools and analysis to grow business revenues streams and compete more effectively in their markets. (See Figure 26.) Big data is playing a role in many industries. We are looking at big data to help us provide better, safer patient care by learning from past episodes of care and outcomes, says the manager of a healthcare facility. As more EMR/EHR systems come online, we believe the medical community will demand access to big data. While most respondents believe their big data efforts will deliver significant benefits, there are barriers for many as well. A large segment, 43%, say the lack of budget holds them back, while 35% are also concerned with a lack of skills. About a third cite both data governance issues as well as lack of urgency from business management, two highly inter-related issues. The ability to help business users connect with the data that is available to them is an important emerging role for data professionals. (See Figure 27.) Top management support is essential, states a respondent whose department has the backing of the CEO and COO, to move forward in business intelligence and big data initiatives, in order to streamline and more accurately monitor and maintain our data flow. This helps ensure that the agency s federal, state, and local regulatory requirements are met as well protecting our grants, and funding streams, the respondent explains. Interestingly, when the organization addresses big data opportunities through formal programs, they are far more likely to be concerned with issues such as data governance, skills, and technology requirements. The presence of such programs points to management's sense that big data is a priority. (See Figure 28.) While many respondents believe that their current technology is capable of helping them manage and capitalize on big data, current solutions in place also are often in need of an overhaul. More than a third of respondents, 36%, say faster querying of the data is a challenge they are attempting to get their arms around. Another 43% are concerned about providing faster access to the larger datasets that may be proliferating around them. (See Figure 29.) What strategies are respondents organizations pursuing to manage and capitalize on big data on an enterprise level? Onethird indicate they are working to extend and optimize their current infrastructure, while about one-fourth are adopting new frameworks such as Hadoop and in-memory databases. (See Figure 30.) When it comes to leadership in big data initiatives, IT departments are taking the lead among a large segment of respondents, as cited by 42%. One-third say senior business executives such as the CEO or CFO are taking charge of the effort. (See Figure 31.) What data management technologies do respondents organizations currently use or plan to adopt within the next 24 months to manage the big data challenge? Two-thirds are employing relational databases to meet the challenge. Close to half, 47%, are employing business intelligence tools. (See Figure 32.) As discussed in the previous section, skills availability is a critical challenge for organizations seeking to better manage or capitalize on their big data assets. There are three ways to address this challenge either automate more, train more, or recruit more. For many respondents, automation and technology are considered the best paths. Close to four-fifths, 59%, say they plan to equip staff with easy-to-use analytic tools. Close to half, 47%, intend to train business users with self-service business intelligence tools. (See Figure 33.)

20 20 Figure 26: Big Data Opportunities Perform predictive analytics 62% Better understand our customers 53% Grow business revenue streams 42% Compete more effectively 41% Lower business operational costs 37% Expose new economic value & opportunities 35% Faster time to market new offerings 27% Other 2% Figure 27: Big Data Business Barriers Lack of budget 43% Lack of skills 35% Data governance issues 31% Not a business management priority 31% Unsure of technology requirements 29% Unsure of the value 32% Other 3%

21 21 Figure 28: Big Data Programs Impact on Big Data Business Barriers Formal big data program No big data efforts Lack of budget 41% 44% Lack of skills 42% 27% Data governance issues 45% 20% Not a business management priority 17% 47% Unsure of technology requirements 33% 22% Unsure of the value 26% 39% Figure 29: Current Solutions Technology Barriers Faster querying of the data 51% Faster access to large data volumes/sets 43% Faster acquisition and storage of data 36% Capacity to store more data 35% Faster access to non-traditional data types 33% Capacity to store non-traditional data types 33% Cost of storage for larger or more varied 29% datasets

22 22 Figure 30: Enterprise Big Data Strategies Extend and optimize current infrastructure 33% We don t have an enterprise strategy for 26% big data at this time Adopt new frameworks such as Hadoop 24% and in-memory databases Establish a head of big data/analytics 22% strategy Implement training and education in 22% big data and analytics Purchase applications that address specific 15% big data opportunities Establish a cross-enterprise committee/ 13% center of excellence Don't know/unsure 22% Other 1%

23 23 Figure 31: Big Data Corporate Leaders IT department 42% Senior business management (CEO, CFO) 33% Line-of-business 22% No one is in charge at this time 21% Cross-enterprise committee/center 11% of excellence Don't know/unsure 13% Figure 32: Big Data Technologies Adopted or To Be Adopted in Next 24 Months Relational databases (e.g., Oracle, DB2, etc.) 64% Business intelligence tools 47% Hadoop/MapReduce 28% In-memory platforms 21% Other Open Source technologies 20% NoSQL databases 20% R 12% Don t know/unsure 17% Other 4%

24 24 Figure 33: How Skills Base Will Be Expanded Over Next 12 Months Equip staff with easy-to-use analytic tools 59% Train business users with self-service 47% BI tools Train BI analysts in data mining 43% Hire data scientists 19%

25 25 DEMOGRAPHICS Figure 34: Primary Job Titles Database administrator (DBA) 17% Data architect 13% IT consultant 11% Programmer/developer 9% Director/manager of IS/IT or 9% computer-related function Analyst/systems analyst 8% Project manager 7% Chief information officer/chief technology 6% officer/vp of IT Systems administrator 5% Executive management level for the business 5% Manager of a business unit 3% Other 7%

26 26 Figure 35: Number of Employees (Include all locations, branches, and subsidiaries) 1 to 100 employees 27% 101 to 500 employees 14% 501 to 1,000 employees 8% 1,001 to 5,000 employees 12% 5,001 to 10,000 employees 10% 10,000+ employees 26% Don t know/unsure 2%

27 27 Figure 36: Primary Industries IT services/consulting/system integration 17% Government (all levels) 12% Financial services 10% Software/application development 10% Business or consumer services 6% Education (all levels) 6% Manufacturing 6% Retail/distribution 6% Healthcare/medical 6% Utility/telecommunications/transportation 5% Insurance 3% High-tech manufacturing 2% Other 10%