Analytics Industry Trends Survey. Research conducted and written by:

Transcription

1 Analytics 2014 Industry Trends Survey Research conducted and written by: Lavastorm Analytics, the agile data management and analytics company trusted by enterprises seeking an analytic advantage. June 2014

2 Table of Contents Executive Summary Key Findings About Lavastorm Appendix I: Methodology Appendix II: Description of Survey Respondents 2

3 Executive Summary Analytics 2014 is Lavastorm Analytics second in a series of annual surveys that explores how analytic technologies are being applied to solve business challenges. This year s survey has been updated to reflect a growing focus on the analysis of big data. This survey report is based on the survey responses Lavastorm Analytics received from 495 active participants in the analytics community, including business analysts, technologists, data analytics professionals, managers, and C-level executives. Survey participants came from a broad variety of industries. The top findings were: Analytics is Growing Despite Inefficient Analytic Supply Chains The majority of survey respondents (64.4%) indicated that their company is increasing investment in Despite this, organizations are still finding it difficult to progress an analytic insight to an implemented business improvement. This is likely hindered by the complexity of the analytic supply chain characterized by the group of people that work on either data or analysis starting with data acquisition and progressing to insight and resulting business action. The multiple handoffs and specialists involved in the analytic supply chain slows progress. In addition, many businesses use multiple analytic tools, demonstrating that few tools are perfectly suited for complex and multifaceted jobs. Both of these trends are driving interest in self-service or data discovery tools, which are simpler to use and bypass a good deal of the traditional analytic supply chain when business users have new questions to answer. The World is Dividing into Big Data Haves and Have Nots Big data is maturing, but the majority of companies are still sitting on the sidelines watching, planning and evaluating. Only 12.6% respondents in our survey report that their company has completed several big data projects that are now in production. Those respondents with big data projects in production appear to be trying to widen the analytic gap between themselves and their competitors. They are more likely to be increasing their investment significantly (38.7%) compared to companies that are not yet working with big data (21.8%). Interestingly, the number one concern for people in organizations that are just experimenting or planning for big data is the shortage of analytic professionals, which could indicate that organizations yet to take the big data plunge may be held back because of the lack of big data skills available to them. 3

4 Big Data Is Currently the Domain of the Data Scientist, But Needs More Business Involvement for Long-term Success Our survey indicated that analysts (business or data analysts) are in the dark when it came to big data technologies and the work that their company was doing to leverage those sources. Data Scientists, who are generally in R&D or IT groups, are much more involved. The best business results will only be obtained, however, when the business is heavily involved in the planning and analysis. Data Quality Concerns On the Rise Many more people are involved with or concerned with data quality, likely due to increased use of new data sources, especially big data sources. In this year s survey 48 percent of respondents are working on data quality, a significant increase over the 27 percent that reported working on data quality last year. 4

5 Key Findings Analytics is Growing Despite Inefficient Analytic Supply Chains There is a general consensus that investment in analytics is increasing. This is consistent with other research that indicates that analytics and business intelligence will remain a top priority of CIOs in 2014 and several years to come. As you can see in Figure 1, 64.4% of respondents indicate their company is investing more in 2014, with 20.5% indicating their company is increasing their investment significantly. Figure 1: How would you describe your company s investments in analytics (tools, people, data sources, etc.) in 2014? Not Sure 8.5% Decreasing Investment Significantly 1.0% Decreasing Investment Moderately 2.6% Increasing Investment Significantly 20.5% Staying the Same 23.4% Increasing Investment Moderately 43.9% Despite the increased investment that is expected, businesses are struggling to turn analytic insights into business improvements. As Figure 2 shows, 20.5% of respondents indicated that turning insights to action is their biggest challenge, while 13.8% indicate that building trust in insights is the major challenge. So investment is increasing even in the face of that difficulty. These challenges show that analytics is a complex operation having as much to do with human interaction and communication capabilities as technology. 5

6 Figure 2: What is your biggest challenge in analytics today? Publishing Results 1.8% Requirements Gathering 5.1% Other 4.5% Turning Insights to Action 20.5% Access to the Data 11.4% Resources 11.4% Gleaning Insights from the Data 16.3% Building Trust in Insights 13.8% Manipulating / Integrating Data 15.2% Difficulty progressing insights to business improvements is likely hindered by the complexity of the analytic supply chain, or the group of people that work on either data or analysis all the way from data acquisition through insight to business action. Analytic processes typically require multiple people with specialized roles (data scientist, data analyst, business analyst, etc.) to collaborate if they are to complete an analytic task. For example, people in our survey who identified their role as data scientist more often resided in a research department (27.4%) than in any other department (Figure 3). They need to work with business leaders, therefore, to get their discoveries out of their department so that it can be acted upon. That s one reason why 25.8% of data scientists said building trust in insights and another 22.6% said turning insights to action were their major challenges (see Figure 4). Likewise, more analysts (those people in our survey who identified their role as a business analyst or data analyst ) reside in IT departments (16%) and again need to work with business leaders to enable decisions and action. (see Figure 5.) 6

7 Figure 3: For data scientists only, what department do you reside in? Other 17.7% Research 27.4% Finance 4.8% Executive/Sr. Management 4.8% Operations 6.5% Services 8.1% Marketing 9.7% IT 11.3% Engineering 9.7% Figure 4: For data scientists only, what is your biggest challenge in analytics today? Requirements Gathering 3.2% Resources 9.7% Building Trust in Insights 25.8% Manipulating /Integrating Data 11.3% Gleaning Insights from the Data 12.9% Access to the Data 14.5% Turning Insights to Action 22.6% Figure 5: For analysts only, What department do you reside in? Other (please specify) 18.9% IT 16.0% Engineering 2.8% Executive/Sr. Management 2.8% Sales 2.8% Marketing 15.1% Finance 5.7% Operations 10.4% Research 13.2% Services 12.3% 7

8 Contributing to analytic inefficiencies is the respondent reliance on multiple analytic tools, sometimes within the same department. As Figure 6 shows, people use a wide array of tools to analyze data. According to the survey, respondents use more than 4 analytics tools each. As Figure 7 shows, the tools most often used by analysts (business and data analysts) are general purpose tools, including Microsoft Excel, SQL and Microsoft Access. This is driven by several factors, including: Respondents have complex, multi-faceted roles to play and, therefore, that they must use specialized tools for different aspects of their job; Respondents don t have the perfect tool for the job and must use several tools as a compromise solution. For example, analysts (business and data analysts) indicated that one of their major challenges is manipulating/integrating data (Figure 8). This is likely because they are dependent on general purpose analytic tools (Microsoft Excel, SQL, and Microsoft Access) which are not designed for the complex data environments we have today. 8

9 Figure 6: What analytic tools are you currently using? Check all that apply. Microsoft Excel SQL R Microsoft Access Tableau SAS SPSS Python SAP BusinessObjects Cognos Microsoft BI Toad QlikView Matlab Lavastorm Analytics Teradata Oracle OBIEE Rapid Miner Hive TIBCO Spotfire MicroStrategy Pentaho StatSoft/STATISTICA SAP NetWeaver BW ACL Alteryx Information Builders Actuate (Birt) Logix Panorama Necto Other (please specify) 17.5% 17.1% 14.8% 12.4% 11.6% 9.3% 9.3% 9.3% 8.7% 8.3% 7.9% 6.5% 5.9% 5.1% 5.1% 3.7% 3.3% 3.0% 3.0% 2.6% 1.8% 0.6% 0.2% 15.2% 24.8% 24.2% 32.7% 36.6% 45.5% 72.4%

10 Figure 7: For analysts (business and data analysts only), what analytic tools are you currently using? Check all that apply. Microsoft Excel SQL Microsoft Access R Tableau SAS SPSS Python Cognos SAP BusinessObjects Microsoft BI Toad QlikView Other (please specify) Lavastorm Analytics MicroStrategy Rapid Miner Teradata TIBCO Spotfire Oracle OBIEE ACL Hive Matlab StatSoft/STATISTICA Alteryx Information Builders SAP NetWeaver BW Pentaho Actuate (Birt) Logix Panorama Necto 26.4% 20.8% 19.8% 16.0% 13.2% 13.2% 13.2% 12.3% 12.3% 10.4% 6.6% 6.6% 6.6% 6.6% 5.7% 4.7% 4.7% 2.8% 2.8% 1.9% 1.9% 1.9% 0.9% 33.0% 39.6% 42.5% 53.8% 78.3%

11 Figure 8: For analysts (business and data analysts) only, what is your biggest challenge in analytics today? Other (Please Specify) 5.7% Publishing Results 0.9% Requirements Gathering 5.7% Resources 7.5% Turning Insights to Action 22.6% Building Trust in Insights 10.4% Access to the Data 11.3% Manipulating/Integrating Data 19.8% Gleaning Insights From the Data 16.0% The organizational complexity is a major reason why organizations have shifted and continue to shift to a self-service approach and why data discovery requirements are expected to drive the majority of the new license spend in BI in years to come. The self-service emergence started with business groups that wanted to take matters into their own hands and cut through a great deal of the organizational issues, such as IT s minimal bandwidth to respond to BI requests, that were preventing them from answering key business questions. We expect the self-service trend to continue with business users taking on more complex analytic challenges themselves, such as integrating more diverse data sources, or taking them on in a collaborative effort with IT. Figure 9 shows that 30 percent of respondents in our survey were investing in self-service analytic tools for business users. Overcoming these organizational challenges should improve analytic results and help companies put analytics into action. Companies that take a top-down data or analytic view of their operations and design a more streamlined analytic process should have a more responsive organization and better insights. For example, tools that are designed for specific roles could help simplify the tool chest for analysts and tools that enable collaboration could bridge the organizational gaps that slow down action. 11

12 Figure 9: For people whose companies are investing in analytics in 2014, what area is your company investing in 2014? Check all that apply. Predictive Analytics 59.0% Dashboards Big Data 43.8% 43.5% Reporting Data Management (Including Integration, Quality, ETL, and Enrichment) Data Exploration and Discovery Interactive Visualization Tools Self-Service Analytic Tools for Business Users Performance Analytics Text Analytics Web Analytics Databases: Relational Process Analytics Databases: Non-relational (Including Columnar) Location Analytics Streaming Analytics 37.9% 35.3% 31.2% 30.3% % 25.9% 23.0% 20.5% 19.2% 15.1% 12.9% 12.0% Olap Search Interfaces None Other (Please Specify) 7.3% 5.4% 2.8% 2.5% With Big Data s Maturing, the World is Dividing into Big Data Haves and Have Nots By all accounts, big data is maturing, but the majority of companies are still sitting on the side lines watching, planning and evaluating. Our survey indicates that maturity is still low. As you can see in Figure 10, only 12.6% report that their company has completed several big data projects that are now in production. Most (54.1%) are in the experimenting or learning/planning phases, while 20.9% are not working with big data at all. 12

13 Figure 10: Describe the maturity of your company s current efforts to use big data tools such as Hadoop or to analyze big data sources, such as unstructured data sources and non- relational databases, such as MongoDB. In Production - We have completed several projects using big data sources and tools and those projects are currently in production and are part of an ongoing business process 12.6% Don t Know 12.4% Learning or Planning We have not started any projects to work with big data sources or tools to date, but are tracking the evolution and application of the technologies 31.1% Not Participating - We don t have a current interest in using big data sources or tools. 20.9% Experimenting - We are experimenting with big data sources and tools, but those projects are not in production and are not part of an ongoing business process 23.0% More importantly, those that have big data projects in production appear to be trying to widen the analytic gap between themselves and their competitors. They are more likely to be increasing their investment significantly (38.7%) than companies that are not yet working with big data (21.8%). See Figure 11. Not surprisingly, companies that have big data projects in production are investing most often in additional big data initiatives (66.1%) compared with companies that are less mature when it comes to big data (45.5% investing in big data in 2014). See Figure

14 Figure 11: For companies with big data projects In Production vs companies that are not, how would you describe your company s investments in analytics (tools, people, data sources, etc.) in 2014? % % 38.7% Those in Production Response Percent % 19.5% Those in Planning or Experimenting Response Percent 14.5% 10 0 Increasing Investment Significantly Increasing Investment Moderately Staying the Same 1.6% 3.0% 1.6% 1.5% Decreasing Investment Moderately Decreasing Investment Significantly 4.8% 5.3% Not Sure Figure 12: For companies with Big Data projects in production vs companies that are not, what area is your company investing in 2014? Check all that apply. 14 Big Data Predictive Analytics Dashboards Reporting Interactive Visualization Tools Text Analytics Data Management (including Integration, Quality, ETL, and Enrichment) Web Analytics Performance Analytics Self-Service Analytic Tools for Business Users Data exploration and Discovery Databases: Non-Relational (including Columnar) Databases: Relational Process Analytics Streaming Analytics Location Analytics Search Interfaces OLAP Other (please specify) None Those in Production - Response Percent Those in Planning or Experimenting - Response Percent

15 In addition, a full 41% of the respondents that say that they are in the Planning or Experimenting phase with big data have not yet acquired any specific big data tools for their initiatives (Figure 13). Interestingly, the people in organizations that are just experimenting or planning for big data believe the shortage of analytic professionals will have the biggest impact on the analytics industry in 2014 (Figure 14). Meanwhile those respondents from companies with big data deployments did not expect as much impact from that trend (Figure 15). Probably this indicates the confidence that those organizations that are ahead of the curve are also positioned to stay there from a skills perspective. While organizations yet to take the big data plunge may be held back because of the lack of big data skills available to them. Figure 13: For respondents whose company is Planning or Experimenting with big data, what big data tools do you currently use? Check all that apply. 45.0% % 35.0% 34.6% % % 1 5.0% 15.0% 12.0% 11.3% 3.4% 6.8% None Hadoop Don t Know Hive MongoDB HortonWorks Other (please specify) 15

16 Figure 14: For respondents whose company is Planning or Experimenting with big data, in 2014, what one development or trend will occur or progress and have the biggest impact on the analytic community? Check all that apply. Increased use of Mobile Analytic Solutions 4.5% Shortage of Appropriate Analytic Tools and Training 6.0% Increased Data Variety (including the use of Unstructured Data or New Data Sources) 7.5% Greater Adoption of Self-Service and Data Discovery Tools 8.6% Increased Data Volumes (including Volume from Machine-Generated Data) 8.6% Increased Use of Cloud-Based Analytic Solutions 9.0% Other 2.3% Shortage of Analytic Professionals 16.2% Shifting Responsibility for Analytics from IT to the Business 14.7% Maturation of Big Data Strategies and Products 12.8% Increased Focus on the Data Scientist Role 9.8% Figure 15: For respondents whose company is in Production with big data, in 2014, what one development or trend will occur or progress and have the biggest impact on the analytic community? Check all that apply. Shortage of Appropriate Analytic Tools and Training 1.6% Other 4.8% Shortage of Analytic Professionals 6.5% Increased Data Variety (including the use of Unstructured Data or New Data Sources) 6.5% Increased Data Volumes (including Volume from Machine-Generated Data) 8.6% Greater Adoption of Self-Service and Data Discovery Tools 9.7% Increased use of Cloud-Based Analytic Solutions 11.3% Maturation of Big Data Strategies and Products 16.1% Shifting Responsibility for Analytics from IT to the Business 16.1% Increased use of Mobile Analytic Solutions 12.9% Increased Focus on the Data Scientist Role 11.3% 16

17 Big Data is Currently the Domain of the Data Scientist, but Needs to Involve the Business More for Long-term Success Our survey indicated that analysts (business or data analysts) are in the dark when it came to big data technologies and the work that their company was doing to leverage those sources nearly 73% of analysts did not know what big data tools were being used (Figure 15), or were not currently using big data tools, compared to almost 39 percent of data scientists (Figure 17). Figure 16: For analysts (business and data analysts) only, what big data tools do you currently use? Check all that apply % 47.2% % % 25.5% % 15.0% 1 5.0% None Don t Know Hadoop 6.6% Hive 4.7% MongoDB 0.9% HortonWorks 6.6% Other (please specify) Figure 17: For data scientists only, what big data tools do you currently use? Check all that apply % 46.8% % 33.9% % 24.2% % % 12.9% 5.0% Hadoop None Hive MongoDB 4.8% Don t Know 1.6% HortonWorks Other (please specify) 17

18 The troubling aspect of this is that big data projects may be treated as a science project and not a business-impacting capability. The best business results will be obtained when the business is heavily involved in the planning and analysis. Gartner and other industry analyst firms have indicated that the vast majority (80% by some estimates) of BI/analytic projects fail and the lack of business involvement is a primary culprit. For best results, organizations should learn from history or risk repeating it. This is likely why most data scientists consider their challenges to be turning insights to action. It s likely that this reflects the maturity of organizations with big data and that data scientists are involved earlier in the process. Our survey shows that while both data scientists and analysts are primarily involved in analysis, they go about it quite differently. As shown in Figures 17 and 18, data scientists focus much more on building statistical models (88.7% to 46.1%) and on research and development (69.4% to 47.8%). Figure 18: For data scientists only, which of the following functions do you personally perform as a regular part of your job? Check all that apply. Data Analysis Statistical Modeling 91.9% 88.7% Research and Development 69.4% Import Data from a Data Warehouse (or from IT) Data Quality Assessment and Improvement Gather Requirements Combine Data from a Warehouse with other Data not in the Warehouse Filter or drill down into Dashboards/Interactive Reports Publish or otherwise provide Data to other Departments 59.7% 59.7% 56.5% % 40.3% Load Data into the Data Warehouse 25.8% Other (please specify) 8.1%

19 Figure 19: For analysts (business and data analysts) only, which of the following functions do you personally perform as a regular part of your job? Check all that apply. Data analysis Gather requirements Data quality assessment and improvement Publish or otherwise provide data to other departments Research and development Statistical modeling Import data from a data warehouse (or from IT) Filter or drill down into dashboards/interactive reports Combine data from a warehouse with other data not in the warehouse Load data into the data warehouse Other In addition, it likely reflects a requirement for greater technical skills to use big data tools. Our survey (see Figures 20 and 21) indicates that data scientists use statistical packages and programming languages, whereas analysts use mostly Microsoft Excel (78%), SQL (54%), and Microsoft Access (42%). More technical tools would lead organizations to call on data scientists for assistance. 19

20 Figure 20: For analysts (business and data analysts) only, what analytic tools are you currently using? Check all that apply. Microsoft Excel SQL Microsoft Access R Tableau SAS SPSS Python Cognos SAP BusinessObjects Microsoft BI Toad QlikView Lavastorm Analytics MicroStrategy Rapid Miner Teradata TIBCO Spotfire Oracle OBIEE ACL Hive Matlab StatSoft/STATISTICA Alteryx Information Builders SAP NetWeaver BW Pentaho Actuate (Birt) Logix Panorama Necto Other (please specify) 26.4% 20.8% 19.8% 16.0% 13.2% 13.2% 13.2% 12.3% 10.4% 6.6% 6.6% 6.6% 6.6% 5.7% 4.7% 4.7% 2.8% 2.8% 1.9% 1.9% 1.9% 0.9% 12.3% 33.0% 39.6% 42.5% 53.8% 78.3%

21 Figure 21: For data scientists only, what analytic tools are you currently using? Check all that apply. R 71.0% Microsoft Excel 64.5% SQL Python SAS Tableau 41.9% 40.3% 37.1% 33.9% SPSS Matlab 25.8% 25.8% Hive Microsoft Access Rapid Minder Toad TIBCO Spotfire StatSoft/STATISTICA Microsoft BI Teradata Oracle OBIEE QlikView Pentaho SAP BusinessObjects SAP NetWeaver BW MicroStrategy Information Builders Alteryx ACL Panorama Netco Logix Lavastorm Analytics Actuate (Birt) 17.7% 16.1% 12.9% 9.7% 6.5% 6.5% 6.5% 4.8% 4.8% 3.2% 3.2% 3.2% 1.6% 1.6% 1.6% 1.6% 1.6% Other 29.0%

22 Data Quality Concerns On the Rise The growing use of new data sources, especially big data sources, is increasing the variability and quality issues that analytic value chains are facing and forcing companies to put more resources towards data quality improvement efforts. As shown in Figure 22, 48% of respondents are working toward data quality, a significant increase over the 27% that reported working on data quality last year. Figure 22: Which of the following functions do you personally perform as a regular part of your job? Check all that apply. Data analysis Gather requirements Data quality assessment and improvement Publish or otherwise provide data to other departments Research and development Statistical modeling Import data from a data warehouse (or from IT) Filter or drill down into dashboards/interactive reports Combine data from a warehouse with other data not in the warehouse Load data into the data warehouse Other As companies are trying to integrate these disparate sources, it is forcing them to tackle the data quality problem head on. Issues such as mismatched data, varying formats and incomplete data are more critical given the greater variety and differences between the data sources and the fact that some of this data is being analyzed for the first time by many organizations. This is why 35 percent of survey respondents who said they were increasing analytic investments in 2014 said their company is also investing in data management (including integration, quality, ETL and enrichment). See Figure

23 Figure 23: For only those respondents whose company is increasing their investment in 2014, what area is your company investing in Check all that apply. Predictive analytics 59.0% Dashboards Big data 43.8% 43.5% Reporting Data Management (Including Integration, Quality, ETL, and Enrichment) Data Exploration and Discovery Interactive Visualization Tools Self-Service Analytic Tools for Business Users Performance Analytics Text Analytics Web Analytics Databases: Relational Process Analytics Databases: Non-Relational (Including Columnar) Location Analytics Streaming Analytics 37.9% 35.3% 31.2% 30.3% % 25.9% 23.0% 20.5% 19.2% 15.1% 12.9% 12.0% OLAP Search Interfaces None Other (Please Specify) 7.3% 5.4% 2.8% 2.5%

24 About Lavastorm Lavastorm is the agile data management and analytics company trusted by enterprises seeking an analytic advantage. The company s data discovery platform provides IT with control over data governance, while empowering business professionals and analysts with the fastest, most accurate way to discover and transform insights into business improvements. Lavastorm s solutions have identified business improvements worth billions of dollars for some of the largest corporations in the world. For more information, please visit: For more information on Lavastorm Analytics or to download a desktop edition of the Lavastorm Analytics Engine, our data analytics software for business analysts, please visit or respectively. Appendix I: Methodology In order to identify the trends in the market, we conducted our research within major analytic communities, including LinkedIn s Lavastorm Analytics Community Group, Data Science Central and KD Nuggets. These communities have a global reach of over 75,000 analytic professionals who play a variety of roles in analytics today and represent all of the major industries. This survey was conducted online using an electronic survey tool. A total of 495 business analysts, technologists, data analytics professionals, managers and C-level professionals were polled across a broad variety of industries including financial services, telecommunications, healthcare and software & internet. After the survey data was collected, the data was analyzed to identify differences between segments of the population. Every survey participant was encouraged to answer every question. 24

25 Appendix II: Description of Survey Respondents The following charts describe the backgrounds of those professionals who completed the survey by role, department and industry. How would you classify your role? 492 Responses Revenue Assurance Manager 2.0% Software Developer 2.0% Solution Architect 2.0% C-level/Executive 2.2% Marketing Analyst 3.9% BI Professional 5.3% Other 8.3% Data Analyst 14.6% Manager/Director 14.4% VP/Director 6.1% Business Analyst 6.9% Consultant 11.4% Data Scientist 12.6% What department do you reside in? 492 Responses Other 16.1% Billing 1.2% Sales 1.6% Engineering 4.7% Finance 7.3% IT 20.1% Marketing 13.0% Services 7.3% Research 12.6% Executive/Sr. Management 7.7% Operations 8.3% Other includes: Analytics (2.8%), Risk (1%) and Audit (.8%) 25

26 What is your company s primary industry? 492 Responses Utilities 1.4% Transportation 1.4% Entergy 2.0% Insurance 2.4% Retail 3.3% Government 3.5% Manufacturing 4.5% Other 16.1% Education 7.3% Finance 8.5% Healthcare 8.9% Telecom 15.9% Services 13.4% Computer Software 11.4% Other includes: Consulting (2.4%) and Marketing (1%). Which of the following functions do you personally perform as a regular part of your job? Check all that apply 492 Responses Data analysis Gather requirements Data quality assessment and improvement Publish or otherwise provide data to other departments Research and development Statistical modeling Import data from a data warehouse (or from IT) Filter or drill down into dashboards/interactive reports Combine data from a warehouse with other data not in the warehouse Load data into the data warehouse Other

27 Lavastorm Analytics, All Rights Reserved.