THE STATE OF THE DATA WAREHOUSE

Transcription

1 March 2015 Sponsored by

2 Introduction As the volume and types of business data have increased at a phenomenal pace, and the cost to store that data has plummeted, businesses have looked to data analytics to gain new insights into their customers and operations. The buzz about Big Data, including the use of technologies like Hadoop, has dominated the news. But what does that mean to the future of data warehousing? Are companies abandoning their data warehouses in favor of new technologies? And what is the impact of new paradigms created by cloud computing? The following report, sponsored by Snowflake Computing, is based on a survey of 319 individuals with responsibility for data initiatives. The goal of the survey was to understand experiences, challenges and trends in data warehousing and data analytics. Key Findings Data warehouses still critically important, but challenges remain 99% say their data warehouse is important to their business operations 70% are increasing their investment in data warehousing 97% face challenges with their current data warehousing solutions Scalability and cost are top challenges Plenty of interest in Big Data, but so far investment is limited 91% have considered a Big Data investment Only 11% have a pilot in place, and only have fully deployed their Big Data initiative 91% have concerns about Hadoop 96% say Hadoop will not replace their existing data warehouse Only 12% have easy access to Hadoop expertise; in contrast, 93% have easy access to SQL expertise Cloud-based data warehousing desirable, especially for data in the cloud 93% see value in the potential benefits of cloud data warehousing Scalability and reduced overhead most valuable 32% have a cloud-based data warehouse today; 16% use only a cloud-based data warehouse 49% of those with data already in the cloud have a cloud data warehouse 79% of those with data in the cloud bring it into the data warehouse, although only 9% bring all data in Sponsored by

3 Detailed Findings Data warehouses still critically important, but challenges remain When asked about the importance of their data warehouse to business operations, almost all of those responsible for data initiatives (99%) indicated that it was important, including almost three-quarters (72%) who characterized it as very important. Only a very few (1%) described their data warehouse as not important. How important is your data warehouse to business opera3ons? 72% 27% 1% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Very important Important Not important Even with the significant amount of discussion of alternative Big Data platforms, data warehousing investment continues to grow. A large majority of companies (70%) are increasing their data warehousing investment, while only a very small number of companies () are reducing their investment in data warehousing. How is the level of investment in your company s data warehouse changing? Not changing 26% Investment is decreasing Investment is increasing 70% Page 3

4 That said, data professionals do see room for improvement in their existing data warehouse. The vast majority of data professionals surveyed (97%) face challenges with their current solutions for data warehousing. Top challenges cited included scalability (43%), cost to maintain (38%), cost to purchase (37%), and the difficult learning curve for business users (37%). What challenges do you face with your current data warehousing solu4ons? Concerned about ability to scale to handle projected data and user growth 43% Very expensive to maintain and operate Very expensive upfront purchase costs Challenging for business users to learn 38% 37% 37% Users complain it is too slow Difficult to find experjse to manage Takes too long to deploy new projects 32% 30% 33% Not able to handle all of the different types of data we have 19% We have no challenges with data warehousing 3% 0% 10% 1 20% 2 30% 3 40% 4 50% Plenty of interest in Big Data, but so far little investment To understand the impact of Big Data initiatives, we first examined the state of those initiatives. We discovered that while most data stakeholders are interested in Big Data only 9% have not considered a Big Data initiative most are still investigating or planning to investigate. Just 16% have made an investment in Big Data, including who are fully deployed and a further 11% that are in a pilot. Most companies, 70%, are interested in Big Data but haven t yet made any investment. That does include 29% who are currently investigating their options, but the largest number is those who are intrigued, but haven t yet done any work to find out more (41%). What investment is your company making in Big Data? 4 40% 3 30% 2 20% 1 10% 0% 11% Fully deployed Have a pilot in place Currently inves<ga<ng We are interested, but haven t inves<gated yet 29% 41% Have inves<gated and decided not to pursue 9% Not being considered Page 4

5 Hadoop not a practical solution for most enterprises Big Data is rarely discussed without talking about Hadoop, the most common foundational technology for these initiatives. It appears that concerns about Hadoop may be an important factor in the slow pace of investment in Big Data. The vast majority of data stakeholders (91%) indicated that they have concerns about using Hadoop in their organization, with the biggest concern by far being lack of expertise (71%). Do you have any concerns about using Hadoop in your organiza6on? Lack of experise with Hadoop Don t have enough data to bother No compelling business need Hadoop is too complex Other I have no concerns 9% 18% 2 23% 71% 0% 10% 20% 30% 40% 50% 60% 70% 80% It is not surprising that lack of expertise is a challenge for any new technology like Hadoop. In contrast, SQL expertise is common. The vast majority of data stakeholders surveyed can easily access SQL expertise (93%), while only a few (12%) have easy access to Hadoop expertise. Do you have easy access to SQL exper3se? No 7% Do you have easy access to Hadoop exper2se? Yes 12% Yes 93% No 88% Page 5

6 This difference in access to Hadoop expertise is even more pronounced for smaller companies, whereas SQL expertise is accessible to almost everyone. Among the companies in our study with fewer than 1,000 employees, only have access to Hadoop skills, while most (92%) have easy access to SQL expertise. Access to SQL or Hadoop exper1se by company size 100% 9 93% 92% 80% 60% 40% 20% 22% 13% Have easy access to SQL exper@se Have easy access to Hadoop exper@se 0% More than 5,000 1,000-5, ,000 When asked directly about opinions on Hadoop as a possible replacement for the data warehouse, only a very small number () reported that they thought Hadoop could eventually replace their existing data warehouses. Most data stakeholders (6) believed Hadoop would not impact their data warehouse, and any investment would be fully complementary. About a third (32%) did think that Hadoop could replace some but not all of what was done today using a data warehouse. Do you consider Hadoop as a possible replacement for exis6ng data warehouses? Yes, Hadoop will completely replace our need for data warehouses Hadoop will replace some of what we do with our data warehouse, but not all 32% No, Hadoop will be complementary to our data warehouses 6 Page 6

7 Cloud-based data warehouse seen as valuable Data stakeholders see potential value in having a data warehouse that is in the cloud. The majority (93%) see benefits in cloud data warehousing with the right capabilities. A cloud-based data warehouse with the ability to scale on demand (66%) and deliver lower administrative overhead (59%) would deliver the most benefit. Which of these poten-al benefits of a cloud- based data warehouse would be valuable to your organiza-on? Ability to scale on demand 66% Less administra=ve overhead 59% Reduced cost 53% Ability to easily share access to data Simplify access for business users 47% 47% Op=mized for structured and semi- structured data 27% None of these would be valuable 7% 0% 10% 20% 30% 40% 50% 60% 70% Because of this potential value, cloud data warehousing is no longer something that only early adopters are using. Almost a third of companies (32%) are already using cloud-based data warehousing, including 1 who have exclusively adopted cloud and do not support an on-premises data warehouse. Which of the following are currently in use in your organiza4on? On- premises data warehouse 86% Cloud- based data warehouse 32% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Page 7

8 What concerns have kept the rest from using a cloud data warehouse? When we asked those data stakeholders who did not have a cloud data warehouse, the top reason was that they had concerns about security (6). It is worth noting that some of these who do not have a cloud-based data warehouse today do plan to adopt the technology in the near future. This includes 16% who indicated that they want a cloud-based data warehouse but haven t gotten around to it yet, as well as multiple participants who used the other option to indicate they were actively in process with a cloud-based data warehouse initiative. who wrote in as an other option saying that they were actively in process with a cloud-based data warehouse initiative. Why doesn t your company use a cloud- based data warehouse? Concerned about security of data in the cloud 6 Technically complicated to move our data 26% Not sabsfied with current cloud- based data warehouse offerings We d like to but haven t go?en around to it 16% 19% Have never thought about it Other 11% 10% 0% 10% 20% 30% 40% 50% 60% 70% n = do not have a cloud data warehouse Cloud data warehouse a natural fit for cloud data Cloud services have traditionally caused challenges for data and business intelligence professionals because cloud data needed to be brought back into the datacenter to be available for general use. However, this has clearly changed. Almost two-thirds of participants in this survey (62%) have data in the cloud either in SaaS applications like Salesforce.com or in public infrastructure clouds like Amazon AWS. These participants were asked how that data fit into their data warehouse initiative. Most companies with data in the cloud are bringing at least some of their cloud data into a data warehouse (79%). Only a small portion of those (9%) are bringing all cloud data into the data warehouse. Most (5) have less than half of their cloud data in their data warehouse. 3 30% 2 20% 1 10% 0% Percentage of cloud data brought into a data warehouse 33% 21% 22% 1 9% No data Less than % More than half All data n = have data in the cloud Page 8

9 Interestingly, there is a clear correlation between companies who have data in the cloud, and those who have a cloudbased data warehouse. When we consider only companies who have data in the cloud, almost half have a cloud data warehouse (49%), while very few companies who don t use data in SaaS applications or public infrastructure clouds have adopted cloud for data warehousing () Have cloud data warehouse 60% 49% 40% 20% 0% No cloud Cloud data Survey Methodology and Participant Demographics In February 2015, technology and analytics stakeholders with responsibility for corporate data initiatives were invited to participate in an online survey on the topic of data analytics and warehousing. Participants were asked a series of questions about their role in data initiatives as well as specific questions on experiences, challenges, and trends with data warehousing, Big Data, Hadoop, and cloud. A total of 319 individuals completed the survey. All had professional responsibility for data initiatives. Participants represented a wide range of geographies, company sizes, role, and vertical industries. All participants worked at companies that used data warehousing. Role in data ini+a+ves Company Size IT manager responsible for delivering data ini;a;ves IT staff responsible for implemen;ng and opera;ng data infrastructure IT execu;ve with data ini;a;ves in my porfolio 27% 46% 51% More than 5,000 21% 100 1,000 42% BI or Analy;cs Technology Owner 11% BI Business Owner 8% 0% 10% 20% 30% 40% 50% 60% 1,000 5,000 37% Ver$cal Industry Have business data in the cloud Technology Financial Services Manufacturing Healthcare Educa6on Government Non- Profit Services Energy and U6li6es Retail Food and Beverage Hospitality and Entertainment Telecommunica6ons Transporta6on Media and Adver6sing Pharmaceu6cal Other 1% 2% 2% 2% 2% 1% 1% 0% 2% 6% 8% 10% 12% 1 16% 8% 9% 12% 13% 1 1 No 38% Yes 62% Page 9

10 About Dimensional Research Dimensional Research provides practical market research to help technology companies make their customers more successful. Our researchers are experts in the people, processes, and technology of corporate IT and understand how corporate IT organizations operate. We partner with our clients to deliver actionable information that reduces risks, increases customer satisfaction, and grows the business. For more information visit. About Snowflake Computing Snowflake Computing, the cloud data warehousing company, was founded in 2012 by a team of experts who hold over 120 patents in the fields of database architecture, data warehouses, query optimization and parallelization. Snowflake was created with the vision to reinvent the data warehouse, bringing together all users, all data and all workloads in a single repository. The company is backed by leading investors including Redpoint Ventures, Sutter Hill Ventures and Wing Ventures. Snowflake is headquartered in Silicon Valley and can be found online at Page 10