TAKING THE ENTERPRISE S BIG DATA TEMPERATURE

Transcription

1 TAKING THE ENTERPRISE S BIG DATA TEMPERATURE Big data promises much, but it s still fighting for support within the enterprise Big data is a huge concern for IT departments the world over, forecast to drive $120 billion in IT spending in 2013, according to Gartner. Technologies are advancing and allowing for quicker deployments and results across wider sets of data, but these advances are happening in a time of financial uncertainty and a widely discussed skills shortage in the data science arena. Big Data Republic and SAP teamed up to survey our engaged end-user community to see how prepared organizations are to make use of big data in their operations. We had more than 200 responses to the survey, the majority coming from companies with more than 1,000 employees. This whitepaper aims to take an overall view of how ready enterprises are to embrace big data, how far along their data journey they are at the moment, and how confident they are of next steps. To do so, we will occasionally compare different verticals and their current readiness for big data. By looking at these results, you should get a good idea of where your values and infrastructure stack up against your contemporaries. 1» Taking the Enterprise s Big Data Temperature

2 What does big data success look like? Before we define how prepared organisations are to leverage big data, we thought it best to determine their end needs. The overall uses of big data are as wide and varied as the data itself, so any discussion of impediments or progress needs to be hung against a backdrop of aims and goals. Most surprising from our research is that, despite media focus on real-time benefits from big data, it accounts for just over 1% of businesses reasons for investment in their systems. What Does Success Look Like? Instead, the major focus on leveraging data comes from a need to increase efficiency in the business unit. Throughout the last 18 months of increasing big data hype, this is one element discussed least, with a few cases of governmental needs for efficiency making headlines. These efficiencies are coupled with (and followed closely in our results by) cost savings which is where the hyped vision of big data benefits starts to come to the fore. The aim to reduce outlay and get more out of money spent, with a combined survey response of 59.13%, fits with the current austere financial climate % 27.78% 23.41% 31.35% 1.19% Increased customer retention Cost savings Innovation and insight Increased efficiency of the business unit Faster return of practical insight Then come the more specific and widely hyped big data benefits of producing innovation and insight (23.41%) and increased customer retention (16.27%). Given a safer and more aggressive financial climate, these areas may well start to grow. The low score for increased customer retention can be partially explained away by sectors like government, which are not directly targeting customer retention, taking part in the survey. The 1.19% indicating faster return of practical insight (the velocity in big data s 3Vs) not only indicates a market focused on cost reduction and efficiency increase, but it also acts as a pointer to big data s maturity. As a relatively fresh concern for IT departments, enterprise-level big data can be seen as relatively immature still taking steps to align the basics (cost, efficiency) before progressing onto the more advanced (increased retention, innovation, and insight) and, finally, the most demanding (real-time results). Vision of success by vertical This lack of interest in instant returns is crystallised when we focus on three of the biggest vertical subsections among survey takers: government, healthcare, and financial markets. Each of these three returned an emphatic 0% focus on fast return of practical insight. Delving further into their responses shows us different focusses in differing verticals. Vision of Success by Vertical Government Healthcare Financial Markets Increased customer retention 8.82% 16.67% 25.00% Cost savings Innovation and insight Increased efficiency of the business unit 29.41% 22.22% 43.75% 20.59% 27.78% 25.00% 41.18% 33.33% 6.25% The fact that a quarter of respondents from the financial markets have a focus on increased customer retention bears out our theory of vertical-specific targets decreasing the overall focus on this area. However, it is not the financial market s main aim, with that honour going to its clear focus on saving cost through data initiatives, with a minimal interest in increasing efficiency. The spread of focus that we can see across various industries reinforces the importance of big data in the future of the enterprise data has the power to revolutionise every area of business, and it is up to these businesses to decide what is most important to them. 2» Taking the Enterprise s Big Data Temperature

3 In contrast to the financial market s goals, government responders are much more keenly focused on increasing efficiency. Given today s economic climate, that s hardly a surprising statistic, with pressure from treasurers and voters to get efficiency down to a fine art. As a result we also see this focus tied in with a 29.41% focus on cost savings. Happily, it s not just about doing the same things more cheaply within government, as about a fifth of respondents also indicated a desire to produce insight and innovation through their data sets. The vertical which seems to want to leverage the widest set of data-driven benefits is healthcare, with a relatively even spread across cost savings (22.22%), innovation and insight (27.78%), and increased efficiency (33.33%). Again, this is most likely in response to market influences for these medically focussed organisations but it may also reveal more democracy among differing departments: It s not just those in charge of the bottom line setting the agenda. As the basics of cost savings, efficiency, and innovation are mastered, expect to see attentions turn to faster return of practical insight. This real-time ability in big data (velocity within the three V s) should essentially act as an amplifier to these other needs enabling the enterprise to respond faster and launch innovations sooner, cut costs more quickly, and reshape the business for efficiency as the needs arise. Confidence by vertical Having defined the overall (and a few industry-specific) needs of data initiatives, it s time to examine how confident these organisations are of achieving the potential benefits they have identified for themselves. On first view, there s an overwhelming confidence among organisations toward improving their data capabilities. A combination of 16.06% extremely confident and 54.62% confident marks a bullish approach to creating an environment to yield results from data. Confidence in Improvement Of the combined negative responses of 29.32%, almost half cited limitations of budget as a reason for their conservative views. This sits with tighter budgets during a period of financial depression. Ironically, it also sits uncomfortably with the quarter of businesses established as looking to drive cost savings putting them in the uncomfortable position of having to spend money to save money % 54.62% 26.51% 2.81% Extremely confident Confident It could go either way Not confident A third of the negative responders blame decision makers in their organizations for lacking project understanding. This underscores a wider staffing issue surrounding big data. The focus in this area has so far focussed on the foreseen shortage in the data scientist role. This sexiest job of the 21st century has accumulated a great deal of press interest, but it s not the only staff development needed. Lack of big data awareness in the senior roles of organizations will also need to be addressed, as an overall big data readiness needs to be established before major investment can be undertaken. Within those under pressure to deliver big data results (easily created from the C-level down when hype is as high as it currently is) lack of time to implement the teams and infrastructure can become an inhibiting factor. However, this isn t seen as a major issue at the moment, with only 17.39% indicating a lack of time to implement as a reason. Reasons for Doubts in Ability to Improve 49.28% 33.33% 17.39% Budget constraints Decision-makers' understanding/lack of understanding about the project Lack of time to properly implement new data management systems 3» Taking the Enterprise s Big Data Temperature

4 Impediments facing government and healthcare Impediments: Healthcare vs Government Healthcare Government The fact that we have mastered data management Budget constraints aren't letting us progress The data management systems we have don't offer the functionality we need We just don't have the talent to make use of our data 23.53% 6.06% 41.18% 72.73% 23.53% 12.12% 11.76% 9.09% Focussing on two of the noisiest public players taking advantage of big data, healthcare seems most confident, with around a quarter feeling they have mastered their data needs. Around the same number of healthcare respondents indicated a lack of confidence in their current data management systems. There is a 41.18% share of healthcare respondents indicating budget constraints are stifling progress by far the biggest impediment in the healthcare space. This figure seems relatively low, however, when compared to the 72.73% of government-based responders complaining about budget constraints. Concern about budgets reduces the portion of respondents feeling confident about their mastery of data management to a measly 6.06%. Meanwhile both industries are on about equal footing when it comes to being concerned about lack of talent, hovering around the 10% mark. While budgets are a huge concern, this last statistic indicates that the media hype surrounding a talent shortage is either overemphasized, or less weighty than other concerns enterprises face. Size as impediment Company Size as Impediment Size of Company Reasons for impediment The fact that we have mastered data management 19.05% 27.27% 15.00% 15.60% Budget constraints aren't letting us progress 40.48% 57.58% 60.00% 54.61% The data management systems we have don't offer the functionality we need 26.19% 12.12% 20.00% 23.40% We just don't have the talent to make use of our data 14.29% 3.03% 5.00% 6.38% It s clear that these businesses know what they want to be achieving, so let s take a look at how their makeup impacts their ability to improve. With all the talk of a data scientist talent gap, it s the smallest firms that see a lack of talent as a big blocking point. Interestingly, they have less of an issue with budget than larger organizations, attributable to their flexibility, plus the general open-source nature of big data solutions. These companies are lean from a tech implementation point of view, but they are aware of what they need to make themselves competitive, and therefore highlight the lack of clout they have to bring in the top talent. There will be key data team members who are happy to switch to smaller organisations to embrace the freedom and agility offered in such environments, but for the large part, money talks when seeking data scientists and advanced analysts. At the other end of the scale, the larger organizations put their emphasis on budget constraints, as cultural and IT big data changes become more cumbersome the bigger the organisation is. By the time an organisation reaches the 500+ size it is safe to assume it has done so on the back of serious investment in what would now be considered legacy IT, 4» Taking the Enterprise s Big Data Temperature

5 so any investment wouldn t just go into new solutions but also into converting or integrating old systems. Conversely, these bigger organizations see the lack of available talent as a very minor threat. Annual Revenue as Impediment Annual Revenue 0-$10M $11M-100M $101M-$1B >$1B Reasons for impediment The fact that we have mastered data management 18.33% 20.41% 19.23% 14.46% Budget constraints aren't letting us progress 53.33% 48.98% 59.62% 51.81% The data management systems we have don't offer the functionality we need 15.00% 26.53% 19.23% 26.51% We just don't have the talent to make use of our data 13.33% 4.08% 1.92% 7.23% When we expand this research to not focus on business size, but on annual revenues, we see a likely reason for this lack of concern. The 0-$10M businesses regard lack of talent as a relatively serious concern at 13.3%. From $11M upwards however, this concern is negated, possibly for one of two reasons. The first being that companies don t reach that level of revenue without first establishing a talented and dynamic data team, capable of handling this step into big data. The second is that, even though there may be somewhat of a talent shortage, with that level of revenue these companies can simply cherry pick the team they want from competitors and other fields. C-suite involvement vs. company size C-suite Involvement in Different Sized Companies Company Size ,000+ They are heavily involved and hands-on in all areas of the project 26.83% 25.00% 10.00% 19.05% They set the agenda and leave the rest up to the data team 14.63% 30.56% 30.00% 32.65% They look to the data team for guidance 31.71% 25.00% 30.00% 23.81% Not at all involved 26.83% 19.44% 30.00% 24.49% With the current heightened media focus on big data it would make sense to assume the C-level is driving data initiatives across the board. However, drilling into survey responses we can see that C-level involvement varies depending on company makeup. CEOs of smaller companies are quick to put faith in their data teams with the majority of responses (31.71%) from organisations with less than 99 members of staff looking to the data team for guidance when it comes to decision making. When companies are in startup mode, this could be explained by the data team and the C-level being one and the same. Somewhat surprisingly, the C-suite in bigger companies is more likely to impose views down the way, setting an agenda for the data teams. One would hope that this approach is still able to establish achievable business goals for the data team to work from, but there is a possibility of over-involvement from C-level stifling progress that the data team could be achieving by setting unrealistic goals and not allowing room for experimentation and discovery. The good news for bigger organisations (1000+) is their leaders have less focus on being hands-on in all data projects (19.5%) than those in even the smallest organisations (26.83%). Types of data to use When asked whether better mining of internal data or incorporation of external data sets offered the most advantage to their companies, respondents indicated the former, with a 67.89% majority. However, on examining C-suite involvement in data projects, further subtleties become apparent. Types of Data to Benefit From 67.89% 32.11% Better mining of our internal data Incorporation of more external data sets 5» Taking the Enterprise s Big Data Temperature

6 C-Suite Data Value Better mining of our internal data Incorporation of more external data sets They are heavily involved and hands-on in all areas of the project They set the agenda and leave the rest up to the data team They look to the data team for guidance 62.50% 74.65% 63.49% 69.49% 37.50% 25.35% 36.51% 30.51% Not at all involved Appreciation of internal data is highest among companies whose C-suites set the data agenda but are largely hands off. When a C-suite becomes more heavily involved, the data plans become more ambitious, resulting in the highest percentage of organizations looking for external data sets to use. This focus on external and internal data sets will likely balance out as big data systems and techniques mature. As businesses strive to justify expenditure on big data, increased results from the company s existing data streams will be of paramount importance. For the more advanced organisations, which start to maximise their use of internal data, continued improvement will depend on either collecting new sets of data or expanding out further and bringing in new sets of data to mix with what they already have. Industry vs. efficiency leverage State of Big Data Progress by Industry Government Healthcare Manufacturing We re great at it; don t see how we could improve 6.25% 0.00% 13.33% We re pleased with what we re doing but know we could get better 68.75% 72.22% 66.67% We have a long way to go 21.88% 27.78% 20.00% We re awful; any change would be an improvement 3.13% 0.00% 0.00% Different industries are currently indicating differing levels of progress in terms of leveraging data. Of those approached to take this survey, members of manufacturing industries showed the highest level of outright positivity with 13.33% of these respondents claiming their data sets were being leveraged as efficiently as they could wish. As manufacturing has a legacy of being close to its data and using data sets to drive efficiency, this is hardly surprising. It s reassuring to know that they are scaling these data-driven processes up to take advantage of new big data opportunities, with another 66.67% of respondents indicating they are pleased in general with their ability to leverage data. In comparison, 6.25% in government and 0% in healthcare indicated they had extreme confidence in their ability to transform their businesses through data. Although these two stats might indicate government has the upper hand over healthcare, 3.13% of those public sector workers claimed to be totally lacking in big data capability, compared to 0% in healthcare. By examining across these three verticals, we see the general immaturity surrounding big data, with a healthy dose of realistic ambition. In each industry, the percentage indicating they have a long way to go ranges from 20% up to just under 28%. While this implies businesses are not where they want to be regarding big data competencies and abilities, it also shows an awareness of their position which is characterised by an understanding of what big data implies and the benefits implicit when big data solutions are properly harnessed. Given unlimited budget Examining budget plans for organisations going forward, we can compare their ideal spends against the next step their budgets will allow. Given an unlimited budget, companies reveal an even spread of needs across new infrastructure, new talent, and training existing talent. 6» Taking the Enterprise s Big Data Temperature

7 Given Unlimited Budget vs Next Spend Budget Next Spend Increased spend on data management infrastructure 37.35% Just improvements and tweaks to our existing setup 47.98% Increasing the number of data scientist/business analytics professionals in the organization 29.32% Switching from our existing data platform to another platform entirely 16.13% Further training of the talent we already have 33.33% Adding some third-party functionality to our existing platform 21.37% We don t have a platform, so investing in an established platform provider is next for us 4.84% We don t have a platform, and we are examining the options offered by some startups 3.23% Examining the actual spends planned, the majority of organisations feel they are close to achieving what they need, only planning to spend on tweaks. In a slightly more involved manner, one fifth are looking to add third-party solutions to their standing architectures. Absolute beginners were also represented in the survey with 4.84% having no platforms in place, but with plans to invest in this area. A further 3.23% of platform-free organisations are investigating startup solutions. The view from the boardroom vs. the view from the coal face Opinion of Senior Management vs Rest Confidence in legality Snr. Mgmt Non-Mgmt Very confident 23.97% 26.23% Relatively confident 56.20% 49.18% Neither confident nor unconfident 17.36% 18.85% Not confident 2.48% 5.74% Impediment Snr. Mgmt Non-Mgmt The fact that we have mastered data management 20.00% 16.13% Budget constraints aren't letting us progress 44.17% 61.29% The data management systems we have don't offer the functionality we need 25.83% 18.55% We just don't have the talent to make use of our data 10.00% 4.03% Despite positive findings in skill sets and big data infrastructure, when we turned our attention to awareness of legal concerns around data, it became apparent there is still much room for improvement. There also seems to be a disconnect between senior management and other staff when it comes to confidence in this area: The more senior staff see themselves marginally more secure than the nonmanagement teams. Such disparity continues when we examine views on budget. While both groups see it as the majority constraint on improvement, non-management teams see it as a huge issue compared to their senior peers. Not surprisingly, the talent don t see a lack of talent as such a big issue, while management does, though management is more confident of mastering data management than their employees are. Conclusion Looking at the results of this survey, it is clear that, although we are in the immature stages of big data, enterprises are working hard to reap rewards and adjust to take advantage in the future. There are major themes of concern these businesses want to overcome, but somewhat surprisingly, they focus most on budgets and less on the much hyped data scientist shortage. Between major verticals like manufacturing and healthcare, individual teams are already defining their goals, and examining the technical stack available to them. In these early stages, data teams are focussing on minimizing costs and driving efficiencies, but as the technologies mature, expect pioneers to start yielding results in innovation and real-time reporting. 7» Taking the Enterprise s Big Data Temperature