Big Data and Cloud Computing - The Global Challenge

Transcription

1 RESEARCH PAPER Is your business ready for Big Data? A discussion of the cultural, operational and technical elements required to make Big Data work for your business August 2014 Sponsored by

2 Contents Executive summary p3 Big Data comes of age p3 1. A culture of success p5 2. Allowing for failure p7 3. Structures of responsibility p7 4. The skills gap p8 5. Integration p5 Conclusions p10 About the sponsor, Talend p11 This document is property of Incisive Media. Reproduction and distribution of this publication in any form without prior written permission is forbidden. 2 Computing research paper sponsored by Talend

3 Executive summary The Big Data market has been through a real transition in the last twelve months. Whilst it is still far from being considered a universal requirement, awareness of the benefits that advanced data analytics can bring has increased significantly. Driving this transition have been factors such as the increasing maturity of Infrastructure-as-a-Service (IaaS) platforms and an improved understanding of what Big Data can offer. Hype about the possibilities of Big Data is being replaced with case studies showcasing the benefits to the bottom line in organisations in the public, private and third sectors. Computing undertook a comprehensive research programme during the first quarter of 2014 and categorised research participants as either advanced or less advanced based on whether or not their companies had gone as far as setting up trial or production environments. This paper discusses the characteristics of advanced companies and sets out the cultural, operational and technical elements that are likely to predict the success of a Big Data project. Big Data comes of age 2014 has been the year that the market for Big Data analytics has matured, moving away from the phase that many categorise as maximum hype. Eleven percent of participants in Computing s survey had actually deployed a Big Data solution and that solution was live in a production environment. This is still a relatively small figure perhaps reflecting some of the challenges inherent in Big Data projects and the fact that the hype surrounding Big Data has only recently peaked. However, a further nine percent of participants were trialling Big Data technologies and 24 percent were actively investigating the possibilities. So, what factors are driving the transition from Big Data theory to practice? Computing asked, What changes have you noticed over the last 12 months with respect to Big Data that are most likely to impact on your business or on the market in general? The results are shown in Figure 1, see page 4. Top of the responses was the increasing maturity and take up of Infrastructure-as-a-Service (IaaS) with 30 percent of all respondents to this question citing it. Certainly, cloud services can and are being used to achieve faster delivery of projects, reduced capital expenditure and increased flexibility. However, the cloud is not without limitations. Fifty-eight percent of the respondents when asked whether Big Data analytics inevitably requires cloud computing replied that the answer was, maybe but the cloud needs to become more secure and transparent. Computing research paper sponsored by Talend 3

4 Fig. 1 : What changes have you noticed over the last 12 months with respect to Big Data that are most likely to impact on your business or on the market in general? Increasing use of infrastructure as a service Better understanding of the possibilities offered by unstructured data Moving from analysing historical data to real-time analytics Wider awareness and a more defined understanding of Big Data Less hype, more real use cases of companies exploiting Big Data Wider variety of vendors offering Big Data solutions Maturity of big data tools bringing them into mainstream Open access/open data more datasets available from government and other sources Wider availability of open source tools and a more diverse breadth of technology None of these 30% 30% 29% 27% 19% 16% 14% 14% 13% 13% Other 1% Other popular answers to the question of what has changed in the last twelve months were better understanding of the possibilities offered by Big Data and moving from analysing historical data to real-time analytics. The beneficial effect on business decision making of being able to monitor the competition, social media, blogs, product or service reviews on websites, press coverage etc. is difficult to overstate and can deliver significant competitive advantage over less agile competitors. A little further down were wider awareness and a more defined understanding of Big Data and less hype and more real use cases of companies exploiting Big Data. These points are of course related and together mean that Big Data is being viewed as a credible reality for increasing numbers of businesses. 4 Computing research paper sponsored by Talend

5 Then there is the data itself. While fairly low in the list of big changes in Figure 1, the release of government data sources as part of initiative like Open Data are beginning to have a profound effect on healthcare, infrastructure planning and scientific concerns. The public sector is also helping to set a Big Data standard, with millions of pounds of funding being made available to unlock its potential. The funding will cover areas from medical research, arts and humanities, economic, social and environmental research. An example of projects already live include the 100,000 Genomes project whereby the DNA of 100,000 people suffering from cancers and other diseases will be sequenced, enabling research into more effective treatments. The perception that Big Data hype has peaked and that an increase in the availability of case studies is driving the initiation of increasing numbers of projects is key in the ongoing maturation of this market. Computing categorised the 20 percent of participants who had or were in the process of implementing Big Data tools as advanced with the remainder of participants being categorised as less advanced. Results were analysed to determine the key differences between the two categories. We found that there are five key areas which are likely to determine whether or not businesses are ready to make the most of the opportunities that Big Data can deliver. 1. A culture of success Advanced respondents had a number of qualities in common. The first of these centred on the culture and vision emanating from top of the company. Unless an organisation is committed to a data-led future with the vision, roadmap, people and skills in place, Big Data isn t going to add value. Indeed, some participants commented that many traditional Business Intelligence (BI) systems are still woefully underused, suggesting that in these organisations The data-led vision has to come from the top Big Data will have very little to offer. The data-led vision has to come from the top, and support from the board for Big Data projects is essential if they are to get off the launch pad as the following panelist emphasises: If there aren t very senior people driving it, the company stands no chance of making good use of it. No matter how good the skilled persons are in understanding data and exploiting data, they haven t got political clout CIO, Large retail Computing asked research participants to rank how accurately a number of statements currently described their organisations with respect to ways of treating data. The results can be seen in Figure 2, page 6. The biggest areas of difference between advanced and less advanced users illustrate how sharply attitudes to data differ between the two types of organisation. Forty-four percent of advanced users said our overall strategy, driven by the business, is to find new opportunities by making the most our data compared with just 28 percent of less advanced firms. An even bigger divergence of opinion occurred with the question of competitiveness. Big Data is a key differentiator to ensure success for us and our clients said 27 percent of advanced users compared with a mere three percent of those less advanced. Computing research paper sponsored by Talend 5

6 Fig. 2 : Which of the following best describes your organisation in terms of dealing with (big) data at the moment? Big Data is seen as an IT issue, the rest of the business is not convinced 17% 4% Our overall strategy, driven by the business, is to find new opportunities by making the most our data 28% 44% Big Data is generally seen as hype or something that doesn t concern us 10% 0% Big Data is a key differentiator to ensure success for us and our clients 27% 3% Some of us believe that Big Data is something to consider, but there is no clear business case 23% 6% The organisation is fragmented with different divisions dealing with their own data issues with no overall consensus on Big Data Large volumes and variety of data is a reality to us, however we don t have the tools or resources to make the most of the opportunities 14% 13% 5% 6% Key: Less advanced Advanced 6 Computing research paper sponsored by Talend

7 2. Allowing for failure A culture in which innovation is hampered by fear of failure is not one that s ready for Big Data, as the following panelists emphasised when asked for their advice on how to make a project more likely to succeed. Be brave. Use trial and error. Don t worry about failing. CIO, Finance Don t be afraid to fail, to make that investment and for it not to go right first time. Team Leader, Research The second point is crucial. The amount of time a CFO considers acceptable to see a return on investment in a Big Data project is likely to be one of the major factors in determining whether a project succeeds. However, the ability and willingness to do this varies according to the size of business and the industry in which they operate, and failing to demonstrate any return on investment for unspecified periods of time is unlikely to be realistic even in the most enlightened of businesses. The best way to show tangible progress within a defined time frame is to start small and choose your technology carefully. Open source platforms reduce the initial costs of a project and therefore allow more time for projects to deliver. Choose a Big Data platform that enables you to focus on one or two key areas rather than requiring you to move all of your data to do anything meaningful with it. This reduces risk because the need to move huge amounts of data from one platform to another is eliminated. Ideally, a Big Data platform should allow tools such as Hadoop to run in a sandboxed environment so analysts can experiment allowing the degree of trial and error necessary for such a project to thrive. The combination of these steps provides a transparency of process and increases the likelihood that project owners can demonstrate results against a business plan if not full return on investment. 3. Structures of responsibility Open source platforms reduce the initial costs of a project and therefore allow more time for projects to deliver Structures of responsibility for data varied considerably between advanced and less advanced organisations. Advanced participants were more than twice as likely to separate initiatives such as Big Data from day-to-day technical operations. This approach makes sense because it avoids the strategic being buried by the relentless day-to-day firefighting so typical in many organisations. The following panelist can clearly relate to this scenario. The vast majority of IT [budget] is spent on just basic service management. Just keeping things going. What we don t have are the real clever people who are sitting there doing enterprise architecture, who are being innovative because all of that money is focused on just keeping the lights on. Computing research paper sponsored by Talend 7

8 Another way to sink a Big Data project is to view it as a purely technical project which IT teams have sole responsibility for delivering. By some margin, top of the list of triggers for investigating Big Data was the business need/clear business opportunity. As we have seen, Big Data has the potential to transform many aspects of business so it is imperative that it is not confined to the IT department. A collaborative approach is key to success. Advanced respondents were more likely to have created dedicated teams made up of field experts, business experts, technology experts and business intelligence people. Fifty-five percent of respondents overall stated that to exploit Big Data to its full potential there is a need to create multi-disciplinary teams and a further 26 percent believed that Big Data should at least be the remit of several departments working together. The separation of Big Data from day-to-day IT operations was only happening for 24 percent of our respondents overall but it was the reality for 51 percent of our advanced participants. This suggests that some degree of separation from daily IT along with interdepartmental cooperation is an excellent start when planning a Big Data project. 4. The skills gap The skills gap is a real and pressing problem for all parties involved with Big Data. Computing asked respondents which Big Data solutions they were using and then asked them whether they had in-house skills for these solutions. The gap was stark as can be seen in Figure 3. Fig. 3 : Which big data technologies are you considering using / have skills in-house? Considering or using Have skills None of these 33% 23% Cloud-based Big Data services 26% 13% NoSQL databases 25% 15% Next-generation datawarehouses/analytics databases 24% 13% Data integration and data quality platforms and tools 24% 22% In-memory databases as applied to Big Data 23% 11% Hadoop software 21% 8% Application development platforms applied to Big Data 17% 11% Analytic and transactional applications and services, as applied to Big Data use cases 15% 13% Advanced analytics and data science platforms as applied to Big Data 15% 13% Non-Hadoop Big Data platforms 12% 6% 8 Computing research paper sponsored by Talend

9 The biggest gaps existed where respondents were using Hadoop and cloud-based Big Data services. In-memory databases applied to Big Data and next generation data warehouses/data analytics were also areas where businesses were struggling to recruit and keep skills. We also asked a question about what companies perceived to be the biggest barrier to Big Data for their business and lack of skills/finding the right skills topped the list, cited by 34 percent of respondents along with cost. A deeper analysis of these results shows up differences again between advanced and less advanced users. Advanced users are more likely to be using or trialling in-memory databases. However, this is one areas where a particularly sharp skills gap exists. There is however less of a skills gap with Hadoop among the advanced group. This probably reflects the emergence of Hadoop as the de facto standard platform for storing and processing large and varied data sets. Advanced companies are more likely to have been busy strengthening their skills in this area. It was the less advanced users that pushed the Hadoop skills gap rating up. Only four per cent said they could count on these skills among their in-house staff. There is no magic way of closing the skills gap, and consultancies and other channel partners are finding ways to help organisations to at least bridge the divide to some degree. However, businesses can help themselves by making an assessment of skills a significant part of the decision making process when choosing their platform. The need for coding skills can be reduced by using a platform with a drag-and-drop front end and which can generate code running natively in Hadoop, for example. 5. Integration We have already established that a gap in skills and cost are some of the biggest barriers to Big Data. We will now turn to the third largest issue integration. Overall 18 percent of our respondents had concerns here but it was more likely to worry those who were categorised as advanced with 24 percent stating that it was a concern. The issue of integration covers numerous potential pitfalls. In many businesses data exists in multiple formats, both structured and unstructured and it is often scattered across different locations both internal and external. The challenge is getting these disparate blocks of data to work together. The following quotation from one of our panellists neatly summarises the challenge facing many businesses: You want to spend 20 percent of your time on integration and 80 percent on analysis, but most organisations grapple with 80 percent on integration and 20 percent on analysis and that s not where you want to be. It s to do with data coming from a lot of different sources with a lot of different business rules around that with a lot of different definitions that tend to mean the same thing but are not the same thing, but you need to make it uniform CIO, Finance Without integration there is a natural limit on how meaningful analysis can be. Unfortunately, a characteristic shared by many businesses not ready for Big Data is a static, siloed approach to data. Organisations who are ready expect data analytics to facilitate their business objectives rather than being trapped by technology that cannot deliver the insights required. They know what their data structures are and they know how to join the dots from both internal and external data sources. Collaboration also tends to be a characteristic of more advanced businesses. Whether a business is advanced or less so, any proposed Big Data solutions need to integrate with an existing data infrastructure. The prospect of having to migrate existing data to a new platform is a process fraught with risk and cost and is likely to act as a massive brake on a project getting off the ground. Computing research paper sponsored by Talend 9

10 Conclusions The Big Data market has developed at quite a pace in the last twelve months and certainly 2014 looks set to be the year that many businesses make the move from theory to practice. This is no surprise the benefits that Big Data can bring are more widely understood than ever before better decision making, the identification of new trends and opportunities, improved responsiveness to customer needs and an overall sharpening of competitive edge are all there for the taking. However, whether the technology, process and people involved in Big Data can be combined to create these opportunities depends very much on the insight and environment of the company deploying it. The research undertaken by Computing found some significant differences between organisations that had put Big Data solutions in place on either a trial or production basis and those that had yet to do so. We found that in order to make the most of all of the opportunities that Big Data can present and to make a Big Data project likely to succeed, businesses need to assess themselves in four key areas culture, structures of responsibility, skills and data integration. Are you Big Data ready? Culture Allows for trial and error Has a vision and understanding of the opportunities Has an understanding that they always need to be looking at the next stage/future developments Understands the value of the assets possessed Budgeting is geared to the future without immediate expectancy of rewards Buy in from the top Structure of responsibilities Separation of Big Data from day-to-day operations Creation of dedicated teams encompassing operational experts, sales and marketing, business intelligence, technology and finance Skills Has access to skills to work data to maximum effect Integration Has technology in place to connect internal and external data Has the technology in place to process and analyse both structured and unstructured data Has a good understanding of data structures and how to connect it Is willing to consider to open up software and data to third parties Supports collaboration between interested parties internal and external If at least some of these criteria can be fulfilled organisations stand a good chance of being able to deliver a Big Data project which fulfills the expectations of the business and customers alike. 10 Computing research paper sponsored by Talend

11 About the sponsor, Talend Data-driven organizations use Talend s integration solutions to gain instant value from their data, by delivering timely and easy access to all historical, live, and emerging data. Through native support of the modern big data platforms including Hadoop and NoSQL, Talend s zero footprint solution takes the complexity out of integration and equips IT to be more responsive to the demands of the business. Based on open source and supported by the broadest ecosystem in the industry, Talend s unified solutions portfolio is provided under a predictable subscription model and includes big data, data integration, data quality, master data management (MDM), enterprise service bus (ESB) and business process management (BPM). Unlike legacy integration architectures that simply cannot scale to efficiently support always more complex needs, Talend s scalable, future-proof solutions address all existing and emerging use cases of integration. Contact Talend: Visit: Computing research paper sponsored by Talend 11