Big Data Insight Group 1 st Industry Trends Report

Transcription

1 Big Data Insight Group 1 st Industry Trends Report Sponsored by: Understanding the business benefits and strategic implications of big data

2 team HEAD OF RESEARCH & STRATEGY: Caroline Boyd Editor: Dominic Pollard Associate editor: Mark Young MEDIA PARTNERSHIPS: Hannah Mitchell Business Partnership Managers: Please contact for details of upcoming events Milly Blundell Owen Gregory Charlotte Tite Research assistant: Tosin Arogundade Design: Optic Juice Managing director: Please contact for sponsorship opportunities Emma Taylor Director: Ranald Lumsden Search for us on Twitter and LinkedIn Telephone: +44 (0) Registered company and publisher name: Nimbus Ninety Ltd Registration Number: Registered in England & Wales Office address: 10 Greycoat Place, London, SW1P 1SB Registered business address: 16 Northfields Prospect, Putney Bridge Road, London, SW18 1PE Copyright Nimbus Ninety Ltd 2011 While every action is taken to ensure the information within this report is accurate, the publisher accepts no liability for any loss occurring as a result of the use of that information. All rights reserved. No part of this report may be published or stored in a retrieval system without the written prior consent of the publisher. Welcome to the 1st Big Data Insight Group Industry Trends Report, focusing on understanding the business benefits and strategic implications of big data. Big data analytics promises to revolutionise the way organisations gain insight and value from their data. The capability is here for organisations of all shapes and sizes to exploit the everincreasing amount of data we collect. Yet many are still struggling to realise the full value of the data they have at their disposal. This report is designed to help senior executives and decision makers to understand the potential of big data. It aims to achieve this with a range of advisory, thought-provoking articles as well as valuable insight into the state of the marketplace. Throughout the spring of this year, the Big Data Insight Group conducted its own independent research to establish just what people are doing with big data. The 300 responses came from a range of senior personnel, including individuals in business, IT, finance and marketing functions, and across a variety of different industry sectors offering a broad cross-section of big data users. The results of the survey, with detailed analysis, are included in this report. The report also includes two columns from leading big data academics, interviews with the CTO of leading app company Shazam, Jason Titus, and David Boyle, of EMI Music and zeebox. There is also a feature on the democratisation of data, leading case studies from the social media website Tagged and the data cloud provider doubleiq, as well as a snapshot of how different industry sectors are exploiting the power of big data analytics. Whether you re already using big data tools or keen to learn how to squeeze maximum value out of the data you generate and store, the 1 st Big Data Insight Group Industry Trends Report is sure to help you on your journey. Thank you to everyone who took the time to complete our survey. This has helped to form a critical piece of independent research which provides an in-depth perspective of the latest trends within the rapidly developing world of big data. Please contact us if you would like to discuss any of the successes or challenges of your own big data projects, or if you have any opinions you would like share. We d love to hear your thoughts. Yours sincerely, Emma Taylor, Founder and managing director, Big Data Insight Group April 2012

3 contents The Introduction Big data is still confusing for many, cut through the hype and understand the benefits of it with an introduction to our 1 st Industry Trends Report The Survey The Big Data Insight Group presents the findings of our 1 st Industry Trends Survey, with in-depth analysis from a panel of industry experts The Experts View: David Chan and Mark Whitehorn Two columns by leading academics on the rise of big data and what organisation will need to do to exploit all it has to offer Interview One: Jason Titus Pitch perfect Jason Titus is the CTO of the song identification app Shazam. With the app receiving over a million downloads a week, he tells Dominic Pollard how the company is now using the data at its disposal to create opportunities for expansion and new revenue streams Interview Two: David Boyle The self confessed data geek Having held senior insight positions in a range of different organisations, David Boyle shares with Mark Young how to make the most out of your data The Feature: The democratisation of data Mark Young explains how the emergence of new avenues of data, combined with the mass availability of the advancing technologies for analysis, has seen data become democratised providing a range of new opportunities for all enterprises, large and small alike Snapshot: Industry leaders Dominic Pollard examines how some organisations are leading the way with big data within their respective sectors Sponsor Case Studies Social network: Tagged When its data analytics system struggled to keep pace with its expansion, Tagged turned to a big data solution Information management systems provider: doubleiq Having adopted EMC s Greenplum Database, doubleiq can answer data analysis queries up to 300 per cent faster 3

4 The Community of Data Scientists, Passionate About Data Science. As a non-profit organization, we are dedicated to the free, open, dissemination of data science. Created by Data Scientists for Data Scientists, we act as a forum for discussions and the exchange of ideas. To learn more, visit: datasciencelondon.org Data Science London Where Data Scientists meet Data Scientists

5 introduction An introduction to... The 1 st Big Data Insight Group Industry Trends Report Cut through the hype and realise the benefits Big data is the buzz term in the business world. But what the term actually means is still a mystery to many. The report s editor Dominic Pollard offers an introduction into 2012 s hottest market trend. The rise in the amount of data we store and manage has been exponential. Every day, we supposedly create 2.5 quintillion bytes of data which means that 90 per cent of the data in the world today has been created in the last two years alone. Quite simply, the data which organisations are storing and managing is increasingly significantly day-by-day. But, the question remains what exactly is big data? Like any new technological term there is an element of hype that must be disposed of so that the real benefits and implications can be understood. For the purpose of simplicity, the widely accepted definition of big data is any amount of data which is too large for your existing IT Infrastructure to be able to store. The three V s But big data is not just about size. There are often cited to be three Vs to big data which distinguish it from traditional business intelligence (BI). The first is volume, as just stated, and the sheer size of the data sets is a critical factor. However, the variety of the data and the velocity of the analysis must also be acknowledged as being fundamental aspects of its definition. Today organisations are gathering data in multiple forms from multiple sources. The rise of social media along with the analysis of images and YouTube videos have opened up new realms of possibilities to extract value from data sources that have previously been too large or complex to have been exploited. Complex, unstructured data can now be analysed and visualised in engaging ways, allowing you to make sense of data which had previously been incomprehensible to the human brain. Velocity refers to the speed at which the analysis can take place. Where previously it may have taken hours, days or even longer to garner any useful answers, big data because of developments in the infrastructure, tools and techniques used for analysis allows real time or near real time responses to vastly more intricate data queries. Actionable insight Big data is now recognised as a term for using masses of structured and unstructured data to gain actionable insights which had hitherto not been possible. Cheaper storage, distributed file systems which allow you to spread computationally intensive processes over potentially limitless machines, cloud computing, and advancing open source software have all contributed to big data s emergence in the mainstream. The benefits big data can deliver, as this report will illustrate, have the potential to revolutionise a business. Through such things as improving the efficiency of internal processes, predicting trends, gaining unparalleled insight into existing or potential customers, getting an immediate and accurate view of the market, and opening new revenue streams, big data can give an organisation competitive advantage in what remain very difficult economic times. However, despite everything big data promises, it remains an enigma to many individuals. This report will examine why this is the case, explore the maturity of the marketplace and establish just whatcan be achieved through big data analytics. 5

6 HIGH-PERFORMANCE Big data has met its match. There s a better and faster way to get value from the constant barrage of big data that s been hitting you. SAS High-Performance Computing simplifies the analysis of very large data sets so you can solve complex business problems with a triple whammy of advanced analytics, processing power and speed. SAS and EMC Greenplum are taking on big data in a big way. SAS High-Performance Analytics will soon be available on the EMC Greenplum Data Computing Appliance. sas.com/bigdata for a free white paper on big data SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. indicates USA registration. Other brand and product names are trademarks of their respective companies SAS Institute Inc. All rights reserved. S90205US.0312

7 Survey big data The 1 st Big Data Insight Group Industry Trends Survey Understanding the business benefits and strategic implications of big data The survey was conducted in February and March It was completed by 300 senior Business, finance and IT personnel from a broad range of industry sectors including financial and professional services, retail, manufacturing, telecommunications, pharmaceutical, charities and the public sector. They represent companies of all sizes, from single site SMEs to global, industry leading blue chip organisations. Fig 1 How much data does your organisation manage and store? (%) Currently By 2015 Big data, although not entirely about the size of data sets as the introduction to this report outlined, has come about because of the exponential rise in the amount of data organisations are storing and managing. It comes as little surprise, therefore, to see in fig. 1 that over the next three years there is going to be a shift towards more organisations storing over 1,000 terabytes (one petabyte) of data. Of the 300 people surveyed, the number of respondents which stated that they were managing and storing over one petabyte rose from 16% currently to nearly double that (29%) in three years time. In essence this highlights the trend that the data we are collecting and storing is increasing day-by-day. In explaining this rise in data and the sources organisations now collect it from, Chris Roche, a senior director at EMC, says: If you look at your individual life, you are on social media websites, you might put videos up on YouTube, you are probably sending more s than you ever have done in your life, and you could be using storage products like Dropbox. Then you realise just how much data you re generating on your own. For businesses, all of this extra data affords new opportunities for extracting insight and value. However, David Boyle, senior vice president of insight at EMI Music and director of insight at zeebox, warns that the Key findings: 67% are either unfamiliar with big data or are still in the education process The ownership of big data initiatives currently rests predominantly with the IT department (47%) Less than a quarter of respondents (23%) believe they will require new personnel to execute a big data strategy sheer volume of data can be dangerous. The more data you store the more complex the systems become for managing it. It can also become slower, more costly and more confusing to analyse, Boyle says. We don t have many problems storing mass quantities of data anymore, but there are certainly difficulties in ascertaining which bits to interrogate and analyse. When establishing the status of the respondents big data initiatives, fig. 2 illustrates how the majority of people (83%) are actively researching, sourcing, implementing or running big data solutions this is split between 50% of respondents who are researching big data in one form or another and 33% who have projects implemented or are in the process of being implemented. Only 17% said they were unfamiliar with what big data is. This may be a reflection of an audience who are likely to complete a survey about big data. Nevertheless, it does indicate that big data is on the radar of most organisations regardless of their size or sector. 7

8 Survey big data Fig 2 What is the current status of your big data initiative? Still unfamiliar with what big data is - 17% Researching and sourcing solutions - 50% Implementing or implemented - 33% Personnel When asked whether or not their organisation would require new personnel to embrace big data strategies (fig. 3), the most common selection by respondents was that they were not sure (46%). This is indicative of the relative immaturity of the market. Only 23% stated that they believed they would need new personnel whereas a significant proportion (31%) appear confident they can execute a big data strategy with their existing staff. Graham Oakes suggests that this is because people are doing techled pilots and that IT people are the ones who are experimenting with the technologies. As such many are unaware if they will require new personnel for a business focused strategy. Everything I am seeing with big data, Oakes says, shows that you will only start to see real benefits when you get collaboration between the technical people, the data scientists and the business functions this is when you can start asking sophisticated business questions and get a solid understanding of how you can get answers to the right people. Fig 4 However, as Graham Oakes, business process efficiency expert and principal at Graham Oakes Ltd, says: Many of those who are researching are actually in the early stage of working out what is meant by big data rather than researching specific projects. David Boyle reiterates this point as he states: I would clump the 50% who are researching with the 17% who said they were unfamiliar with big data. In reality this says to me that 67% are unsure about big data and are still in the educational process. This could mean that the big data market is not as advanced as the graph may seem to suggest at first glance. Nevertheless, fig. 2 demonstrates that although many organisations are still at the start of their big data journeys something which is not surprising, given that big data only recently emerged as a mainstream business tool the growth of interest in the subject has been rapid. Roche believes that the results here show that many are seeing it as important to their organisation, whereas only a few months ago it wasn t even a term people had heard of. Examining what the main barriers are to big data Main barriers by industry sector Retail Public sector Financial services Lack of relevant skills 49% Lack of tools and technologies 39% Costs 36% Lack of understanding 56% Costs 52% Lack of relevant skills 51% Lack of relevant skills 42% Lack of understanding 40% Lack of tools and technologies 37% Professional services Costs 35% Compliance 33% Lack of understanding 28% Chris Roche stresses that it is ultimately going to be more a question of skills than personnel. He says: In terms of how you get these new skills there is a myriad of different models. You Fig 3 Will/did your organisation require(d) new personnel to execute your big data strategy? Yes - 23% No - 31% Not sure - 46% can train up your own people and there are initial training programmes around data science emerging or, alternatively, you can recruit new people. Otherwise, you could also get those skills from a managed service provider. Organisations will need new skills but it could be their current staff that is retrained to deliver them. This may prove to be a popular approach; many analysts predict that we will face a shortage of individuals with relevant skill sets (discussed on page 13) as demand increases. Indeed this is supported by fig. 4 which examines the perceived barriers people have to adopting big data. Lack of relevant skills and lack of understanding are 8 THE 1 st Big Data Insight Group INDUSTRY TRENDS REPORT

9 Survey big data common across the industry sectors. The fact that the retail and financial services sectors cite skills as the biggest issue suggests that they are further ahead with their big data initiatives and have now got to the stage where the skills and personnel they have within their organisation are preventing them from achieving maximum value from their data analytics. As Oakes maintains: The further on people get, the more they will realise that skills are the issue. David Boyle believes that ultimately the biggest barrier to big data across all industry sectors will be a combination of both personnel and skills. He says: There are few people who have the skills to understand, shape and run the whole project. People need to ask the right questions and then have the ability to identify, gather, store and analyse the relevant data to get the answers to these questions. Graham Oakes suggestion that those who are experimenting with big data are running predominantly technology-led IT projects is supported by fig. 5. Nearly half (47%) of respondents said that big data analytics lay within the remit of the CIO or IT personnel of their organisation. The other 53% was divided among various departments and job roles including marketing, business intelligence (BI) personnel and data architects and scientists. This certainly indicates that it is IT departments who are having a strong influence on the adoption of big data by being the first to embrace the new tools and technologies. David Boyle expects the responsibility to shift toward business departments over the next year or two something he believes is essential if organisations are to get the most out their data. As businesses begin to understand and embrace big data you can expect to see a more even spread of responsibility across the business functions, he says. The IT department might be the first to hear about the technology and test it, but when the benefits of big data become clear to the organisation the projects will become properly organised and you will see other personnel starting to get involved. Chris Roche supports this view as he emphasises: The future of analytics is not any one person s remit. Everybody needs to be involved the data scientist, the data architects, business people, marketing and IT. It isn t just one department either; to get the value out of your data it has to be a team sport. Drivers The opportunities which big data presents to an organisation differ depending on various factors. The sort of data they have at their disposal and the value they wish to extract from it will be dictated by an organisation s business objectives. Fig. 6 shows how the driving factors to adopting big data vary by industry sector. What stands out is both what the different sectors want to achieve from their data analytics but also the different stages of maturity each sector finds itself in. Main Drivers By Industry Sector Retail Public Sector Financial Services New insight into customers 64% More targeted marketing campaigns 57% Better planning and decision making 53% Better planning and decision making 78% Extract more value from old data 76% New insight into customers 62% Fig 5 Who is responsible for owning big data analytics within your organisation? Marketing CTO BI personnel Data scientists or data architects CIO or IT personnel Fig 6 Examining what the main drivers are for big data Extract more value from old data 68% Competitive advantage 51% Better planning and decision making 46% Professional Services Extract more value from old data 67% New insight into customers 57% Competitive advantage 52% The retail industry appears far more mature as the majority of respondents from organisations in this sector stated that they wanted new insight into customers (64%) and more targeted marketing campaigns (57%). Graham Oakes feels that this shows that retailers are asking the right questions. This is an area where you can see ROI because if the effectiveness of a campaign goes up by two per cent then you ve proved its benefits, he says. The public sector, the findings would suggest, is at an earlier stage of its big data journey as extracting more value from old data features prominently (76%). This is indicative of a desire to adopt new methods as

10 Survey big data Fig 7 What tools does your organisation use to perform data analytics? Existing BI tools - 50% New big data tools - 15% A combination of existing BI and new big data tools - 35% a means of getting the maximum value from existing data and assets which have hitherto not been exploited. Alternatively, it could show that many organisations are in an experimental stage of a project in which they are using old data to test new tools, technologies and techniques. David Boyle states: Both fig. 4 and 6 illustrate that people aren t taking an active or thorough enough approach to their data analytics. This is not a surprise but just highlights that this is a new market which people are trying to get to grips with. The tools and technologies Big data is not, of course, just about the rise in the amount of data we have, it is about the ability we now have to analyse these data sets. It is the development in open source tools and technologies, including such things as Distributed Files Systems (DFSs), which deliver this ability. The survey findings (fig. 7) illustrate that people are beginning to adopt these new tools, again supporting claims that IT departments are experimenting with the new technologies in the early stages of their big data journeys. Half of the respondents said they are using big data tools within their organisation, either exclusively or in combination with their existing business intelligence tools. The other half of organisations are still solely using traditional and existing BI tools to perform their data analytics. The impression EMC s Chris Roche gets from these results is that people want choice. He says: When it comes to tools, I think people want to be able to use any analytic tool in their kit bag, whether they re writing code or using an analytics partner or some form of SaaS, they want to have the choice. It comes down to whatever the best tool is for the job in hand. I am a great believer that if you ve made an investment in something and it does the job why would you buy a second tool? People want to make the most out of their investment and big data should not change that. David Boyle states that it is important not to become overly fixated on the technology and urges that a pragmatic, business focused approach remains of paramount importance. He believes: If you re not disciplined about the questions you ask at the beginning of the project and you re not careful about how you analyse the data then you will just use new tools to throw masses of data at. You have to ask the right business questions and you do that by being smart about which data you use and how you analyse that data you can t just implement new tools and expect them to act as a silver bullet. Currently over half (59%) of respondents stated that it takes them hours or even longer to get responses to their data analytics queries (fig. 8), although these queries are not necessarily big data ones. Only 19%, meanwhile, claimed to be able to get real time insight from their data. Graham Oakes is confident that as testing progresses further we can expect to see the time it takes to get responses drop. He says: This shows that people have still got projects in pilots and the figures are a reflection of the time it is taking for them to find the data they want and Fig 8 Typically, how quickly do you get responses to your data queries? (%) to load and analyse it. But I would expect that people are going to want to get more of the responses to their data queries down to minutes at most. For Chris Roche, speed is only relative to the task in hand. He says it is important to acknowledge that there are situations, such as performing analysis for annual reports, in which you can afford to wait hours or even days to perform analysis on data sets. He believes people need to ask if you got quicker responses, could that time differential help your company perform more successfully? If the answer is yes, he states, then you have a case for 10 THE 1 st Big Data Insight Group INDUSTRY TRENDS REPORT

11 Fig 9 Has your organisation yielded tangible benefits from big data analytics? Yes - 20% No - 39% Not sure - 41% adopting new big data tools and technologies. Far greater speed of responses is one of the key, although not defining, characteristics of big data. The fact that less than half of respondents (41%) can get responses to their data analytics in minutes or less supports what we see elsewhere in the survey results people are still in the very early stages of trying to understand how they can exploit big data tools. As such, they are yet to realise the benefits it has to offer, namely real time insight into data through the use of DFSs and intelligent visualisation and dashboarding. All the survey analysts were confident that as people know the ability is there to do this we can expect to see more and more organisations achieve much quicker responses to their data analytics, if they deem it is something which would add value to their business processes. Proving the results The immature state of the big data market can be seen in fig. 9 which demonstrates just how few respondents (20%) have seen tangible benefits from big data analytics within their organisation. This could be the result of a number of factors. Firstly, as established, there are only a small number of organisations who have big data projects implemented from which they can receive any benefits. Secondly, it illustrates the nature of the projects which people are currently implementing. If they are, as suggested, executing technology pilots as a means of familiarising themselves with the new tools that are available then tangible benefits to the organisation are not yet the objective. More so, it is to gain a better understanding of the technology. Therefore, the lack of business focus to big data projects makes it far more difficult to realise any business benefits. Furthermore, only 14% of the respondents who stated they had big data projects implemented said they are yet to see positive return on investment (ROI) from them (fig. 10). As Graham Oakes says: This all suggests people are struggling to show benefits from projects because they are in the early stage of implementation. Chris Roche adds: There are so many people saying no and not sure in figs. 9 and 10 that this would suggest that they didn t have a clear goal when they began their projects, while many have simply not started any big data projects yet. But for those who have but aren t seeing any benefits you must ask do they have clear business benefits they are trying to achieve or are they just playing around with the technology? Only when the market matures and the people with the relevant skills from a range of business functions begin to play a role in asking the right questions will people begin to realise the potential advantages big data has to offer. There is a risk to this approach, though, as Oakes surmises: With a tech pilot your aim is to understand the technology but it is hard to prove the business benefits from that. Hopefully people will come out of this first round of piloting saying that they understand the technology, they know more about what skills they ll need and can then set up some business focused pilots. Then the second round of pilots will offer far better results. The worst case scenario is people come out of these pilots without being able to show clear ROI and they then think big data is not for us. Reflecting on the results of the survey, David Boyle says that at the moment the state of the market reads like a story of missed opportunities. Big data should and can be quick, cheap and powerful when done properly, he emphasises. If the right people can ask the right question then people can start sifting through all this data to choose the correct, clean parts of it and analyse and display this in an interactive way. The findings of this research suggest that the market, though rapidly evolving, is still an immature one. There may still be issues with defining big data and beyond this uncertainty remains over the skills, tools and personnel required to execute a successful big data strategy. With many trying to educate themselves in these areas we can expect to see the market changing dramatically over the Survey big data Fig 10 Has your organisation seen a positive return on investment from your big data project? coming months and years as people gain a far greater understanding of what the benefits of big data are as well as best practice tips on how to overcome the obstacles to realising them. What shines out from these findings is that big data is not simply about new tools and technologies to deal with increasing amounts of data. It is about taking an intelligent approach to using that data to answer clear, predefined, business orientated objectives from which an organisation can reap the rewards. Yes - 14% No - 26% Not sure - 60%

12 COLUMN DAVID CHAN The world is changing our mindsets must change with it The rise in the amount of data we are generating and storing has been meteoric of late. But while the world of data is developing rapidly, the ways that we manage it are not keeping pace. David Chan says our paradigms need to change. A column by David Chan, director, Centre for Information Leadership, City University London david.chan.1@city.ac.uk Big data has arrived; there can be little doubt about that. Its arrival has come not only with a meteoric rise in data but with the advances in the tools and technologies we use to analyse it. However, technology is never a silver bullet and before we can reap the benefits of big data we must change our mindset for how we approach it. Paradigms are the way we make sense of the world. They are the perspectives that we use to understand what is happening around us. But, of course, if we don t have the right information or enough of it, our understandings can be built on falsehoods. For instance, the ancient Greeks understood thunder and lightning to be the god Zeus throwing his thunderbolts. They didn t know about the interaction of warm air, water vapour and static electricity. The prevailing paradigm for data analytics is static modelling the search for rules that explain variation in data. We examine randomised data sets and look for patterns between the key points of difference. If we can find these same patterns in other data sets then we know we have a model. We then use that model to make predictions. A real world example of this can be seen in the way that banks weigh up the risk of giving somebody a loan. They analyse the applicant s financial history and lifestyle and try to see whether the patterns fit somebody that usually meets their repayments on time or somebody that defaults. Static modelling like this is fine, in theory. Indeed, it works perfectly well when there are fixed relationships that are discoverable through detailed analysis. But what happens if what we are seeking to model does not work in the same way? The butterfly effect is common in mathematics. This is where small changes in the original conditions have hugely disproportionate effects on the outcome. So if you went back in time a thousand years and changed something very small, the world could potentially be a very different place today. In data analysis, the effect can often be that the model we use to gain reliable information out of our data no longer works when applied to the rapidly evolving environment around us. Technology is never a silver bullet and before we can reap the benefits of big data we must change our mindset Unfortunately, these small changes are becoming increasingly common as our data sets get bigger and bigger. Furthermore the sources that we now capture all this data from consist of complex, mutually interacting systems and this makes them prone to feedback. We think of the problem as noise if you re listening to a conversation and then somebody else talks over them, it s easy to mistake what the person you re speaking to has said. This noise is especially common now that we are using unstructured data. For instance, if we look at surveys or polls, to what extent are the answers affected by the way the respondents wish to appear? Although we can account for identified factors, how do we know there are not other factors that have significant impacts? The effects we are trying to predict may well be swamped by other signals. Therefore we need to change the way that we manage our data and the paradigms that we use to understand it. New technologies like Complex Adaptive Systems are specifically designed to help with noisy data. But without the right approach, they aren t enough. We need to use big data analytics as a more accurate means of adapting a system to cope with the data being received from these constantly evolving markets. It s new, and not generally well known, but it is the answer. The data for extremely indepth insight has arrived, and so have the tools; we now just need to change our mindsets to unlock the value they can offer. 12 THE 1 st Big Data Insight Group INDUSTRY TRENDS REPORT

13 COLUMN MARK WHITEHORN New skills for the new tools The age of big data is upon us. In order for organisations to revolutionise their data analytics they will need the right personnel. Mark Whitehorn explains. A column by Mark Whitehorn, professor and chair of analytics at the School of Computing, University of Dundee The idea that business intelligence (BI) solutions have to be tailored to a business requirements is old news every organisation is unique and as such their analytics strategies must be customised. The same is true for the effective handling and analysis of big data, only more so. Hardware, software and personnel are the three major components. Obviously you need the kit to hold the big data and the software to analyse it. Exactly what this might be will depend on the data you are going to work with and the kind of analysis you want to perform. However, a much greater challenge is likely to be finding the right people to execute your big data analytics. It is easy to think of big data as completely separate from The challenge is likely to be finding the right people to execute your big data analytics normal data, that being smaller sets of structured data. After all, we ve given it a whole new name. In reality, big data is almost always cross-analysed with existing tabular data. So, in order to make the most of big data analytics, you ll need all the usual suspects in terms of BI skills - a good understanding of relational and multi-dimensional design and structures, programming languages for databases, and information presentation skills. Excellent communication skills are also high on the list, both verbal and written. So, that s the BI side covered, but what else should you have on your big data skills wish-list? A good background in data, statistics, data mining and algorithm design is also going to prove essential. The statistics are vital because we need to know, not only whether two sets of data are different, but whether that difference is really significant. Data mining covers a collection of techniques for finding patterns in data and can reveal information hidden deep in a sea of data, such as clusters of customers with similar buying behaviour. In addition you need the skills to deal with the particular technology you intend to use, likely to be a NoSQL system of some kind - perhaps MapReduce or Hadoop. All of these relatively specific skills should be coupled with experience and, last but certainly not least, imagination. Such is the range of skills and experience required to execute big data analytics, it makes sense to think in terms of assembling a team rather than seeking all of these skills in one individual. Big data is a new field and not yet well understood in the way that, say, transactional data is understood. As you read this, organisations in a range of sectors, private and public, are working on the fundamental research that will crystallise into new techniques and applications for handling big data. In ten years time it will be obvious exactly how big data was going to be exploited but it s far from obvious now. However, even if we are not yet exactly sure how to extract the best results, it s already abundantly clear that the greatest rewards will go to the pioneers of whom there are broadly two types. There are the early adopters of new techniques and applications as they are developed, commercialised and packaged. They will do well. But the true pioneers are the ones who blaze the trail, who develop the techniques the richest rewards will be theirs. The challenge, difficult though by no means impossible, is to find those people with these skills and experience to enable your organisation to reap these rewards. Ultimately, we need people who are switched on, clued up and able to see opportunities where no-one else has seen them. Can you, for example, analyse Twitter data and spot potential customers who don t even know they need your product? I don t know, and in all probability you don t either, and there s no formula to tell you whether it s even possible. However, the skills, attitudes and aptitudes listed above, coupled with a good dose of imagination, will let you lead the way. 13

14 INTERVIEW JASON TITUS Pitch perfect For a decade now Shazam, the song identification mobile phone app, has been settling bets, discovering new artists and unearthing guilty pleasures for its users. With over a million and a half new users each week, the company s recent growth has been exponential and its big data analytics have revealed a range of new possibilities for expansion. Dominic Pollard visited the company s London headquarters to speak to its CTO Jason Titus and find out about the magic behind the technology. 14 THE 1 st Big Data Insight Group INDUSTRY TRENDS REPORT

15 INTERVIEW JASON TITUS Shazam is a mobile phone application that allows the user to find out the artist and name of any song they hear. Within just a few seconds, that nagging question of oh, what is this song? can be answered at the touch of a button. CTO Jason Titus says Shazam is like a unique, magical experience in reality, the service is built on the innovative use of data and technology. Released back in 2002, the service reached one billion tags (the term the British born and based company uses when someone Shazams a song) nine years on. The second billion arrived less than 12 months later. Product development, enabled by big data, has also increased considerably since the start of the decade. The most recent advances include the introduction of real time play-along lyrics, a halving of the time it takes to identify a song and the ability to connect the app with Facebook and share tags with friends. Once a song is tagged, the user is also now given options including watching a related YouTube video or buying the song on itunes. We provide regular updates to record labels on what is being tagged and at the end of the year we do a report to give our predictions for up and coming bands Jason Titus, CTO, Shazam With over 1.5 million new people using the Shazam app every week across all of the major mobile phone operating systems, it s fair to say that the company is hitting all the right notes among its user base. And with all that activity comes masses of data, which Shazam analyses to make the product even better while opening up entirely new revenue streams. Titus explains Shazam s technology in layman terms. To simplify, he says, the app listens to the frequency and amplitude of the music, finding interesting points within the music to create a unique signature and then sends that to the server to be compared to all the records of songs we have. It s highly optimised; we ve spent a lot of time trying to make what is a computationally intensive thing very time efficient. We are resistant to noise or pitch shifts. On the radio, for example, the music is up to five per cent faster or slower. We have to account for this in the core algorithm. What s hot, what s not? The data that Shazam generates and analyses is of great importance, too. Shazam s song-tagging charts are an accurate precursor to the US Billboard charts a few weeks later. There are a lot of people who want to know what the latest musical trends are, explains Titus. We provide regular updates to record labels on what is being tagged and at the end of the year we do a report to give our predictions for up-and-coming bands. These are commercial arrangements that we have instigated. For the partner it s about seeing who s had a trajectory which would suggest they are going on to big things. Through the collection of billions of geographic tags, Shazam can also identify what is popular in a specific area and can therefore analyse what particular subcultures are emerging around the world. Shazam s partners can then use this information to tailor their offerings. The company can also report back to advertisers, music promoters and other media producers on how successful their musical accompaniments have been. So, for instance, if a band performs on a talk show Shazam can report in near real time how many of its users tagged the song and feed this information back to the show, helping advertisers define the audience and its inclinations. Titus and his colleagues are now looking to expand the product further into advertising as well. Examples the CTO gives include a car advert or a movie trailer. If you were to Shazam one of these adverts, the app would be able to identify what you were watching and provide 15

16 INTERVIEW JASON TITUS We are definitely a business in transition. We are starting to do some very interesting analysis with real time tools Jason Titus, CTO, Shazam the user with additional information, release dates and purchasing options for that product. The company has already covered 50 advertising campaigns in the US and is looking to become even more active in this arena. Clearly then, Shazam is building an increasing influence over the development of the music and media industries, and, thanks to its data, its opportunities for expansion and monetisation within horizontal markets are vast. It is a shining example of how you can take the data you have at your disposal and use it to elevate the company to new heights. However, Titus stresses that making a great user experience remains the company s absolute priority. The predictive data that is collected inconsequentially is an added bonus which they are able to use analytical tools to exploit to both Shazam and the music industry s benefit. An orchestra of harmonic technologies Shazam uses an initiative called Demo Day to help it find new technologies, an initiative that Titus carried over from his previos role; for the six years previous to joining Shazam in January 2010, Titus worked as vice president of communications at Yahoo!. He left after coordinating the development of the new unified Mail and Messenger experience across desktop. Demo Day was a particularly popular technique the American-based Internet giant used and Titus has taken across the pond to Shazam s London headquarters. The initiative allocates each developer 15 per cent of their working week to explore new technologies and experiment on their own projects. Then, once every couple of months, the developers present what they ve created to the rest of the company. Whether it s an improved user interface or an optimisation to the app s backend, the initiative lets the creative juices flow and, in turn, the company gets a steady influx of new tools and ideas across the entire development team. Demo Day is something that s been particularly successful and the developers seem to like it a lot, Titus says. Over the last year there have been many product improvements which have come out of it. They might not have been on our roadmap but from what they ve done we have gone away and said we should really look into doing that. The app itself is, of course, technologically rich. But beyond the wizardry that makes sounds Shazamable, the company is also harnessing other progressive technologies, including cloud, in a wealth of other ways. Approaching new tools and technologies in an open and experimental way has been an integral part of allowing the company to exploit the true value and insight from the data it collects. We are definitely a business in transition, Titus expresses openly. We are starting to do some very interesting analysis with real time tools. Going forward we are evaluating which ones we will use whether that s Hadoop or a mix of other tools. Our pace of development has improved dramatically over the last year, in part because we have started to leverage much more of the open source code that s out there. Only through clear business objectives and the right tools to support them can the company get such a range of people across the globe singing from the same song sheet in such perfect harmony. The direction and focus of the company, enabled by the intelligent and strategic use of its data, is assisted by that sprinkling of magic that underpins the whole operation. Thanks to this, Shazam has ensured that it won t be hearing the dulcet tones of the fat lady singing any time soon. 16 THE 1 st Big Data Insight Group INDUSTRY TRENDS REPORT

17 INTERVIEW DAVID BOYLE Howdy David Boyle: partner David Boyle is what you might call a data fanatic. At the moment he holds two jobs: senior vice president of insight at the record label EMI and director of insight at start-up company zeebox creators of an application which combines TV viewing with live data about the show and social media. With Facebook and Twitter functionality built into the application, users can comment on the TV programmes they watch and also see what their friends are watching. But the really smart bit is that the application brings up a host of hyperlinks for more information on a myriad of different things that are happening at that very moment. And its scope is boundless. Say, for instance, a soap storyline is playing out, the user might be presented with links to the actors profiles, more information on the discussion points they are engaging with or the music playing in the background. zeebox can monitor users journey through these links, the comments they make on the social media sites, and their overall TV viewing and engagement patterns. Yet while many companies would look to use this information solely to tempt advertisers onto the application with increased demographic insight, Boyle sees wider opportunities. We look to set up partnerships with people Having held senior insight positions in a wide range of different industries, David Boyle has learnt a few things about the best ways to use data. He talks to Mark Young about the importance of partnerships, scalable infrastructure and transparency. that can get actionable insight from the data, he says. That might be an advertiser, but equally it might be a charity. Someone like an aid agency would, for instance, stand to benefit greatly from knowing firstly where a crisis is featured within television shows and secondly how different audiences react to the information or portrayal. The premise, like the dashboards zeebox creates for presenting the information, is the same for any type 17

18 INTERVIEW DAVID BOYLE of organisation advertiser, charity or anything else alike and that means the process is easily replicated for multiple different partners and projects. The only difference is that some organisations are going to pay a premium for that information and others aren t, says Boyle. Other than that, Boyle is coy when it comes to talking about the commercialisation of this data. Instead of cold hard cash he prefers to bring the conversation back round to one key word: partnership the modus operandi that he sees as the only truly effective way to operate with data. At any rate, I m not sure that direct monetisation is the correct thing to do with data, he says. Partnerships to make use of the data in the right way are a lot more productive. It s far better to help people understand and draw conclusions. Then they ll have a lot more benefit and they will then help us, whether that s with promotion, advertising or money itself. You don t get much value from data unless you really understand it and you know how to make use of it a cold data transaction won t yield anywhere near as much value as a true partnership. This ideology certainly served him well at EMI. As little as five years ago the music label was struggling. Like most of its peers, EMI initially resisted the effects of the digitalisation of music instead of creating new revenue streams in this evolving environment. A high profile takeover by a venture capitalist firm was seen by many as a last-ditch effort to save the company and a long shot at that after all, what do a bunch of suits know about running a record label? How to stop one going under, apparently. Terra Firma introduced a culture within EMI where data is at the heart of every decision. It was then Boyle introduced partnerships. When I joined in 2009 we weren t really talking to consumers and weren t using them to help us make decisions, he explains. We were collating data and then sending that to the business decision makers and telling them how things should be done. So we reinvented it from scratch with a focus on buy-in and partnering with the business helping them to make decisions based on their needs, rather than tell them they are making wrong ones based on the data. If EMI was slow on the uptake of Internet music trends to begin with, it is fully on board now. Take the popular song streaming service Spotify through which EMI, like the overwhelming majority of record labels, makes all of its recordings available at practically no cost to the user. EMI does get a monetary return for this but, perhaps more importantly, it garners swathes of customer trend data. In the same way as Google can create profiles for the people using its search engine, EMI can track You don t get much value from data unless you really understand it and you know how to make use of it David Boyle, senior vice president of insight, EMI and director of insight, zeebox individual journeys (without personal details) through Spotify. Unlike the label s executives that preceded him a few years ago, Boyle rejects the notion that digital music channels result in fewer revenues. The Spotify model engages people with music and they then go out and buy more of it and a bigger range than they would have before. And we have the data to help them find things they want to engage with. Everybody wins. Rinse, repeat Boyle also advises that data projects need to be scalable and repeatable. All too often insight is a slow process, he says, and the project is a big one-off with only certain people and departments involved. Every time you do something there will be big set up costs. Boyle s team at EMI sends out packs of research results to a wide range of people around the business every day. These are also available to all staff everywhere in the world online. Included is a wealth of different information relating to lifestyle, attitudes, media engagement in the widest possible sense, affinities, songs, price expectation, reactions to different versions of songs and anything else that any individual around the business has asked for. If you ve got enough scale you can ask anything you want in research, he says. His final piece of advice is that companies need to ensure that they aren t creepy with people s data. The best way to do that is transparency not selling personalised data, only sharing trends, being open about where data is going and who it is going to. When you get good at working with data, there may be a cross to bear. Making visualisations for data is Boyle s favourite part of his job but it s also the least important and he now gets to do it less and less. Still, with untold levels of data to play with as zeebox continues its assault on the social TV market with its partner-led paradigm, there should be enough to keep his love affair with consumer insight alight for a while yet. (An extended version of this interview is available at 18 THE 1 st Big Data Insight Group INDUSTRY TRENDS REPORT

19 FeaturE democratisation of data Data is now available on a mass scale to all enterprises, large and small alike. Those that make the most intelligent use of it to piece together the puzzle will thrive, not just survive. Mark Young explores. Our world has been enriched by data in innumerable ways. However, while its effects are far reaching, there has only been an exclusive list of companies who have been at the forefront of big data. Up until now, complex data capture and analysis has been the reserve of governments, academics and mega rich corporations. They alone have had the resources necessary to implement the intricate and expensive infrastructure big data requires. But things are changing. Data is becoming democratised a wealth of it is now available for one and all. For instance, with user-interface-friendly web analytics tools, many of them available for free, anyone running a website can review the hit rates and demographics of their audience. Moreover, they can analyse what users do during their visits, how long they stay, what they click and where they move onto next. In the same vein, there s a host of social media monitors that can report exactly what s being said about a company by its audience across the online chatosphere. Furthermore, we are now in the age of the Internet of things. Countless everyday objects are now equipped with radio frequency identification (RFID) tags and wireless Internet technology. This allows them to continuously log data to a 19

20 FeaturE democratisation of data Challenge: Size With the floodgates open, you can quickly go from having no data to finding yourself in an ocean of it. This can render your efforts counterproductive. Because you need to spend so much time sorting through the data you ll never get any value from it. A strong plan of action that is linked to business objectives is essential. What sets companies like Tesco apart, says Bernard Marr, is that while they may have tonnes of data, they use less than 10 per cent of it. What they do use is tightly aligned to their strategy. They spend a lot of time asking themselves what do we need to know? before they go off and drown in it. Widening the scope The advances in big data allows any business of any size rather than just those that can invest millions into it to make decisions based on facts, rather than perceived wisdom. It therefore offers competitive advantage to those that embrace it. As well as improving processes, data allows companies to introduce completely new revenue lines and even begin to influence the shape of the vertical industries it touches. Shazam and zeebox, as you will have read in this report, are perfect cases in point. These companies partner with others that can benefit from their data by understanding what users react to and how they converse about it. network on their performance and their surroundings data that can be used for wider contextual understanding. It s not just their own data that a company can benefit from either. Many other organisations, including government bodies, are making their data publicly available in the interests of transparency. In fact, the UK government has appointed a Public Sector Transparency Board with an agenda to make all public data accessible and open for commercial use, in any lawful way. Part of this is the midata scheme a voluntary set of standards with 26 corporate founding signatories including the likes of British Gas, Google, Royal Bank of Scotland and Visa, which includes a commitment to sharing data for commercial gains. This is actively encouraged as part of the government s private sector growth agenda. With data sources markedly increased, the democratisation is furthered by the mass availability of low cost technologies for working with the data. The advent of cloud-based computing and storage, complemented by distributed file and computing systems like Apache Hadoop and Google MapReduce, which spread computational tasks over potentially limitless processors across the globe, means that elaborate mathematical queries can be carried out in near real time on a pay-as-you-go pricing model, ostensibly opening it out to the masses. For John Paul Tointon, managing director of fast growing cloud recruiter JPE Recruitment, the rising mobility and interoperability of data is driving democratisation. Now you can enter or monitor data within the palm of your hand, he says. Since the 1980s people have been talking about the next killer app. Now we are going to see the mega app which will allow people to automate the entry of data across all platforms, filter relevant data, and automatically disperse it to the right people in the organisation so they can action it. Now we are seeing that come to fruition. Challenge: Context An abundance of data at your fingertips could also offer you false confidence in your convictions. You can always find data to prove just about any point you want to make, but you must ensure that you approach your analysis objectively. Dr Neil McBride, a Reader in information technology management at De Montfort University, says that organisations must be careful in assessing the context of the data they are using. The critical issue is interpretation, says Dr McBride. We understand things within the context of our own culture. You have to understand the reasons that led somebody to create their data and their intentions for doing so. A hospital might, for instance, report certain figures to ensure that they are meeting targets but that data may not be strictly accurate or may have disregarded other extenuating factors. If a business makes strategic decisions based on that data, they may be doing so erroneously. You need to consider the wider picture. Unfortunately, that kind of deep analytical understanding may be in short supply, as Mark Whitehorn outlines in this report. Indeed, McKinsey predicts that by 2018 there will be a shortage of 1.5 million managers and analysts with the necessary skills to turn big data into strategic information. 20 THE 1 st Big Data Insight Group INDUSTRY TRENDS REPORT