Big Data: What defines it and why you may have a problem leveraging it DISCUSSION PAPER

Transcription

1 DISCUSSION PAPER

2 1. Enterprise data revolution One of the key trends in the enterprise technology world at the moment - and one that has been steadily growing in influence and importance in the past few years - is Big Data. Technologies that can help firms take advantage of this trend are set to be one of the biggest drivers of growth in the IT industry in the coming years, as recognition grows of the impact better use of information can have on a business. Big Data is moving beyond the scope of experimentation and early adoption to become a 1 key consideration at the heart of many new IT projects. Figures from Gartner estimate more than four out of ten companies have already invested in Big Data solutions or intend to do so within the next 12 months. As a result of this, the market is expected to be worth $23.6 billion by a compound annual growth rate of 31.7 per cent and around seven times faster than the overall information and communications 2 technology market. However, many companies may still be unsure of exactly what defines Big Data, what separates it from previous information-gathering solutions and what they need to be doing to make the most of it. The key reason why Big Data is such a high priority for businesses across all industries is due to the need to manage the ever-growing amounts of information that are available to firms. Every day, users around the world generate around quintillion bytes of data and this is only set to grow as more people get online and the number of connected devices boom. By 2017, it has been estimated that global traffic from mobile devices alone will climb to 134 exabytes a year 4 - a 13-fold increase on 2012 levels. Also by that year, half the world's population will be connected to the internet and total global IP traffic will be three times 5 higher than today. Much of this will be in the form of unstructured data from social media sites, pictures, videos, sensors, location trackers and mobile phone signals, to name just a few. As a result, organisations will often find it extremely difficult to get this information under control and use it to benefit the business. There are essentially two challenges involved when it comes to working with Big Data that create problems for firms when it comes to leveraging this for real-world results. The first of these is the actual task of gathering the information. This involves ensuring that data warehousing solutions are up to the task and that digital assets are stored in a logical, consistent and easy-to-access form. Manufacturing opportunities, deployment, activation and ongoing management of millions of devices, and the analytical opportunities arising from massive streams of potentially real-time information, all represent huge untapped business opportunities. - Stephen Prentice, Vice-President and Fellow, Gartner 1 Gartner: Gartner Survey Finds 42 Percent of IT Leaders Have Invested in Big Data or Plan to Do So Within a Year (03/13) 2 IDC: New IDC Big Data Technology and Services Forecast Shows Worldwide Market Expected to Reach to $23.8 Billion in 2016 (01/13) 3 IBM: Big at the speed of business 4 Cisco: Cisco Visual Networking Index Forecast Projects 13-Fold Growth in Global Mobile Internet Data Traffic from (02/13) 5 Cisco: Cisco's Visual Networking Index Forecast Projects Nearly Half the World's Population Will Be Connected to the Internet by 2017 (05/13)

3 However, this is only half the challenge, as it will then be essential for firms to analyse this information carefully in order to draw out usable insight than can help inform their decision-making and improve key business operations. As many firms now rely on real-time access to this insight, speed is also an issue. While many firms may have the capability to assess information manually, the time it will take to do this means the results are often outdated and irrelevant by the time they are produced. It is in converting raw data into insight where many investments in Big Data tend to fall down, as companies often do not have the technical knowledge or business understanding to make Big Data work for them. Gartner research estimates fewer than ten percent of enterprises have a clear strategy for how they will use their information, with the market research firm adding companies will need to reconsider how 6 they view data in order to get the best results. Enterprises must be willing to innovate in the way they perform decision support and analytics. This is not a technology challenge, but rather a process and change management challenge. - Hung LeHong, research vice-president, Gartner 7 When discussing Big Data operations, enterprise users will often see the challenges referred to in terms of the 'three Vs' of information. These are volume, variety and velocity and, between them, they sum up the major hurdles that must be overcome if companies are to make a success of their investment in Big Data. So when planning a Big Data strategy, companies need to be asking questions to ensure they have a clear understanding of all these challenges and a clear roadmap for meeting them. 6 IDC: New IDC Big Data Technology and Services Forecast Shows Worldwide Market Expected to Reach to $23.8 Billion in 2016 (01/13) 7 Gartner: Gartner Says Less than 10 Percent of Enterprises Have a True Information Strategy (06/13)

4 2. Volume - Drowning in data The first consideration for many firms - and what is often the most visible result of Big Data strategies - is how they will deal with the sheer amount of information available to them in today's environment. Every minute, 639,800GB of data is transferred around the globe. This includes 204 million s sent and received, 100,000 new Tweets, 47,000 app downloads and 30 hours of video uploaded to 8 YouTube. Therefore, one of the first questions businesses must ask themselves is: More than two-thirds of the data generated from what IDC refers to as the 'Digital Universe' in 2012 was created by consumers, so businesses that deal with these customers stand in a great position to be able to utilise Big Data. The company estimates that around a third of this information will have tangible business value if exploited properly, but despite this, only a tiny fraction 9 is currently analysed. By some estimates as little as one per cent is effectively studied in order to offer usable insight. "Do we have the facilities in place to cope with this?" What some of the world s biggest websites handle every minute One thing goes without saying, the amount of data that we are generating and storing is increasing exponentially. Exactly how much by is a matter of contrasting stats and figures, but one of the most commonly stated facts is that today we generate as much electronic data every single day as everything that had been created prior to Dominic Pollard, Editor, Big Data Insight Group10 For many businesses, the solution to this will be to ensure that they are able to effectively filter out the large quantities that will not be relevant to their needs and focus on what will provide the best results for their specific requirements. But how do firms know which of the vast array of data is the most relevant? This leads into the next key focus area - variety. 8 Intel: What Happens in an Internet Minute? (03/13) 9 IDC: The Digital Universe in 2020: Big Data, Bigger Digital Shadows and Biggest Growth in the Far East (12/12) 10 Comms Express: Data volumes 'continuing to rise' - Data & Server Racks (09/12)

5 3. Variety - A world of information When you think of data, where do you it imagine it is coming from? In the current environment it is not just tried and tested sources, as what separates Big Data from traditional business intelligence is not only the amount of information, but the increasingly wide range of sources that businesses will have to deal with. It is no longer just about gathering details such as customer profiles, buying habits and market trend information. A great deal of the information firms now have available is unstructured and goes far beyond simple text entries or figures. This is particularly true for enterprises with a strong social presence, as the huge amount of details that can be gathered from these sources - such as Tweets, status updates, photos and videos - can provide a level of insight that was not previously possible. However, what this does mean for companies that are able to harness this data effectively is there are many uses to which the information can be put. The digital universe is made up of images and videos on mobile phones uploaded to YouTube, digital movies populating the pixels of our high-definition TVs, banking data swiped in an ATM, security footage at airports and major events such as the Olympic Games, subatomic collisions recorded by the Large Hadron Collider at CERN, transponders recording highway tolls, voice calls zipping through digital phone lines, and texting as a widespread means of communications. - IDC, The Digital Universe in Being able to gather and unify data from a range of different sources and types enables firms to reassess marketing campaigns, spot emerging trends, aid in product design and highlight any areas where the customer experience can be improved. And this does not just come from traditional gadgets, or even new technologies such as tablets and smartphones. One especially disruptive trend set to have a huge impact on the amount of data collected in the coming years is the 'Internet of Things'. This will involve an increasingly diverse range of embedded devices and communications - many of which will involve machine-to-machine transactions sending and receiving huge variety of data about a huge variety of factors. The diversity of this information will pose a major challenge. The sheer number of potential applications using this is so broad is it easy for firms to become overwhelmed, resulting in them missing immediate opportunities because they do not have the right tools. Therefore, even if businesses have made steps to cope with the increasing volume of data, being unable to gather and analyse information from multiple, non-traditional sources is often where Big Data projects run into problems. Being able to understand all of this and pick out the most appropriate information for your business will be essential to success. However, this alone will not guarantee the best outcome unless it can be done in a timely fashion, which is where the third of the big three Vs comes in. Manufacturing opportunities, deployment, activation and ongoing management of millions of devices, and the analytical opportunities arising from massive streams of potentially real-time information, all represent huge untapped business opportunities. - Stephen Prentice, Vice-President and Fellow, Gartner IDC: The Digital Universe in 2020: Big Data, Bigger Digital Shadows and Biggest Growth in the Far East (12/12) 12 Gartner: Gartner Says Potential Size and Diversity of the Internet of Things Mask Immediate Opportunities (07/13)

6 4. Velocity - real-time results for an evolving world Big Data: What defines it and why The speed at which information can be analysed and converted into usable insight could make or break a Big Data project. Think about how long it has taken your firm in the past to turn raw information into actionable results. The chances are it was a lengthy process, but this is no longer good enough - it is no use being able to interpret information to find the right answers if it takes so long to do so that the results are already out of date by the time they are presented. Businesses now operate in a real-time world where being able to make the right decisions instantly and adapt the second conditions change is vital. Therefore, Big Data analytics solutions need to reflect this culture and ensure companies are able to gain immediate access to their data. The performance of data solutions is said to be the biggest challenge for organisations, cited by 24 per cent of firms, with being able to get real-time access just behind, at percent. Key reasons why businesses need real-time data in their organisations Good Big Data solutions can also dramatically cut the time it takes to process large amounts of information. For example, the European Space Agency needed the ability to gather and analyse as many as five billion records - each one about 300 bytes - on the fly. With Big Data solutions, it was able to achieve this in around 12 and a half hours - a time frame that simply would not have been possible in a pre-big Data era. Real-time data provides an accurate view of what is happening in the business right now. It therefore makes sense that so many would identify decision-making and accountability as their top drivers for real-time data since both are exceedingly dependent on timely and accurate information. - Ziff Davis, Strategic IT Survey on Data Management13 13 Ziff Davis: Strategic IT Survey Results: Data Management (11/12)

7 5. Meeting these changes - taming the wild west of Big Data Being able to handle the volume, variety and velocity of information that is coming into businesses can help firms unlock the full potential of their data and ensure companies are able to derive value from their intelligence - which can be thought of as the fourth V that is necessary to make Big Data work. Tied into this is yet another V - veracity. In other words, how can you be sure that the data you're using is reliable and bringing real value to your business? IBM figures suggest as many as a third of business leaders don't trust the information that is central to their decision-making, so gaining reassurances about 14 the quality of data is also vital. [Businesses] tend to think that the amount they harvest, the speed they gather it and the range of information is important; but volume, velocity and variety don't matter as much as employing better governance over the data collated, or understanding what is needed to grow the business or mitigate risk - Eddie Short, Head of Business Intelligence, 15 KPMG Organisations that use the wrong data, misinterpret information or do not ask the right questions could end up making poor decisions and doing their company more harm than good. However, getting this right will require investment in not only the latest technology solutions, but in ensuring firms have the right staff on hand to make the most of these tools. With so many solutions and strategies offered for dealing with Big Data, many analysts have likened the current situation to the 'Wild West', with many enterprises scrambling to get involved with the technology before they have a real idea of how the tools will work for them. Therefore, understanding how to harness this information and having the right technologies and expertise in place will become an essential part of how any firm does business in the coming years - no matter what sector they are in. Firms need to look at each of the key points that define Big Data and ask themselves if they are ready. Can you cope with increased volume? Do you understand how the increasingly-connected world will affect your data gathering? Are you responding quickly enough? Whether it is marketing studying customer demographic data to create more tailored advertising campaigns or healthcare providers analysing patient details to help with diagnoses and the formulation of personal treatment plans, Big Data will be central to the future of everyday operations, so the sooner businesses look to deploy and get to grips with the technology, the better-placed they will be. 14 IBM: Big at the speed of business 15 KPMG: Businesses at risk from poorly harnessed Big Data, says KPMG (06/13)

8 About InterSystems Big Data: What defines it and why InterSystems is a global software leader that provides an advanced platform for the rapid development of breakthrough applications. Developers can seamlessly embed and extend the platform s capabilities into new and composite applications for use on-premise, in the cloud, and via Internet-connected devices. These capabilities include workflow, business process management, business rules, data integration, real-time analytics, and more. Due to the cohesive, robust design of InterSystems platform, applications benefit from exceptional levels of interoperability, flexibility, and performance. InterSystems provides 24x7 worldwide support to millions of users around the world. Please contact us directly for more information: InterSystems UK InterSystems House, Tangier Lane, Eton, Windsor, Berkshire SL4 6BB England Voice: +44(0) [email protected]