Why small data is the answer to big data s problems.

Transcription

1 Why small data is the answer to big data s problems. Sponsored by Executive Summary With the huge focus on big data over the past year, it can be seen that data itself is becoming a new type of corporate asset that represents a key basis for competition. But big data is only part of the story. What we re finding is that small data is actually playing as important, if not a more important role, than big data. With most organisations struggling to handle the volumes of data they already have, the priority isn t overlaying more data, but using existing data to deliver better business outcomes. With the proper tools, protections and incentives this can now happen, and small data can become a shared network of knowledge and insight that grows every minute of every day. It can be used to deliver better customer experiences and help the organisation react faster to market forces and opportunities. For most organisations, harnessing small data is the most effective route to better productivity, faster innovation and greater agility. Key take-out points Thanks to the big data phenomenon, many businesses are looking at more effective ways of extracting value from data. This is putting small data in a favourable perspective. Much of your structured data will already be available to your business in desktop locations and in spreadsheets for example. People should be able to access, compile and start using this information quickly and easily. In reality, you re probably attempting to access this small data already, but without fast, effective and cost efficient ways to do this you re not leveraging the full potential from your data.

2 Data is the new oil Industry experts are likening the big data boom to the early days of big oil. Just as oil became crucial to the development of the industrialised economy, data is now critical to the modern information age. As with the big oil boom, where entire industries grew to source, mine, extract, analyse, refine, distribute and sell oil, similar service industries are growing to meet our data management needs. As a comparison it makes perfect sense, because data is the life-blood of the organisations and institutions that we ve established around the world to serve our society. An article recently published by McKinsey 1 paints a picture of a fictitious organisation operating today. It explains how reams of data flowing from financial transactions and customer interactions are cascading into the organisation at unparalleled rates from new devices and multiple points along the value chain. Sensors embedded in process machinery are collecting operations data, while marketers are scanning social media channels or using location data from smartphones. Data exchanges are networking the supply chain partners, and employees are swapping best practices on corporate wikis. The article ends with the conclusion that all of this new information is laden with implications for leaders and their enterprises. What we re seeing is a game-changing trend caused by massive investments in our ability to collect, integrate and analyse data. By constantly testing, bundling, synthesising and making information instantly available from the store floor to the CFO s office businesses can become far nimbler and much more responsive. Data becomes a competitive asset McKinsey suggests that, Over time, big data may well become a new type of corporate asset that will cut across business units and function much as a powerful brand does, representing a key basis for competition. 2 This point of view is backed up by emerging academic research that suggests that companies who use data and business analytics to guide decision-making are more productive and experience higher returns on equity than competitors that don t McKinsey Global Institute report, Big data: The next frontier for innovation, competition, and productivity 2. McKinsey: Oct 2011: Are you ready for the era of Big Data? 3. Erik Brynjolfsson, Lorin M. Hitt, and Heekyung Hellen Kim, Strength in numbers: How does data-driven decision making affect firm performance?, Social Science Research Network (SSRN), April In this study, the authors found that effective use of data and analytics correlated with a 5 to 6 per cent improvement in productivity, as well as higher profitability and market value

3 The challenge facing organisations is how to use this data. We need to start thinking about whether we re prepared to exploit data s potential and to manage the opportunities and threats it can pose. Success in making better data-driven decisions will demand not only new skills but also new perspectives on how data should be managed, mined, analysed, and shared. To avoid being left behind, companies are rushing to cash in on the information they glean from customers and vendors are stepping up to help. The venture capital community has taken note of this trend: Over the past year, investors have poured hundreds of millions of dollars into start-ups that promise to exploit caches like Facebook s reported 100 petabytes of data. 4 Big data has some big problems too. Big data is only part of the story. According to the Harvard Business Review 5, big data doesn t work if you ignore the small things that matter. And now we re finding that small data is actually playing as important a role, if not a more important role, than big data. The hype about big data is obscuring the real issue of enabling intelligent and instantaneous analysis to provide optimal insights for business decisions. CIOs need to ensure they re looking at these high-volume, high-velocity challenges in the right way: as business enablers, not tech projects. 6 Even though they are armed with the hyper-intelligent weapons of big data, many companies haven t done much to improve the experiences they deliver to their customers, nor are they finding that they can react any faster to market forces or opportunities. Negative interactions with companies are as common as ever. A reported 86% 7 of consumers have switched companies after having a bad customer experience. Most Australian organisations can barely handle the volume of structured data they already have, let alone handle the overlaying of unstructured data onto it. They would deliver better customer experiences if they spent less time worrying about big data and more time making good use of the information that s already available to them. 4. Jacques Bughin and Michael Chui, The rise of the networked enterprise: Web 2.0 finds its payday, mckinseyquarterly.com, December Big Data Doesn t Work if You Ignore the Small Things that Matter via HBR.org by Robert Plant on 10/4/12 6. Forbes: The Top 10 Strategic CIO Issues For Big Data Doesn t Work if You Ignore the Small Things that Matter via HBR.org by Robert Plant on 10/4/12

4 Small data is the answer to big data s problems According to Adam Murray, General Manager at C3 Products, small data can be defined as the structured data that organisations generate internally or obtain from external entities, such as the Australian Bureau of Statistics. This data is usually already available to people, on their desktops or in spreadsheets, for example. The problem is accessing and using it in meaningful ways. The era of small data begins is an E-conference blog by Stanford Graduate School 8. It discusses a situation in which small data greatly improves the capacity and performance of governments, non-government institutions and commercial organisations: Timeconsuming forms and other inefficient data practices can be eliminated; public health and education can be improved by leveraging the power of more accurate and complete data; and in a commercial situation, small data can provide the ability to transform advertising into a more respectful, less disruptive industry that rewards people for their purchases. The blog ends with the conclusion that companies who play by these new rules will be rewarded with access to rich and robust data otherwise unavailable, giving them competitive advantages. Small data in action We want this paper to deliver as much practical information and insight as possible. So in this section we ll discuss five situations where small data has been used to solve real organisational or operational challenges. When quality is more important than quantity A major element of many organisations business intelligence strategy is being able to access accurate information from non-source systems in a timely manner. In 2009, the Office of Environment and Heritage NSW 9 commenced a Business Intelligence Program to improve the department s reporting and analytical capabilities. The organisation is responsible for a geographic spread across 6.8 million hectares of land in 800 different locations throughout the state. Using small data systems it is now able to develop a fast, easy, flexible platform to capture and upload data data it already possessed, but was previously unable to access and use in intelligent ways. Amongst other things, the organisation can now track wildfires day to day across the entire state. This information can then be fed through to the NSW Fire and Rescue Services for a prompt, and often life saving, response. 8. The era of small data begins Stanford Graduate School of Business E-Conference blog 9. C3 Case Study for the Office of Environment and Heritage NSW, 2012

5 When depth of insight, not breadth of data, is required The vast majority of organisations don t actually need more data, they simply need more variables to apply to the data they already have. This type of fat data is rich, deep and full of insightful information about the organisation. It enables people to learn more, look further and use information in more varied ways. Reducing the amount of data being used in analytics and honing in on specific signals is an effective way to develop smart data. Being shrewder with the data already available, rather than going out and gathering more, is the best way to avoid what Nassim Taleb, author of Antifragile, calls noise. 10 When groundwork matters more than guesswork Big data is about estimating and predicting trends. Small data is a solid foundation on which people can make better, faster, more informed decisions. In a world where yesterday s data is like yesterday s news and users are accustomed to instant updates, small data offers more immediate decision-making support. Small data contextualises business operations and allows organisations to make informed decisions using quality information that pertains directly to their business. In an interview with Fast Company, 11 data planner Dimitri Maex described a project for Las Vegas s Paris Hotel in which his team ran through all of the positive reviews on Trip Advisor and categorised what people enjoyed about the hotel. After analysing this small data, Maex changed the advertising copy and website homepage to feature fountain views and views of the strip. This small change resulted in a 20% increase on the return of its online advertising. 12 When a better return on investment is important Whereas big data requires substantial storage and a plethora of tools, processes, protocols and systems to manage it, small data systems can be set up quickly and cost effectively. In the example given previously for the Office of Environment and Heritage NSW, the organisation was intelligently using its data within days. In the case of Terry White Chemists 13, the business needed a quick and easy stock master data management solution to deliver its online shopping offering. Within a day it was uploading and merging information from different sources. The solution immediately started delivering a return on the investment because it removed the need for manual stock master data management. 10. Taleb, N. Antifragile: Things that Gain from Disorder, Making The Information Firehouse Manageable For Data-Driven Decisions, Fast Company, Baer, D. Making the Information Firehouse Manageable for Data-driven Decisions. Fast Company. fastcompany.com/ /making-information-firehouse-manageable-data-driven-decisions 13. C3 Case Study for Terry White Chemists, 2012

6 When you need to acquire and combine data from different sources Acquiring and combining different types of information from different sources, or the same information from different sources, usually requires an entire transformation process. The NSW Public Service Commission (PSC) 14 needed a tool to collect workforce profile data from all state government agencies in NSW. What made this more of a challenge was that these agencies had hundreds of thousands of rows, and were running over 100 complex validation rules against this data. The small data solution developed in this case was able to process this data in a matter of minutes, providing, ultimately, a single source of truth. How do you start embracing small data? Many businesses are renewing their focus on the value of data. While it s exciting to salivate over the new technologies behind big data, the reality is that most people and organisations simply need quick and easy access to the information they already have and to start using it in intelligent and productive ways. Before you go any further in your search for a big data or small data solution, ask yourself some questions: 1. How much is bad data costing my organisation? Industry analysts have studied the long-term costs of inaccurate data entry and have concluded that it simply costs too much money not to have a cleansing solution at the beginning of your data collection chain If much of the data my organisation needs on a daily basis is already in spreadsheets, how confident am I that the processes we have in place for acquiring, compiling and managing this data are quality controlled? One study in early 2011 of nearly 1,500 people in the UK found that 57% of spreadsheet users have never received formal training on the spreadsheet package they use. 72% said that no internal department checks their spreadsheets for accuracy. Research studies estimate that roughly 94% of spreadsheets deployed in the field contain errors C3 Case study for The NSW Public Service Commission (PSC), The real cost of bad data: The rule Melissa Data 16. Stephen G. Powell, Kenneth R. Baker, Barry Lawson ( ). A Critical Review of the Literature on Spreadsheet Errors

7 3. How efficient are the processes and systems for pulling all this data together? Can it be faster and more reliable? With proper tools, protections and incentives, people can use information that wasn t previously available to them, they can gain access to usable data and they can contribute their data to bigger projects in this way, creating outcomes that are bigger and more profound than their data alone. Small data can actually become a shared network of knowledge and insight that grows every minute of every day. For most organisations, this is the most effective route to a more productive, innovative and agile organisation that maintains its relevance in a rapidly changing information-driven age.

8 unleash the potential of your smalll data start today! Integrity from C3 Products is a purpose-built data acquisition and validation tool. It allows you to pull-in data from hard to reach places, then to collate, validate TO FIND OUT MORE, OR TO ARRANGE A SHORT DEMO, CLICK HERE, OR CALL THE INTEGRITY TEAM TODAY ON and make it easy for others to use. Because Integrity automates this entire process it is the fastest and most efficient way to unleash the potential of your small data. c3integrity.com