Big Data: Key Trends and Practical Implications.

Transcription

1 Analytics. Consultancy. Engineering Big Data: Key Trends and Practical Implications. Tel: +44 (0) Contact:

2 Introduction The Evolution Of The Big Data Landscape The early evolution of the Big Data world was driven by a core group of large Internet companies (Google, Facebook, Twitter) who had huge volumes of data and no legacy infrastructure. They were able to recruit top data engineers and build their own technologies for dealing with the data. Not only were those large players heavy users and innovators for Big Data technologies, but some of those engineers also left and formed their own startups. The current Big Data landscape reflects a maturity point where a much wider group of organisations have adopted or are looking to adopt Big Data approaches and technologies. Many large companies are starting to invest, but do not have a clean slate in terms of infrastructure, which adds complexity. The business case is also more difficult for areas where the existing infrastructure is not broken and still copes. The understanding is that it will need replacing, but it is typically not going to be an overnight process, especially where critical operational systems are involved. As we will explore further down, it is also worth noting that successful implementations of Big Data approaches are not simply about buying a piece of kit and implementing a new piece of technology such as Hadoop, but rather depend on a combination of technologies, people and processes that work together in harmony.

3 5 Key Trends in Big Data 1. Breadth & Scope Starting with perhaps the most obvious, we can see there continues to be more players joing the market, while many of the existing offerings have matured. What does this mean for organisations? No single stack solutions are likely to cover all your needs and certainly won t compete with best of breed solutions. This is not a surprise given the variety of topics and specialisations in the landscape: as disciplines are still maturing in many areas, it would be misplaced to expect one solution to cover everything. Common understanding and language is required to navigate the various areas within disciplines - for example within Analytics you might have Tableau listed in Visualisation, but not in BI; headings for Big Data technologies include the fairly generic Analytics Platforms, including Microsoft which would drill down into a solution architecture diagram all of its own, alongside Statistical Computing, Machine Learning and Data Science. For a business compiling a shopping list it is not easy to understand which tools are likely to be best for their needs or if they will need multiple tools to get the job done. Do you need proprietary tools or will open source work for you? For example, you have open source Stats and Machine Learning tools like R, KNIME and Weka to rival SAS, SPSS and Azure to name a few examples. The open source tools aren t fledglings either - R has 3 times the number of users as SAS or IBM s SPSS who have been using it as their primary Data Mining / Analytics tool in the past year. 2. Data Integration strategy Data integration strategy is a hot topic - many large organisations acknowledge the challenge of still having data in siloes and having a number of data integration challenges to overcome in providing joined up customer analytics across the customer journey. Aspirations in this area appear to be running ahead of the core thinking required on data architecture, plumbing and governance. There is no shortage of data, but rules and logic are now needed to drive automation and effective use of that data. While building the strategy in these areas is fundamental to long term analytics success, it is not a simple thought given the number of moving parts, number of internal stakeholders involved, potential politics and the variety of data types being collected that need a common lexicon to allow for meaningful integration. It s also a project that needs balanced cooperation between business functions and is seldom successfully led by one without sufficient representation from others. 3. Return on Analytics From an analytics perspective, senior management will invariably want to understand the likely returns from their investments in terms of tools, resources and process change. As the landscape continues to mature, a shiny new piece of kit will cease to suffice as a business case for new implementations, especially if there is already an organisational migraine occurring from the data strategy challenges mentioned above. To establish clear returns from analytics investment, analytics strategy needs to be closely married to data strategy, with very well-articulated objectives and use cases. Business cases for new tools and application in isolated siloes are increasingly unlikely to fly.

4 4. Infrastructure Innovation The underlying infrastructure has had some time to mature given that Google published work on MapReduce around 10 years ago, leading on to the creation of Hadoop. However, there still continues to be plenty of innovation and improvements in this area. Over the past year we have seen Apache Spark become very popular and successful thanks to its many fold (10 to 100 times) speed increase over Hadoop s MapReduce in some use cases as well as the fact that it is easier to use. 5. Machine Learning & AI Is a really hot topic at the moment as it addresses the question of how to sift the data and find valuable actionable insights for the business that will actually drive ROI as well as just sounding cool. This is not a new discipline (IBM were developing machine learning programs for game playing in 1952 and in the 90s the machine learning became popular again as Computer Science and Statistics began to intersect). As a result of the exponential growth of data we are now seeing, statistical AI is rightly at the heart of making sense of the exabytes of data being generated and increased need for automated learning and the sheer speed of data flows. Organisations now need to find, analyse and validate complex patterns in the sea of Big Data. This is where Data Science fits in, in terms of developing and implementing models to mine the vast amounts of data available to the organisation Areas of consideration for those looking to implement Big Data solutions: We still hear a lot of presentations about the various Vs of Big Data, but less talk about the next level of practical implications. The challenges for organisations are intrinsically around these key points: 1. Storage, Speed and Cost Fundamental building blocks for your data management strategy, the increasing maturity of Big Data technologies allow cheaper, faster and more agile solutions. That doesn t have to mean ripping out legacy infrastructure in one hit or making a massive one-time investment in a new stack. 2. Security You may have created a decentralised network of repositories or a centralised data lake with infinite depth, but ultimately you re responsible for the safety of customer data and the quality of data that is fed back to customers. In many ways data is like blood and as an organisation you re responsible for a blood bank, so you have a serious responsibility to check the quality of what s coming in, where and how it is stored, making sure it is not contaminated, and that the right type goes to the right person when they need it. 3. Data Strategy & Governance Including a consistent data lexicon and overarching strategy across the organisation. This comes in the category of boring but important, without it your business will look like a badly functioning 1980s robot with arms and legs operating to a separate set of instructions, rather than an agile organism with a central nervous system. 4. Customer Relevance You may have created a big enough data lake to drown planets in, but is it feeding the right rivers, streams and tributaries with the best customer insights at the touchpoints where they are needed, at the exact time needed? And how are you checking that is optimised?

5 5. Analytics Approaches & Algorithms Which to apply to which data, when and where across the business? And do they join up? Does your Voice of Customer analytics talk to your Customer Value analytics and your Channel Attribution model? Which models need to be real-time, which need reinforced learning, where are your business rules best managed? How do your analytics choices across the customer journey interact with your data strategy? 6. Customer Trust If data is the life blood of the organisation and that data is willingly donated and received back via transfusion by our customers, then we need to have a very solid and trusting relationship with those customers. Increasingly, it is understood that free is seldom truly free, as it is often more of a barter agreement for blood / data. However do businesses really mean it, when they say that they take your privacy oh so seriously. Do they believe in treating the customer fairly? Is their fulfilment of legal obligations a poor excuse for a more fundamental approach to trust that will surely be needed for good quality data to keep flowing in the ever expanding data world. 7. People & Skills We can t leave without a mention of this most important aspect. If data is the blood of the organisation, it s people that make up its body. Nanobots or flesh and bones, no organisation exists without them, well at least until AI develops that far. Data Science usually implies skills across a range of disciplines including: Data Engineering, Machine Learning, Statistical Data Modelling, BI, Visualisation, Business Analytics and Digital / Social Analytics. Finding individuals with the right hard skills along with communication and business facing skills is difficult in today s environment as those people are in short supply. Finally, are we still talking about Big Data? Some would argue that to continue talking about Big Data is to miss the point. The initial hype, some years ago now, was fundamentally about the plumbing needed to make good use of the data, then the technologies needed were relatively new and the theories employed (which weren t for the most part new) were mostly unfamiliar to many decision makers. Arguably, in today s world, if you are still having the discussions internally then you are already way behind. Like it or not, data is continually produced by every organisation and individual at an increasing pace, depth and quantity and variety. As the area continues to mature it is likely that the somewhat clumsy term Big Data will gradually get dropped.

6 About Lynchpin Lynchpin is an independent analytics consultancy. For over 10 years we have been helping organisations use data to make the best decisions, with capabilities spanning consultancy, analytics and data engineering. Lynchpin provides a completely joined up service and works with key stakeholders to achieve profitable outcomes. We pride ourselves on being vendor agnostic with an impeccable understanding of every available analytics technology and extensive experience of implementing and integrating them. Our team of analysts interpret data and use algorithms to find patterns and correlations to help you understand and better target your customers. Our services include customer, marketing and attribution modelling, optimisation and testing, data visualisation, measurement design, data transformation and analytics implementation. We work with over 25 organisations from different sectors including MTV, Standard Life, HSBC, Nationwide, John Lewis, Ralph Lauren, Hearst Magazines and Canon Tel: +44 (0) Contact: Info@lynchpin.com