How To Turn Big Data Into An Insight

Transcription

1 mwd a d v i s o r s Turning Big Data into Big Insights Helena Schwenk A special report prepared for Actuate May 2013 This report is the fourth in a series and focuses principally on explaining what s needed in the insights layer of a Big Data platform. For more information about how the insight layer relates to other parts of a Big Data platform please refer to the corresponding papers in this series: Navigating the Big Data infrastructure layer and Navigating Big Data business analytics. For more information about the opportunities and challenges posed by Big Data for organisations today, please refer to the first paper in the series, Unlocking the potential of Big Data. This is a special report prepared independently for Actuate. For further information about MWD Advisors research and advisory services please visit MWD Advisors is a specialist advisory firm which provides practical, independent industry insights to business analytics, process improvement and digital collaboration professionals working to drive change with the help of technology. Our approach combines flexible, pragmatic mentoring and advisory services, built on a deep industry best practice and technology research foundation.

2 Turning Big Data into Big Insights 2 Summary Understand the business requirement first The promise of a Big Data platform is that it takes data in its rawest form and converts it into consumable, actionable information Applying Big Data Insights within a business context Data visualisation tools can bring Big Data to life Your Big Data initiative needs to be framed by a clear understanding of how your data and analytic insights can be tied to a particular business goal, challenge or potential opportunity that needs addressing. It s important that there s a business context to your Big Data initiative, something that sets the scene and can therefore guide your choices about the right Big Data Insight technology for your organisation. The concept of a Big Data platform provides a technology framework for taking data in its rawest form, transforming it and putting it in a format where it can be consumed and acted upon by decision makers. Three core layers are required to support these capabilities: the lowest layer is responsible for the storage and organisation of data; the middle layer is where the analysis of that data occurs; and the upper layer is where data insights are discovered and consumed. This report focuses on the third: the insight layer. A business user s role, the type of analysis being performed and the context of use are all key factors determining how Big Data insights are served up to end-users. For example, self-service tools that enable users to build queries and develop reports are typically the preserve of more advanced users, whereas mainstream or more casual business users typically require a more context and role-based delivery interface where Big Data insights are integrated as part of an operational workflow or process. Your Big Data Insight technology choices need to take this business context into account. Data visualisation tools are becoming a valuable way to discover insights within your Big Data. They are adept, for instance, at helping users identify patterns, trends, outliers and relationships within the data. Successful data visualisation efforts also require a fresh approach to insight delivery, where visualisations are used to tell a story about the data and provide an environment for users to interactively explore and probe the data further.

3 Turning Big Data into Big Insights 3 Technology cost and sophistication driving the Big Data train As outlined in our previous report, Unlocking the potential of Big Data, in spite of the headlines and vendor rhetoric, the ability to manage growing volumes of data is not a new phenomenon for organisations today. In fact, many early adopters of BI and data warehousing (especially in the retail, telecoms and financial services industries) have long been accustomed to capturing and managing large volumes of data. Yet in spite of this we still see the rise and rise of Big Data as a seemingly relatively new concept so what has changed? Through their own technology innovations, web and social data-driven businesses such as Google and LinkedIn have shown us how to process Big Data sets (in their case web searches) on massively scalable storage and computing platforms using commodity hardware. Their technology expertise and success is the inspiration behind open source Big Data technologies such as Apache Hadoop and its ecosystem of tools (which we introduce in more detail in the second report of this series, Navigating the Big Data infrastructure layer). The challenge of processing certain kinds of Big Data has also driven other technology innovations related to massive parallel processing architectures, in-memory analytics, columnar databases and complex event processing platforms. All of these pieces bring more choices to organisations that want to advance their use and management of Big Data. Similarly, enhancements in predictive analytics, text mining and advanced data visualisation tools make the exploitation of Big Data more straightforward, by making it easier to discover hidden or interesting patterns that, in turn, can be used to enhance productivity, drive efficiencies and growth, and create a sustainable competitive advantage. Figure 1: Drivers of broader Big Data adoption Source: MWD Advisors But it s not only technology developments spurring the advancement of Big Data; as figure 1 shows the deployment economics of technologies are equally important. In particular, the decreasing cost of storage and memory, alongside the scalability of cloud computing platforms and appliances together with the growing influence of open source tools brings the promise of lower cost and more affordable Big Data platforms. The opportunities of Big Data are opening up to a wider audience, as it becomes more economically feasible to exploit, manage and leverage Big Data especially for those organisations that may have been priced out of this activity previously.

4 Turning Big Data into Big Insights 4 A Big Data platform has three layers Most of the commentary around Big Data has focused on the type of data under management whether structured or multi-structured (defined as data stored and organised in a multitude of formats, including text, video, documents, web pages, messages, audio or social media posts, and so on), or real-time or data in-motion. However, before any decision can be made about what kinds of information and technology capabilities are required there needs to be agreement and buy-in about what you want to achieve from your Big Data initiative. At the very least it needs to be framed by a clear strategy that helps outline how data and analytics can be tied to a particular business challenge or potential opportunity that needs addressing. This in turn provides the starting point from which organisations can assess the technical implications of their Big Data effort, for example by examining how data can be transformed from its raw state to a point to where it can be consumed and acted upon. To support this end to end process a Big Data platform needs to provide capabilities for: Capturing, processing and storing data Exploring and applying advanced analytics techniques Discovering and consuming insights. Today these capabilities are supported by a multitude of technology components some of them are relatively new, while others are based on existing technologies and architectures. In figure 2 we bring these concepts together as part of an overall Big Data platform with three layers. The lowest layer is concerned with organising and storing data; the middle layer is where the analysis of that data occurs; and the upper layer is where data insights are discovered and consumed. Figure 2: Capabilities of the three Big Data layers Source: MWD Advisors Although these capabilities aren t necessarily new to BI and data warehousing practitioners, it s become apparent that the old models for storing and analysing data don t necessarily apply to all Big Data assets. Not only is the amount of data vast and potentially more time-sensitive in nature, but the variety of data to be managed can be far greater and this is markedly changing the requirements of the technology needed. This report focuses principally on explaining what s needed in the insight layer of a Big Data platform. Please refer to the other papers in this series for an explanation of the other two layers.

5 Turning Big Data into Big Insights 5 Getting to grips with the Big Data insights layer Delivering actionable insights to business decision makers or end-users is often seen as the last mile challenge of any Big Data initiative. Being able to store and analyse all of your Big Data is one thing, but utilising Big Data for actionable insights can prove to be an equally, if not more challenging activity. In other words, when a trend, risk or opportunity has been discovered, how easy is it to put these big insights into the hands of people that need to action it? Our research suggests that a large part of the problem lies with making data easily digestible to business users in way that they can leverage and act on it in a confident and informed way. That said, any organisation looking to leverage Big Data insights firstly needs to have a clear idea of what they are trying to achieve through its use. As we discussed in the third report in this series, Navigating Big Data business analytics, many of the factors dictating how to approach your Big Data analytics initiative apply here. First and foremost there needs to be an inherent understanding of the business objective, problem or challenge you are trying to address; these might be quite prescriptive, such as reducing churn or improving fraud detection, or pertain to other business improvement programs such as better customer experience analysis, market targeting, behavioural and customer lifecycle analysis, for example. However, what s important is that there s a business context for your Big Data initiative, something that sets the scene and can therefore guide your choices about the right Big Data insight technology. Big Data insights technology options for users As shown in figure 3, Big Data insights can be leveraged in a variety of ways, where each option is typically determined by the type of user using the tool, their skills level and also the type of insights being discovered. For example, more advanced and professional users such as line-of-business managers, power users or IT workers are more likely to be more at home slicing and dicing datasets, creating reports and drilling down into data as supported, for example, through reporting and query tools. In contrast, more casual and novice users such as line-of-business and front-line workers often need to be shielded from the complexities of the raw data and calculations that sit behind it, and prefer to have Big Data insights pushed to them. This could be via an analytic application or equally these insights could be surfaced during an interaction or operational business process as part of a routine business task. Alternatively, other types of users perhaps those who are more analytically orientated may find that visually exploring the data as a way of identifying hidden patterns and trends offers up a more compelling way of consuming and digesting Big Data insights. Whatever the type of user and insight delivered, one of the most popular methods of deploying these insights is through a dashboard and/or scorecard as they provide a very effective way of summarising, visualising and displaying business results and insights according to a person s role. Figure 3: Big Data insight technology options Reporting & visualisation tools Dashboards & scorecards Analytic Embedded operational Source: MWD Advisors

6 Turning Big Data into Big Insights 6 Self-service and automating insights From another perspective, organisations looking to assess their options for Big Data insights can broadly focus their efforts across two technology domains: those that can be used by business and/it users in more of a self-service capacity to generate or discover insights, and those technologies, and systems that generate insights but also deliver them to business users so they can be consumed as part of a decision-making process. The table below provides examples of these different modes of insight together with the associated user type and supporting Big Data insight technology. Table 1: Modes of insight and supporting Big Data technology options Mode of insight Technology option Example user Key facts Self service Query & reporting tools Line-of-business and/or IT user These tools are the mainstay of traditional BI deployments. If data is available in a SQL or MDX compliant format such as an OLAP cube, data mart or data warehouse then it is relatively easy to point a query or reporting tool at the data source and serve up a set of results to an end user. While support from these tools has always been geared around supporting structured data, there is a growing emphasis from vendors to support the querying and reporting of multi-structured data sources. For example some vendors provide connectivity to Hadoop via Hive, so users can query and explore data in a Hadoop cluster. Dashboards & scorecards Executive and senior manager and/or line-ofbusiness and operational managers A popular way of organising, aggregating, presenting and monitoring business metrics in objects such as tables, graphs, charts, diagrams, and maps. Each dashboard delivers a role-based view tailored to a particular business audience such as executive or operational manager. A scorecard on the other hand uses the principles of a dashboard by presenting information according to a particular prescribed view or methodology, such as a balanced scorecard. Data visualisation tools Line-of-business and/or IT user Visualisation tools are very good at simplifying complex relationships found within both structured and multi-structured Big Data. They work by querying and modelling the underlying data source (in many cases leveraging an in-memory computing engine) before presenting a visual analysis of the data to users. These tools are particularly suitable towards a more exploratory style of analysis by virtue of their ability to spot trends, identify anomalies, recognise patterns and communicate data in a more meaningful and visual way. Please refer to the other paper in this series: Navigating Big Data Business Analytics for a more detailed explanation of this analysis style. Automating insights Analytic Line-of-business users These bring together an integrated set of tools and capabilities such as data schemas, business views, algorithms and predefined reports and dashboards that can accelerate the time it takes to deliver insights from your Big Data platform to end users. In particular are designed to address a particular business pain point or maximise an opportunity; examples include web analytics or a click stream analysis application.

7 Turning Big Data into Big Insights 7 Embedded analytic Decision management Analytics-as-aservice Operational and/line-of-business workers Operational and/line-of-business workers Operational and/line-of-business workers Much like analytic these are designed to target a particular business challenge or opportunity but they differ from their analytic application counterparts by virtue of the fact that insights are embedded as part of the workflow of an operational application, such as a CRM, SCM or Marketing Automation system. Although this is a development area for vendors we are starting we see the emergence of these types of for example around social media analytics where the integration of customer sentiment insights is embedded as part of CRM customer service workflows. These take analytic a stage further by using the insight generated, such as a predictive model score, to help guide or automate a business decision for example in support of activities such as next best action, fraud detection, or credit card approvals. If the insights are generated on event streaming data such as credit card transaction data then this decision automation happens in real time, otherwise insights are typically called as part of a decision service as and when they are needed. Decision management is an evolving discipline that requires a significant configuration and development effort by bringing together a range of technologies including predictive modelling, business rules and optimisation techniques to serve up insights to both users and. In a growing number of cases we also expect to see Big Data insight to be delivered as an application service. This primarily cloud based model describes the process of extracting insights from Big Data and delivering an action of set of actions (such as guidance or recommendations) directly to business decision makers. So for example there could be a service around customer churn, where churn scores are linked to a set of actions that recommend which high value customers to focus on and the type of actions that could prevent these customers from churning. Differences in maturity and adoption As we have outlined, Big Data insight technologies are principally designed to deliver actionable information to business users according to their role and business need. As the table above demonstrates, today there exist a wide variety of choices about how these insights are served up; but equally the technological maturity of each of these technology options also needs to be taken into account when considering your Big Data insights layer. For example, support for query and reporting tools is well established in enterprises both large and small where it s used against a structured data store such as a data warehouse. However, support for using these same standard BI tools against multi-structured Big Data file systems or data stores such as Hadoop HDFS, Hive or a NoSQL database is not as advanced. Although BI vendor support is improving, we believe this lack of interoperability between BI tools such as reporting and dashboarding tools and multi-structured data stores is currently one of the obstacles preventing wider scale adoption of Big Data insight technologies, especially where organisations want to harness their existing BI tool investments. Similarly, as Big Data practices grow and mature over the next two to five years, we expect packaged that include pre-configured content, models and industry best practice, to become a more viable deployment option as organisations look to simplify and lower the implementation risk and cost, and speed up the time to value of their Big Data deployments. Equally, the need to bring business insights generated by a Big Data platform into the context of an operational decision and workflow such as whether to offer a customer discount or not will likely spur the adoption of that embed insights within the operational layer of both front and back office systems and processes. While the integration between newer Big Data technologies and current analytics environments is still a work-in-progress for both organisations and vendors alike, it remains an important development area especially since the real value of Big Data will only come when organisations can act on insights obtained from a complete, timely and joined up view of their business, encompassing all forms and types of data and analysis.

8 Turning Big Data into Big Insights 8 Key considerations when planning your Big Data insight investment Utilise and harness your existing investments in reporting, dashboarding and visualisation tools where possible to bring the value of structured Big Data to business users. These tools predominately work against structured Big Data stores such as SQL based in-memory or MPP databases; however, expect to wait for the same level of tool support to be provided against other Big Data stores such as Hadoop HDFS or a NoSQL database. Accordingly, press your vendor about their plans for Big Data BI tool integration support. Give due consideration to how you present Big Data insights to end-users. A user s role, the type of analysis being performed and the context of use are all key factors determining what delivery interface is used. Self-service BI tools are typically the preserve of more advanced users, whereas front line workers or casual users will require a more context and role-based delivery interface where Big Data insights are integrated as part of an operational workflow or process. Think about data visualisation tools as an effective way for communicating and delivering Big Data insights to users. If configured and designed correctly they provide a valuable tool for illustrating patterns, trends, and relationships within the data. Successful data visualisation efforts also require a fresh approach to insight delivery, where visualisations are used to tell a story about the data and provide an environment for users to interactively explore and probe the data further. Support for pre-packaged Big Data is evolving. Most support thus far has concentrated on packaging up tools and content that deliver insights sourced from structured data. Purpose built appliances in particular that pre-configure both software and hardware components together including delivery interfaces are becoming a recognised and popular deployment model for Big Data insights. Expect to see this support to extend to more multi-structured data environments in the short to medium term.