CIO considerations for Big Data

Transcription

1 An Enterprise Strategy Program paper CIO considerations for Big Data Abstract This paper describes the Microsoft vision for Big Data, discusses key industry trends, and evaluates how to obtain real business value from investments in Big Data. To highlight the potential that Big Data holds for organizations of all sizes, this paper presents common business scenarios that include customer analytics, risk management, science and research, and IT business innovation. Author Achim Granzen, Architect, Microsoft Services Publication date November 2012 Version 1.0 We welcome your feedback on this paper. Please send your comments to the Microsoft Services Enterprise Architecture IP team at

2 Acknowledgments The author would like to thank the following people who contributed to, reviewed, and helped improve this paper. Contributors Susan Conway, Ulrich Homann, Mike Wise The information contained in this document represents the current view of Microsoft Corporation on the issues discussed as of the date of publication. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS DOCUMENT. Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation. Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property. The descriptions of other companies products in this document, if any, are provided only as a convenience to you. Any such references should not be considered an endorsement or support by Microsoft. Microsoft cannot guarantee their accuracy, and the products may change over time. Also, the descriptions are intended as brief highlights to aid understanding, rather than as thorough coverage. For authoritative descriptions of these products, please consult their respective manufacturers Microsoft Corporation. All rights reserved. Any use or distribution of these materials without express authorization of Microsoft Corp. is strictly prohibited. Microsoft and Windows are either registered trademarks of Microsoft Corporation in the United States and/or other countries. The names of actual companies and products mentioned herein may be the trademarks of their respective owners. Microsoft Proprietary and Confidential Information Page ii

3 Table of contents 1 Executive summary Microsoft vision - Democratize Big Data Business motivation Big Data opportunities The evolving Big Data platform Key trends Enabling business value out of Big Data investments Sensemaking Gaining insight from Big Data Insight decisioning Innovating your business with Big Data Key scenarios Customer analytics Risk and performance management Science and research IT business innovation Call to action: Getting started with a Big Data strategy Our approach to Big Data and Business Intelligence References and resources Microsoft Proprietary and Confidential Information Page iii

4 1 Executive summary Enterprises can now generate, capture, and buy large volumes of data from a variety of sources and then use analytical methods to gain business insight, support business processes, improve competitive advantage, and generate profit. The convergence of data availability and analytical innovation is commonly referred to as Big Data, and many organizations are now starting to use it to produce insights that are relevant to tactical and strategic business issues. However, more often than not the use of Big Data starts with a lab approach in IT departments that is driven by the eagerness of IT staff to examine and understand the latest cutting edge technology. To obtain real business value out of investments in Big Data, a strategic approach is required to join technical possibilities with business goals. The Microsoft Enterprise Strategy Program makes resources and offerings available that can help you get started. Microsoft Proprietary and Confidential Information Page 1

5 2 Microsoft vision - Democratize Big Data Microsoft envisions all business users having the ability to gain actionable insights from virtually any data, including insights that were previously hidden in unstructured data. Microsoft is expanding the vision of Business Intelligence to provide business insight as a service layer for applications to increase the richness and variety of the Big Data experience. This layer can serve as a new platform to provide insight into structured and unstructured data of any volume by creating unified and intuitive approaches to discovering, gathering, storing, indexing, exploring, analyzing, and performing selfservice visualization of Big Data. Figure 1. Aspects of Big Data The Microsoft vision of democratizing Big Data aims to provide your organization with the following three key capabilities: Manage data of any type or size Flexible data management layer that supports all data types structured, semi-structured, and unstructured data at rest or in motion Data management and analysis function that can be performed on-premises, in the cloud, or using a hybrid approach Enrich your data with the world s data Enrichment layer for discovering, transforming, sharing, and governing data Deeper insights that combine an organization s data with data and services from external sources Gain insight from any data Compelling suite of tools to help users gain insight from analytics Enable insight decision making for everyone from data scientists to casual users Microsoft Proprietary and Confidential Information Page 2

6 3 Business motivation Figure 2. Overview of the Big Data Customer Analytics Framework 3.1 Big Data opportunities Big Data delivers game-changing benefits using a new approach to data acquisition, management, and visualization for the emerging Big Data platforms. Organizations have always produced significant amounts of unstructured data from sources such as medical images, blogs, radio-frequency identification (RFID) tags, and locality sensors. Historically, organizations threw away most of the data they could collect to avoid what were once considered excessive costs of managing such a data deluge. Spurred by plummeting storage and computation costs coupled with a new understanding of the inherent value of previously discarded data, organizations are demanding new types of business insight from every bit of data they can access using cost-effective and scalable methods. Examples of new insights include: Understanding user behavior and online interactions Identifying trends and popular topics in social media sentiment analytics Optimizing and targeting advertising campaigns Discovering medical epidemiological trends (such as identifying the next flu outbreak) Identifying financial fraud within public sector transactions Such insights are critical in providing competitive advantages to organizations as well as improving tactical decisions and controlling costs. To achieve such insights, organizations must invest to create a platform that accommodates the requirements of Big Data. Microsoft Proprietary and Confidential Information Page 3

7 3.2 The evolving Big Data platform The initiatives launched in the IT labs of Web 2.0 companies are now evolving into platforms with potential for broad adoption in all industries. The design, operation, and use of a state of the art Big Data platform has become easier, and the user circle has widened from data scientists in IT labs to business analysts, information workers, and, increasingly, information consumers. Eventually, nearly everyone in your organization can analyze and make more informed decisions with the right tools, as described in the following example. Example: Using social media to conduct brand research and promotion Organizations can conduct brand research and promotion using social media to understand and gain insight into the opinions of customers and the market (public) about the organization and its products and services. This research can produce immediate, mid-term, and long-term impact on the top line, such as market research, brand building, brand protection, product development, and customer service. Insight obtained from brand research can be used at many levels: Executive level. Determine new markets, customer segments, products, and services. Marketing director/cmo level. Develop an organizational brand and overall marketing message. Marketing campaign managers level. Optimize segments, channels, and campaign for highest returns. Call center agents. Provide better service to customers by better understanding their motivations and opinions. Although its initial uses are mostly tactical, Big Data has the potential to drive real business innovation. After integrating Big Data analytics into operations to improve day-to-day decision making, your organization will be poised to start innovating with Big Data (see the Innovating your business with Big Data section later in this paper). Microsoft Proprietary and Confidential Information Page 4

8 4 Key trends Figure 3. The volume and impact of data is rapidly increasing A set of broad industry trends are putting pressure on traditional data management and Business Intelligence platforms and tools. These trends are as follows: Increasing data volumes The annual growth of worldwide information volume is roughly 50 percent and continues to rise. This explosion of new data is driven by a full range of traditional and nontraditional sources that include sensors, devices, and tools that monitor and catalog content on the Internet, such as bots and crawlers. According to an IDC study, 1 the amount of digital information created and replicated is forecasted to hit 1.8 zettabytes (1.8 trillion gigabytes) in 2011 and is predicted to grow by a factor of 44 during the forecast period. Increasing complexity of data and analysis The real growth in data comes from unstructured data in a wide variety of documents, streaming data, and click-stream data. The success of search engine providers and e-retailers who have unlocked the value of unstructured data has debunked the presumption that 80 percent of such data has no value. The business requirement to store, analyze, and mine a combination of structured and unstructured data is becoming the new norm. Changing economics and emerging technologies Cloud computing and commodity hardware have dramatically reduced the acquisition cost of computational and storage capacity and is fundamentally changing the economics of data processing. To create platforms for tackling massive data processing tasks, architects are complementing commodity hardware with new, distributed parallel processing frameworks (such as Hadoop and MapReduce) and a rich ecosystem of tools. These trends provide a variety of opportunities for organizations to obtain business insight that supports effective decision making and innovation in business processes, products, and services. 1 IDC Digital Universe Study, sponsored by EMC, June Microsoft Proprietary and Confidential Information Page 5

9 5 Enabling business value out of Big Data investments 5.1 Sensemaking Gaining insight from Big Data To obtain pertinent insight, traditional data platforms require you to pre-identify and structure data of interest, and to define and apply a data model. However, with the increasing volume and types of data, people cannot reliably anticipate which data will be valuable or decide which data can be discarded without risking the loss of insight. With the explosion of potentially meaningful data some structured, some unstructured (such as signals, streaming, social, interaction, and transactions) new types of analyses are needed. The cycle that holds the greatest promise for gaining knowledge from Big Data is based on the concept of sensemaking developed by Pirolli and Card in 2005 within the intelligence analysis community. Figure 4. Obtaining value from all data sources Microsoft Proprietary and Confidential Information Page 6

10 Figure 5. The sensemaking cycle is useful for identifying valuable data and questions. Sensemaking uses continuous feedback and defines interdependent relationships to create a context that provides information about data and supports analysis of the data. Big Data expands this cycle to the unmodeled domains of unstructured content. 2 One of the most significant aspects of sensemaking is how it transforms traditional analytical processes: Traditionally, data comes to you. The data is then structured through an information management process (such as extract, transform, and load) after which enterprise stakeholders can analyze it. Now, knowledge and insight come to you. You define your business objectives and then allow the sensemaking process to map them to data in real time, drawing out knowledge that may form the basis for immediate action, such as modifying product prices in real time. Sensemaking is emerging within Big Data environments as a powerful method for culling promising information from torrents of data, as well as helping identify questions that business decision makers should ask to meet their business objectives. 5.2 Insight decisioning The connections between an organization, its data, and its processes defines the technologies that support traditional structured data as well as unstructured data and advanced analytics within an insight decisioning platform. The following figure illustrates the conceptual model for an insight decisioning platform. 2 Klein, et al, A Data/Frame Theory of Sensemaking. June, 2008 Microsoft Proprietary and Confidential Information Page 7

11 Figure 6. Conceptual model of an insight decisioning platform An insight decisioning platform consists of four different types of components: consumption, insight, integration, and analysis. The nature of the insight decisioning process calls for Big Data consumption components from cloud environments as well as internal business data. The insight platform may use a hybrid environment or a pure cloud environment that processes and transfers refined Big Data results into an on-premises analysis environment. Resources that are available in the cloud enable organizations of all sizes to enrich their data sources, often using a pay-as-you-go model that allows them to apply analytics to selected sets of data, without incurring significant cost. Insight components provide the following features: Hot stream data (alert information such as customer credit issues); cold stream data (latent information, such as unstructured data from search engines that track banking service inquiries); and multiple decision points about a sale or upsell. Alert features, such as those provided by StreamInsight have the ability to process time series data in near real-time. Integration components combine results from the Big Data interrogation with results of similar interrogation of structured business data. The data analyst facilitates the process by defining the mappings of the unstructured and structured findings to create an integrated result for presentation and general consumption. Analysis components provide a business analyst with results to review, such as a customer s ability to make loan payments, or upsell opportunities. Microsoft Proprietary and Confidential Information Page 8

12 5.3 Innovating your business with Big Data A typical approach for innovating your business with Big Data might consist of the following three components. Big Data foundation A foundational infrastructure is necessary to use, manage, and maintain Big Data. The infrastructure can be integrated into existing Business Intelligence and operational systems, and may be deployed on-premises, in the cloud, or as a combination of the two. The infrastructure should also include a sandboxed environment for employees to acquire new skills and experiment with data. Key benefits include: Enable subsequent Big Data activities Enable development of innovative, customer-specific approaches and solutions Enable ad hoc analysis and discovery for simple, tactical use cases Big Data operational analytics Big Data enriches predictive analytics to improve day-to-day operations and tactical business decision making. Big Data operational analytics use near real-time data (hot stream) and non-real-time data (cold stream) to help ensure agile and well-founded business action. Key benefits include: Improve operations in all targeted business areas, building on richer, more accurate and detailed business insight (for example, achieving high-yield campaigns using customer social media behavior) Become more agile in business operations and tactical decision making Big Data innovation Big Data drives business innovation. Using closed-loop marketing techniques, Big Data can enhance branding and drive sales. Big Data can also reveal opportunities for market-driven products and services, and identify new business areas and markets for an organization to pursue. Key benefits include: Widen the idea pipeline Shorten the reaction time to changing business and sales conditions Increase the agility of decision making processes and strategic decision making Microsoft Proprietary and Confidential Information Page 9

13 6 Key scenarios There are four main business scenarios in which Big Data is starting to be used as an extension to the traditional Business Intelligence and analytical systems: Figure 7. Big Data scenarios Microsoft Proprietary and Confidential Information Page 10

14 There are additional business scenarios for Big Data, particularly in online search, advertising and managing social networks. However, this section discusses business scenarios beyond web analysis scenarios. For overview purposes, this section presents cross-industry scenarios, although some scenarios might be more common in certain industries. For example, customer analytics is especially important for retail and telecommunications companies, whereas financial services organizations focused more on risk and performance management. Also, science and research is a hot scenario for the energy industry to support the search for natural resources. And pharmaceutical companies might consider all four scenarios of equal priority. 6.1 Customer analytics Going beyond classic customer analytics that use internal, structured data, Big Data customer analytics can create a multifaceted view of a single customer, with actionable business insights from such diverse sources as point-of-sale transactions, loyalty data, online/web activity, lifestyle information, market research, demographic data, marketing channel responses, direct communications, and social media. Classic customer analytics tools (such as segmentation, churn, and cross-/up-sell) are enriched and complemented by analysis of text, sentiment, marketing, and advertising to enable closed-loop processes for optimizing marketing techniques and portfolios, predicting retail behavior, improving customer service and satisfaction, conducting brand research, protecting brand value, and developing new products and services. Sample use cases involving customer analytics include: Multichannel campaign management and predictive customer analytics Social media brand promotion and sentiment analytics New product and services development 6.2 Risk and performance management Risk and corporate performance management involves primarily financial and operational risk/portfolios with a primary goal of obtaining an optimal state that yields the highest benefits (financial, safety, and so on). A key application is trading and investment portfolio management, specifically in high-frequency trading and short-term investment, and in financial risk management for mid and long-term (insurance). Sample use cases: Energy trading and risk management (ETRM) Improved actuarial analysis Computerized high-frequency trading and portfolio optimization Microsoft Proprietary and Confidential Information Page 11

15 6.3 Science and research Science and research involves both the public sector and private sector and uses varied data source such as meteorological data, fundamental research such as subatomic research, astronomy, medical/pharmaceutical/genetics research, and natural resources exploration and exploitation. In a wider sense it also deals with areas such as real-time monitoring in manufacturing and production. Sample use cases of science and research include: Natural resources exploration and prospecting Pharmaceutical clinical trials Meteorology and the natural disaster research 6.4 IT business innovation IT organizations across all industries are constantly being evaluated against the services and benefits they provide to their enterprises. Simply keeping the lights on has become an outdated metric for many IT organizations: they are now under pressure to innovate and participate as a business enabler. Big Data and cloud technologies provide unique opportunities for IT organizations to examine and redefine their current services, establishing new creative ways for helping employees do their business better while enhancing operational efficiency. Sample use cases of IT business innovation include: Develop new innovative IT services to enable internal/external business clients Streamline IT operations and modernize technology platforms to exploit today s opportunities Microsoft Proprietary and Confidential Information Page 12

16 7 Call to action: Getting started with a Big Data strategy Microsoft helps customers assess their approaches to Big Data as part of the Enterprise Strategy Program (ESP). This program focuses on helping enterprises realize value from their IT investments and consists of four core elements: Enterprise architects. Dedicated to the customer and charged with accelerating customers toward their business goals. Microsoft network. Subject matter experts from across all areas of Microsoft, including product groups, research and development, internal IT resources, and Microsoft Research. Value Realization Framework (VRF). A framework and methodology designed to deliver on the value proposition of the program. Library. A collection of exclusive intellectual property including comprehensive guidance, reference architectures, implementation information, and worked examples from Microsoft engagements. 7.1 Our approach to Big Data and Business Intelligence Our approach focuses on how you can programmatically extend your traditional Business Intelligence infrastructure to leverage Big Data techniques, aligning the right technologies to help enable and obtain measurable benefits from your Big Data strategy. The Big Data Workshop is specifically designed to help you get started with a strategic, business valuefocused approach to Big Data. On a high level, the workshop provides the following: Rapidly models the Big Data environment in your enterprise Identifies gaps in your current Big Data/Business Intelligence infrastructure Finds opportunities to generate value using Big Data techniques and tools Proposes initiatives to address those opportunities Augments the initiatives with metrics that measure adoption and realization of opportunity value For more information about the Microsoft Enterprise Strategy Program, contact your Microsoft account representative or visit Microsoft Proprietary and Confidential Information Page 13

17 8 References and resources Sources referenced in this white paper include: IDC Digital Universe Study, sponsored by EMC, June Klein, et al, A Data/Frame Theory of Sensemaking. June, 2008 Additional resources from Microsoft include: Conway, Susan. Obtaining Insight from Big Data. Microsoft, 2011 Conway, Susan. Making Decisions with Insight: A process and platform for generating business insight. Microsoft, Wise, Mike. CIO considerations for Business Intelligence: Obtaining competitive insight. Microsoft, Microsoft Proprietary and Confidential Information Page 14