An Enterprise Framework for Business Intelligence

Transcription

1 An Enterprise Framework for Business Intelligence Colin White BI Research May 2009 Sponsored by Oracle Corporation

2 TABLE OF CONTENTS AN ENTERPRISE FRAMEWORK FOR BUSINESS INTELLIGENCE 1 THE BI PROCESSING NEEDS OF THE BUSINESS 1 TYPES OF INFORMATION WORKERS 3 BI TECHNOLOGY REQUIREMENTS 4 PUTTING THE PIECES TOGETHER: AN ENTERPRISE BI FRAMEWORK 7 AN EVOLUTIONARY APPROACH TO DEPLOYING A BI SYSTEM 8 Brand and product names mentioned in this paper may be the trademarks or registered trademarks of their respective owners. Copyright 2009 BI Research, All Rights Reserved.

3 AN ENTERPRISE FRAMEWORK FOR BUSINESS INTELLIGENCE The business intelligence (BI) and data warehousing industry is changing rapidly as organizations strive to exploit business intelligence to improve business performance, while also at the same time trying to manage increasing data volumes and the need to provide business users with faster and easier access to information. Organizations need to build BI systems that can handle technology and product changes To support constantly changing BI technologies and information needs, organizations need to design a flexible enterprise BI framework that incorporates the different kinds of BI processing, information workers and BI technologies that organizations must support if they are to fully realize the business benefits that business intelligence offers. This paper presents a BI framework that can be used to develop and deploy a new BI system, or to enhance an existing one. This flexible and adaptable framework enables BI applications and the underlying data warehousing environment to evolve and adapt to the constantly changing technology and product landscape. The paper includes the results of a survey that asked BeyeNETWORK site visitors about the different approaches and technologies their organizations use to plan, build and grow their BI systems to exploit the business value business intelligence and data warehousing offers. There were 217 responses to this survey. THE BI PROCESSING NEEDS OF THE BUSINESS Three key elements need to be considered when designing and developing a BI system: BI processing needs, the information worker audience and the technologies that need to be provided. This first section discusses the BI processing needs of organizations. There are three main modes of BI processing: strategic BI, tactical BI and operational BI. One of the key distinguishing features between these three processing modes is the time period over which each of them analyzes and reports on information. This time period can best be explained in terms of the workflow shown in Figure 1, which illustrates how business intelligence aids decision making. Closed-loop processing enables business users to see the effects of business decisions In Figure 1, data flows from operational applications that run day-to-day business processes to BI applications that monitor and analyze the data to provide insight about the actual business performance of those processes. Business users then employ collaborative applications to share and evaluate the results produced by the BI applications. This evaluation may result in users making changes to a business process (e.g., a modified marketing campaign) or a business plan (e.g., an updated sales forecast). The positive or negative effects of those changes can then be measured by the BI applications to create a closed-loop decision workflow. Copyright 2009 BI Research, All Rights Reserved. 1

4 Figure 1: Closed-loop decision workflow BI agility can vary from near real time to weeks and months depending on business need The speed at which business users need to do this closed-loop processing determines over what period of time data is analyzed and how responsive the BI environment needs to be. This speed can be expressed in terms of the elapsed or action time between a business event occurring that requires action and the user taking that action. In the case of fraud detection, for example, the required action time maybe a matter of seconds, whereas in a marketing campaign, the action time could be a few hours, days or weeks. Figure 2 outlines the action times supported by the three modes of BI processing. Other variables, such as the types of users supported and the types of data required, are also shown in the figure. Figure 2 is intended to illustrate general differences, rather than absolute boundaries, between the three modes of processing. Strategic and tactical BI processing represent the types of solutions that organizations have been developing for many years. In this environment, data flowing from operational processes is captured into a data warehouse for use by BI analytical applications. Users run these analytical applications at periodic intervals to collect data analytics for the specific data points or key performance indicators (KPIs) they are interested in sales in a particular region, for a specific product, during a specific time period would be an example. One industry trend here is to supplement these data analytics with content analytics created by analyzing unstructured data. Operational BI is the fastest growing BI segment Operational BI processing focuses on the performance of a specific operational business process, rather than on business unit or enterprise-wide KPI data points. Operational BI is one of the fastest growing areas of business intelligence. As action time requirements approach near real time, operational BI processing becomes driven by business events occurring in operational systems, rather than by users requesting information at periodic intervals. Examples of events that may trigger operational BI processing include credit card transactions (to detect potential fraud) and customer calls to a call center (to monitor call center call backlog and response times). Copyright 2009 BI Research, All Rights Reserved. 2

5 To process business events and support so-called event analytics, the BI environment must either enable the loading of low-latency event data into a data warehouse or the embedding of BI processing into the operational processes that handle events. Figure 2: The three modes of BI processing Strategic BI Tactical BI Operational BI Action time Weeks to months Days to weeks Intra-day to daily Types of users Executives & business analysts Business analysts & line-of-business managers Line-of-business managers & operational users Focus of analysis Type of data Enterprise Line of business Business process Historical Historical Real time, low latency, historical Mode of operation User driven Data centric User driven Data centric User & event driven Process centric Business objective Achieve long-term business goals Manage tactical initiatives Manage business processes TYPES OF INFORMATION WORKERS Two main types of information worker consumers and producers Business intelligence specialists, analysts and vendors often use the term information worker to describe a business user that makes use of the information produced by BI applications. There are two main types of information worker: information consumers and information producers. Information consumers range from senior and middle managers to call center staff. A common characteristic of these information workers is that they do not use corporate data and IT information-handling tools throughout their working day. These types of users represent the bulk of the information workers in an organization. Information consumers are a large user audience who need easy to use tools Information consumers need to discover information that enables them to do their jobs more efficiently and make more informed business decisions. Information discovery involves locating and retrieving information of interest, and then filtering and organizing it so that it is easy to navigate and explore. Some information consumers, such as business managers, may do some limited analysis of the discovered information, while others, such as call-center staff, just use the information as is. Information producers create the information used by information consumers. In addition to using information creation and discovery tools, information producers also use analysis techniques to enhance, aggregate and report on information. Copyright 2009 BI Research, All Rights Reserved. 3

6 Information producers are more knowledgeable about the BI technologies and corporate data Information analysis extends information discovery by enabling information producers to apply their knowledge and expertise to discovered information and enhance its value by putting it into a business context. Enhancing information in this way makes it easier and faster to use by information consumers in their jobs. Information consumers may do additional analysis of the information, but this is likely to be limited in scope. Examples of information producers include BI specialists, and data and business analysts. An important consideration is that there are different types of information consumer. The skills and requirements of a call center representative are different from those of a senior manager (a VP of Sales, for example). The former is a pure information consumer, whereas the latter may also have some information producer skills. It is important therefore to map job roles to both skill and technology requirements. Information needs to be made easier to consume As organizations deploy business intelligence to a wider audience of information consumer they will need to focus on making the BI system easier to use, and the information produced easier to consume by putting it into a business context. BI TECHNOLOGY REQUIREMENTS Figure 3: Main components of a decision workflow The next task in the design of a BI system concerns the BI technologies that it needs to provide. These technologies can be categorized according to how they support each of the activities of the decision workflow shown in Figure 3. This figure is a more detailed version of the workflow of Figure 1. The six main activities of a decision workflow are: Discover the operational data and other business content that can aid business decision making. Access the operational data and business content required to make business decisions. Integrate the retrieved operational data and business content into a shared data store, such as a data warehouse, as required. Copyright 2009 BI Research, All Rights Reserved. 4

7 Analyze operational data, business content and data warehouse information to produce analytics, alerts and recommendations that aid decision making. Deliver the results of analytical processing to business users and applications. Share the results with other users and collaborate to determine what actions need to be taken to resolve any business issues identified by the results. A wide range of technologies supports these six activities. These technologies can be broken down into two broad groups (see Figures 4a and 4b): data management technologies (discover, access and integrate activities) and BI application processing technologies (analyze, deliver and share and collaborate activities). Figure 4a: Data management technology requirements Activity Discover Access Integrate Technology Faceted data search Business glossary Data analysis & modeling Data profiling EII data federation Data quality management & services ETL data consolidation Changed data capture & propagation Data replication Metadata management & reporting Figure 4b: BI application processing technology requirements Activity Analysis Deliver Share & collaborate Technology Production reporting Ad hoc analysis Advanced analytics (statistics, time series, etc.) Data & text mining Predictive modeling Performance management scorecards Operational BI (alerts, event analytics, embedded BI services, etc.) Content analytics Metadata management & reporting Dashboards Portal integration Web interface & rich Internet application features Advanced visualization Office product integration Mobile computing support Collaborative & social computing integration Copyright 2009 BI Research, All Rights Reserved. 5

8 Figure 5a: Percentage of survey respondents rating a data management technology as very important or somewhat important Figure 5b: Percentage of survey respondents rating a BI application processing technology as very important or somewhat important Source: BeyeNETWORK Source: BeyeNETWORK The BeyeNETWORK survey asked respondents about the importance of the technologies listed in Figure 4 to their organizations. The results are shown in Figures 5a and 5b. Figure 5a shows the results for data management technologies and Figure 5b the results for BI application processing technologies. 54% of survey respondents need to support data warehouses in excess of 10 terabytes of data The results for the data management part of the survey show the interest and importance of data quality management technology. Respondents to this part of the survey were also asked about the types of data they needed to integrate into a data warehouse. Some 51% needed to integrate unstructured data, 59% application package data, and 69% application event and web data. The respondents were also asked about the amount of data their data warehouse environments needed to support. Some 54% of respondents indicated a size in excess of 10 terabytes of data. Copyright 2009 BI Research, All Rights Reserved. 6

9 The results for the BI application processing part of the survey indicate the drive by organizations toward making BI technology more usable. Scores for web-based interfaces, executive dashboards and office product integration demonstrate this trend. As discussed earlier, the results show operational BI to be a key industry direction. 78% of respondents require ease of use features for less experienced users Survey respondents were also asked about several other BI application processing requirements. Items that received high scores here were spreadsheets (73% of respondents), ease of use for less experienced users (78%) and low-cost licensing for a large number of users (79%). PUTTING THE PIECES TOGETHER: AN ENTERPRISE BI FRAMEWORK Figure 6: An enterprise BI framework Now that we reviewed the three modes of BI processing (strategic, tactical, operational), types of information workers (information producers and information consumers), and the data management and BI application processing technologies that support these BI processing modes and information workers, we are now in a position to discuss an enterprise BI framework that can be used to design and build a flexible BI system. An example of such a framework is shown in Figure 6. The framework in Figure 6 is a more detailed version of the decision workflows illustrated in Figures 1 and 3. The framework shows how data flows between operational, BI analytical and collaborative applications. At the center of the diagram are the traditional BI applications that produce tactical and strategic data analytics. The BI industry is evolving to support embedded BI and event analytics The left-hand side of the figure shows how embedded BI services receive operational events and produce event analytics. It also illustrates how embedded BI services interact with operational applications to process service requests. As an example, a Copyright 2009 BI Research, All Rights Reserved. 7

10 service request could ask a BI service to access customer lifetime value scores from a data warehouse for use in a call center application. Support for content analytics is also an important trend A BI system must be able to interact with collaborative and office applications The right-hand side of Figure 6 depicts how analytics are produced from unstructured business content. In some cases, content analytics may be produced directly by content analysis applications, while in other situations the unstructured content may be converted into a structured or semi-structured format and stored in a data warehouse for use by tactical and strategic BI applications. At the bottom of Figure 6 are the data management services that handle the data discovery, access and integration activities of a decision workflow. At the top of the figure are the collaborative applications that receive data, event and content analytics, which enable business users to share information and expertise and determine what actions are required to handle different business situations. These collaborative applications also interact with office and social computing applications as well as business portals and other types of web interfaces. AN EVOLUTIONARY APPROACH TO DEPLOYING A BI SYSTEM The enterprise BI framework shown in Figure 6 can be used to construct a BI system that can evolve with the constantly changing BI technology and product landscape. There are many approaches to constructing such a BI system. Regardless of the approach for selecting products, a BI system must be able to support different approaches and handle technology changes Some organizations, for example, may select and integrate best of breed products, while others may opt for using a single platform from a single supplier. Some companies may focus on open source solutions in order to reduce software costs. It is also likely, given the current recession, that IT budgets will become restricted, and as a result some business units may deploy their own short-term BI solutions. Given this dynamic environment, it is important that the BI system remain flexible to be able to support and integrate these various approaches. The BeyeNETWORK survey showed that 52% of respondents are building their BI systems using best of breed products, while 25% said they use a single suite of products from a single supplier. Most of the remainder (21%) also use a single suite of products, but employ best of breed products to fill in any missing capabilities. Another approach to filling in these gaps is with open source software. About 50% of respondents were using, or planning to use, an open source data management or BI product. The direction of the BI vendors over the past few years has been to create a single suite of BI products through both product development and product acquisition. As a result, the level of integration in these product suites varies depending on how recent the products were acquired and on how much effort is required to incorporate the acquired products into the BI suite. The level of product integration is a key distinguishing feature between vendors When asked if they thought the BI industry benefited from the large number of acquisitions over the past few years, 9% of respondents strongly agreed, 44% partly agreed, but nearly half were neutral or felt is was not beneficial. These results suggest that customers are still unsure about this industry trend and are waiting to see how successful vendors are at integrating acquired products. Regardless, 43% of respondents said they intend to reduce and consolidate the number of BI-related Copyright 2009 BI Research, All Rights Reserved. 8

11 products they use, and 22% were considering it. The level of integration in a product suite and its ability to incorporate other best of breed solutions are key distinguishing features between different vendor solutions. Organizations are trying to reduce the number of BI products they use Organizations are embracing new BI and data warehousing technologies It is important to be able to evolve a BI system to handle different types of BI processing, information workers and technologies One approach to building and evolving a BI system using an enterprise BI framework is to create a list of recommended products that can be updated as the market changes. This approach helps control product proliferation and enables the organization to efficiently manage the move toward supporting the complete enterprise BI framework shown in Figure 6. Among our survey respondents, 66% have a recommended product list. Not all companies enforce such a list, but twothirds of those that do maintain a list enforce it and have a policy that deviations have to be justified. The results of the BeyeNETWORK study clearly demonstrate that organizations see the importance of bringing new and evolving business intelligence and data warehousing technologies into their organizations. In the data management and data warehousing area companies were especially focused on data quality management, supporting large data warehouses, and on expanding their data warehousing environments to support web and unstructured data. For business intelligence, the areas of focus were on operational BI, and on making business intelligence more usable through web-based interfaces, executive dashboards and integration with office products. Improved usability enables business intelligence to be deployed to a wider user audience, which leverages the investment made in an organization s BI and data warehousing system. To exploit new BI and data warehousing developments, organizations need a flexible BI system that can absorb new technology developments and support new types of BI processing without a major impact to existing users. This paper has presented an enterprise BI framework that includes the key elements of such a system. The framework can support the three main modes of BI processing, different kinds of information workers, and the main data management and BI application processing technologies that exist today. It will take time for organizations to build a BI system that supports this framework. The objective of the framework is not to document a short-term objective, but to provide a long-term goal for a BI system that organizations can gradually move toward and can adjust as products and technologies change. About BI Research BI Research is a research and consulting company whose goal is to help companies understand and exploit new developments in business intelligence and data integration. When combined, business intelligence and a sound data integration infrastructure enable an organization to become a smart and agile business. About the BeyeNETWORK The BeyeNETWORK focuses on business intelligence, performance management, data warehousing, data integration and data quality, serving these communities with unparalleled industry coverage and resources. Copyright 2009 BI Research, All Rights Reserved. 9