CRITICAL FACTORS IN DEVELOPING A DATA WAREHOUSE

Transcription

1 INFORMATION MANAGEMENT: STRATEGY, SYSTEMS, AND TECHNOLOGIES CRITICAL FACTORS IN DEVELOPING A DATA WAREHOUSE Duane E. Sharp INSIDE Are Companies Realizing A Return On Their Investment?, Internal Access To The Corporate Data Warehouse, Building the Data Warehouse, Focusing on the Real Problem, Selecting the Right Data Warehouse Champion, Using Detailed Historical Data, Applying Technology, Trust The Data INTRODUCTION Data warehousing has become one of the most significant technologies of the past decade, and has permeated virtually every business sector, from retailing to finance, in one form or another. The International Data Corporation (IDC) estimates that revenue from the total worldwide data warehouse software market, including data access, warehouse management/storage, and data transformation/warehouse generation, will grow at a compound annual growth rate (CAGR) of 30.8%, during the period from 1995 to the year Worldwide market revenue was $1.4 billion in 1995; based on this forecast, it will grow to $5.4 billion by the year This growth pattern is a certain indication that data warehousing is well beyond the stage of early adoption and has been accepted by pragmatic businesses as a proven technology for enhancing their business operations. As an example of the proliferation of this major corporate application of information technology, it is worth noting that NCR Corporation, a world leader in data warehousing technology, had over 500 data warehousing installations worldwide in 1997, in a broad range of business sectors. ARE COMPANIES REALIZING A RETURN ON THEIR INVESTMENT? A recent IDC ROI study, published as The Foundations of Wisdom: A Study of the Financial Impact of Data PAYOFF IDEA Although the past decade s experiences have proven that data warehousing can provide a company with high ROI, developing a warehouse is one of the most complex projects a company can undertake. This article outlines a five-step program for increasing the likelihood of success when developing a data warehouse. Auerbach Publications 1999 CRC Press LLC

2 Warehousing by Stephen Graham, interviewed 62 sites that have successfully implemented a corporate data warehouse. The study covers a wide range of industries, including financial services, health care, telecommunications, retail, government, and manufacturing. The average initial investment by the surveyed sites was $2.2 million. The major finding of the study is that organizations recouped their initial investment within an average of 2.3 years. The average return on the initial investment over 3 years was more than 400%, dramatic confirmation that data warehousing can be a good investment. INTERNAL ACCESS TO THE CORPORATE DATA WAREHOUSE A data warehouse takes time-oriented data from multiple applications and organizes it according to subjects meaningful to the corporation or business. Corporations, concerned with informing their decision makers, are pursuing this strategy for two major reasons: 1. Reduced complexity. The data in the decision-support database or warehouse is made available in a form that is relatively easy to understand. 2. Improved performance. The warehouse can be tuned to provide better performance and faster response to complex queries and analysis. BUILDING THE DATA WAREHOUSE From a qualitative perspective, according to the IDC survey, the key benefits of a corporate data warehouse are: More streamlined systems administration; and More productivity for internal analysts. Building a data warehouse is one of the most complex processes a corporation can undertake. It will change the corporate decision-making process without necessarily reengineering the corporation. Traditionally, corporate decisions have been based on the analysis of data, without detailed information to support the data. Corporations analyzed data from reports and made decisions based on limited information. Data warehousing changes this process dramatically, by quickly transforming all available detailed data (irrespective of volume) into meaningful business information. The end results are timelier and better informed business decisions. Experience based on successful data warehouse implementations points to five critical factors, which are essential for a successful implementation. The following analysis of these factors decribes why they are important to any data warehousing project. Focus on a Real Problem It is a fundamental axiom that a successful data warehouse implementation needs to be based on solving a real business problem, and the cor-

3 poration will have to solve this problem. A data warehouse which does not address a critical business problem is destined for failure. The business problem selected must have senior management backing which correlates with the desire to solve the problem. Most successful data warehouses are cross-functional, because the ROI increases with both the breadth of data they hold and the impact they have on the business. Business problems that have been solved by a data warehouse solution include: Credit card risk management; Sales and inventory management; Supply chain management; Exposure management; and Target marketing. Solving these business problems requires large volumes of data from many business functions and, in some cases, even from outside sources. It also involves structuring the data based on input from end users who will use the system, as to what data is important and how it should be presented. History has shown that if a data warehouse is built without end-user input, end users will not use it and the development exercise will be a spectacular, expensive failure. Information technology professionals alone cannot build a data warehouse: user organizations must be involved from the beginning. There are several approaches to implementing a data warehousing system. One solution which is often applied to solving a business problem is the so-called packaged data warehouse. A packaged solution is usually a single-vendor solution, with a pre-defined front-end application, a standard database management system, and an industry-generic database design. It often fails because it does not solve the critical business problems of a corporation; however, it is implemented to prove the concept. Since it is designed to meet a variety of requirements, it usually fails to address the specific needs which are always a part of any organization s data warehousing business. The data warehousing solution which is most likely to be successful is one that provides a solution to critical business problems, specific to the organization for which it is designed, with significant end-user involvement and senior management support. Select the Right Data Warehouse Champion The second critical success factor is acquiring a strong champion for the data warehouse implementation. The complexities of the implementation are enormous, ranging from maneuvering the project through the corporate political environment to gaining consensus among cross-func-

4 tional business users with different objectives. Usually, a data warehouse champion has to spearhead the project to ultimately make the data warehouse successful. The data warehouse champion is typically a fairly senior business user with a strong understanding of the information technology environment. He must understand the political landscape, have the capability to bring tough issues to a consensus, and should report to a senior corporate sponsor during the data warehouse implementation. Meeting these criteria is the best way to ensure that the champion will prove to be a real champion when the chips are down. The champion must be firmly convinced that a data warehousing solution will meet the requirement and solve the defined business problem, to the extent of betting his or her reputation on the implementation. He will also ensure that the right team of professionals is involved in defining the business problem to be solved, and ultimately in developing the data warehousing solution that will meet the requirements. A key element in the champion s involvement is the requirement to challenge information system specialists, to work with them for the benefit of the corporation, and to represent the business users in defining methods of access and presentation of the wide range of information to be derived from the data warehousing system. Use Detailed Historical Data The foundation of every successful data warehouse is the detailed historical data on which it is based, and this is the third critical success factor. One approach, which has been used by information systems departments to manage the volume and complexity of the issues associated with navigating through weeks, months, or even years of detailed transaction data, is summary data structures. Although these elements have often become a preferred strategy for implementing decision support systems, they frequently become a detriment to the data warehousing system and its original intent. Summary data structures inevitably fall short of meeting requirements, for several reasons: Obscuring data variations: Because they are only summaries of information, they tend to obscure important data variations, masking important variations in corporate data which may point to problems, indicating areas where successful techniques have been applied in the past and may be applied in the future. Complex maintenance: Another deficiency of summary structures is that their maintenance can be fairly complex and quite resource intensive, requiring a significant amount of updating to reflect adjustments to transaction data.

5 Single static scheme: A final deficiency of summary structures is that they are usually created using a single static scheme for organizing transaction level details into a coherent and manageable information format. This limitation ultimately causes the summary structure to fall short because it prevents the business user from viewing the data in a manner conducive to a discovery process. In short, most of today s business problems or opportunities cannot be identified using a few, limited static views of the business activity. Summary data tables do have a place in data warehouse design. However, they should not be considered as an alternative to storing detailed data, but rather as a technique for solving some very well-defined performance problems. Apply Technology The fourth critical success factor is that a successful data warehouse implementation will apply technology to the business problem. One technology which has been applied to the data warehousing solution is symmetric multi-processor (SMP) computer hardware supporting a relational, multi-dimensional, or hybrid database environment. More advanced solutions use massive parallel processor hardware (MPP). In a decentralized data warehouse architecture, this solution will probably employ middleware to coordinate wide area access. Furthermore, it will entail the use of graphic user interface (GUI) application tools (either developed in-house or purchased off-the-shelf) and online analytical processing (OLAP) tools to present the volumes of data in meaningful formats. There are a broad range of architectures which can meet the requirements of a data warehousing system, and the selection of the right technology is a critical factor. However, architectural issues should only be approached when the business problems to be addressed are clearly understood. The technology should always be applied as part of the solution. Evaluations of different technology can consume significant amounts of time and energy. It is better to work with vendors that can provide references which relate to an organization s requirement. Other sources of information are technical publications, seminars and conferences, and research organizations that have conducted studies and evaluated a variety of different issues around data warehousing. Knowledgeable individuals in organizations that have implemented a data warehousing solution are also a major and extremely useful information resource. Trust The Data History Does Not Lie The fifth and final critical success factor in a data warehousing implementation is realizing that historical data is a strategic asset, since it is a

6 source of corporate truths that do not forget or deceive. Human perception and memory can be faulty, and the data warehousing system should not be entrusted to a process which relies on the human memory. Precise point-in-time readings of key business indicators can help recreate a thumbnail sketch of past business events. They can also forecast the success of a future event, potentially reducing the probability of recreating previous business disasters. However, data alone will not solve a business problem. Specialists with specific information system skills will be needed to scrub, load, access, and present the gigabytes and terabytes of transaction data generated each year by the business. Statisticians and business analysts can interpret the business information distilled from all the detailed data, and provide the business analysis and predictive models that project future business trends. While history does not lie, it can sometimes mislead. Inconsistencies, incomplete or absent metadata definitions, data loss from media corruption, or unnecessarily restrictive retention cycles, are potential serious threats to the quality of data residing in a data warehouse. CONCLUSION There are no guarantees with data warehousing implementation; however, the probability of success will be increased significantly if these five critical success factors are addressed. Therefore, it is important to consider these critical factors long before the first query is run or the first gigabyte of data is loaded. A data warehouse system is one of the most complex applications which an organization can implement, since it involves the core business of the organization and a large part of its transaction and business history. It will have a dramatic impact on each information user. While it is not in itself a solution to a business problem, it provides a means to a solution, one which will involve the entire organization in a major cultural change. This change will enable employees to use detailed information as a key to knowledgeable corporate decision-making. Duane E. Sharp is president of SharpTech Associates, a Canadian company specializing in the communication of technology. An electronic engineer with more than 25 years of experience in the IT field, he has authored numerous articles on technology and a textbook on interactive computer terminals, and he has chaired sessions at Comdex Canada. He can be reached at desharp@netcom.ca.