HELSINKI UNIVERSITY OF TECHNOLOGY T Enterprise Systems Integration, Data warehousing and Data mining: an Introduction

Transcription

1 HELSINKI UNIVERSITY OF TECHNOLOGY T Enterprise Systems Integration, Data warehousing and Data mining: an Introduction Federico Facca, Alessandro Gallo,

2 Federico Facca, Alessandro Gallo: Data warehousing and Data mining. Data warehousing and Data mining: an Introduction Abstract The purpose of this paper is giving a short introduction to the concepts of Data mining and Data warehousing and an explanation of their general possibilities and a short description of their uses in the field of Enterprise System Integration. Some references to free material are given to deepen these topics.

3 Federico Facca, Alessandro Gallo: Data warehousing and Data mining. 1 TABLE OF CONTENTS 1 Introduction Background of the research Objectives of the research Scope of the research Research methods What is data warehousing? What is data mining? How are they related? Why use them? Data warehousing Data mining Use in enterprise system integration Data warehousing Data mining How do data warehousing and data mining work? Data warehousing Data mining Product related with data warehousing and data mining IBM ( Oracle ( Sybase ( Angoss, knowledge engineering ( Other relevant data mining products Conclusions Summary Feedback about this exercise References and bibliography Related websites... 10

4 Federico Facca, Alessandro Gallo: Data warehousing and Data mining. 2 1 INTRODUCTION 1.1 Background of the research Data mining and data warehousing are two of the most important instruments for market management, but they re also used in many other fields: risk management, scientific research, medical research, insurance planning strategies, fraud management, banking strategies, game playing strategies (Toronto Raptors, NBA). Together they can be very useful instruments for ESI: often enterprises store a great amount of data in different databases, storing them all together and processing them using a data warehouse is the first thing to do in order to achieve useful information from them. This report is organized in sections: What is data warehousing? A short definition of data warehousing is given What is data mining? In this section short definition of data mining and its field of research are given. How are they related? The scope of this section is to explain in few words how data warehousing and data mining interact. Why use them? Here the reader can find an explanation of motivation to use this kind of solution instead of others. Use in ESI. In this section some considerations on their use from ESI point of view are given. How do data warehousing and data mining work? The purpose of this section is to give some technical advice on data warehousing and data mining power. Products related with data warehousing and data mining. Short overview on the current products in commerce. References. 1.2 Objectives of the research The main objective of this report is to explain what are data mining and data warehousing. Other objectives, that can be considered sub objectives, are to give an overview of their use, to try to explain how they work, to give some references in order to have a better understanding of the matter. 1.3 Scope of the research The scope is to give an overview of data mining and data warehousing, without deepening into too technical details as this paper is just an introduction to these topics and the point of view is related with ESI, not with databases and knowledge engineering. Anyway, for further details the reader can consult references. 1.4 Research methods The main research method is literature study.

5 Federico Facca, Alessandro Gallo: Data warehousing and Data mining. 3 2 WHAT IS DATA WAREHOUSING? The volume of data that a company collects may be very large, like also the databases may be numerous. In such a case, a system that makes easier and faster the process of retrieval information is needed. This instrument is a Data Warehouse. A common definition of data warehouse: A Data Warehouse is a repository of integrated information, available for queries and analysis. Data and information are extracted from heterogeneous sources as they are generated [...]. This makes it much easier and more efficient to run queries over data that originally came from different sources. [Inmon, W.H. 1992]. A data warehouse is a database in which are stored data from the other databases of the company, after that these data have been pre-processed in order to make them more accessible. 3 WHAT IS DATA MINING? Data mining is a method for data processing, nowadays it could be considered the powerful one. Data mining is also known as Knowledge Discovery in Databases KDD, and it can be defined as a method for retrieving information from data ( The nontrivial extraction of implicit, previously unknown, and potentially useful information from data [Frawley W., Piatetsky-Shapiro G. and Matheus C. 1992]). Information and data is not the same thing: data is just something stored somewhere; information is something richer. Data mining becomes a hot topic in the last years thanks to increase of computing power: previous data, which have been compiled and never analysed, have been analysed and the data mining techniques have been improved. The power of data mining is the ability to achieve not visible information stored in the data. Data mining finds patterns to classify data into information. None of other traditional data process methods is so unrelated with human way of thinking: data mining doesn t need a guide to achieve information: there s no need to say to it what to search, that s way it can find precious information previously unknown. 4 HOW ARE THEY RELATED? Data mining is useful especially if there s a great amount of data to analyse: the biggest and the most complete data repositories actually are data warehouses. So the link between these two things is very clear. Data Selection Target Data Data Mining Categories of risky customers KNOWLEDGE Customer Products Orders Product Inventory Product price Customer likely to buy new products Categories of profitable customer Data Warehouse

6 Federico Facca, Alessandro Gallo: Data warehousing and Data mining. 4 Figure 1. From data to knowledge using data mining 5 WHY USE THEM? 5.1 Data warehousing In a company where there are different databases, organized in different ways according to the needs of the single department or unit of the enterprise, the retrieval of the useful information for the strategy or other high level decisions, like marketing or customer service decisions, may be a difficult and slow process. On the other hand, the databases of an enterprise are often based on different systems, like mainframes and old systems, called legacy systems, and newer systems (for example built upon serverclient architecture). So, in order to provide an instrument that can support high level decisions and give the right information at the right time, integration of databases and pre-processing of the great amount of data are needed. These are the functions that a data warehouse implements. There is another task that data warehouse can perform. It could be useful not only to retrieve information, but also for create new knowledge from the available data. In fact, data warehouse is often used like a support for the activity of data mining. 5.2 Data mining There are a lot of methods for processing data, but most of them are deeply related with the ideas and way of thinking of the people who are using them. They need to be guided in some way by human intelligence. Also data mining can work in this way, but it can work also in a more independent way from human minds. This is very useful if there s not a concrete idea of the information to be found. This feature in some field of research could be very important: discover previous unconsidered relation between some diseases and other factors, for example, can lead to find a new approach to the study of these diseases. 6 USE IN ENTERPRISE SYSTEM INTEGRATION One of the main purpose of enterprise system integration is knowledge management, and it can be split in three categories: knowledge acquisition, knowledge organization and knowledge deployment. The first and the second category are related with data mining and data warehousing. As previous said, data mining is a powerful instrument for information retrieving, and this is directly related with knowledge acquisition. As regards knowledge organization, one of the functions of data warehousing is storing data in order to support business analysis and management decision-making. So, the use of data warehousing and data mining can help the ESI process, but, on the other hand, the process of creating a data warehouse and, then, performing data mining has to be led by the business policy of the company, especially during the preprocessing of the data

7 Federico Facca, Alessandro Gallo: Data warehousing and Data mining. 5 of the different databases, in phases like elimination of not useful data and aggregation. 6.1 Data warehousing Often information is split in different database according to the needs of the different components of the company. The marketing division has its own database, with a structure to fulfill its needs, and so on for the sales division, the product development division,... Data stored in these way are not very helpful for the management purpose and for having a complete overview of the company. So through data warehousing is possible to process and combine data in an automated way in order to fulfill needs previous unsatisfied. This is needed for developing a decision support system. Data Warehouse Customer Products Orders Product Inventory Product price Product price Product inventory Product Price Change Customer orders Product price Avaiable inventory Customer Profile Product price Marketing programs Inventory database Order processing database Marketing database Figure 2. Data warehouse at work 6.2 Data mining This instrument can be a very important help for discover new information that can support the planning of new strategies for the company, the analysis of current strategies, the development of new products, and so on. One of the most important fields, related with ESI, in which data mining is used, is CRM (Customer Relationship Management).

8 Federico Facca, Alessandro Gallo: Data warehousing and Data mining. 6 CRM is a process that manages the interactions between a company and its customers. The primary users of CRM software applications are database marketers who are looking to automate the process of interacting with customers. Data mining applications automate the process of searching the mountains of data to find patterns that are good predictors of purchasing behaviors. After mining the data, marketers must feed the results into campaign management software that, as the name implies, manages the campaign directed at the defined market segments. Data mining helps marketing users to target marketing campaigns more accurately; and also to align campaigns more closely with the needs, wants, and attitudes of customers and prospects. If the necessary information exists in a database, the data mining process can model virtually any customer activity. The key is to find patterns relevant to current business problems. 7 HOW DO DATA WAREHOUSING AND DATA MINING WORK? 7.1 Data warehousing Data warehousing is something more than a second copy of data, otherwise it would be a simply backup database. Creating and maintaining a data warehouse implies other operations, which can be classified in: extraction, consolidation, filtering, cleansing, transformation, aggregation and updating. Extraction: periodical download of new data from various databases. Consolidation: combination of data from different databases in order to perform data analysis. Filtering: elimination of data not needed for analysis. Cleansing: finding and repairing errors due to data manipulations. Transformation: modification of data in order to make them consistent. Aggregation: summarization of data into appropriate units for analysis. Updating: adding new data. 7.2 Data mining There are a lot of techniques related with data mining, but the general process can be described using the following steps: Identification of the problem.

9 Federico Facca, Alessandro Gallo: Data warehousing and Data mining. 7 Data preparation: before applying data processing techniques, data needs to be manipulated in order to choose the relevant ones. Creation of data mining patterns: using different techniques is possible to obtain different patterns. The patterns are obtained by selecting a training set of data (a subset of existing data used to create the pattern) and by testing them using other subsets of data called testing sets. Testing sets and techniques are needed in order to avoid problem like overfitting: the pattern fits efficiently the data given but is not useful for other set of data, as it is too tied up with training set data. To choose between different patterns generated with different techniques a valuation of the kind of errors that the patterns are likely to generate is needed. The choice of the technique is driven by the goal that is to be achieved: for example, fraud recognition in an assurance company suggests the use of a technique of classification (data mining is used to find rules useful to classify in categories, like safe and not safe, the costumers, using age, profession and other parameters), products sales analysis in a supermarket needs a technique of associations recognition (collected data are used to find new relations between products). Other techniques are, for example, clustering and regression. 8 PRODUCT RELATED WITH DATA WAREHOUSING AND DATA MINING Data warehouse market is increasing a lot in 1997 it was around $3.5 billion and it increased till $8 billion in 1998 [Metagroup survey] and now is considered a stable market with valued products. Data mining products market is still developing a lot, there are more than 200 tools but only few of them are used in large contests. Here are presented data on few diffused solutions for data warehousing and data mining (in many cases they re the leader in database marketing). In the case of data mining solutions is interesting to point out that some of them are based on an owned implementation of JDM API specification and API, which should contain an implementation of the most important algorithms of data mining. 8.1 IBM ( IBM gives to its customer a complete and integrated solution for data warehousing and data mining. DB2 Solution Family: DB2 Universal Database $20, DB2 Intelligent Miner for Data $60,000.00

10 Federico Facca, Alessandro Gallo: Data warehousing and Data mining. 8 DB2 Warehouse manager $10, Example of customer: Sogei uses IBM solutions for security and CRM solution [ 4.ibm.com/software/ data/solutions/pdf2/sogei.pdf] 8.2 Oracle ( Oracle, like IBM, extends its database products with options that allow data warehousing and data mining. The data mining solution is not tied up with Oracle s database, actually can be used with the most diffused databases. Oracle9i Solution Family: Oracle Database Enterprise Edition Oracle Partitioning Oracle Data Mining Example of customer: Lowestfare.com is a full-service provider of discount travel products and services. This company uses Oracle solutions to manage its customer data in order to improve its CRM. [ 8.3 Sybase ( Sybase offers databases products and a data warehouse solution compatible with other database solutions. Sybase solutions: Adaptive Server Enterprise $3, Industry Warehouse Studio $60, Example of customer: Telecel, a portuguese leading telecommunications operator, used Sybase solutions to reengineer its data warehouse to accommodate new business lines and changing technical environments. [

11 Federico Facca, Alessandro Gallo: Data warehousing and Data mining Angoss, knowledge engineering ( Angoss provides specific solutions for data mining that can be used with almost any database solutions. Angoss solutions: KnowledgeSTUDIO KnowledgeSEEKER KnowledgeSERVER Example of customer: Bank of Montreal used KnowledgeSTUDIO to analyse a large amount of customer data, with different purposes, like customer segmentation, analysis of credit risk and predictions of defaulting on mortgage payments. [ 8.5 Other relevant data mining products Data Mining Business Solutions by DigiMine ( Xchange 8 by Xchange ( Clementine by SPSS ( 9 CONCLUSIONS The value of an instrument or a technology is directly tied with the importance of the problems that it can help to resolve or the relevance of the results that it provides. So, according to this consideration, data warehouse and data mining are of actual and strategic importance, since they are related with problems which are actual and strategic. The reason for which data warehouse and data mining are so popular nowadays is perhaps that, in a world where information is a so important resource for an enterprise, they can create and make more powerful this resource, working not only by themselves, but also integrated with other instrument or in a wider philosophy, like CRM. In order to better understand what they can do, and also what they cannot do, it is useful to see at how they work, which are the relation between them and which are the link between them and the needs of a company, like enterprise integration or marketing research. Data warehouse makes information more accessible and useful for the whole company, processing the data collected in the different existing databases, data mining create new knowledge operating with algorithms that classify or find relations between these data. Knowledge and information, that are central topics for enterprises health, are also the issues to which data warehouse and data mining look.

12 Federico Facca, Alessandro Gallo: Data warehousing and Data mining SUMMARY All the information that was needed to write this report was collected from Internet sites and books concerning datawarehouse and datamining (see the references). The phase of collection of the material was partially done also during the final phase of drawing up. The material was found first looking for definition, description and explanation of the concepts of data warehouse and data mining, then searching real products descriptions and concrete examples of the use of these technologies in some companies, in order to give some references for a deeper study. 11 FEEDBACK ABOUT THIS EXERCISE The subject of this report was not totally unknown to the authors, collecting the material gave new knowledge about the topic and an interesting view on concrete cases of enterprises that use and enterprises that provide data warehouse and data mining instruments. 12 REFERENCES AND BIBLIOGRAPHY Frawley, W.; Piatetsky-Shapiro, G.; and Matheus, C Knowledge Discovery in Databases: An Overview. AI Magazine, 13(3): Inmon, W.H EIS and the data warehouse: a simple approach to building an effective foundation for EIS. Database Programming & Design, 5(11): Berson,A., Smith, S., Thearling, K Building Data Mining Applications for CRM. McGraw Hill. Norris, G., Dunleavy, J., Hurley, J. R., Balls, J. D., Balls, J., Hartley, K. M E- Business and ERP: Transforming the Enterprise, John Wiley & Sons, 216 pp., ISBN: Palace, B Data Mining. teacher/technologies/palace/index.htm. Steven, A Information Systems: Foundation of E-Business, 4/e. Pearson Education, 576 pp., ISBN Steven, A Information Systems: a management perspective, third edition. Addison-Wesley Educational publishers. ISBN RELATED WEBSITES AJx Data Mining Web Page

13 Federico Facca, Alessandro Gallo: Data warehousing and Data mining. 11 Data Warehouse Resources Online data mining system KDnuggets The Data Warehousing Information Center The Data Warehousing Institute