Intellectual Property / Copyright Material

Transcription

1 What is Data Mining? Author: BALWANT RAI Organization: Evaltech, Inc. Evaltech Research Group, Data Warehousing Practice. Date: 02/27/04 Abstract: In this paper we will be going through data mining. The process of extracting previously unknown, valid and actionable information from large databases and then using the information to make crucial business decisions is called data mining. Data mining activities are usually performed by three different classes of users - executives, end users and analysts. Virtually any process from pharmacology to customer service can be studied, understood, and improved using data mining. The top three end uses of data mining are, not surprisingly, in the marketing area - customer profiling, targeted marketing, and market-basket analysis. At last we will be going through different types of data mining techniques. Intellectual Property / Copyright Material All text and graphics found in this article are the property of the Evaltech, Inc. and cannot be used or duplicated without the express written permission of the corporation through the Office of Evaltech, Inc. Evaltech, Inc. Copyright 2004 Page 1 of 7

2 DATA MINING The process of extracting previously unknown, valid and actionable information from large databases and then using the information to make crucial business decisions is called data mining. Data Mining is the process of extracting knowledge hidden from large volumes of raw data. The importance of collecting data that reflect your business or scientific activities to achieve competitive advantage is widely recognized now. Powerful systems for collecting data and managing it in large databases are in place in all large and mid-range companies. However, the bottleneck of turning this data into your success is the difficulty of extracting knowledge about the system you study from the collected data. Human analysts with no special tools can no longer make sense of enormous volumes of data that require processing in order to make informed business decisions. Data mining automates the process of finding relationships and patterns in raw data and delivers results that can be either utilized in an automated decision support system or assessed by a human analyst. All complex questions that can probably be answered if information hidden among megabytes of data in your database can be found explicitly and utilized. Modeling the investigated system, discovering relations that connect variables in a database are the subject of data mining. Modern computer data mining systems self learn from the previous history of the investigated system, formulating and testing hypotheses about the rules which this system obeys. When concise and valuable knowledge about the system of interest had been discovered, it can and should be incorporated into some decision support system which helps the manager to make wise and informed business decisions. DATAMINING USERS AND ACTIVITIES Data mining activities are usually performed by three different classes of users - executives, end users and analysts. Executives need top-level insights and spend far less time with computers than the other groups. End users are sales people, market researchers, scientists, engineers, physicians, etc. Analysts may be financial analysts, statisticians, consultants, or database designers. These users usually perform three types of data mining activity within a corporate environment: episodic, strategic and continuous data mining. In episodic mining we look at data from one specific episode such as a specific direct marketing campaign. We may try to understand this data set, or use it for prediction on new marketing campaigns. Analysts usually perform episodic mining. In strategic mining we look at larger sets of corporate data with the intention of gaining an overall understanding of specific measures such as profitability. Hence, a strategic mining exercise may look to answer questions such as: "where do our profits come from?" or "how do our customer segments and product usage patterns relate to each other?" In continuous mining we try to understand how the world has changed within a given time period and try to gain an understanding of the factors that influence change. Why use data mining? Data mining uses algorithms to sift through huge volumes of information for the purpose of detecting patterns hidden in the data. Understanding these patterns quickly leads to improved business intelligence Data might be one of the most valuable assets of your corporation - but only if you know how to reveal valuable knowledge hidden in raw data. Data mining allows you to extract diamonds of knowledge from your historical data and predict outcomes of future situations. It will help you optimize your business decisions, increase the value of each customer and communication, and improve satisfaction of customer with your services. In all these cases data mining can help you reveal knowledge hidden in data and turn this knowledge into a crucial competitive advantage. Evaltech, Inc. Copyright 2004 Page 2 of 7

3 Today increasingly more companies acknowledge the value of this new opportunity and turn to Megaputer for leading edge data mining tools and solutions that help optimizing their operations and increase your bottom line. DATA MINING APPLICATIONS Virtually any process from pharmacology to customer service can be studied, understood, and improved using data mining. The top three end uses of data mining are, not surprisingly, in the marketing area - customer profiling, targeted marketing, and market-basket analysis. Customer profiling : In customer profiling, characteristics of good customers are identified with the goals of predicting; who will become one and helping marketers target new prospects. Data mining can find patterns in a customer database that can be applied to a prospect database so that customer acquisition can be appropriately targeted. For example, by identifying good candidates for mail offers or catalogs direct-mail marketers can reduce expenses and increase their sales. Targeting specific promotions to existing and potential customers offers similar benefits. Market basket analysis Market-basket analysis helps retailers understand which products are purchased together or by an individual over time. With data mining, retailers can determine which products to stock in which stores, and even how to place them within a store. Data mining can also help assess the effectiveness of promotions and coupons. Another common use of data mining in many organizations is to help manage customer relationships. By determining characteristics of customers who are likely to leave for a competitor, a company can take action to retain that customer because doing so is usually far less expensive than acquiring a new customer. Fraud detection is of great interest to telecommunications firms, credit-card companies, insurance companies, stock exchanges, and government agencies. The aggregate total for fraud losses is enormous. But with data mining, these companies can identify potentially fraudulent transactions and contain the damage. Financial companies use data mining to determine market and industry characteristics as well as predict individual company and stock performance. Another interesting niche application is in the medical field: Data mining can help predict the effectiveness of surgical procedures, diagnostic tests, medications, service management, and process control. What can Data Mining do for you? Identify your best prospects and then retain them as customers. By concentrating your marketing efforts only on your best prospects you will save time and money, thus increasing effectiveness of your marketing operation. Predict cross-sell opportunities and make recommendations. Whether you have a traditional or web-based operation, you can help your customers quickly locate products of interest to them - and simultaneously increase the value of each communication with your customers. Learn parameters influencing trends in sales and margins. You think you could do this with your OLAP tools? True, OLAP can help you prove a hypothesis - but only if you know what questions to ask in the first place. In the majority of cases you have no clue on what combination of parameters influences your operation. In these situations data mining is your only real option. Segment markets and personalize communications. There might be distinct groups of customers, patients, or natural phenomena that require different approaches in their handling. If you have a broad customer range, you would need to address teenagers in California and married homeowners in Minnesota with different products and messages in order to optimize your marketing campaign. Evaltech, Inc. Copyright 2004 Page 3 of 7

4 Reasons for the growing popularity of Data Mining Growing Data Volume The main reason for necessity of automated computer systems for intelligent data analysis is the enormous volume of existing and newly appearing data that require processing. The amount of data accumulated each day by various business, scientific, and governmental organizations around the world is daunting. According to information from GTE research center, only scientific organizations store each day about 1 TB (terabyte!) of new information. And it is well known that academic world is by far not the leading supplier of new data. It becomes impossible for human analysts to cope with such overwhelming amounts of data. Limitations of Human Analysis Two other problems that surface when human analysts process data are the inadequacy of the human brain when searching for complex multifactor dependencies in data, and the lack of objectiveness in such an analysis. A human expert is always a hostage of the previous experience of investigating other systems. Sometimes this helps, sometimes this hurts, but it is almost impossible to get rid of this fact. Low Cost of Machine Learning One additional benefit of using automated data mining systems is that this process has a much lower cost than hiring an army of highly trained (and payed) professional statisticians. While data mining does not eliminate human participation in solving the task completely, it significantly simplifies the job and allows an analyst who is not a professional in statistics and programming to manage the process of extracting knowledge from data. Different DM Technologies and Systems It would be very instructive to discuss various existing approaches to data mining while stressing out the following three vital criteria: Control of the significance of obtained results Transparity of developed empirical models and their interpretability Degree of search process automatisation and ease-of-use DATA MINING TECHNIQUES:- Genetic Algorithms Genetic algorithms also generate rules from data sets, but do not follow the exploration oriented protocol of rule induction. Instead, they rely on the idea of mutation to make changes in patterns until a suitable form of pattern emerges via selective breeding. Figure The genetic cross- over operation is in fact very similar to the operation breeders use when they cross-breed plants and/or animals. The exchange of genetic material by chromosomes is also Evaltech, Inc. Copyright 2004 Page 4 of 7

5 based on the same method. In the case of rules, the material exchanged is a part of the pattern the rule describes. Let us note that this is different from rule induction since the main focus in genetic algorithms is the combination of patterns from rules that have been discovered so far, while in rule induction the main focus of the activity is the dataset. Genetic algorithms are not just for rule generation and may be applied to a variety of other tasks to which rules do not immediately apply, such as the discovery of patterns in text, planning and control, system optimization, etc. Decision Trees Decision Trees are one of the most popular data mining techniques in use today. Decision trees use innovative, well-proven techniques derived from statistical and artificial intelligence research to automatically find correlations and groupings in data. Decision trees express a simple form of conditional logic. A decision tree system simply partitions a table into smaller tables by selecting subsets based on values for a given attribute.users can browse these correlations and examine relationships in context to gain valuable in sights into the data. Rather than the user requesting what dimensions to view, the algorithms of the decision tree root out what dimensions are affecting the business goal. As the tree grows, the factors that affect different groupings within the data are discovered and visualized by the user. The correlations found by the decision tree algorithms are used to infer business rules. These rules form the basis for a predictive model. One of the principle advantages of decision trees, as compared to other data mining techniques, is that the rules and thus the models can be stated in plain English. The interactive capability of decision trees is useful for building predictive models. Once the decision tree algorithms have generated a list of correlated variables (at any given node), the modeler chooses which split to use. His or her domain knowledge plays a critical role in the model creation process. Interactive decision tree creation effectively combines the human intelligence and domain expertise with the pedantic computational power of decision tree algorithms. All data types are dealt with carefully. Algorithms automatically change when the user selects a continuous dependent variable or categorical dependent variable. If an independent variable is continuous, the variable is automatically pre-categorized. Five different algorithms are available to do the precategorization, including a heuristic algorithm. Cross Tabulation Cross tabulation is a very basic and simple form of data analysis, well known in statistics, and widely used for reporting. A two dimensional cross-tab is similar to a spreadsheet, with both row and column headings as attribute values. The cells in the spreadsheet represent an aggregate operation, usually the number of co-occurrences of the attribute values together. Many cross-tabs are effectively equivalent to a 3D bar graph which displays co-occurence counts When dealing with a small number of non-numeric values, cross-tabs are simple enough to use and find some conditional logic relationships (but not attribute logic, affinities or other forms of logic). Cross-tabs usually run into four classes of problems: first when the number of non-numeric values goes up, second when one has to deal with numeric values, third when several conjunctions are involved, and fourth when the relationships are not just based on counts. Neural Networks Neural Networks are nearly as popular as decision trees. While they are not interactive and they do not explain their results, they do tend to produce models with better predictive power than do decision trees. Evaltech, Inc. Copyright 2004 Page 5 of 7

6 Neural Nets Neural nets are a class of predictive modeling system that works by iterative parameter adjustment. Structurally, a neural network consists of a number of interconnected elements (called neurons) organized in layers which learn by modifying the connection strengths (i.e., the parameters) connecting the layers. Neural nets usually construct complex equation surfaces through repeated iterations, each time adjusting the parameters that define the surface. After much iteration, a surface may be internally defined that approximates many of the points within the dataset. The basic function of each neuron is to: (a) Evaluate input values, (b) Calculate a total for the combined input values, (c) Compare the total with a threshold value and (d) Determine what its own output will be. While the operation of each neuron is fairly simple, complex behavior can be created by connecting a number of neurons together. Typically, the input neurons are connected to a middle layer (or several intermediate layers) which is then connected to an outer layer.to build a neural model, we first train the net on a training dataset, then use the trained net to make predictions. We may, at times, also use a monitoring data set during the training phase to check on the progress of the training. Each neuron usually has a set of weights that determine how it evaluates the combined strength of the input signals. Inputs coming into a neuron can be either positive (excitatory) or negative (inhibitory). Learning takes place by changing the weights used by the neuron in accordance with classification errors that were made by the net as a whole. The inputs are usually scaled and normalized to produce a smooth behavior. During the training phase, the net sets the weights that determine the behavior of the intermediate layer. A popular approach is called back propagation in which the weights are adjusted based on how closely the network has made guesses. Incorrect guesses reduce the thresholds for the appropriate connections. Neural nets can be trained to reasonably approximate the behavior of functions on small and medium sized data sets since they are universal approximations. However, in practice they work only on subsets and samples of data and at times run into problems when dealing with larger data sets (e.g., failure to converge or being stuck in a local minimum. It is well known that back propagation networks are similar to regression. There are several other network training paradigms that go beyond back propagation, but still have problems in dealing with large data sets. One key problem for applying neural nets to large data sets is the preparation problem. The data in the warehouse has to be mapped into real numbers before the net can use it. This is a difficult task for commercial data with many non- numeric values. Since input to a neural net has to be numeric (and scaled), interfacing to a large data warehouse may become a problem. For each data field used in a neural net, we need to perform scaling and coding. The numeric (and date) fields are scaled. They are mapped into a scale that makes them uniform.this is not a very difficult task. However, non-numeric values cannot easily be mapped to numbers in a direct manner since this will introduce unexpected relationships into the data, leading to errors later. To be used in a neural net, values for nonscalar fields such as City, State or Product need to be coded and mapped into new fields, taking the values 0 or 1. This means that the field State which has values is no longer used. Instead, we have new fields, each taking the value 0 or 1, depending Evaltech, Inc. Copyright 2004 Page 6 of 7

7 on the value in the record. For each record, only one of these fields has the value 1, and the others have the value 0. In practice, there are often 50 states, requiring 50 new inputs. Now the problem should be obvious: What if the field City has 1,000 values? Do we need to introduce 1,000 new input elements for the net? In the strict sense, yes, we have to. But in practice this is not easy, since the internal matrix representations for the net will become astronomically large and totally unmanageable. Hence, by-pass approaches are often used. Some systems try to overcome this problem by grouping the 1,000 cities into 10 groups of 100 cities each. Yet, this often introduces bias into the system, since in practice it is hard to know what the optimal groups are, and for large warehouses this requires too much human intervention. In fact, the whole purpose of data mining is to find these clusters, not asks the human analyst to construct them. The distinguishing power of neural nets comes from their ability to deal with smooth surfaces that can be expressed in equations. These suitable application areas are varied and include fingerprint identification and facial pattern recognition. However, with suitable analytical effort neural net models can also succeed in many other areas such as financial analysis and adaptive control. Eventually, the best way to use neural nets on large data sets will be to combine them with rules, allowing them to make predictions within a hybrid architecture. Belief Networks Belief networks (sometimes called causal networks) also rely on co-occurrence counts, but both the graphic rendering and the probabilistic representation are slightly different from agents. Belief networks are usually illustrated using a graphical representation of probability distributions (derived from counts). A belief network is thus a directed graph, consisting of nodes (representing variables) and arcs (representing probabilistic dependencies) between the node variables. Each node contains a conditional probability distribution that describes the relationship between the node and the parents of that node. The belief network graph is acyclic, meaning that there are no cycles. The arcs in belief network denote the probabilistic dependencies between the nodes, rather than impacts computed from the cross-tabs. Evaltech, Inc. Copyright 2004 Page 7 of 7