Hard hats for data miners: Myths and pitfalls of data mining

Transcription

1 Hard hats for data miners: Myths and pitfalls of data mining T. Khabaza SPSS Advanced Data Mining Group Abstract The intrepid data miner runs many risks, such as being buried under mountains of data or vanishing along with the mysterious disappearing terabyte. This paper debunks some myths and sketches some hard hats for data miners. 1 Introduction Data mining is a business process, finding patterns in your data which you can use to do your business better. Through data mining we gain insight into a business problem; this insight may be of use in itself, but it also helps us to gain the other benefits of data mining, such as a predictive capability. This paper is about the practice of data mining; it is not a research paper, but reports lessons learned through solving practical business problems and through contact with many data mining users and potential users. There are many myths and misconceptions about data mining, and holding these misconceptions leads data mining users to run specific risks. The first half of this paper lists some common misconceptions about data mining, corrects them, and describes the risks to which they can lead. The second half of the paper lists other common problems or pitfalls of data mining, with their symptoms and cures. 2 Myths and misconceptions about data mining 2.1 Myth #1: Data mining is all about algorithms The ordinary business-person, attending a typical data mining conference, reading its proceedings, or even reading only the contents page of such a

2 14 Data Mining III proceedings, could be forgiven for thinking that data mining is all about advanced data analysis algorithms. This misconception might be summarised as all you need for data mining is good algorithms; the better your algorithms, the better your data mining, and its corollary advancing the state of the art in data mining means advancing knowledge of algorithms. To hold this view is to misunderstand the data mining process completely. Data mining is a business process, involving many elements such as formulating business goals, mapping business to data mining goals, acquiring, understanding and pre-processing the data, evaluating and presenting the results of analysis and deploying these results to achieve business benefit, as well as the modelling component. (A good explanation of this process can be found in the emerging industry standard process model CRISP-DM [1].) In their extreme form, the consequences of holding this misconception are disastrous for a data mining project, and such a project will fail to produce any useful results. In practice, this occurs only in the narrowest, most academic of projects, where useful results for the business are not absolutely required. In any project where there is a requirement for the results to benefit the business, the data miner who holds this misconception is forced to discard it, at least partially, and face the need for a broader view of the data mining process. This is not to denigrate those parts of data mining research which develop or improve data mining algorithms. Algorithms play a key role in data mining, and new or improved algorithms are one way in which the art of data mining advances. The problem occurs when we focus mainly or solely on algorithms and ignore the other!)()-!)5~0 of the data mining process. 2.2 Myth #2: Data mining is all about predictive accuracy Above I have rejected the notion that data mining is all about modelling algorithms, but within that part of data mining which is about algorithms, how can we judge the quality of an algorithm? Readin< data mining research literature might lead us to suppose that the main criterion for judging an algorithm is the predictive accuracy of the models it generates. This view completely misrepresents the role of algorithms in the data mining process. It is true that in order to be useful a predictive model should have some degree of accuracy, because this reflects whether the algorithm has really discovered patterns in the data. However, many other properties of an algorithm or a model affect its usefidness; examples include whether the model can be understood by the analyst, and whether it requires technical knowledge to understand the model or apply the algorithm. Considering the properties (other than predictive accuracy) which the data mining process requires of algorithms, we can see the likely consequences of holding this mistaken view: algorithms will be produced which can be used only by technology experts. These algorithms will have only the most limited role in a process which is driven by business expertise.

3 Data Mining III ~5 2.3 Myth #3: Data mining requires a data warehouse Data mining practitioners often hear statements like we are not ready for data mining yet, we need to build our data warehouse first. Such statements are based on the view that data warehousing is a pre-requisite for data mining. This is a subtle misconception about the relationship between data warehousing and data mining. It is true that data mining can benefit from the warehoused data being well organised, relatively clean, and easy to access. These benefits can accrue if the warehouse has been constructed with data mining specifically in mind, and with knowledge of the requirements of the data mining envisaged. If it has not, the warehoused data may be less useful for data mining than the source operational data, or in the worst case completely useless (for example in cases where only summary data is warehoused). To avoid this risk, it is usefhl to perform pilot data mining projects using operational data in order to determine the correct content and organisation for the warehouse. It is misleading to state that data mining requires a data warehouse; a more accurate summary of the relation would be that data mining can benefit from a data warehouse, but that to construct such a warehouse often requires data mining. 2.4 Myth #4: Data mining is all about vast quantities of data Early explanations of data mining in the computing press often start with statements like We now collect more data than ever, yet how are we to gain benefit from these vast data stores?. To focus on the size of data stores provides a convenient introduction to the topic of data mining, but subtly misrepresents its nature. Data mining becomes useful when data becomes too large or too complex to analyse by eye, that is anything larger than a few tens of examples and a handful of attributes. Many usefil data mining projects are performed on small or medium-sized datasets, for example containing only hundreds or thousands of records. Apart ffom its convenience in popular explanations, an association of data mining with vast datasets is also connected with the recent emphasis on performance and scalability of data mining tools. This drive to extend the reach of data mining tools to large data is perfectly justified there are many large datasets which it benefits us to mine. However it would be a mistake to believe that these large datasets are the sole focus of data mining. Holding this erroneous belief would lead us to produce tools which sacrifice usability for scalability, whereas in fact both aspects are essential. To quote a customer of a leading data mining tool: other data mining tools optimise machine time, but this tool optimises my time. Whether the datasets are large or small we much strive to optimise the user s time, and this may be assisted by scalability and performance.

4 16 Data Mining III 2.5 Myth #5: Data mining should be done by a technology expert Data mining technology, particularly modelling techniques, is of an advanced sort, and its workings are unlikely to be understood by the wider IT community. Some would claim that this means they should be applied only by technology experts who understand their workings. (This claim may be influenced by a historical association with statistical modelling algorithms, which are more open to misinterpretation than most data mining algorithms.) In fact, the very reverse is true, because of the paramount importance of business knowledge in data mining. When performed without business knowledge, data mining usually produces nonsensical or useless results (see pitfall #3 below). It is therefore essential that data mining is performed by someone with extensive knowledge of the business problem, which is very seldom combined with knowledge of the technology. It is the responsibility of data mining tool providers to ensure that tools are accessible to business rather than technology experts. It behoves the data mining community at large to make clear to potential users that data mining provides insight and useful suggestions, rather than mathematical certainty. 2.6 Myth #6: Neural networks are opaque and consequently useless an over-simplistic view of data mining Myth #6 is a relatively specific misconception about one family of modelling techniques (neural networks) which arises from a broader misunderstanding about the data mining process. One sometimes encounters the view that neural networks are not very useful in data mining because one cannot discover why they make the predictions that they do, or the rules that they use. This means that their predictions cannot be justified, and that they will not contribute much insight. While this argument reflects a correct emphasis on understandability of models and the insight produced by data mining, the conclusion about the disutility of neural networks is erroneous, and the argument reflects a mistaken view of the data mining process, possibly related to myth #1. This mistaken view regards data mining as a rather simple process: take the data, apply a modelling technique, use the results. This omits the iterative nature of the data mining process, and the way in which many techniques are used together to produce a result. Neural networks are used in a variety of ways in data mining projects, uses which are not impacted by the opacity of the models. Here are some examples:. Neural networks can be used for attribute selection, either by training them repeatedly with different combinations of attributes, or by using techniques of sensitivity analysis to rank the attributes by their impact on predictions. Neural networks can be used for pattern confiiation because they are particularly powerful pattern finders for many applications, neural networks can be used to confirm that a pattern exists, before spending effort on tuning other techniques to find it.

5 Data Mining III 1 7 Neural networks can be used in conjunction with other techniques, for example to improve the confidence of predictions by discarding those where the neural network disagrees with the predictions of another technique, or by using other techniques to analyse the behaviour of the neural networks. All of these uses of neural networks reflect the fact that the data mining process cannot be summarised as apply a modelling technique and use the results. Data mining facilities form a toolbox, whose contents are used in varied and sometimes surprising ways to solve a problem. 3 Pitfalls of data mining and their cures 3.1 Pitfall #1: Buried under mountains of data Data mining should be an interactive, iterative process where the analyst applies substantial business knowledge and is engaged with the data. However, those who hold myth #4 (that data mining is about vast quantities of data) often suppose that this process must be applied to all of the available data. This can lead to attempts to mine volumes of data for which the available hardware and software cannot provide an acceptable interactive response (for example, building a model within a few minutes). The data mining process becomes sluggish, and by the time a question is answered, the analyst cannot remember why it was asked. It is hard to feel that this process is generating insight. The cure for this malaise is usually some form of sampling. For example, if we have a million customers and a 20 /0 annual attrition (or chum ) rate, we need not plot our graphs or build our models using the fill million examples, or even half a million (leaving, say, half for independent results validation). Consider the following questions and answers: Q: How many chum profiles do we expect to find? A: Maybe ten. Q: How many examples of each profile do we need? A: Maybe a thousand. Conclusion: A sample of ten or twenty thousand churners, and an equivalent number of non-churners, will be sufficient for this analysis. Note that this does not mean that we will never encounter the need to build models from millions of examples, only that we should not assume that we must do so if this data is available. One interesting class of cases is those where we wish to find a rare profile. Suppose that we wish to find a specific phenomenon which causes only 1 % of churn. It might be thought that we must build models against the whole dataset in order to find it. However there are other approaches. For example we might find the common chum profiles first, using a relatively small sample to build the models, use these initial profdes to score the entire database, and then focus

6 18 Data Mining III subsequent analysis on the relatively small number who chum but were predicted not to do so. 3.2 Pitfall #2: The Mysterious Disappearing Terabyte This is a common phenomenon, but not always a pitfall. The phrase mysterious disappearing terabyte refers to the fact that for a given data mining problem, the amount of available and relevant data maybe much less than initially supposed. Consider the following scenario: You are a data mining consultant, and your client is a large bank, holding terabytes of data on its customers. There is some concern that the available computing resources will be inadequate to mining this volume of data. The bank wishes to mine information on credit risk. Different types of credit (for example personal loans, business loans, overdrafts) would present different patterns of credit risk, so each data mining project will concentrate on one type of borrower. A number of factors are judged (by the bank s domain experts) to be relevant. Are these factors collected by the bank? Yes, they have looked ahead and started collecting the relevant factors, eighteen months ago. Lots of borrowing has taken place in the intervening time so there should be no problem about data! How many bad debts of the relevant kind have occurred in that time? Plenty - almost a thousand! Thus the relevant data consists of less than a thousand cases of bad debt plus a sample from a plentifid supply of cases of good debt - say 3,000 records in all. Somehow, terabytes of data have softly and silently vanished away, fortunately not (quite) taking the data miner with them (this time). 3.3 Pitfall #3: Insufficient business knowledge I have emphasised previously the crucial role played by business knowledge in data mining. Without it, we can neither recognise useful results nor guide the data mining process towards them. It is sometimes supposed that the end user of data mining can reasonably take the attitude: here is the data, please go away and mine it, and come back with the answers. When a data mining project is organised in this way, at best the project will take many long and costly iterations to produce useful results, and at worst the results will be gibberish and the project will fail. This pitfall can only be avoided by involving the end user, and more specifically someone with a detailed knowledge of the business, at every stage of the data mining process. Ideally the data miner should be part of that business, but if a data mining consultant is used then the consultant should literally sit next to someone with the required business knowledge who understands the question under consideration. For this to work, a highly interactive data mining environment with good response time is required.. (A data mining consultant with general knowledge of the relevant industry is not sufficient detailed knowledge of the specific business is needed.)

7 Data Mining III Pitfall #4: Insufficient data knowledge In order to perfonm data mining we must be able to answer questions like what do the codes in this field mean?, and can there be more than one record per customer in this table?. In some cases this information is surprisingly hard to come by for example because the data expert has left the organisation or moved to another department, or in the case of legacy systems there may be no data expert at all. This problem is exacerbated when the database or data warehouse management is outsourced the external supplier is even less motivated than the user organisation to maintain the information in case it is needed in future. There is no simple cure for this problem. IT departments should be made aware of the need to maintain information about the organisation s databases, and when a data mining project is proposed we should consider how much data knowledge is available, and any risks caused by its absence or scarcity. 3.5 Pitfall #5: Erroneous assumptions, courtesy of the experts Business and data expertise are crucial resources for data mining, but that does not mean that the data miner should accept unquestioningly every statement of the experts. One benefit from data mining is that organisations discover surprising facts about their data and about their business. The data miner should seek to confirm the truth of experts statements so far as they relate to the data. Typical examples of erroneous or misleading statements would include: No customer can hold accounts of both these types. No case will include more than one event of this type. Only the following codes will be present in this field. Statements like this should be verified by examining the data. Data mining tools should make this easy. It is particularly important to check these issues when processing of the data will depend on them, so that mistakes in these assumptions can be spotted before they lead to errors in the treatment of data. 3.6 Pitfall #6: Incompatibility of data mining tools The data mining process requires a wide range of facilities, so it might be supposed that a wide variety of tools will be used. This can lead to a high overhead in switching contexts and converting data between different formats. At its worst this can lead to the omission of necessary steps, and even mild cases can seriously interfere with the exploratory character of data mining. The most readily available solution is to use a data mining toolkit in which all the required facilities are present in an integrated form. However, no toolkit will provide every possible facility, especially when the individual preferences of analysts are taken into account, so toolkits should also be open, and interface easily with other available tools and third-party options.

8 20 Data Mining III 3.7 Pitfall #7: Locked in the data jail-house In addition to openness with regard to tools, data mining systems should be open with regard to data. Some data mining tools require the data to be held in a proprietary format which is not compatible with commonly used database systems. (This is sometimes referred to as the data jail-house.) This can result in large overheads to transfer data into the format required, and difficulty in deploying the results into an organisation s systems. A good data mining tool will interface to your data via common standards. 3.8 Pitfall #8: Disorganized data mining This common pitfall is often a consequence of the apply the algorithm, use the results misconception (see myth #6). The data mining takes place in an ad-hoc manner, with no clear goals and no idea of how the results will be used. The consequences can be unusable results. To produce useful results, it is necessary to have clearly defined business and data mining goals, formulated early in the project, along with deployment plans. A simple way of ensuring this is to use a standard process such as CRISP-DM [1]; this ensures the correct preparation for data mining, and provides a common language for communication of methods and results. Data mining tools should support standard process models. 4 Conclusions Data mining is a business process, requiring extensive business knowledge and best practiced by, or in very close collaboration with, business experts. Data mining uses a variety of different kinds of techniques, and should not be focussed mainly or exclusively on modelling algorithms and their predictive accuracy. Each technique can play a variety of roles. Data miners should make intelligent decisions about the amount of data required, assuming neither that all of an organisation s data will be relevant, nor that all the available data will be required. Effective data mining requires flexible and interoperable techniques; this requirement is best met by integrated, open toolkits, which can interface to data via open standards. The data mining process can be characterised by interaction and engagement with the data in an iterative fashion. A standard data mining process model such as CRISP-DM helps to ensure the correct preparation for and use of data mining, and should be supported by data mining tools. References [1] Chapman, P., Clinton, J., Kerber, R., Khabaza, T., Reinartz, T., Shearer, C. and Wirth, R. CRISP-DM I. O Step-by-step data mining guide, CRISP-DM Consortium, 2000, available at org.