Data Mining Chapter 1: Introduction Fall 2011 Ming Li Department of Computer Science and Technology Nanjing University

Transcription

1 Data Mining Chapter 1: Introduction Fall 2011 Ming Li Department of Computer Science and Technology Nanjing University

2 Advances in computer technology Computer Hardware Super computers and high performance PCs Various huge volume storage media Many types of data collecting devices Software Techniques Development of system software (e.g., OS, DB) Multimedia data and corresponding processing software The Internet DATA Integrating different types of resources (hardware, software, data, )

3 Why data mining? So many data to use Swear to mine them out! dig, Dig, DIG Data Too many data, can t find useful information. How can I do? Data are rich but information is poor

4 What can be mined out from data? Knowledge, hidden relationships, underlying reality, Information that should be potentially useful for decision making or understanding the nature of the task. Let s see some concrete examples

5 Example I: Mining supermarket transactions

6 Example II: Mining the valuable customers? 2G? 3G

7 Example III: Mining network intrusion patterns Decide whether the current access is intrusion. Building models based on the historical records using predictive models, such as decision tree, neural networks, Anomaly detection

8 Example IV: Mining gene data stage(1-3) stage(4-6) stage(7-8) stage(9-10) stage(11-12) stage(13-16) Identifying gene function, through gene expression Finding key genes Identifying gene expression patterns Identifying gene interactions Finding similar genes

9 Example V: Mining medical data CAD (Computer-Aided Diagnosis) Systems to help improving the diagnosis of doctors based on the historical cases.

10 Example VI: Mining the web Mining the web structure, web content, web usage, and web logs to indentify interesting patterns for web search, user behavior identification, Page rank, Learning to rank,

11 Example VII: Mining financial data Fraud detection Stock trends discovery

12 Example VIII: Mining usage data Mining the usage behavior pattern to enable more natural Human-Computer Interaction

13 Other successful data mining applications Sports: The Advanced Scout system analyzes the logs of NBA games to uncover the interesting pieces of information which might go unnoticed by coaches. If player A is on the floor, player B s shot accuracy decrease from 75% to 30% Transportation The department of railway, China, has been uses data mining techniques to analyzing the passengers during the Chinese new year to find the key issues that affecting the transpiration throughput since Steel industry Baostell Group Corporation began to use data mining to quality insurance in 1995

14 Top data mining fields Out of the top 20 field, nearly half of them (9) are related to finance/commerce. 8 of these field are among the top 10 field. Telecom, bio-information, medical, health, are also top fileds. [KDNudges Poll, 2009]

15 Salaries of data miners [KDNudges Poll, 2009]

16 Financial crisis effect on data mining [KDNudges Poll, 2009]

17 So, what is data mining after all Data mining is the analysis of (often LARGE) observational data sets to find unsuspected relationships and to summarize the data in novel ways that are both understandable and useful to the data owner. [D. Hand et al., Principles of Data Mining] 数据挖掘是通过对 ( 大规模 ) 观测数据集的分析, 寻找确信的关系, 并 将数据以一种可理解的且利于使用的新颖方式概括数据的方法

18 So, what is data mining after all Large: Small data set does not requires data mining. Large data causes problems Observational data: Not experimental data. Assumes no control over the data collection process. Unsuspected relationships: The relationships or patterns should be correct and significance. Novel Common sense is useless Understandable The mining results will present to user for decision making Useful The mining results should be useful to the users.

19 How large can the data set be? [KDNudges Poll, 2011]

20 Relationship to KDD Data Mining is an important step of KDD

21 Systematic view of data mining Subject: Data Tools: Data mining Models or pattern structures: the nature and structure of the representation to be used in the data Score function: defining how well the model fits the data Optimization and search method: how to search model and patterns that optimizes the scoring function Data management strategy: handling data access efficiently during the mining process.

22 Data types in mining tasks Flat data: Table or data matrix Attribute, Feature, Variable, Fields Example, instance, record, entity, case, object

23 Data types in mining tasks Text data

24 Data types in mining tasks Web data

25 Data types in mining tasks Multimedia data audio images video

26 Data types in mining tasks Temporal and spatial data

27 Data types in mining tasks Compared to 2009 Decline of spatial data / audio / text (free-form) Increase of social network data / web content data / XML data [KDNudges Poll, 2010]

28 Data mining components: Models and patterns Models A model structure is defined as a global summary of a data set It makes statements about any point in the full measurement space Patterns Pattern structures is defined as describes a local structure relating to a relatively small part of the data It make statements only about restricted regions of the space spanned by the variables.

29 Data mining components: Score function Score functions quantify how well a model or parameter structure fits a given data set How to choose score function? The choice of score function would precisely reflect the utility (i.e., the true expected benefit) of a particular predictive model. The score function should: Have certain semantics, which is usually related to errors, risks, costs. Be stable Be easy to evaluate An example of the squared error

30 Data mining components: Optimization and search methods Optimization and search method seeks the parameter values of the models according to the optimal value of the score function. How to search? Finding the optimal values of parameters in models is typically cast as an optimization problem Mathematical optimization techniques: e.g, Gradient decent, LP, QP, SDP, Heuristic search techniques. e.g, Evolutionary computing, Particle Swarm Optimization, Ant Colony Optimization,

31 Data mining components: Data management strategies Data management strategies take care of the ways in which the data are stored, indexed, and accessed How to manage data? Utilizing the state-of-art database or data warehouse techniques for managing the large scale data. Distributed and parallel computing may be required to increase throughput. Appropriate data structures Properly index the stored data Design efficient query algorithms Precomputing if possible

32 Data mining tasks Exploratory Data Analysis Interactive and visual Question: If it resides in high dimensional space, how to visualize the data while keeping its property.

33 Data mining tasks Descriptive Modeling Describe all of the data collected. Question: How to characterize the general properties or underlying structures of the data

34 Data mining tasks Predictive Modeling Perform inference on the collected data in order to make predictions Question: how to construct the mapping from the input space to the output space

35 Data mining tasks Discovering patterns and rules Detecting anomalies. Question: how to be sensitive to these weak of signals? It would be a huge star rather than a galaxy! Finding association rules Question: how to find the co-occurrence of two items from across a huge data collection Buying diaper Buying beer

36 Data mining tasks Retrieval by Content Input query and ask for information relevant to the query Question: how to measure relevance? CBIR System

37 High-level picture of data mining

38 Data mining vs. Statistics: The difference Data scale: Data mining are usually conducted on huge volumes of data, while hundreds of data points are regarded as big data set. High dimensional data The data to be mined are usually of high dimensionality, such as images, text,., while statistics does not consider the dimensionality of data Data format Various forms of data can be mined. Such as text, audio, video,, while most of the data in statistics are flatted

39 Data mining vs. Statistics: The difference Data incompleteness: It is likely for the real-world data to be mined to have missing values (e.g., salary, age) and noise, while traditional statistics seldom considers it. The characteristics of the specific data Data mining cares about the characteristics of the data to be mined (e.g, unequal cost, imbalanced data distribution, distribution changing, ), while traditional statistics seldom considers it. The role Data mining is the ``secondary data analysis techniques (the data have been already collected) while statistics is the `` primary data analysis techniques because it controls the data it collects.

40 Three perspectives of data mining Machine Learning Statistics Practical Data analysis techniques Data Mining Data analysis method with Mathematically validity shown Data management techniques Database For more information, please refer to [Z.-H. Zhou, AIJ 03]

41 Let s move to Chapter 2