Data Mining. Dr. Saed Sayad. University of Toronto

Size: px

Start display at page:

Download "Data Mining. Dr. Saed Sayad. University of Toronto 2010 [email protected]. http://chem-eng.utoronto.ca/~datamining/"

Gary Barnett
10 years ago
Views:

1 Data Mining Dr. Saed Sayad University of Toronto

2 Data Mining Data mining is about explaining the past and predicting the future by means of data analysis. 2

3 Data Mining Statistics AI & Machine Learning Data Mining Database & DW 3

4 Data Mining Applications CRM Banking Credit Scoring Direct Marketing/ Fundraising Fraud Detection Retail Insurance Telecom Manufacturing Science Health care/ HR Medical/ Pharma Government applications Other e-commerce Biotech/Genomics Web Travel/Hospitality Security / Anti-terrorism Junk / Anti-spam Investment / Stocks Entertainment/ Music Gambling Source: KDnuggets.com

e-commerce Biotech/Genomics Web Travel/Hospitality Security / Anti-terrorism Junk email / Anti-spam Investment

5 Data mining activity in 2007 compare to 2006 somewhat lower 4% much lower 5% much higher 20% about the same 41% somewhat higher 30% Source: KDnuggets.com 5

6 Data Mining Steps 1 Problem Definition 2 Data Preparation 3 Data Exploration 4 Modeling 5 Evaluation 6 Deployment 6

7 CRISP-DM Process Model CRoss-Industry Standard Process for Data Mining Source: 7

8 1. Problem Definition Understanding the project objectives and requirements from a business perspective and then converting this knowledge into a data mining problem definition with a preliminary plan designed to achieve the objectives. Source: 8

definition with a preliminary plan designed to achieve the objectives.

9 2. Data Preparation Data DSN Data Text ETL Modeling Data 9

10 3. Data Exploration Data Exploration Univariate Analysis Bivariate Analysis Average, StDev, Min, Max,... Bar, Line, Pie,... Charts Correlation Z test,... Combination Charts 10

11 Data Exploration - Univariate 11

12 Data Exploration - Bivariate 12

13 4. Modeling Classification Regression Clustering Association Bayesian Linear Regression Hierarchical A Priori Decision Tree Robust Regression K-Means Logistic Regression Neural Network SVM 13

Priori Decision Tree Robust Regression K-Means Logistic

Data Mining: Classification & Regression Frequency Table Covariance Matrix Similarity Functions Neural Networks Others OneR Linear Regression KNN Perceptron SVM Bayesian

14 Data Mining: Classification & Regression Frequency Table Covariance Matrix Similarity Functions Neural Networks Others OneR Linear Regression KNN Perceptron SVM Bayesian LDA (Z Score) Back Propagation GA Decision Tree PCA/PCR RBF Markov Chains Logistic Regression HMM Robust Regression Scalable Methods 14

LDA (Z Score) Back Propagation GA Decision Tree PCA/PCR RBF Markov Chains Logistic

Modeling - Classification Age f Responder e.g., Y or N http://chem-eng.

15 Modeling - Classification Age f Responder e.g., Y or N 15

16 Modeling - Regression Age f Amount Purchased e.g., $

17 Modeling - Clustering Income Age 17

18 Association Rules Market Basket Analysis 18

19 5. Evaluation Charts Stats Gain Chart Lift Chart K-S Chart Confusion Matrix Mean Square Error Variables Contribution 19

20 Predicted Negative Predicted Positive Evaluation - Confusion Matrix Positive Cases True Positive Negative Cases False Positive CM False Negative True Negative 20

Negative Cases False Positive CM False Negative

21 Evaluation Gain Chart Responder% 100% 45% 10% Population% 10% 50% 100% 21

22 6. Deployment SQL VB JAVA HTML 22

23 Data Mining Team Modeler Domain Expert DBA Analyst 23

24 Data Mining Software Vendors SAS SPSS Data Mining KXEN Angoss KNIME 24

25 Case Study

CONTENTS PREFACE 1 INTRODUCTION 1 2 DATA VISUALIZATION 19

CONTENTS PREFACE 1 INTRODUCTION 1 2 DATA VISUALIZATION 19 PREFACE xi 1 INTRODUCTION 1 1.1 Overview 1 1.2 Definition 1 1.3 Preparation 2 1.3.1 Overview 2 1.3.2 Accessing Tabular Data 3 1.3.3 Accessing Unstructured Data 3 1.3.4 Understanding the Variables and Observations