Decision Trees and other predictive models. Mathias Lanner SAS Institute

Size: px

Start display at page:

Download "Decision Trees and other predictive models. Mathias Lanner SAS Institute"

Agnes Porter
9 years ago
Views:

1 Decision Trees and other predictive models Mathias Lanner SAS Institute

2 Agenda Introduction to Predictive Models Decision Trees Pruning Regression Neural Network Model Assessment 2

3 Predictive Modeling The Essence of Data Mining Most of the big payoff [in data mining] has been in predictive modeling. Herb Edelstein 3

4 Predictive Modeling Applications Database marketing Financial risk management Fraud detection Process monitoring Pattern detection 4

5 Predictive Modeling Training Data Training Data case 1: inputs case 2: inputs case 3: inputs case 4: inputs case 5: inputs Numeric or categorical values 5

6 Predictive Modeling Score Data Training Data case 1: inputs case 2: inputs case 3: inputs case 4: inputs case 5: inputs Score Data case 1: inputs case 2: inputs case 3: inputs case 4: inputs case 5: inputs????? Only input values known 6

7 Predictions Training Data case 1: inputs case 2: inputs case 3: inputs case 4: inputs case 5: inputs Predictions Score Data case 1: inputs? case 2: inputs? case 3: inputs? case 4: inputs? case 5: inputs? 7

8 Predictions Training Data case 1: inputs case 2: inputs case 3: inputs case 4: inputs case 5: inputs Predictions Score Data case 1: inputs case 2: inputs case 3: inputs case 4: inputs case 5: inputs????? 8

9 Predictive Modeling Essentials new case Predict new cases x 3 x 4 Select useful inputs Optimize complexity 9

10 Predictive Modeling Essentials new case Predict new cases x 3 x 4 Select useful inputs Optimize complexity 10

11 Three Prediction Types Training Data case 1: inputs case 2: inputs case 3: inputs case 4: inputs case 5: inputs Predictions Decisions Rankings Estimates 11

12 Decision Predictions Training Data case 1: inputs case 2: inputs case 3: inputs case 4: inputs case 5: inputs Decisions primary secondary tertiary primary secondary Trained model uses input measurements to make best decision for each case. 12

13 Ranking Predictions Training Data case 1: inputs case 2: inputs case 3: inputs case 4: inputs case 5: inputs Rankings Trained model uses input measurements to optimally rank each case. 13

14 Estimate Predictions Training Data case 1: inputs case 2: inputs case 3: inputs case 4: inputs case 5: inputs Estimates Trained model uses input measurements to optimally estimate value. 14

15 Model Essentials Predict Review new case Predict new cases Decide, rank, estimate x 3 x 4 Select useful inputs Optimize complexity 15

16 Model Essentials Select Review new case Predict new cases x 3 x 4 Select useful inputs Optimize complexity 16

17 Curse of Dimensionality 1 D 2 D 3 D 17

18 Input Selection Redundancy Irrelevancy 18

19 Model Essentials Select Review new case x 3 x 4 Predict new cases Select useful inputs Optimize complexity Decide, rank, estimate Eradicate redundancies irrelevancies 19

20 Model Essentials Optimize new case Predict new cases x 3 x 4 Select useful inputs Optimize complexity 20

21 Fool s Gold My model fits the training data perfectly... I ve struck it rich! 21

22 Model Complexity Too flexible Not flexible enough 22

23 23 Data Splitting

24 Training Data Role Training Data case 1: inputs case 2: inputs case 3: inputs case 4: inputs case 5: inputs Validation Data case 1: inputs case 2: inputs case 3: inputs case 4: inputs case 5: inputs Training data gives sequence of predictive models with increasing complexity. 24

25 Validation Data Role Training Data case 1: inputs case 2: inputs case 3: inputs case 4: inputs case 5: inputs Validation Data case 1: inputs case 2: inputs case 3: inputs case 4: inputs case 5: inputs Validation data helps select best model from sequence 25

26 Validation Data Role Training Data case 1: inputs case 2: inputs case 3: inputs case 4: inputs case 5: inputs Validation Data case 1: inputs case 2: inputs case 3: inputs case 4: inputs case 5: inputs Validation data helps select best model from Sequence. 26

27 Model Essentials Optimize new case x 3 x 4 Predict new cases Select useful inputs Optimize complexity Decide, rank, estimate Eradicate redundancies irrelevancies Tune models with validation data 27

28 Agenda Introduction to Predictive Models DECISION TREES Pruning Regression Neural Networks Model Assessment 28

29 Predictive Modeling Tools Primary Decision Tree Regression Neural Network Specialty Dmine Regression MBR AutoNeural Rule Induction DMNeural Multiple Model Ensemble Two Stage 29

30 Predictive Modeling Tools Primary Decision Tree Regression Neural Network Specialty Dmine Regression MBR AutoNeural Rule Induction DMNeural Multiple Model Ensemble Two Stage 30

31 Predictive Modeling Tools Primary Decision Tree Regression Neural Network Specialty Dmine Regression MBR AutoNeural Rule Induction DMNeural Multiple Model Ensemble Two Stage 31

32 Model Essentials Decision Trees new case Predict new cases Prediction rules x 3 x 4 Select useful inputs Split search Optimize complexity Pruning 32

33 Simple Prediction Illustration Analysis goal: Predict the color of a dot based on its location in a scatter plot

34 Model Essentials Decision Trees new case Predict new cases Prediction rules x 3 x 4 Select useful inputs Split search Optimize complexity Pruning 34

35 Decision Tree Prediction Rules 40% leaf node root node < interior node < < % 70% 55%

36 Decision Tree Prediction Rules new case 40% < < < % 70% 55%

37 Decision Tree Prediction Rules new case Decision = Estimate = % < < < % 70% 55%

38 Model Essentials Decision Trees new case Predict new cases Prediction rules x 3 x 4 Select useful inputs Split search Optimize complexity Pruning 38

39 Decision Tree Split Search left right Calculate the logworth of every partition on input

40 Decision Tree Split Search left right 53% 42% 47% 58% max logworth( ) 0.95 Select the partition with maximum logworth

41 Decision Tree Split Search left right 53% 42% 47% 58% Repeat for input. max logworth( )

42 Decision Tree Split Search left right 53% 42% 47% 58% max logworth( ) bottom top % 35% 46% 65% max logworth( )

43 Decision Tree Split Search < Create partition rule from best partition across all inputs

44 Decision Tree Split Search < Repeat process in each subset

45 Decision Tree Split Search left right 61% 40% 39% 60% logworth( ) 5.72 Select the partition with maximum logworth on input

46 Decision Tree Split Search left right 61% 40% 39% 60% bottom top 38% 55% 62% 45% Repeat for input. logworth( ) 5.72 logworth( )

47 Decision Tree Split Search < < Create second partition rule

48 Decision Tree Split Search Repeat to form maximal tree

49 49 Demo Decision Tree

Agenda. Mathias Lanner Sas Institute. Predictive Modeling Applications. Predictive Modeling Training Data. Beslutsträd och andra prediktiva modeller

Agenda. Mathias Lanner Sas Institute. Predictive Modeling Applications. Predictive Modeling Training Data. Beslutsträd och andra prediktiva modeller Agenda Introduktion till Prediktiva modeller Beslutsträd Beslutsträd och andra prediktiva modeller Mathias Lanner Sas Institute Pruning Regressioner Neurala Nätverk Utvärdering av modeller 2 Predictive