Software Course and the Case Practice Introduction of Credit Risk Data

Transcription

1 Software Course and the Case Practice Introduction of Credit Risk Data Cheyu HUNG / 洪哲裕 StatSoft Holdings, Inc., Taiwan Branch November 27, 2013 Making the World More Productive Headquarters: StatSoft, Inc East 14th Street Tulsa, OK USA +1(918) info@statsoft.com Australia: StatSoft Pacific Pty Ltd. Brazil: StatSoft South America Ltda. Bulgaria: StatSoft Bulgaria Ltd. Chile: StatSoft South America Ltda. China: StatSoft China Czech Rep.: StatSoft Czech Rep. s.r.o. France: StatSoft France Germany: StatSoft GmbH Hungary: StatSoft Hungary Ltd. India: StatSoft India Pvt. Ltd. Israel: StatSoft Israel Ltd. Italy: StatSoft Italia srl Japan: StatSoft Japan Inc. Korea: StatSoft South Korea Netherlands: StatSoft Benelux Norway: StatSoft Norway AS Poland: StatSoft Polska Sp. z o.o. Portugal: StatSoft Iberica Lda Russia: StatSoft Russia S. Africa: StatSoft S. Africa (Pty) Ltd. Spain: StatSoft Iberica Lda Sweden: StatSoft Scandinavia AB Taiwan: StatSoft Taiwan UAE/Egypt: StatSoft Middle East United Kingdom: StatSoft Ltd. Copyright StatSoft, Inc., StatSoft, StatSoft logo, and STATISTICA are trademarks of StatSoft, Inc.

2 Credit Risk Data The application for this data Variables in the Credit Risk data Next steps for the Data Mining project Copyright StatSoft, Inc., StatSoft, StatSoft logo, and STATISTICA are trademarks of StatSoft, Inc. 1

3 Applications Practically all data need some preparation work. Handling missing data and outliers. Selecting important variables. Sampling Classification tasks have many uses Classify a variable with 2 or more groups Find probability of a predicted classification Copyright StatSoft, Inc., StatSoft, StatSoft logo, and STATISTICA are trademarks of StatSoft, Inc. 2

4 Application for Credit Risk Data A financial institution needs a way to decide if and how much credit to extend to customers who apply. This is our business need Automated Neural Networks The goals of the data mining project include: Determining the variables that are best predictors of credit risk, Finding a high performance predictive model that classifies customers, Deploying that model to make decisions on credit application, Updating the model as we collect more data. Copyright StatSoft, Inc., StatSoft, StatSoft logo, and STATISTICA are trademarks of StatSoft, Inc. 3

5 Next Steps for the Data Mining Project The project goals are expressed and data are available. The next step is to understand the data. We will do this by reviewing the data graphically. Later the data needs prepared using data cleaning tools. From there, we move to modeling, evaluation and deployment. Copyright StatSoft, Inc., StatSoft, StatSoft logo, and STATISTICA are trademarks of StatSoft, Inc. 4

6 Introduction to Credit Risk Data In this session, we discussed the business need and application for the data. We reviewed the variables We outlined the next steps for the data mining project. Copyright StatSoft, Inc., StatSoft, StatSoft logo, and STATISTICA are trademarks of StatSoft, Inc. 5

7 Next Steps With an understanding of the application, we are ready to start working with the data. The next session will look at query and import options to bring the data into STATISTICA from a database or external format. From there, we will start graphical review and data cleaning. Copyright StatSoft, Inc., StatSoft, StatSoft logo, and STATISTICA are trademarks of StatSoft, Inc. 6

8 Initial Graphical Review Review the data graphically to reveal issues with the data that will need addressed in the data cleaning phase. Explore tools to mark or remove invalid points. Copyright StatSoft, Inc., StatSoft, StatSoft logo, and STATISTICA are trademarks of StatSoft, Inc. 7

9 Data Cleaning for Outliers Detecting outliers Graphically Statistical tests Handling outliers What caused the outlier? What can be done to clean the data? Copyright StatSoft, Inc., StatSoft, StatSoft logo, and STATISTICA are trademarks of StatSoft, Inc. 8

10 Detecting Outliers Graphically A box plot can show outliers in continuous data. A histogram can show outliers in categorical data. Copyright StatSoft, Inc., StatSoft, StatSoft logo, and STATISTICA are trademarks of StatSoft, Inc. 9

11 Detecting Outliers with Statistical Tests Grubbs test Normal distribution Percentiles Tukey Descriptive Statistics Grubbs Test p-value Variable Statistic Age Copyright StatSoft, Inc., StatSoft, StatSoft logo, and STATISTICA are trademarks of StatSoft, Inc. 10

12 Remove the outliers Handling Outliers Are the outliers data entry errors? Are the outliers due to entries in the data set that don t belong there? Keep the outliers Are the outliers legitimate points that simply have extreme values? Copyright StatSoft, Inc., StatSoft, StatSoft logo, and STATISTICA are trademarks of StatSoft, Inc. 11

13 C&RT for Classification What is C&RT Misclassification cost Stopping conditions Cross Validation Surrogates Suppose these terms above are known and learned Example in STATISTICA Copyright StatSoft, Inc., StatSoft, StatSoft logo, and STATISTICA are trademarks of StatSoft, Inc. 12

14 C&RT Classification and Regression Trees A nonparametric data mining algorithm for generating decision trees. Splits are made by variables that best differentiate the target variable. Each node can be split into two child nodes Stopping rules govern the size of the tree. Copyright StatSoft, Inc., StatSoft, StatSoft logo, and STATISTICA are trademarks of StatSoft, Inc. 13

15 Misclassification cost Misclassification is inevitable since no model is perfect. Some misclassifications are worse than others, so STATISTICA allows you to account for this with misclassification costs. Observed Good Credit Observed Bad Credit Predicted Good Credit Correct % X margin lost (by unit) Incorrect % X margin lost (by unit) Predicted Bad Credit Incorrect % X margin lost (by unit) Correct % X margin lost (by unit) Copyright StatSoft, Inc., StatSoft, StatSoft logo, and STATISTICA are trademarks of StatSoft, Inc. 14

16 Stopping Conditions Variety, but choose one is enough Decision Tree Pruning Misclassification error or Deviance Select a minimum number of cases for a node to be considered for splitting. Select a maximum number of total nodes. FACT direct stopping Select the fraction of objects for determining if a node should be split. Copyright StatSoft, Inc., StatSoft, StatSoft logo, and STATISTICA are trademarks of StatSoft, Inc. 15

17 Cross Validation Cross Validation is a method to prevent over fitting the data and failing to generalize to new data. V-Fold Cross Validation Good for smaller data sets, when holding out a test sample is not feasible. Repeats the analysis on V different random samples taken from the data and compares the resulting trees. Train Test Sample Cross Validation Test sample data is used to determine if the right size tree was found based on how well the tree performs on the test data. Copyright StatSoft, Inc., StatSoft, StatSoft logo, and STATISTICA are trademarks of StatSoft, Inc. 16

18 Surrogates In deployment stage, surrogate splits are used in place of the actual split variable when its value is missing. The surrogate is the next best split variable. Each split can have multiple surrogates Copyright StatSoft, Inc., StatSoft, StatSoft logo, and STATISTICA are trademarks of StatSoft, Inc. 17

19 CHAID Trees What is CHAID Analysis options Exhaustive CHAID Example in STATISTICA Copyright StatSoft, Inc., StatSoft, StatSoft logo, and STATISTICA are trademarks of StatSoft, Inc. 18

20 CHAID Chi Square Automatic Interaction Detection Performs multi-level splits where C&RT uses binary splits. Well suited for large data sets. Commonly used for marketing segmentation studies. Copyright StatSoft, Inc., StatSoft, StatSoft logo, and STATISTICA are trademarks of StatSoft, Inc. 19

21 CHAID Analysis Options ANOVA type design Misclassification costs Cross Validation Bonferroni adjustment Risk estimate s (credit scoring for model bu Dependent variable: Credit Rating Options: Cate gorical response Risk estimate Standard error Train Copyright StatSoft, Inc., StatSoft, StatSoft logo, and STATISTICA are trademarks of StatSoft, Inc. 20

22 CHAID Analysis Options Stopping parameters Minimum cases for a node to be split Maximum number of total nodes Probability for merging and splitting Copyright StatSoft, Inc., StatSoft, StatSoft logo, and STATISTICA are trademarks of StatSoft, Inc. 21

23 Exhaustive CHAID Optional to proceed or NOT More computationally intensive, for large data sets may require extended computations. Performs more thorough merging and testing of predictor variables to find the best split candidate. Typically produces better performing classification trees. Copyright StatSoft, Inc., StatSoft, StatSoft logo, and STATISTICA are trademarks of StatSoft, Inc. 22

24 Boosted Trees What are boosted trees? Analysis options Stopping parameters Examples in STATISTICA Copyright StatSoft, Inc., StatSoft, StatSoft logo, and STATISTICA are trademarks of StatSoft, Inc. 23

25 Boosted Trees The idea of Boosted trees is to build a sequence of simple trees, weighting them inversely by misclassification. The final classification for deployment is based on voting from these simple trees. Copyright StatSoft, Inc., StatSoft, StatSoft logo, and STATISTICA are trademarks of StatSoft, Inc. 24

26 Boosted Trees Analysis Options Learning Rate Number of additive terms Random test data proportions Subsample proportions Seed for random number generator Copyright StatSoft, Inc., StatSoft, StatSoft logo, and STATISTICA are trademarks of StatSoft, Inc. 25

27 Boosted Trees Stopping Parameters Minimum number of cases Maximum number of levels Minimum number in child nodes Maximum number of nodes Copyright StatSoft, Inc., StatSoft, StatSoft logo, and STATISTICA are trademarks of StatSoft, Inc. 26

28 Random Forests For Classification What is Random Forests classification? Analysis options Example in STATISTICA Copyright StatSoft, Inc., StatSoft, StatSoft logo, and STATISTICA are trademarks of StatSoft, Inc. 27

29 Random Forests Random Forests builds a series of trees. Each tree predicts a classification. The Random Forest predicts the classification predicted by the most trees. Copyright StatSoft, Inc., StatSoft, StatSoft logo, and STATISTICA are trademarks of StatSoft, Inc. 28

30 Random Forest Analysis Options Number of predictors - optimal setting is log2(m+1) Number of trees Sampling proportions Stopping parameters Copyright StatSoft, Inc., StatSoft, StatSoft logo, and STATISTICA are trademarks of StatSoft, Inc. 29

31 Comparing Performance across Models Generating deployment code Rapid Deployment results Lift and Gains Charts Copyright StatSoft, Inc., StatSoft, StatSoft logo, and STATISTICA are trademarks of StatSoft, Inc. 30

32 Generating Deployment Code Optional topic Many Data Mining and Statistics tools in STATISTICA have the ability to generate deployment code. STATISTICA Visual Basic C/C++ language PMML Script Deployment to STATISTICA Enterprise Save PMML deployment code as.xml Copyright StatSoft, Inc., StatSoft, StatSoft logo, and STATISTICA are trademarks of StatSoft, Inc. 31

33 Rapid Deployment Optional topic Load multiple data mining models Make predictions on new data Generate lift and gains charts comparing models Write predictions back to data file Copyright StatSoft, Inc., StatSoft, StatSoft logo, and STATISTICA are trademarks of StatSoft, Inc. 32

34 Lift and Gains Charts Optional topic Lift chart Shows the effectiveness of the model compared to no model. Gains chart Shows the percentage of observations correctly classified for the given category, in this case, bad. Copyright StatSoft, Inc., StatSoft, StatSoft logo, and STATISTICA are trademarks of StatSoft, Inc. 33

35 Voting Across Models What is voting or bagging? Instability or results in small datasets Reviewing results C&RT Input Data CHAID Random Forest Vote Final Prediction Boosted Trees Copyright StatSoft, Inc., StatSoft, StatSoft logo, and STATISTICA are trademarks of StatSoft, Inc. 34

36 What is Voting or Bagging? Data Mining offers a variety of model building tools, so a large number of models can be created in a given project. None of these models will fully capture the underlying relationship of the data. Using an ensemble of models together to determine the final prediction is called voting or bagging. Copyright StatSoft, Inc., StatSoft, StatSoft logo, and STATISTICA are trademarks of StatSoft, Inc. 35

37 What is Voting What or Bagging? is Voting or Bagging? C&RT CHAID Input Data Random Forest Vote Final Prediction Boosted Trees Copyright StatSoft, Inc., StatSoft, StatSoft logo, and STATISTICA are trademarks of StatSoft, Inc. 36

38 Instability of Results in Small Data Sets Data Mining can model complex relationships between variables. Without ample data, instability can be an issue. Using multiple models with voting combats the problem of instability in results. The ensemble of models typically outperforms individuals. Copyright StatSoft, Inc., StatSoft, StatSoft logo, and STATISTICA are trademarks of StatSoft, Inc. 37

39 Rapid Deployment gives predictions based on voting. That output can generate other results. Summary of Deployment (Error rates) (CreditScoring) BoostTreeModel ExhaustiveCHAIDMo RandomForestMode TreeModel VotedModel del Error rate Copyright StatSoft, Inc., StatSoft, StatSoft logo, and STATISTICA are trademarks of StatSoft, Inc. 38

40 Q/A Welcome to mail us! Copyright StatSoft, Inc., StatSoft, StatSoft logo, and STATISTICA are trademarks of StatSoft, Inc. 39