Virtual Site Event. Predictive Analytics: What Managers Need to Know. Presented by: Paul Arnest, MS, MBA, PMP February 11, 2015

Transcription

1 Virtual Site Event Predictive Analytics: What Managers Need to Know Presented by: Paul Arnest, MS, MBA, PMP February 11,

2 Ground Rules Virtual Site Ground Rules PMI Code of Conduct applies for this virtual presentation. The Virtual Attendees are expected to: Participate for a minimum of 40 minutes. Login information will be verified. Answer the question pertaining to the presentation correctly in the survey in order to obtain the PDU credit (1). Respond to the survey within 48 hours (By Friday February 13, 2015) of participation in order to obtain the PDU credit. 2

3 Predictive Analytics What Managers Need to Know 3

4 Predictive Analytics A NEW ENVIRONMENT 4

5 Definition Predictive Analytics: Techniques that quantify potential outcomes or events based on past data Not descriptive analysis and descriptive statistics Not techniques that enable end-users to perform individual data discovery or to customize reports 5

6 Convergence Once restricted to specialized statistics organizations, advanced modeling techniques are moving into the IT mainstream Stat/Analytics Shop IT 6

7 Concepts/Buzzwords Machine learning Supervised learning Unsupervised learning Response variable Target variable Dependent variable Left hand side variable Explanatory variable Independent variable Right hand side variable Logistic regression Random forest, etc. Sensitivity Specificity 7

8 Tool independence Predictive techniques use mathematical algorithms that are independent of particular tools SAS, R, Stata, SPSS, many more Use specialized tools for model development It is possible to implement models using general software tools, i.e., Java,.Net 8

9 Don t be intimidated Your stat/analysis package is programmed to do the heavy math You ll discover that most internal stat shops are using a small set of models and techniques over and over again Most of the work: Understanding what you want to accomplish Understanding the data Organizing the data 9

10 Understand the results Predictive analytics produce a probability of a characteristic or behavior based on a detailed analysis of past characteristics or behaviors Probability is 100% Certainty Model accuracy depends on similarity of past conditions to present 10

11 Predictive Analytics HOW IT WORKS AND WHAT TO EXPECT 11

12 Logistic regression Workhorse procedure for predictive analytics Supervised technique 12

13 Step 1 Identify a known population that exhibits the characteristic you want to predict dependent, target or response variable plus a known population that does not You may take the whole population ( big data ) or a sample Use 80% or 90% of the sample as the training data set Withhold the remainder for validation 13

14 Step 2 Construct a hypothesis ( null hypothesis ) Select variables expected to distinguish target population independent or explanatory variables 14

15 Step 3 Run a logistic regression against the variables Logistic regression will calculate the likelihood (predictive odds) that the independent variables are associated with the dependent variable 15

16 Step 4 Test the hypothesis on the withheld sample and the broader population Caution: It s critical to identify the target characteristics accurately 16

17 Logistic regression: targets Target: Workers Compensation Fraudsters Target High Incidence Organization Dr on CMS Ineligible List High Risk Occupation Psychological Impairment Imperceptible Physical Impairment Linda Rebecca Samuel Stephen Amanda Hugh Francesco Allen Eric Gail Joseph Derek Kevin

18 Logistic regression: general General population of covered workers Target High Incidence Organization Dr on CMS Ineligible List High Risk Occupation Psychological Impairment Imperceptible Physical Impairment Linda Rebecca Samuel Stephen Amanda Hugh Francesco Allen Eric Gail Joseph Derek Kevin

19 Results Maximum Likelihood Estimates: Fraud likelihood = (intercept) (multiple cases) (CMS ineligible) (rep disciplined) (psychological) (imperceptible physical) 19

20 Interpretation Positive coefficients mean all factors contribute to likelihood of fraud Coefficients reflect the actual weight the model places on each factor Intercept ( ) means this model predicts a 12% likelihood of fraud if no modeled factors present 20

21 Test of model accuracy C-statistic (probability outcome is better than chance) = indicates an acceptable model 0.80 indicates a strong model the closer to 1 the better Visually represented as ROC curve 21

22 Considerations Accuracy only as good as the target population sample Sum of the terms = logit of the predictive probability of the model translates into odds a claim is fraudulent Conversion of coefficient of the target variable logit(p) to probability p = 1 1+ e logit(p) 22

23 Logit transformation If all factors present, logit(p) = = = 92% probability of fraud p logit(p) p logit(p) p logit(p) p logit(p)

24 LR weaknesses All potential fraud factors combined into a single equation With many independent predictor variables, characteristics can cancel each other out Logistic regression has a hard time weighting interactions between individual variables Must be programmed explicitly Requires additional data manipulation 24

25 LR weaknesses (ctd) In rare-event modeling with a large number of predictive variables, logistic regression can produce many false positives Difficult to differentiate rare events from normal events when the rare events occur with extremely low frequency Bad solution is to boost the sensitivity of the model 25

26 Other supervised methods Decision tree mitigates the problem of numerous weak predictors overwhelming a strong predictor (logistic regression) Sorts observations of the dependent variable into buckets corresponding to its available classification values Conditional selection into paths ( branches ) Priority determined by frequency of characteristics 26

27 Decision tree example High Incidence Organization Left-Facing Arrows: Value = Characteristic is absent Right-Facing Arrows: Value = Characteristic is present 0 = No Fraud 1 = Fraud Misclassification Rate = 23.08% 4F/10N 9F/3N Imperceptible Physical Impairment Psychological Impairment Purity 4F/5N Purity 7F/3N 5 cases = 0 0 cases = 1 Doctor on CMS Ineligible List 0 cases = 0 2 cases = 1 Imperceptible Physical Impairment 1F/3N 3F/2N 4F/1N 3F/2N Psychological Impairment High Risk Occupation High Risk Occupation High Risk Occupation Purity Tie Tie Purity 3F/1N Purity Tie 2F/1N 2 cases = 0 0 cases = 1 1 case = 0 1 case = 1 2 cases = 0 2 cases = 1 0 cases = 0 1 case = 1 Doctor on CMS Ineligible List 0 cases = 0 1 case = 1 1 case = 0 1 case = 1 Doctor on CMS Ineligible List Imperfect Purity Purity Tie 1 case = 0 2 cases = 1 0 cases = 0 1 cases = 1 0 cases = 0 1 case = 1 1 case = 0 1 case = 1 27

28 Beyond decision tree Decision tree may overweight highfrequency but insignificant characteristics Boosted decision tree and random forest are techniques to improve on the results of the basic algorithm based on misclassification rates Neural networks model all possible combinations and select the best ones based on misclassification rates 28

29 Unsupervised methods K-means cluster Consider it a generalization of logistic regression Identify a set of independent variables Transformations likely required, as above Procedure tries to identify a set of statistically significant clusters based on the selected variables Can tease out meaningful characteristics 29

30 Predictive Analytics SOME BEST PRACTICES IN DATA MANAGEMENT 30

31 Data best practices Understand your data What does it represent How does it enter your data warehouse Check data for suitability Missing values? Do target and individual predictors correlate? Ensure that data cleansing and transformation steps are documented and repeatable for model re-estimation 31

32 Counterintuitive-ness The more independent variables, the less predictive value each individual variable, or characteristic, has, on average 32

33 Counterintuitive-ness (ctd) In rare event modeling, even a very accurate model can produce disproportionately large false positives Example: Target population 1% in a population of 1,000,000 (10,000 targets). If predictive model has a 10% false positive rate (90% accurate): Target General population 10, ,000 True positives: 9,000 True negatives: 891,000 False negatives: 1,000 False positives: 99,000 33

34 Takeaways for success 1.Clearly identify target variable 2.Limit predictor variables 3.Know the model data and manage it data management is most of the work 4.Know how to measure model performance 5.Set goals and expectations for the model 6.Monitor model performance and adjust/ re-estimate as necessary 34

35 Thank you/questions Paul Arnest 35