On Generating High InfoQ with Bayesian Networks

Transcription

1 On Generating High InfoQ with Bayesian Networks Ron S. Kenett Research Professor, University of Turin Chairman and CEO, KPA Ltd. Based on joint work with Galit Shmueli, Silvia Salini and Federica Cugnata 1

2 Applied statistics is about meeting the challenge of solving real world problems with mathematical tools and statistical thinking 2

3 Mathematical Tools Statistical Thinking 3

4 Mathematical Tools Statistical Thinking Real World Problems 4The mathematical statistician

5 Mathematical Tools Statistical Thinking Real World Problems 5The proactive statistician, G. Hahn

6 Presentation topics Bayesian networks Information Quality Robustness of network Sensitivity of predictions information knowledge numbers data findings statistical analysis Examples from: Customer surveys Risk management of telecom systems Monitoring of bioreactors Managing healthcare of ESRD patients Kenett, Ron S., Applications of Bayesian Networks (2012). Available at SSRN: 6or

7 Information Quality (InfoQ) The potential of a particular dataset to achieve a particular goal using a given empirical analysis method g X f U A specific analysis goal The available dataset An empirical analysis method A utility measure InfoQ(f,X,g) = U( f(x g) ) Kenett, R.S. and Shmueli, G. (2014) On Information Quality, Journal of the Royal Statistical Society, Series A 7(with discussion), Vol. 177, No. 1, pp. 3-38,

8 Domain Space Knowledge Analytic Space Information Quality Bayesian networks InfoQ Robustness of network Sensitivity of predictions Goals InfoQ(f,X,g) = U(f(X g)) 1. Data resolution 2. Data structure 8 Data Quality Primary Data Analysis Quality Secondary Data - Experimental - Experimental - Observational - Observational 3. Data integration 4. Temporal relevance 5. Chronology of data and goal 6. Generalizability 7. Operationalization 8. Communication

9 f = Bayesian Networks Judea Pearl 2011 Turing Medalist 9

10 10 Causal calculus, Counterfactuals, Do calculus, Transportability, Missing data models, Causal mediation, Graph mutilation, External validity, Causality in missingness,.

11 Customer Surveys Behaviors Recommendation Loyalty Loyalty Index Attitudes & Perceptions Experiences & Interactions Overall Satisfaction Perceptions Equipment and System Sales Support Technical Support Training Supplies and Orders Software Add On Solutions Customer Website Purchasing Support Contracts and Pricing System Installation 11

12 Customer Surveys Goals Goal 1. Goal 2. Goal 3. Goal 4. Goal 5. Goal 6. Goal 7. Goal 8. Goal 9. Goal 10. Decide where to launch improvement initiatives Highlight drivers of overall satisfaction Detect positive or negative trends in customer satisfaction Identify best practices by comparing products or marketing channels Determine strengths and weaknesses Set up improvement goals Design a balanced scorecard with customer inputs Communicate the results using graphics Assess the reliability of the questionnaire Improve the questionnaire for future use 12

13 Bayesian Network Analysis of Customer Surveys 13 Kenett, R.S. and Salini S. (2009). New Frontiers: Bayesian networks give insight into surveydata analysis, Quality Progress, pp , August.

14 14

15 20% BOT12 Set up improvement goals 15

18 Robustness and Sensitivity Analysis of Bayesian Networks How are the factors in the model affecting the target variable levels? Which factors affect customer satisfaction and how How are factors affecting the distance between conditioned distributions and observed data? Which factors will have the largest impact if changed 18

19 Network learning Robustness of Network bnlearn: 1) Grow-Shrink (GS) algorithm, 2) Incremental Association (IAMB) algorithm, 3) Interleaved-IAMB (Inter-IAMB) algorithm, 4) Fast-IAMB (Fast-IAMB) algorithm, 5) Max-Min Parents and Children (MMPC) algorithm, 6) ARACNE and Chow-Liu algorithms, 7) Hill- Climbing (HC) greedy search algorithm, 8)Tabu Search (TABU) algorithm, 9) Max-Min Hill-Climbing (MMHC) algorithm and 10) two-stage Restricted Maximization (RSMAX2) algorithm for both discrete and Gaussian networks 19

20 20 Robustness Analysis

21 Network estimation Sensitivity to Driver Variables Generate a full factorial experiments based on driver combinations. Analyse the target variable thus generated (overall satisfaction) with respect to the observed one, in order to study the effect that each driver combinations has on its variability. 21

22 Sensitivity Analysis 50,0% 45,0% Target variable: Satisfaction 44,0% 40,0% 35,0% 30,0% 25,0% 26,4% 20,0% 15,0% 10,0% 5,0% 4,6% 9,7% 15,3% 0,0%

23 Sensitivity Analysis 1000 simulated target variable using the BN estimated on the observed dataset. Distribution of simulated target variable for each level of each variable Equipment Sales Technical Support Mean Mean Mean Impact of factors on target variable Suppliers Administrative Support Terms and Conditions Mean Mean Mean of simulate target variable for each level of each variable (ABC) Mean

24 Sensitivity Analysis Compare the observed distribution with the simulated one (1000 runs) using Kolmogorov-Smirnov test Equipment Sales Technical Support Mean of KS Mean of KS Mean of KS Impact of factors on conditioned target variable distribution Suppliers Administrative Support Terms and Conditions Mean of KS Mean of KS Mean of Kolmogorov-Smirnov for each level of each variable (ABC) Mean of KS

25 Information Quality (InfoQ) of Integrated Analysis f(f 1,f 2, f n ) Models f 1 f 2 f 3 f 4. N f Goals g 1 4 g 2 2 g 3 2 g 4 1 N g Kenett R.S. and Salini S. (2011). Modern Analysis of Customer Surveys: comparison of models and integrated 25 analysis, with discussion, Applied Stochastic Models in Business and Industry, 27, pp

26 Goal BN CUB Rasch CC 1 Decide where to launch improvement initiatives 2 Highlight drivers of overall satisfaction 3 Detect positive or negative trends in customer satisfaction 4 Identify best practices by comparing products or marketing channels 5 Determine strengths and weaknesses 6 Set up improvement goals 7 Design a balanced scorecard with customer inputs 8 Communicate the results using graphics 9 Assess the reliability of the questionnaire 10 Improve the questionnaire for future use Kenett R.S. and Salini S. (2011). Modern Analysis of Customer Surveys: comparison of models and integrated 26 analysis, with discussion, Applied Stochastic Models in Business and Industry, 27, pp

27 Goal 1: Identify causes of risks that materialized Goal 2: Design risk mitigation strategies Goal 3: Provide a risk management dashboard 27

28 28 Bayesian network of communication network data

29 Conditioned BN Experimental Array Robustness of Prediction 29

30 Conditioned BN Experimental Array Robustness of Prediction Classification error runs i 1 I ( ( x) s s ) i i 30 BAYESIAN NETWORKS: ROBUSTNESS AND SENSITIVITY ISSUES By Federica Cugnatay, Ron Kenetty,z and Silvia Salini

31 Analysis: Main effect plots of mean GoF Main Effects Plot (data means) for GoF mean 61.0 Lines Extensions Mean of GoF mean Smart phones Analyzing the mean GoF for each factor, the most robust levels are Lines=3, Smart Phone=3, and Extensions=

32 Analysis: Main effect plots of mean GoF 5 4 Contour Plot of GoF mean vs Extensions, Lines GoF mean < > 61.0 Extensions 3 Hold Values Smart phones Lines A central number of lines and a central level for extension is associated with a low mean GoF 32

33 Analysis: Main effect plots of mean GoF Smart phones Contour Plot of GoF mean vs Smart phones, Lines GoF mean < > 62 Hold Values Extensions Lines A limited number of lines and a limited level for smart 33 phone is associated with a low mean GoF

34 Analysis: Main effect plots of mean GoF Contour Plot of GoF mean vs Smart phones, Extensions Smart phones GoF mean < > 61.5 Hold Values Lines Extensions 4 5 A limited level both for extension and smart phone is associated 34 with a low mean GoF

35 Knowledge Goals Information Quality InfoQ(f,X,g) = U(f(X g)) Information Quality Data Quality Primary Data Analysis Quality Secondary Data - Experimental - Experimental - Observational - Observational g X f U A specific analysis goal The available dataset An empirical analysis method A utility measure 1. Data resolution 2. Data structure 3. Data integration 4. Temporal relevance What How 5. Chronology of data and goal 6. Generalizability 7. Operationalization 8. Communication 35

36 Big Data Analytics InfoQ(f,X,g) = U(f(X g)) 1. Data resolution 2. Data structure 3. Data integration 4. Temporal relevance 5. Chronology of data and goal 6. Generalizability 7. Operationalization 8. Communication Russom, P., Big Data Analytics, TDWI Best Practices Report, Q

37 Knowledge 1. Data resolution 2. Data structure 3. Data integration 4. Temporal relevance 5. Chronology of data and goal 6. Generalizability 7. Construct operationalization 8. Communication Data Quality Information Quality Analysis Quality Goals Bayesian Network 37

38 Thank you for your attention What did we cover? Statistical thinking Bayesian networks Information Quality (InfoQ) Robustness of network Sensitivity of predictions