Technosocial Predictive Analytics Creating decision advantage through the integration of human and physical models Antonio Sanfilippo
Overview Motivations, needs and requirements General approach and challenges Responding to challenges Conclusions The work presented has been developed within the context of the Technosocial Predictive Analytic Initiative, a 4-year research program at the Pacific Northwest National Laboratory 2
Why Predictive Analytics and why now? Factors such as globalization and the ever-growing rate of knowledge sharing keep escalating the asymmetric nature of threat vectors Strategic surprises emerging from non-linear relationships between trigger and target events present harder challenges in managing risk Analysts and policymakers increasingly ask for anticipatory techniques to aid decision making Use Predictive Analytics descriptors such as What-if Scenarios and Possible Futures to help counter adversities and maximize opportunities 3
Why do we need computers to perform anticipatory reasoning? The human brain provides a unique framework for memory and prediction The ability to focus in the moment and take quick decisions by insight and intuition make human judgment uniquely effective However, human judgment can lead to fallacious reasoning when biased Lack of knowledge/expertise (Klein 1998) Groupthink (Janis 1972) Positive framing, trust in extreme judgments, neglect of uncorrelated observables (Tversky & Kahneman 1973, 1981) Memory and focus attention limitations (Miller 1956, Heuer 1999) 4
Bridge the gap between human and computer intelligence to enhance human judgment Help analysts and policymakers to make better decisions Provide multidisciplinary knowledge reachback to inform analysis and response during decision making Supplement the expertise of the analyst and policymaker with simulated scenarios generated by integrated computational models Stimulates creative critical reasoning through visual analysis and collaborative/competitive work 5
R&D areas and capabilities Knowledge inputs Support the modeling task by facilitating the acquisition and vetting of expert knowledge and evidence Technosocial modeling Develop new methods for integrating human and physical models Cognitive enhancement Leverage analytical gaming to stimulate creative thinking 6
Challenges Model design & calibration Knowledge coalescence Elicitation and integration of subject matter expertise Elicitation and brokerage of experts judgments Model interoperability Link across diverse models Model Instantiation Use marshaled evidence to apply models to specific problems Analytical integration Work with interoperable models to support analysis and decision-making within a collaborative environment 7
8 Model design and calibration
Modeling annual electricity consumption in 7 US cities (Lu et al., 2010) 9 Portland Salt Lake Phoenix Boulder Billings Vancouver Calgary 5 scenarios: Now; 2050: BAU, Setpoint Change, Lighting Efficiency, A/c Efficiency
Knowledge coalescence PCA of DOE Residential Energy Consumption Survey data (Sanquist et al. 2010) Composite variables RECS parameters Affluence Appliance use Climate Insulation Air conditioning at Home Heating Degree Days.032.081 -.776.278.159 -.091 Cooling Degree Days -.096 -.005.886 -.111 -.064.071 Age House -.115 -.150 -.190 -.385.441 -.010 Oven Use (Electric) -.003.428.006.140 -.066 -.016 Dish Washer Use.448.399 -.011.148 -.118.028 Number Refrigerators.481 -.029.056.225 -.005.147 Number Freezers.003.261 -.052.305 -.040.118 Clothes Washer Use.186.817 -.005 -.062.029.017 Clothes Dryer Use.055.813.060.028.001 -.038 Number Ceiling Fans.231.147.601.307.068 -.080 TV Use -.079.244.071 -.090.028.621 Number Small Appliances.601.209 -.078.188 -.094 -.192 PC Use.522.087.002 -.121 -.109.080 Number Thermostats.289.005 -.029.286.405 -.071 Number Wall/Window ACs -.055.002 -.021 -.159.783.077 Number AC Thermostats.191.121.403.248 -.625 -.084 Light Use.454.167.083 -.109 -.034.218 Sliding Glass Doors.334 -.079 -.178 -.038 -.370.056 Number Windows.595 -.039.017.118.315 -.116 Adequate Insulation.059 -.042.026.722 -.127.015 Type Window Glass.191.060 -.175.613 -.062 -.014 Person at home all day.009 -.146 -.006.116.028.780 Household Members.241.550 -.015 -.206.040.244 Household Income.705.111.023.035 -.102 -.269 10 Total Square Feet.615.043 -.003.285.043 -.109
Relevance of composite and single factors from RECS data analysis to energy consumption Model R R Square Adjusted R Square Std. Error of the Estimate Change Statistics R Square Change Appliance Usage.393 a.154.154.57263.154 Affluence.537 b.288.288.52543.134 Price.640 c.410.409.47876.121 Climate.707 d.499.498.44097.090 Number Hot Water Heaters.749 e.561.560.41281.062 Insulation.758 f.574.573.40687.013 At Home.762 g.580.579.40404.006 Hot Tub (Electric).766 h.586.585.40127.006 Age Youngest Household.768 i.590.588.39956.004 Member Married Household.769 j.592.590.39861.002 Air Conditioning Factor.770 k.594.592.39792.002 Below 125% Poverty.771 l.595.593.39723.002 Pool (Heated by Electricity).772 m.596.594.39677.001 11
Price 0.195 Energy Consumption Behavior 0.224 0.258 0.151 0.022 0.010 0.003 Affluence Appliance Use Climate Insulation Air Conditioning At Home Heating Degree Days -0.411 Cooling Degree Days 0.469 Age House -0.198 0.227 Oven Use (Electric) 0.142 Dish Washer Use 0.094 0.133 Number Refrigerators 0.101 Number Freezers 0.157 Clothes Washer Use 0.272 Clothes Dryer Use 0.270 Number Ceiling Fans 0.318 0.158 TV Use 0.443 Number Small Appliances 0.126 PC Use 0.110 Number Thermostats 0.208 Number Wall/Window ACs 0.403 Number AC Thermostats 0.2132-0.322 Light Use 0.095 Sliding Glass Doors 0.070-0.190 Number Windows 0.125 0.162 Adequate Insulation 0.371 Type Window Glass 0.315 Person at home all day 0.557 Household Members 0.183 Household Income 0.148 Total 12 Square Feet (sqrt) 0.129
13 Wiki-based approach to model discussion
Model Design: From model discussion to the creation of generic (box-n-arrow) models PICTURE OF UN-CALIBRATED ILLICIT TRAFFICKING MODEL TO BE ADDED 14
15 Model calibration using Conjoint Analysis
16 Results of model calibration
17 Model Interoperability
Model interoperability: relating SD and BN models System Dynamics IED Effectiveness Model Bayes net IED delivery model Coalition to C ivilian Casualty Ratio B3 Increased Coalition Patrols Catching Insurgents Cleared and B1 Ineffective IEDs Clearing IEDs Rollu B4 Technical Response R5 Technical Counter Measures IED Race IED Soph istication R3 Technical Improvements Deployed IEDs - - IED Effectiveness - R1 Percent Population Under Control Controlling Population - Gai 18 Recruitment R4 Recruitment
System Dynamics model of IED effectiveness* *Jarman, Brothers, Whitney, Young & Niesen, PSAM 2010 Coa lition to C ivilian Casualty Ratio Increased Coalition Patrols B1 B3 Cleared and Ineffective IEDs Catching Insurgents Rollu Clearing IEDs B4 Technical Response R5 Technical Counter Measures IED Race IED Soph istication R3 Technical Improvements - Deployed IEDs IED Effectiveness - - R1 Percent Population Under Control Controlling Population - Gai 19 Recruitment R4 Recruitment
Bayesian net model of IED delivery (Kill Chain)* 20 *Whitney et al. 2009
Interoperability of SD and BN Models* *Jarman, Brothers, Whitney, Young & Niesen, PSAM 2010 System Dynamics IED Effectiveness Model Bayes net Kill Chain Model B4 Technical Response R5 Technical Counter Measures IED Race Coalition to C ivilian Casualty Ratio The probability of IED placement in the BN is translated to a rate of deployment in the SD model to forecast IED effectiveness IED Soph istication R3 Technical Improvements Recruitment R4 B3 Increased Coalition Patrols Catching Insurgents Cleared and B1 Ineffective IEDs Clearing IEDs - - IED Effectiveness - Deployed Deployed IEDs R1 Percent Population Under Control Controlling Population Recruitment Recruitment - Rollu Gai The probability of recruiting personnel in the BN is a linear function of the rate of recruitment simulated in the SD model Build IED Place IED
Coupling Bayesian Nets and System Dynamic models Integrated SD/BN models can naturally be described as Dynamic Bayesian nets* time k time k1 time k2 time kn SD SD SD SD BN BN BN BN *Jarman, Brothers, Whitney, Young & Niesen, PSAM 2010 See also Mohaghegh et al., 2009 for a different approach 22
23 Model Instantiation: Extracting and vetting evidence from heterogeneous data sources
24 Model Instantiation: Using games to generate evidence (include 3 min video)
Analytical integration
Analytical integration Integrate models within a gaming environment to enable analysts and policy-makers to stress-test the quality of their analyses and plans through role-play Players take on competing or collaborating roles, according to their goals Each role is matched with resources and activities which link to model parameters When an activity is performed, the model changes state and the resources available to players may change 26
A sample game: Improvised Explosive Devices Players take on competing roles Blue team: prevent IED attacks Red team: carry out attacks Resource management flow Players allocate resources across predefined activities Model determines how activities increase/decrease the likelihood of success for red/blue team activities 27
Integrating computer models in analytical gaming System Dynamics The notions of stocks and flows map directly to environmental parameters that change over time SD engines such as STELLA offer interfaces for manipulating input parameters and tracking model change over time Bayesian Nets Players actions modify prior probabilities on event model nodes New probabilities are propagated in the model Evidence nodes are used to broadcast messages to players Model changes are conveyed to players in terms of Ensuing changes to their resources and the game outcome Messages which describe specific events (from evidence nodes) 28
Model Interaction AbnormalMovement of Materials evidence node Prepare site Late night movement of large vehicles headline Move the IED event node
Analytical gaming with interoperable models System Dynamics IED Effectiveness Model Bayes net Kill Chain Model 30
31 Exploring gaming results to distill sequence of events that define classes of strategies
Conclusions TBA 32
Thanks! 33 Email: antonio.sanfilippo@pnl.gov Phone: 509-375-2677 Web Site: http://predictiveanalytics.pnl.gov