Technosocial Predictive Analytics Creating decision advantage through the integration of human and physical models. Antonio Sanfilippo

Transcription

1 Technosocial Predictive Analytics Creating decision advantage through the integration of human and physical models Antonio Sanfilippo

2 Overview Motivations, needs and requirements General approach and challenges Responding to challenges Conclusions The work presented has been developed within the context of the Technosocial Predictive Analytic Initiative, a 4-year research program at the Pacific Northwest National Laboratory 2

3 Why Predictive Analytics and why now? Factors such as globalization and the ever-growing rate of knowledge sharing keep escalating the asymmetric nature of threat vectors Strategic surprises emerging from non-linear relationships between trigger and target events present harder challenges in managing risk Analysts and policymakers increasingly ask for anticipatory techniques to aid decision making Use Predictive Analytics descriptors such as What-if Scenarios and Possible Futures to help counter adversities and maximize opportunities 3

4 Why do we need computers to perform anticipatory reasoning? The human brain provides a unique framework for memory and prediction The ability to focus in the moment and take quick decisions by insight and intuition make human judgment uniquely effective However, human judgment can lead to fallacious reasoning when biased Lack of knowledge/expertise (Klein 1998) Groupthink (Janis 1972) Positive framing, trust in extreme judgments, neglect of uncorrelated observables (Tversky & Kahneman 1973, 1981) Memory and focus attention limitations (Miller 1956, Heuer 1999) 4

5 Bridge the gap between human and computer intelligence to enhance human judgment Help analysts and policymakers to make better decisions Provide multidisciplinary knowledge reachback to inform analysis and response during decision making Supplement the expertise of the analyst and policymaker with simulated scenarios generated by integrated computational models Stimulates creative critical reasoning through visual analysis and collaborative/competitive work 5

6 R&D areas and capabilities Knowledge inputs Support the modeling task by facilitating the acquisition and vetting of expert knowledge and evidence Technosocial modeling Develop new methods for integrating human and physical models Cognitive enhancement Leverage analytical gaming to stimulate creative thinking 6

7 Challenges Model design & calibration Knowledge coalescence Elicitation and integration of subject matter expertise Elicitation and brokerage of experts judgments Model interoperability Link across diverse models Model Instantiation Use marshaled evidence to apply models to specific problems Analytical integration Work with interoperable models to support analysis and decision-making within a collaborative environment 7

8 8 Model design and calibration

9 Modeling annual electricity consumption in 7 US cities (Lu et al., 2010) 9 Portland Salt Lake Phoenix Boulder Billings Vancouver Calgary 5 scenarios: Now; 2050: BAU, Setpoint Change, Lighting Efficiency, A/c Efficiency

10 Knowledge coalescence PCA of DOE Residential Energy Consumption Survey data (Sanquist et al. 2010) Composite variables RECS parameters Affluence Appliance use Climate Insulation Air conditioning at Home Heating Degree Days Cooling Degree Days Age House Oven Use (Electric) Dish Washer Use Number Refrigerators Number Freezers Clothes Washer Use Clothes Dryer Use Number Ceiling Fans TV Use Number Small Appliances PC Use Number Thermostats Number Wall/Window ACs Number AC Thermostats Light Use Sliding Glass Doors Number Windows Adequate Insulation Type Window Glass Person at home all day Household Members Household Income Total Square Feet

11 Relevance of composite and single factors from RECS data analysis to energy consumption Model R R Square Adjusted R Square Std. Error of the Estimate Change Statistics R Square Change Appliance Usage.393 a Affluence.537 b Price.640 c Climate.707 d Number Hot Water Heaters.749 e Insulation.758 f At Home.762 g Hot Tub (Electric).766 h Age Youngest Household.768 i Member Married Household.769 j Air Conditioning Factor.770 k Below 125% Poverty.771 l Pool (Heated by Electricity).772 m

12 Price Energy Consumption Behavior Affluence Appliance Use Climate Insulation Air Conditioning At Home Heating Degree Days Cooling Degree Days Age House Oven Use (Electric) Dish Washer Use Number Refrigerators Number Freezers Clothes Washer Use Clothes Dryer Use Number Ceiling Fans TV Use Number Small Appliances PC Use Number Thermostats Number Wall/Window ACs Number AC Thermostats Light Use Sliding Glass Doors Number Windows Adequate Insulation Type Window Glass Person at home all day Household Members Household Income Total 12 Square Feet (sqrt) 0.129

13 13 Wiki-based approach to model discussion

14 Model Design: From model discussion to the creation of generic (box-n-arrow) models PICTURE OF UN-CALIBRATED ILLICIT TRAFFICKING MODEL TO BE ADDED 14

15 15 Model calibration using Conjoint Analysis

16 16 Results of model calibration

17 17 Model Interoperability

18 Model interoperability: relating SD and BN models System Dynamics IED Effectiveness Model Bayes net IED delivery model Coalition to C ivilian Casualty Ratio B3 Increased Coalition Patrols Catching Insurgents Cleared and B1 Ineffective IEDs Clearing IEDs Rollu B4 Technical Response R5 Technical Counter Measures IED Race IED Soph istication R3 Technical Improvements Deployed IEDs - - IED Effectiveness - R1 Percent Population Under Control Controlling Population - Gai 18 Recruitment R4 Recruitment

19 System Dynamics model of IED effectiveness* *Jarman, Brothers, Whitney, Young & Niesen, PSAM 2010 Coa lition to C ivilian Casualty Ratio Increased Coalition Patrols B1 B3 Cleared and Ineffective IEDs Catching Insurgents Rollu Clearing IEDs B4 Technical Response R5 Technical Counter Measures IED Race IED Soph istication R3 Technical Improvements - Deployed IEDs IED Effectiveness - - R1 Percent Population Under Control Controlling Population - Gai 19 Recruitment R4 Recruitment

20 Bayesian net model of IED delivery (Kill Chain)* 20 *Whitney et al. 2009

21 Interoperability of SD and BN Models* *Jarman, Brothers, Whitney, Young & Niesen, PSAM 2010 System Dynamics IED Effectiveness Model Bayes net Kill Chain Model B4 Technical Response R5 Technical Counter Measures IED Race Coalition to C ivilian Casualty Ratio The probability of IED placement in the BN is translated to a rate of deployment in the SD model to forecast IED effectiveness IED Soph istication R3 Technical Improvements Recruitment R4 B3 Increased Coalition Patrols Catching Insurgents Cleared and B1 Ineffective IEDs Clearing IEDs - - IED Effectiveness - Deployed Deployed IEDs R1 Percent Population Under Control Controlling Population Recruitment Recruitment - Rollu Gai The probability of recruiting personnel in the BN is a linear function of the rate of recruitment simulated in the SD model Build IED Place IED

22 Coupling Bayesian Nets and System Dynamic models Integrated SD/BN models can naturally be described as Dynamic Bayesian nets* time k time k1 time k2 time kn SD SD SD SD BN BN BN BN *Jarman, Brothers, Whitney, Young & Niesen, PSAM 2010 See also Mohaghegh et al., 2009 for a different approach 22

23 23 Model Instantiation: Extracting and vetting evidence from heterogeneous data sources

24 24 Model Instantiation: Using games to generate evidence (include 3 min video)

25 Analytical integration

26 Analytical integration Integrate models within a gaming environment to enable analysts and policy-makers to stress-test the quality of their analyses and plans through role-play Players take on competing or collaborating roles, according to their goals Each role is matched with resources and activities which link to model parameters When an activity is performed, the model changes state and the resources available to players may change 26

27 A sample game: Improvised Explosive Devices Players take on competing roles Blue team: prevent IED attacks Red team: carry out attacks Resource management flow Players allocate resources across predefined activities Model determines how activities increase/decrease the likelihood of success for red/blue team activities 27

28 Integrating computer models in analytical gaming System Dynamics The notions of stocks and flows map directly to environmental parameters that change over time SD engines such as STELLA offer interfaces for manipulating input parameters and tracking model change over time Bayesian Nets Players actions modify prior probabilities on event model nodes New probabilities are propagated in the model Evidence nodes are used to broadcast messages to players Model changes are conveyed to players in terms of Ensuing changes to their resources and the game outcome Messages which describe specific events (from evidence nodes) 28

29 Model Interaction AbnormalMovement of Materials evidence node Prepare site Late night movement of large vehicles headline Move the IED event node

30 Analytical gaming with interoperable models System Dynamics IED Effectiveness Model Bayes net Kill Chain Model 30

31 31 Exploring gaming results to distill sequence of events that define classes of strategies

32 Conclusions TBA 32

33 Thanks! 33 Phone: Web Site: