AIAA Infotech@Aerospace 2010 20-22 April 2010, Atlanta, Georgia AIAA 2010-3481 Probabilistic Evidential Reasoning with Symbolic Argumentation for Space Situation Awareness Glenn Takata 1 and Joe Gorman 2 Charles River Analytics, Inc., Cambridge, MA 02138 A number of research efforts have investigated approaches and technologies to improve space situation awareness. Prior research included investigation of probabilistic methods using structured models such as Bayesian belief networks. Such approaches have a firm foundation in evidential reasoning and are believed to have considerable potential for reasoning about situations of interest in space. However, building the necessary models and specifying the quantitative probabilities linking nodes is a non-trivial effort. Probabilistic models necessitate the estimation of probabilities that relate events. However, situations of interest are rare events and a lack of relevant observations makes it challenging to validate probabilistic models. Research in the elicitation of probabilities has shown that even domain experts cannot effectively generate valid mathematical relationships between abstract entities that form probabilistic models. The size and complexity of probabilistic models that realistically describe situations of interest exacerbates the challenge of defining the required probabilities. Data driven machine learning approaches have been used to address this problem in other domains. However, the lack of exemplars of events of interest limits the applicability of data driven machine learning to the modeling space situations of interest. The result is an increased reliance on elicited knowledge. Technologies are needed that simplify the process of model construction and that can take advantage of multiple information sources to provide space situation awareness. This paper describes an approach, based on Toulmin s theory of argumentation, which provides an intuitive representation that can be used to create and exercise models of space situations. These models are then supplied to a fusion component that uses the Dempster-Shafer theory of belief propagation to estimate confidences for the characterization of these situations of interest. I. Introduction In 2001, the U.S. Space Commission 1 enumerated the potential threats to the U.S. s space infrastructure and called for changes across the U.S. Department of Defense (DOD) in order to mitigate existing vulnerabilities. The Rumsfeld Commission report has since been adopted as the basis for numerous space management and situation awareness programs. A first step in managing any situation is situational awareness. Therefore, the Air Force has focused on improving space situational awareness (SSA) by putting into place systems that monitor the health and condition of satellites and their output, watching for indications that the spacecraft are being affected by natural or artificial means and whether this constitutes an attack. 2 A number of research efforts have been instituted to investigate approaches and technologies to improve SSA. Prior research included investigation of probabilistic methods using structured models like Bayesian belief networks. 3,4 Such probabilistic methods have a firm foundation in evidential reasoning. 5 For SSA probabilistic methods are believed to have considerable potential for reasoning, but building the models and specifying the quantitative probabilities linking nodes 6 is a non-trivial effort. Probabilistic model development is complicated by the lack of relevant instances of exceptional space situations that can be used as bases for reasoning or to estimate the structural parameters. The lack of exemplars limits the applicability of data driven machine learning approaches and increases the reliance on elicited knowledge. The scarcity of available training data puts an enormous burden put on the analyst to generate all the parameters required to specify the model and estimate the probabilistic 1 Senior Software Engineer, Decision Management, Charles River Analytics, Inc., Cambridge, MA. 2 Principal Engineer, Decision Management, Charles River Analytics, Inc., Cambridge, MA. 1 Copyright 2010 by the, Inc. The U.S. Government has a royalty-free license to exercise all rights under the copyright claimed herein for Governmental purposes. All other rights are reserved by the copyright owner.
parameters. A Bayesian network using 10 binary (true, false) variables, for instance, requires the estimation of 1024 probabilities. This burden can be alleviated in special cases when the model has a special structure or when simplifying assumptions can be made. 7,8 However, these models still necessitate the estimation of probabilities to relate events, which would be difficult without hard evidence to validate them. Research in the elicitation of probabilities has shown that even domain experts cannot generate valid mathematical relationships between abstract entities, a problem that gets exacerbated when a large amount of relationships need to be specified. 9 In this paper we present a knowledge-based system that is used for the fusion of multisensor events that has application in the creation and maintenance of space situation awareness. The approach uses symbolic argumentation to implement data fusion at Level 2 (Situation Assessment) of the JDL model. 10,11 The aim of this system was to improve the users ability to make sound decisions through enhanced situation awareness. The system uses uncertain or incomplete information to form situation models that enable decision makers to quickly identify and characterize situations of interest while filtering out irrelevant information. This system uses a high-level graphical knowledge representation based on an intuitive model of human discourse. The system is based on symbolic argumentation, a framework which supports human experts by augmenting and complementing their own cognitive abilities. 12 When combined with methods for reasoning with uncertain and incomplete information the system provides a declarative and robust framework for automated decision-making. The system includes a visual model development tool that facilitates the elicitation of knowledge about situations of interest, enabling users to record and evaluate their decision-making processes using the system s automated capabilities. The following section reviews the principles underlying the argumentation approach. This is followed, in section 3, by a description of the graphical representation of argumentation models. In section 4 discuss our implementation of the model development environment and the argumentation based fusion engine. Section 5 describes how the system is applied in a multi-level fusion environment. We conclude, in section 6, with some observations from our experience with this project and some discussion of further research. II. Argumentation for Fusion The argumentation approach uses Toulmin s 13 theory of argumentation as a basis for the development of a reasoning model and the Dempster-Shafer theory 14 to calculate belief estimates based on this model. A. Argumentation Theory Toulmin 13 described a procedural model of human reasoning, which provided a rich analysis of argument and can serve as a guideline for organizing the information the information user to make decisions under conditions of uncertainty. Toulmin s model decomposes an argument into claim, data, warrant, backing, qualifier, and rebuttal (Fig. 1). The claim is the conclusion for which an estimate of support is required and data represents the evidence that will support the claim. The warrant explains why the data are relevant to the claim. Backing is supplied to support the validity of the warrant in the argument. The qualifier indicates how strong the claim is while the rebuttal represents an exception or counter argument to the claim. In this example, the claim is that a collision will occur. Supporting evidence is supplied through an observation (data) that a debris has been detected within 2.8 miles and the knowledge (warrant), based on prior research, that a near flyby is likely to result in a collision. A counterargument (rebuttal) is supplied by the observation that the debris is moving away from the satellite. This formulation gives a natural structure to argumentation that can be exploited for knowledge elicitation and for confidence estimation. 2
. Figure 1. Example of Toulmin's Theory of Argumentation B. Dempster-Shafer Belief Propagation The influence of evidence on arguments and, ultimately, on the claims is determined through an application of Dempster-Shafer theory. 14 This theory of beliefs supports reasoning under uncertain and incomplete information. Unlike other inferencing paradigms, such as Bayesian networks, Dempster-Shafer theory does not require a complete specification of all the nodes and parameters in a model that incorporates uncertainty in the calculation. This ability to make use of uncertain or incomplete information simplifies the task of building a chain of argumentation and of specifying parameters. Dempster-Shafer theory provides an automated inference technique that generalizes the Bayesian theory of subjective probability. Like Bayesian approaches, Dempster-Shafer reasoning accumulates evidence and updates prior claims in the light of new evidence. Unlike Bayesian methods, Dempster-Shafer theory requires only that the data support or refute the claim. It does not require that the total belief be apportioned over all the possible claims. The simple example of Fig. 1 may be expanded with other claims that are related to, but mutually exclusive with, the Collision event. For instance, there may be a claim of No Collision. If data provided a.50 belief in a Collision, the remaining.50 belief would not be applied to No Collision, but would represent uncertainty in the model. Calculations based on this approach result in upper and lower bounds on the belief in a claim (Fig. 2). The lower bound, support, is the degree to which the evidence supports a proposition and is based on the direct evidence for the claim. The upper bound, Figure 2. Dempster-Shafer Outputs plausibility, is the extent to which the evidence fails to refute a proposition and is based on the total evidence for related but mutually exclusive claims. The actual belief in this claim lies between these bounds. III. Network Representation of Argumentation The Dempster-Shafer model was the basis of our graphical representation for argumentation (Fig. 3). In this representation, a situation of interest is characterized by a set of mutually exclusive claims about an event. These are represented as Characterization nodes in the network. Data that contribute to each characterization can take to form of intermediate nodes that result from the combination of other data or can be leaf nodes that represent simple data and serve as entry points for evidence from external inputs. They are represented as intermediate or leaf Argument nodes, respectively. The influences of these data on each claim and on each other are specified as weighted Edges connecting Arguments to Characterizations and to other Arguments. Warrants are represented as 3
connections between Arguments and Edges, i.e. an Argument, representing backing, can influence the weight on an Edge. Figure 3. Network Representation of Toulmin Argumentation Characterizations can be simple or complex. Complex Characterizations are defined as sets of mutually exclusive simple Characterizations. The network representation does not change. However, increased support for one of the Characterizations will affect the plausibility of the related Characterizations. Complex Arguments are treated in the same fashion. Rebuttals are treated as normal Arguments. However, during evaluation their evidence is used to support the complementary Characterization or Argument. In the case of a simple Characterization or Argument this is simply the opposite Characterization or Argument, e.g. Threat vs. No Threat. In the case of complex Characterizations or Arguments, the evidence is applied to the complementary set of Characterizations or Arguments. This representation that encapsulates the reasoning, is readily understood and is more easily manipulated by subject experts than other probabilistic models. IV. Implementation The prototype system consisted of two parts, (1) a fusion engine that implemented the Dempster-Shafer reasoning, and (2) a modeling environment that assisted in the development and assessment of argumentation networks that characterize situations of interest. The fusion engine was a component in a larger research prototype multi-level fusion system for SSA. The modeling environment was available for stand alone use during model development. Expert knowledge was encoded in graphs that represent argumentation networks. The argumentation network was represented internally as a hierarchical structure of nodes and edges and was stored externally in XML. The fusion implementation consisted of three processes. The first was a data conditioning process which extracted information from the event message for use in association and evaluation. The second was an association process that used the conditioned data to identify the situation and the Argument within that situation which could ingest evidence about that event. The third process applied the evidence from the event to the situation model. Event data was supplied by external applications through a publish and subscribe (PUB/SUB) messaging mechanism. The results of evaluation were published back as situation messages. 4
Figure 4. Modeling Environment The conditioning process was tailored to each data source. For each source, data were extracted and parameters computed that could supply the information needed for the association and evaluation steps. Specifically, these included (1) a time window for the event, (2) a description of the event, (3) an estimate of the evidence provided by the event, and (4) the identification of an asset or target. The event description included specific parameters that were supplied to the rule processor during the association process. The association process relied on a forward-chaining rules implementation to identify a leaf Argument for the incoming event data. An ontology based reasoning system was used, which allowed the rules to take advantage of taxonomic relationships in the data to make its inferences. The rules specified the conditions on event parameters that were necessary for a particular argument. Sensor and other facts from incoming L1 data fusion events were supplied to the reasoner. If an association was deduced, the corresponding situation was used in the evaluation. Confidences supplied with the event were used to calculate evidence for further processing. The evaluation process applied the weight on the Edge to the evidence associated with the source Argument node. The destination node of the Edge used Dempster-Shafer to combine the evidence from all incoming nodes into a support value. It then relayed this support value to subsequent nodes in the graph. This process was performed recursively until the Characterization nodes were updated. The support and plausibility values for each node were then combined into a situation message which was published for use by other applications. The fusion engine was implemented as a plugin, which could be inserted into an OSGI-based application that could supply sensor event tracks and could, in turn, receive situation reports. The fusion engine ingested event track messages and published situation reports for use by other components. Its operation was driven by external situation model and rule definition files. The introduction of new situation models required the modification of these files and little or, in most cases, no modification of the fusion software. This gave the user the ability to quickly apply new models and rules without complex reprogramming. The modeling environment provided the user with a tool to visually build and modify the situation models and to create and edit the association rules (Fig. 4). It consisted of a tool palette and a model editor. In the model editor the argument network was visualized as a set of blue rectangles and green ovals connected by arrows. The rectangles 5
represented characterizations of the situation while the ovals represented arguments. The arrows indicated the influences between nodes and could be weighted to moderate the influences. With this drag-and-drop editor a situation model could be quickly built giving the expert an immediate visualization of her thinking process. The model could be tested using sliders that simulated the evidence applied to the Argument node. The fusion engine plugged into the development environment used these inputs to evaluate the state of the situation and display the results back to the user. A playback tool allowed the user to select a test dataset and apply it to the situation model that was being created. This tool is shown in the lower left of Fig. 4. The behavior of the model was displayed in the editor as the data were processed. This allowed the user to validate the model against external datasets that might have been prepared specifically for validation or might have been obtained from other simulations. This combination of model editor and playback tool gave the user the ability to iteratively refine the situation model. Once completed the model and rules could be saved to external files and used with the fusion engine in other applications. V. Application This system has proved an efficient and intuitive means for eliciting knowledge about situations of interest while simultaneously building the argumentation networks that form the basis for situation awareness. New networks were created and existing networks updated by users with minimal technical support. The Dempster- Shafer based fusion component implemented fusion at Level 2 of the JDL model. 10,11 This model comprises five fusion levels representing increasing refinement of data characterization. Briefly, level 0 involves preprocessing at the sensor or other source; level 1 performs identification and tracking of data relating to a single entity or event; and level 2 iteratively fuses multiple entities into a picture of a situation. Level 3 combines information from multiple levels to perform a threat or impact assessment and level 4 performs process assessment and monitoring in order to optimize the process itself. The modeling environment was used to build situation models based on specific and generalized scenarios describing situations of interest. The models were evaluated with an appropriate set of input data sources that were available in the runtime or simulation environment. Data conditioners were implemented as necessary for the sources. Rules were developed to associate measurands to the arguments. The process of building the model and defining the rules would suggest refinements requiring new arguments and new data sources that would enhance the fidelity of the model and consequently, the operator s understanding of the situation. This could lead to the addition of new data sources and their associated level 0 or level 1 fusion processes or to the refinement of the existing fusion processes to provided better detail for use by the situation model. New data sources would require the development of new conditioners to prepare the data for ingestion by the association process. A player component was used to apply test data to the situation model. Controlled playback permitted the verification and assessment of rule classification incoming events and the correct association of events to model arguments. Once the associations were made, evidence was calculated for each leaf argument and the results propagated to the conclusions. The conclusions were then published back to the editor view. The expert was able to review these results to assess if the evidence produced belief values that conformed to her view of the situation. Based on these results, the expert was able to adjust the model and the rules to fit with her knowledge of the situation. This sequence of playback and adjustment led to incremental refinements of the situation model. Ultimately, the final model was used in a runtime application that was developed to demonstrate not only the Level 2 fusion, but the integration of this and other components implementing a variety of fusion levels of the JDL model. 10,11 The fusion engine was installed in the application as a plugin along with configuration files that contained the model, the rules, and other supporting data. VI. Conclusion Argumentation networks were used in the runtime applications to detect and characterize space situations of interest by fusing data from a variety of ground and satellite based sensors to estimate confidences for the characterization of these situations. The visual development tool gave the expert an intuitive editor to build situation models using her knowledge and to refine these models using playback tools to test the model. The tool and the fusion engine are based only on the data representation and not on any domain specific requirements. Hence, this symbolic argumentation based approach is applicable to other environments and venues that require the fusion of multisource inputs to characterize a situation that is incompletely or imperfectly understood or for which prior data are unavailable. 6
Other military venues for situation awareness, e.g. air or ground operations, are candidates for the application of this technology. However, the technique is readily extendable outside of SA to applications, like vehicle health maintenance or medical diagnostics, which require a similar fusion of multiple streams of data. These venues pose similar types of problems, where the expert is called upon to characterize and act upon an event whose causes are only partially understood. The expert is able to build up a structure of plausible arguments that lead to believable characterizations and consequent decisions. Acknowledgment This work was performed under Government contract number FA9453-08-C-0005 with the Air Force Advanced Research Laboratory. The authors thank our sponsors at AFRL/RV for their support and direction on this project. References 1 Rumsfeld, D., Rumsfeld Commission. 2001. 2 Tirpak, John A., Space Superiority Cannot be Taken for Granted, so the Air Force is Making Plans to Defend It, Air Force Magazine, June, 2006, pp. 42-46. 3 Hanson, A.M., Gonsalves, P., Tse, J., and Grey, R., Automated Data Fusion and Situation Assessment in Space Systems, International Journal on Artificial Intelligence Tools, vol. 13, no. 1, March, 2004, pp. 57-61. 4 Wu, J.K. and Wong, Y.F., Bayesian Approach for Data Fusion in Sensor Networks, 9th International Conference on Information Fusion, 2006, 10-13 July, 2006, pp. 1-5. 5 Schum, D.A. The Evidential Foundations of Probabilistic Reasoning. Evanston, IL, Northwestern University Press, 2001. 6 Pew, R.W. and Mavor, A.S. (eds), Modeling Human and Organizational Behavior, Washington, DC, National Academy Press, 1998. 7 Lemmer, J. F. and Gossink, D. E., Recursive Noisy or- a Rule for Estimating Complex Probabilistic Interactions, IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics, vol. 34, no. 6, 2004, pp. 2252-2261. 8 Wisse, B.W., van Gosliga, S.P., van Elst, N.P. and Barros, A.I., Relieving the Elicitation Burden of Bayesian Belief Networks, Sixth Bayesian Modeling Applications Workshop on UAI 2008. 9 Zagorecki, A. and Druzdzel, M., An Empirical Study of Probability Elicitation Under Noisy-OR Assumption, Proceedings of the Seventeenth International Florida Artificial Intelligence Research Society Conference (FLAIRS 2004), edited by V. Barr & Z. Markov, AAAI Press, Menlo Park, CA, 2004, pp. 880-885. 10 Steinberg, A. and Bowman, C., Rethinking the JDL Data Fusion Levels, Proceedings of National Symposium on Sensor and Data Fusion, JHAPL, 2004. 11 Hall, D.L. and Llinas, J. (eds), Handbook of Multisensor Data Fusion, Boca Raton, FL, CRC Press, 2001. 12 Das, S., Symbolic Argumentation for Decision Making Under Uncertainty, Proceedings of the 8th International Conference on Information Fusion, Philadelphia, PA, 2005. 13 Toulmin, S. E., The Uses of Argument, New York, Cambridge University Press, 2003. 14 Shafer, G. and Logan, R., Implementing Dempster s Rule for Hierarchical Evidence, Artificial Intelligence, vol. 33, no. 3, 1987, pp. 271-298. 7