Open World Target Identification Using the Transferable Belief Model

Transcription

1 Open World Target Identification Using the Transferable Belief Model Matthew Roberts EADS Innovation Works Newport, UK Gavin Powell EADS Innovation Works Newport, UK Dafni Stampouli EADS Innovation Works Newport, UK Abstract The availability of datasets for the comparison of fusion techniques across the board is a key ingredient to the fusion community. Presented in this paper is one such dataset along with a selection of fusion methods applied and evaluated using a set number of prescribed metrics. The dataset is the output of a tracking and classification algorithm that describes a land based vehicle as it traverses across some terrain. This data can be fused over time to obtain a more robust target classification. Evidence theory combination rules are applied to the dataset and an extension to a current combination rule is presented and evaluated. I. INTRODUCTION The Evaluating Techniques for Uncertainty Reasoning Working Group (ETURWG) is an ISIF working group formed to allow a common test bed to be created where fusion algorithms can be tested and evaluated so that a common point of reference can be used for comparison with competing algorithms. As such there is a need for data, scenarios and metrics to be defined and made available. In this paper we describe that dataset and the three scenarios that are encompassed within it. For means of comparison a set of metrics are described and applied to the fusion algorithms that have been used to process that dataset for target classification. Furthermore we propose an extension to the GRP [] combination rule to allow past classifications to have a greater impact on possible future states. This is carried out by eliminating targets from the potential set if it has been shown that the current object is manoeuvring in a manner that is outside of the capable range for a given potential target type. The dataset is based on an open world scenario. The open versus closed world is an important point and has relevance to the fusion rules used and their application. In a closed world we are making the assumption that we uncategorically know every possible outcome, or rather the true state must be within our list of possible outcomes. In some circumstances this is true but often in reality we operate in an open world. In an open world we have a list of possible outcomes but accept that this is not completely exhaustive and that there are some possible states that we do not have in our list, or are even unaware of. The open world is generally more difficult to deal with but as it has more applicability in the real world it has been chosen for this dataset. Section II explains the scenarios and metrics used in this paper. Relevant background material relating to evidential theory is briefly presented in Section III. Section IV describes our proposed solution to the use case. An improvement to the GRP combination rule is given in Section V. Our results are shown in Section VI. Section VII contains our conclusions. II. SCENARIOS The scenarios used in this paper are from a use case for the Evaluating Techniques for Uncertainty Reasoning Working Group (ETURWG). The use case is based on previous work for performing joint tracking and classification with a Wireless Sensor Network (WSN) and the Transferable Belief Model (TBM) [2], [3]. The purpose of the use case is to classify the target for each of the three scenarios. Each scenario consists of a single target moving through an area of interest whilst being monitored by a WSN. Each of the nodes have rangeonly sensors. In the original work, a sensor selection algorithm was used to determine which sensors should be active at each time step, and a particle filter was used to estimate the position and velocity of the target at each step. For the use case, the output of the particle filter is available, and as such, sensor selection and target tracking does not need to be performed. It should be noted that the estimate of the target velocity in all three scenarios is very noisy. There are 5 known target classes: amphibious light tank (); pedestrian; car; light tank (), and bicycle. We have an open world assumption, and as such the simulations also contain an unknown target class a main battle tank (MBT); this is the ground truth target class for Scenario C. The simulated target trajectories used this paper include non-linear motion such as slowing down at bends and junctions whilst travelling along a road. Terrain is assumed to hinder target classes by varying amounts; this is modelled by assigning each terrain type into a terrain class for a given target class. Three terrain classes are used Go, Slow Go, and No Go. Table I shows these assignments; the letters G, S, and N indicate Go, Slow Go, and No Go, respectively. For example, a pedestrian travelling on the terrain types Road and Grass is considered unhindered and can therefore travel up to and including it s Go speed its maximum possible speed. The maximum speed for each target class and terrain type combination is shown in Table II. This has previously been presented in [2], [3].

2 TABLE I TERRAIN CLASS ASSIGNMENTS. Road Grass Water Buildings Marsh Trees Steep Land Target Class estrian G G N N S G S G G S N S S N G G N N N S N G S N N N S N MBT G G N N N S N G S N N N S N Buildings Start Trees Finish Steep land Road Grass Water TABLE II TARGET CLASSES AND THEIR ASSUMED MAXIMUM SPEEDS FOR EACH TERRAIN CLASS. Maximum Speed (ms - ) Target Class Go Slow Go No Go estrian MBT Fig. 2. Target trajectory and terrain for Scenario B. Annotations have been added to the terrain map. The red line is the target trajectory. As shown in [2]. Steep land Start Steep land Trees Buildings Start Finish Buildings Finish Water Grass Road Marsh Road Fig. 3. Target trajectory and terrain for Scenario C. Annotations have been added to the terrain map. The red line is the target trajectory. As shown in [2]. A. Metrics Fig.. Target trajectory and terrain for Scenario A. Annotations have been added to the terrain map. The red line is the target trajectory. As shown in [2], [3]. Scenario A consists of an travelling along a road, over grass, crossing a section of water, and then travelling over grass again. The terrain map for this scenario can be seen in Figure. The is the only target class capable of travelling over water. In Scenario B, a car travels along a road which includes crossing a roundabout (see Figure 2). The car slows down whilst crossing the roundabout and accelerates away from it. Scenario C consists of a MBT that travels along a road and then goes off road (see Figure 3). Due to the known target classes, this unknown class may appear to be a tank whilst travelling along a road. Two metrics are used to compare the performance of the fusion algorithms, which in this case are different combination rules classification accuracy and decision accuracy. Section VI discusses how these metrics are calculated when the ground truth target class is an unknown target class (i.e. Scenario C). Classification accuracy is the mean classification probability for the ground truth target class across the length of the simulation. In certain circumstances, this metric may favour closed world scenarios as the normalisation at each time step can inflate the classification probabilities. Decision accuracy is calculated as the proportion of the total simulation time where the classification probability of the ground truth is more than any other class. This metric is designed to be less susceptible to the inflation that can occur with normalisation. This metric is used with the assumption that the target class is decided purely on the classification probability.

3 III. BACKGROUND Evidence theory was born some years ago through the initial work of Arthur Dempster and Glenn Shafer [4], [5]. There are various forms based upon that original methodology that exist to date. It is this field of techniques that are used to determine the target in the 3 scenarios of the ETURWG use case, and of which we choose to compare and contrast using a prescribed set of metrics. The methods that we apply to the data set fall into various subsections of evidence theory and those being Dempster-Shafer Theory (DST), Dezert-Smarandache Theory (DSmT) [6] and the TBM [7]. The underpinnings of these are well described in their respective literature and as such we suggest the reader follow those references if unfamiliar with any of the approaches. This example utilises the TBM as a means of applying the various combination rules. The TBM relies on working in the space where you have a set that contains all possible outcomes that are to be considered, and also an empty set, which denotes anything that is not included in that set. A basic belief assignment (bba) is created at each time step, and this is where mass, m, is placed on each of the possible subsets of the world and denotes the belief we have that the real true state lies somewhere within that set as shown by: A. Combination rules m : 2 Ω [, ] with A Ω m(a) =. () At the core of any evidential theory technique for the fusion of 2 or more pieces of information is the combination rule. This allows information to be combined in a manner that will enable the system to make a robust and informed decision about the world that is being sensed. The combination rules vary in complexity, usability, and applicability to the problem space. These rules are generally well known and their strengths and weaknesses well documented. We do propose an extension to previous work in defining combination rules and show the new rule s comparative results when applied to the data set. ) Conjunctive combination: The conjunctive rule of combination is generally accepted to be the original rule for combination used within DST. It is capable of working in both the open and closed worlds and its simplicity ensures it is often used to conquer problems. It tends to work best in a closed world, issues relating to this have previously being discussed by the authors [8], [9]. When utilised in an open world this rule will tend toward empty set domination and an unresponsive system. The unnormalised open world conjunctive rule of combination is given by: m 2 (X) = m (A) m 2 (B), (2) A,B 2 Ω A B = X where m 2 is decisive intersection of the two pieces of evidence given by m and m 2. The normalised closed world is given by: m DS (X) = k 2 A,B 2 Ω A B = X where k 2 is the normalisation factor: k 2 = A,B 2 Ω A B = m (A) m 2 (B), (3) m (A) m 2 (B). (4) 2) Disjunctive combination: The disjunctive rule of combination [] is the counterpart to the conjunctive rule of combination and provides a cautious combination of information, as opposed to the decisiveness of the conjunctive. This rule will tend toward vagueness and the placement of mass on the ignorant set as more combinations are performed. We only apply this rule using an open world assumption within this paper. The disjunctive rule is given by: m 2 (X) = A,B 2 Ω A B = X m (A) m 2 (B). (5) where m 2 is the cautious union between the two pieces of evidence given by m and m 2. 3) PCR5: PCR5 [] is a combination rule that uses a more intelligent means of redistributing the conflict when used in a closed world. To give this a means of comparison to the other rules here we have used it in an open world manner. This rule was intended for use within the DSmT framework where the open world is treated in a slightly different manner. The PCR5 rule that we have utilised for this comparison is given by: m P CR5 (X) = m 2 (X) + [ Y 2 Ω \{X} X Y = m (X) 2 m 2 (Y ) m (X) + m 2 (Y ) + m 2(X) 2 m (Y ) m 2 (X) + m (Y ) ], (6) This is generally seen as one of the more impressive combination rules that can be applied to evidence theory. 4) RCR-S: RCR-S [2] is a robust combination rule with symmetric coefficients. This rule is a weighted combination of both the conjunctive and disjunctive combination rules. The weighting factor is given by: α (k) = k k + k 2 and β (k) = k k + k 2 (7) and are applied to the robust combination rule by: m RCR S (A) = k k + k 2 m (A)+ k k + k 2 m (A) (8) and m RCR S ( ) =. RCR-S is the closest neighbour to the GRP rule in terms of its underlying methodology.

4 5) GRP: GRP [] was created to combat the shortcomings of some of the other combination rules that are in circulation. This rule was required to be able to effectively combine information in a temporal manner, where measurements may be received for an infinite amount of time, while still retaining stability, flexibility and an element of memory of the past to effect the way in which current information is combined. This is specifically designed for use where some estimate of state is given and constantly updated with new information. GRP is the arithmetic mean of the conjunctive and disjunctive combination rules, after they have been discounted by some factor. The discounting takes place using: { m α α m(a) if A Ω; (A) = (9) α m(a) + ( α) if A = Ω, to discount the conjunctive rule, and: { m α α m(a) if A Ω, A ; (A) = α m(a) + ( α) if A =, () to discount the disjunctive rule. The amount that we discount by is governed by the precision (lack of vagueness) that the current state/system has. There are various precision metrics [], but the one used within this paper is given by: p (m) = Ω A Ω m(a) A, A 2Ω () The GRP combination rule is: m GRP (A) = m fd(a) + m fc (A), (2) 2 m fd (A) = m f md (A), (3) m fc (A) = m f mc (A), (4) where A 2 Ω, m md = m α m (using Equation ), m mc = m α m (using Equation 9), both discounting equations use α = p (m f ), m f is the bba that represents the existing information, and m m is the bba that represents the incoming information. IV. SOLUTION The solution for the ETURWG use case that is presented here is based on previous work for performing tracking and classification with a WSN and the TBM. The frame of discernment is the set of known target classes. Each simulation consists of a number of discrete time steps. At each of these steps a bba is created and is fused with the bba that represents the existing knowledge the pignistic transform is then used to provide a classification probability for each target class. The conditional bba created at each time step, m C, is created calculated using the Generalised Bayesian Theory (GBT) [3]: m C (A x) = pl(x c i ) [ pl(x c i )], (5) c i A c i Ā where pl(x c i ) is the plausibility of target class c i given the speed of the target, x, and A 2 Ω. The plausibility of a target for a given speed and given terrain class can be calculated using: ( pl(x) = (x x) Betf(x) + dā ) Betf(a) da, x da (6) where Betf is a bell shaped pignistic density function, Betf(x) = Betf( x), and x v x; v is the mode of Betf. x is taken as the mean of the W N most recent speed estimates from the particle filter. The Betf for each target and terrain class combination can be found in Section VI. Equation 6 is calculated for each of the three terrain classes and combined using a weighted sum; the weightings are calculated from the coverage of an ellipse whose size and shape are defined the number of standard deviations, N σ, and the covariance of the current particle filter position estimate weighted covariance, respectively. The centre of the ellipse is positioned at the weighted mean of the particle filter position estimate. For a more detailed explanation of this please see [2], [3]. If a closed world assumption is used, m C is then normalised to remove the mass assigned to the empty set. If this is not possible, i.e. m C ( ) =, then an assignment of m C (Ω) = is used. Once m C has been calculated for the current time step, it is combined with the existing bba that represents the existing knowledge built up over all of the previous time steps, m k. If this is the first time step than the existing bba is taken as m k (Ω) =. The two bbas are fused using a combination rule resulting in m k. The performance of different combination rules is compared in Section VI. At each step the pignistic transform is used to calculate a classification probability (including the for the empty set) [7]: (B) = A Ω m k (A) A B A A Ω. (7) This is then used to calculate the metrics discussed in Section II-A. V. IMPROVED COMBINATION RULE For the ETURWG use case, we have modified an existing combination rule, GRP. The modification was made to allow GRP to classify a target whilst retaining a memory of target classes that are no longer possible. This prevents mass from being assigned to targets that have been previously been determined as having little or no classification probability. The modification should allow this improved rule to classify a target in an open world scenario especially when a scenario consists of different segments with different target behaviours. The remainder of this section describes the improved combination rule, GRP+. At the start of using the GRP+ combination rule, the conditional bba created in the current time step, m C, is conjunctively combined, using an open world assumption, with m F (F) =, where F is the set of all feasible and known target classes; this transfers all of the belief assigned

5 to targets which have been deemed no longer possible onto the remaining masses: m CF (A) = m C F (A) A 2 Ω. (8) The GRP combination rule is then used, but instead of discounting to the ignorant set, mass is moved to the most ignorant but still feasible set F. To achieve this, Equation 9 is modified to be: { m α α m(a) if A F; (A) = (9) α m(a) + ( α) if A = F, In the notation used in Section III-A5, m f = m k and m m = m CF. After an initialisation period, I, a target class, A, is deemed to no longer be possible if (A) is smaller than or equal to a threshold, T. The set of feasible target classes is determined using: F = {A Ω : (A) > T }. (2) If at any time F =, then the set of feasible classes is reset to F = Ω. The set of feasible classes is determined at the end of using the GRP+ combination rule for the next time the GRP+ is used. VI. RESUS Simulations were run for each combination of the 3 scenarios and 7 combination rules: GRP; GRP+; PCR5; RCR- S; conjunctive combination (closed world assumption); conjunctive combination (open world assumption); and disjunctive combination (open world assumption). At the end of each simulation, the classification accuracy and decision accuracy metrics where calculated. When calculating the metrics, the unknown target was assumed to be classified as the empty set. Therefore, for the purposes of the metrics, the empty set was treated like any other target class. This section For all three scenarios, N σ = 5, W = 7, and T =.. The plausibilities calculated at each time step, using Equation 5, are calculated using the pignistic density functions in Figure 4. The classification accuracy calculated for each combination rule and scenario can be found in Table III; Table IV contains the decision accuracy results. Where Table III and IV state DNF for a particular combination of scenario and combination rule, the simulation failed to finish; this was always due to an attempt to normalise a bba during the fusion step where k =. An extra row has been added to both tables to show an average for the 3 scenarios, this provides an overall view of the performance of each combination rule. It can been seen that GRP+ has the highest scoring overall classification and decision accuracy; this is due to the poor performance of the combination rules that use a closed world assumption when trying to identify the target in Scenario C. Betf(x) Betf(x) Betf(x).5.5 tbmterrain Go Speed (ms ) x (a) 4 2 tbmterrain Slow Go PT76 ZiL 44 T62 Challenger Speed (ms ) x (b) 2 tbmterrain No Go Speed (ms ) x (c) Fig. 4. The assumed Betf for the terrain classes Go (a), Slow Go (b), and No Go (c). Previously shown in [3]. Scenario GRP TABLE III CLASSIFICATION ACCURACY GRP+ Combination rule PCR5 RCR-S Conj. (Closed) Conj.(Open) Disj. A DNF.283 B C.2.32 DNF.979 Mean N/A A. Scenario A GRP creates a robust and flexible classification as shown in Figure 5. The issue of this approach is clearly highlighted where its flexibility means that the target is incorrectly classified. More importantly this misclassification is with an object type that should be impossible given the objects previous motion. This is shown after time step 2 where the object is classified as a bicycle even though previously the target was moving at a velocity that is highly unlikely for a bicycle. GRP+ accounts for this by correctly removing the bicycle from the list of possible outcomes, once it understands that the object motion dictates it is not a possible class, as shown

6 Scenario GRP TABLE IV DECISION ACCURACY GRP+ Combination rule PCR5 RCR-S Conj. (Closed) Conj. (Open) Disj Scenario A GRP+ A DNF B C DNF.977 Mean N/A Scenario A GRP Fig. 5. The fused bba for each time step of Scenario A when using the GRP combination rule. in Figure 6. It can be clearly seen that at time step 2 there is now a low, but equal, probability of each target class showing ambiguity. This is as expected as the motion of the object is not pertaining toward any particular class. This improved behaviour results in a better classification and decision accuracy for GRP+ compared to GRP. In Table III, it can be seen that PCR5 and RCR-S score higher than the other rules. This is because these rules are designed to work with a closed world assumption (within the TBM and the redistribution of mass that takes place inflates the values for when compare to GRP and GRP+. This is a safe behaviour to have in this Scenario, but not in open world scenarios. This effect is reduced for the decision accuracy metric because the absolute value of is not important. The use of conjunctive combination with a closed world assumption fails as it cannot cope with when the conflict k =. The conjunctive combination with an open world assumption scores badly due to the convergence toward the empty set; similarly the scores for the disjunctive rule of combination are poor due to the convergence toward the ignorant set Fig. 6. The fused bba for each time step of Scenario A when using the GRP+ combination rule Scenario B GRP Fig. 7. The fused bba for each time step of Scenario B when using the GRP combination rule. B. Scenario B Similar properties are shown in Scenario B as were displayed in Scenario A. The vehicle is moving quite rapidly initially and heavily suggests that it is a car. Because of this other objects that cannot travel at that speed should be removed from the list of possible outcomes. Figures 7 and 8 depict these attributes of the algorithms well and provide a very different result. The notable different between GRP and GRP+ is between time steps where GRP+ is more decided. Notably GRP+ eventually classifies correctly but the flexibility of GRP means that an incorrect classification ensues. Both metrics show a similar performance for the open world approaches discussed above and the sophisticated closed world approaches of PCR5 and RCR-S. This is to be expected as the

7 Scenario B GRP Scenario C GRP Fig. 8. The fused bba for each time step of Scenario B when using the GRP+ combination rule Fig. 9. The fused bba for each time step of Scenario C when using the GRP combination rule. ground truth target is within the set of known target classes. The conjunctive rule of combination with a closed world assumption has achieve a perfect score as it initially believes the target class is a car with % certainty and is unable to change its mind later in the simulation. The conjunctive rule of combination with an open world assumption scores badly due to the convergence discussed in Section VI-A. The disjunctive rule of combination scores better in this scenario compared to Scenario A because during the first part of the scenario, where the target travels along the road quickly, the target s behaviour is very different from the other known target classes. C. Scenario C In Scenario C, the object is moving and then slows down considerably to make a turn and traverses across grass. While traversing across grass this should show that it is very unlikely to be a car, as is shown below by the low probability from time step 2 to 25 as shown in Figure 9. After this period, the car becomes the most likely target class even though the last period of information has suggested that it can t be a car. GRP+ overcomes this issue, as shown in Figure, by retaining some memory of what has previously happened. It removes car from the list of possible outcomes once the object travels across the grass in a manner that is highly unlikely for a car. GRP+ provides a correct classification while GRP incorrectly classifies. Unlike Scenarios A and B, this scenario requires an open world assumption in order to perform well; this can been seen in Tables III and IV where the closed world combination rules do not receive a score of more than for both metrics. It should be noted that a better score for both metrics may have been seen if the PCR5 combination rule had been used within the DSmT framework instead of the TBM. The open world conjunctive combination rule appears to perform very well, Scenario C GRP Fig.. The fused bba for each time step of Scenario C when using the GRP+ combination rule. but it is not possible to determine how much of this is due to noise and the overall metrics for this rule are not as good as GRP and GRP+. VII. CONCLUSION We have presented a vehicle identification use case for the Evaluating Techniques for Uncertainty Reasoning Working Group (ETURWG) and proposed a solution to this that uses the TBM. The use case is based on a previous work, a closed world tracking and classification problem, which has been modified to be an open world classification problem. The use case consists of three scenarios. The solution proposed is not limited to a particular combination rule. A new rule, based on GRP, called GRP+ has been proposed; this rule adds a stronger memory element to move mass away from targets

8 that have previously been eliminated from the classification exercise. The results presented in this paper have shown a good performance for the GRP+ combination rule in comparison to other rules. PCR5 and RCR-S has performed well in the closed world scenarios, but are unable to perform adequately in the open world scenario (within a TBM implmentation). As is to expected, the conjunctive combination rule with a closed world assumption and the disjunctive rule of combination with an open world assumption do not perform well in the scenarios presented in this paper. Although the results for GRP+ combination rule are better than other rules for the use case presented in this paper, there is still room for improvement. Future work should improve upon this combination rule to perform better when the ground truth target class is not within the set of known classes. ACKNOWLEDGMENT The authors would like to thank David Marshall, diff University and Pete Talbot-Jones, EADS both of whom helped to develop the previous work on which the ETURWG use case is based. REFERENCES [] G. Powell and M. Roberts, GRP. A recursive fusion operator for the transferable belief model, in Proceedings International Conference on Information Fusion 2, Jul 2. [2] M. Roberts, Tracking and classification with wireless sensor networks and the transferable belief model, Ph.D. dissertation, diff School of Computer Science & Informatics, diff University, 2. [3] M. Roberts, D. Marshall, and G. Powell, Improving joint tracking and classification with the Transferable Belief Model and terrain information, in Proceedings International Conference on Information Fusion 2, Jul 2. [4] A. P. Dempster, A generalization of bayesian inference, Journal of the Royal Statistical Society, vol. 3, no. 2, pp , 968. [5] G. Shafer, A Mathematical Theory of Evidence. Princeton, NJ: Princeton University Press, 976. [6] J. Dezert and F. Smarandache, An introduction to DSmT, CoRR, vol. abs/93.279, 29. [7] P. Smets and R. Kennes, The Transferable Belief Model, Artificial Intelligence, vol. 66, no. 2, pp , 994. [8] G. Powell, M. Roberts, and D. Marshall, Pitfalls for recursive iteration in set based fusion, in Workshop on the Theory of Belief Functions, Apr 2. [9], biasing issues in the Transferable Belief Model for fusing and decision making, in Proceedings International Conference on Information Fusion 2, Jul 2. [] D. Dubois and H. Prade, Representation and combination of uncertainty with belief functions and possibility measures, Computational Intelligence, vol. 4, no. 3, pp , Sept 988. [] F. Smarandache and J. Dezert, Information fusion based on new proportional conflict redistribution rules, in Proceedings International Conference on Information Fusion 25, vol. 2, Jul 25, pp [2] M. Florea, A.-L. Jousselme, E. Bosséb, and D. Grenier, Robust combination rules for evidence theory, Information Fusion, vol., no. 2, pp , April 29. [3] B. Ristic and P. Smets, Belief function theory on the continuous space with an application to model based classification, Proceedings of Information Processing and Management of Uncertainty in Knowledge- Based Systems, IPMU, pp. 4 9, 24.