Stochastic Dynamic Programming Based Approaches to Sensor Resource Management

Transcription

1 Stochastic Dynamic Programming Based Approaches to Sensor Resource Management R.B. Washburn, M.K. Schneider, J.J. Fox Fusion Technology and Systems Division Alphatech, Inc. Burlington, MA bob.washburn, michael.schneider, Abstract This paper describes a stochastic dynamic programming based approach to solve Sensor Resource Management (SRM) problems such as occur in tracking multiple targets with electronically scanned, multi-mode radar. Specifically, it formulates the SRM problem as a stochastic scheduling problem and develops approximate solutions based on the Gittins index rule. Novel results include a hybrid state stochastic model for the information dynamics of tracked targets, an exact index rule solution of the SRM problem for a simplified tracking model, and use of approximate dynamic programming to extend the index rule solution to more realistic models. Keywords: Sensor resource management, dynamic programming, Gittins index, multi-armed bandit. 1 Introduction Modern sensor suites contain sensors that are agile, allowing fast redirection (e.g., electronic scanning) and fast switching between modes (e.g., radar modes such as SAR and MTI), giving a large number of control options. However, allocation of sensor resources is also subject to a variety of constraints: there may be many things to look at (complex environments with multiple targets), limited time to sense targets because of deadlines on the usefulness of information (urgent requests) or exposure to threat environments, sensors may have a limited field of view, and so on. Developing sensor control algorithms that satisfy such constraints and maximize the value of information obtained is the goal of Sensor Resource Management (SRM) research. Stochastic dynamic programming [1] provides a principled, systematic approach to formulating and solving SRM problems [8]. However, its application requires paying the price of developing mathematical models for the problem formulation and then finding efficient algorithmic solutions. In this paper we describe new results in developing efficient solutions based on stochastic dynamic programming to solve SRM problems such as occur in tracking multiple targets with electronically scanned, multi-mode radar. Specifically, we formulate the SRM problem as a stochastic scheduling (or multi-armed bandit) problem and develop approximate solutions based on the Gittins index rule [2]. Although there has been much research on index rules [7], [6], [3], [9], [5], there appear to be no applications to SRM except for the recent work by Krishnamurthy and Evans [4]. The reason for this is that the SRM problem, formulated as stochastic scheduling, doesn t usually satisfy the conditions necessary for the existence of an index rule solution. Thus, one is forced to develop approximations (e.g., the restless bandit work of [9], [5]). In this paper we show how to relax one of the usual conditions and to develop index rule-based SRM algorithms for kinematic tracking. The new contributions of this paper include: 1. Stochastic scheduling formulation of SRM and a corresponding extension of the index rule solution to more general reward functions, more suitable for SRM problems. 2. Hybrid state stochastic model of information dynamics for tracking problems, which facilitates efficient solutions of stochastic dynamic programs and alleviates the curse of dimensionality. 3. Exact index rule solution for a simple tracking model. This solution provides a basis for efficient approximations of more realistic tracking models. 4. Lookahead approximation of the stochastic dynamic program solution based on an index rule solution, applicable to more realistic tracking models. Section 2 formulates the SRM problem as a stochastic scheduling problem and shows how to extend the index rule solution to more general types of reward functions. This section also discusses generalization of the formulation to controlling sensors with variable dwell times and

2 selectable modes. Section 3 describes hybrid state models of information dynamics to use for tracking problems. These models use discrete-valued random variables to drive a continuous-valued information state. This approach helps reduce the dimensionality of the resulting stochastic dynamic programming equations. In Section 4 we find the optimal index rule solution for a simple version of the tracking problem and use this solution in Section 5 as the basis for approximate solutions of more complex models. Section 5 also describes numerical results of the lookahead approximation for a multi-target tracking example. 2 SRM Problem Formulation 2.1 Stochastic Scheduling Formulation We formulate SRM as a stochastic control problem in which sensor actions Ù Ø are the controls and the controlled state, denoted Ü Ø, is an information state. Conceptually, Ü Ø denotes the accuracy of the information in the track. Most generally, it can be the conditional probability of the target state given previous observations (as in [4]). In our problem Ü Ø will be an estimate of the track error covariance. In this paper we will assume that the sensor action applies to one target at a time and that time is discrete (e.g., Ø ¾ ). The control Ù Ø indicates which target to look at time Ø. The information state is initially Ü ¼, and Ü Ø is given by a stochastic equation Ü Ø Ü Ø Û Ø Ù Ø (1) for Ò, Û Ø are random variables representing uncertainties in target dynamics and sensor measurements. We assume that Û Ø are independent random variables with distribution Ô Û. At each time Ø we must determine an action Ù Ø for the sensor to take depending only on the information state Ü Ø Ü Ø Ü Ò Ø (2) of all targets. The SRM problem is to determine Ù Ø as a function of Ü Ø which maximizes the expected value of the total discounted reward, namely Ò Â ÑÜ «Ø Ê Ü Ø ØÜ ¼ ¼ (3) the optimization is taken over all stationary control policies (i.e., Ù Ø Ü Ø ). The parameter «is the discount factor. The dynamic programming equation is  ÑÜ Ù Ò Ê «Â Û Ù Ô Û Û (4) Ô Û Ò Ô Û (5) The optimal control policy is given by finding the action Ù which achieves the maximum Â. 2.2 SRM and Index Rules In the conventional stochastic scheduling problem (for which there are index rule solutions), one receives the discounted reward «Ø Ê Ù Ø Ü Ù Ø Ø (6) at time Ø. In the SRM problem one receives the reward «Ø Ò Ê Ü Ø (7) In the conventional scheduling problem, the state Ü Ø of a machine evolves via Ü Ù Ø Ø Ù Ø Ü Ù Ø Ø Û Ù Ø Ø (8) and Ü Ø Ü Ø if Ù Ø (9) In the SRM problem (8) is still true, the equation represents the prediction and update of the target track state over one time step. The noise Û Ø expresses uncertainty in how the target state actually changes as well as uncertainties in the measurement of the target state. However, if the sensor doesn t look at the target, the evolution of the state is Ü Ø Ü Ø Û Ø if Ù Ø (10) the equation represents the prediction of the target track state over one time step, without a measurement update. The noise Û Ø expresses uncertainty in how the target state changes in one time step. In general, Ü Û Ü. Thus, the SRM problem differs from the stochastic scheduling problem in two critical respects: 1. The reward depends on unsensed targets (in the conventional scheduling problem idle machines contribute no reward). 2. The state of an unsensed target can change (in the conventional scheduling problem the state of idle machines remain frozen). In some SRM problems it is possible to have Ü Ú Ü (11) so that the state of unsensed targets remains the same, just as in the scheduling problem. For example, this is the case

3 if Ü are statistics of a static parameter (e.g., target classification problems, Ü is a probability vector of possible target types). If (11) is true, we can reformulate the SRM problem with an equivalent reward function so that the reward doesn t depend on unsensed targets, again just like the scheduling problem, and thus we can get index rules which are optimal solutions of the SRM problem. Theorem 1. Suppose that Ü Ø Ü Ø Û Ø Ù Ø (12) for Ò Û Ø are independent for all Ø and Û Ø has distribution Ô Û for all Ø. Suppose that the control Ù Ø ¾Ò and Ù Ø depends on the state Ü Ø. Suppose that for all Ù. Define Ê Ü Then a policy optimizes Ü Û ÙÜ (13) Ê Ü Û Ô Û Û Ê Ü Ø¼ if and only if it optimizes «Ø Ê Ù Ø Ü Ù Ø Ø Ü ¼ Ê Ü Ø Ü ¼ Ò «Ø ؼ Proof. For any control Ù Ø show that for constants ؼ «Ø Ê Ù Ø Ü Ù Ø Ø Ü ¼ ؼ «Ø Ò «¼. Ê Ü Ø Ü ¼ (14) (15) (16) Corollary 2. If the target state remains unchanged when the target is not viewed by the sensor, then there is an index rule which is an optimal solution of the SRM problem. That is, there are functions Ñ Ü, for each target, such that the optimal SRM policy is given by Ù Ø Ö ÑÜ Ñ Ü Ø (17) If Â Ü Å is the solution of Â Ü Å ÑÜ Ê Å Ê Ü (18) «Â Ü Û Å Ô Û Û Ê Ü Ê Ü Û Ô Û Û Ê Ü (19) then the index function is defined by Ñ Ü ÑÒ Å Â Ü ÅÅ (20) Proof. Follows from the theorem above and results for finding index functions in [1]. The result shows that the assumption that idle machines do not contribute to the reward is not a critical assumption. As long as the states of idle machines are frozen, the theorem shows that it is possible to find an equivalent reward such that idle machines do not contribute. Thus, frozen states of idle machines is the key assumption in order to have an optimal index rule solution to the stochastic scheduling problem. Of course, the frozen state assumption is not realistic for tracking SRM problems. However, one can use index rules as the basis for good suboptimal approximations of the more general problem. 2.3 Extension of SRM Formulation It is necessary to extend the problem formulation to model more realistic situations in which sensors have selectable modes and in which different modes require different amounts of time to complete (e.g., consider the difference between SAR and MTI modes for a radar). In such situations we formulate the scheduling problem so that the control Ù Ø and the information state Ü Ø are constant over the time interval Ø. Let Ü denote the information state of target at stage. Stage begins at time and ends at time. At each stage we must determine a control Ù for the sensor to use over the period Ø. In this case the control Ù has the form Ù (21) is the target Ò to look at and is a sensor mode to use. Time ¼ ¼ and is given by (22) so that the time duration of stage depends on the sensor mode used. This control Ù depends only on the information state at the beginning of the state, namely, Ü Ü Ü Ò (23) The information state evolves according to Ü Ü Û Ù (24) We can assume that Û depends only on Ü and Ù and has conditional distribution Ô Û Ü Ù.

4 The SRM problem is to determine Ù Ø as a function of Ü Ø which maximizes the expected value of the total discounted reward, namely  ÑÜ ¼ «Ø Ò Ê Ü Ø ØÜ ¼ (25) the optimization is taken over all stationary control policies (i.e., Ù Ø Ü Ø ). The parameter «is the discount rate. Because the information state is piecewise constant, the total discounted value is Ò «Ø Ê Ü Ø Ø ¼ ¼ «««Ò Ê Ü Thus, we obtain an equivalent discrete time problem with the dynamic programming equation  ÑÜ Ù Ê È «Ù Ò «Ê «Ù Â Û Ù Ô Û Ù Û Ô Û Ù Ò (26) Ô Û Ù (27) 3 Model of Information Dynamics for Tracking Problems 3.1 Modeling Detection Uncertainty To design the SRM algorithm we need to model how the information state responds to sensor actions, as modeled by the equation Ü Ø Ü Ø Û Ø Ù Ø (28) Modeling the information state as a conditional probability distribution of the target state variable is one possible model, but for many problems it gives a large dimensional stochastic evolution equation and a correspondingly large dimensional dynamic program to solve. Thus, we try to model the information dynamics with smaller number of salient statistics in order to obtain a tractable stochastic dynamic program. For tracking problems, in which the SRM algorithm is trying to maximize the quality of kinematic tracking for multiple targets, the track error covariance is an effective information state Ü Ø and hybrid state equations, in which Û Ø takes a finite number of values, provide a low complexity description of the information dynamics. The simplest instance of the model has Û Ø just indicate target detections so that Û Ø takes just two possible values (denote, Û Ø if the target is detected and Û Ø if the target isn t detected). Let denote the probability that target is detected, assuming that the sensor looks at target. Then has the following form. Ü Ü Ø À Ì Ê À Ì É Ì (29) Ü Ü Ø Ì É Ì (30) If the sensor doesn t look at target (Ù ), then Ü Û Ù Ü Ø Ì É Ì (31) whether Û or. Here Ü Ø denotes an error covariance and the matrices,, É, À, Ê model the effect of prediction and update on the error covariance. 3.2 Modeling Association Uncertainty We can also include the effect of misassociations by considering a four state process Û Ø as follows. Suppose that Û can be one of the four states (32) stands for detection, for missed detection, for association, and for misassociation. Then Ü Û Ù has the following form. Ü Ü Ø À Ì Ê À Ì É Ì (33) Ü Ü Ø Ì É Ì (34) Ü Ü Ü Ø Ë Ì É Ì (35) The idea is that Ë ¼ models the bias induced by a misassociation. If the sensor doesn t look at target (Ù ), then Ü Û Ù Ü Ø Ì É Ì (36) A simple model for Ë is Ë ¾ ¾ Ã Ê Ã Ì (37) is a gate size parameter (e.g., or ), is the dimension of the measurement, à is the Kalman gain Ã Ü Ø À Ì Ê À À Ì Ê (38)

5 and Ê À Ü Ø À Ì Ê (39) is the predicted measurement covariance. The probabilities of the discrete events are given by ÈÖ Û Ø È (40) ÈÖ Û Ø È (41) ÈÖ Û Ø È (42) ÈÖ Û Ø È (43) For ¾ the association probability models are given by and È ¾ ¾ ¾ ¾ ¾ ¾ (44) È ¾ (45) Ö Ø Ê (46) is the desired gate size, Ê is the predicted measurement covariance, and is the expected density of confuser objects in the measurement gate of target. Note that is the expected number of confusers in the one-sigma predicted measurement gate. This is sometimes referred to as the contention for the target. 4 Tracking SRM Index Rule 4.1 Target Location Example Suppose that each target has a location and Ü Ø is the root mean square error of its location estimate. Define the goal set to be ¼Æ (47) for some Æ ¼. Define the individual target track reward function as Ê Ü Ü ¾ (48) denotes the indicator function, equal to if the condition is true and equal to ¼ otherwise. The sensor looks at just one target, Ù Ø at time Ø. If the sensor looks at target and the target is detected, then Ü Ø Ü Ø ¾ ¾ ¾ (49) Assume that is the probability of detecting the target, if the sensor looks at it. If the target is not detected or if the sensor doesn t look at target, then Ü Ø Ü Ø (50) Let denote the initial location error i.e., Ü ¼. 4.2 Equations for Index Function Transform Ü to Þ Ü ¾. Then we can express the reward as Ê Þ Þ ¾ Æ ¾ (51) with the evolution equation Þ ¾ if target is detected Þ Û (52) Þ if target is not detected To find the index rule to a sum of individual target rewards, we need to replace the reward function Ê Þ with Ê Þ defined by Ê Þ Ê Þ Û Ô Û Û Ê Þ if Æ ¾ ¾ ÞÆ ¾ ¼ if ÞÆ ¾ ¾ or Þ Æ ¾ To find the index function corresponding to Ê Þ,we solve Â Þ Å ÑÜ Å Þ ¾ Æ ¾ ¾ Æ ¾ «Â Þ ¾ Å «Â Þ Å (53) by successive approximation, noting that the value function at any iteration is piecewise constant. The successive approximation iterations converge to the optimal value function Â Þ Å ¼ Å Å Þ ¾ Å Þ ¾ for ¼ and 4.3 Index Rule «Å if Å «Å if Å «(54) «(55) «The index function Ñ Þ is defined as the smallest Å such that Â Þ ÅÅ (56) For Þ ¾, this is the smallest Å such that Å Å. Thus, Ñ Þ «(57)

6 for Þ ¾. Similarly, for Þ ¾, wehave Ñ Þ ¼ (58) In terms of the parameters Æ and, we can express the index function as Ñ Þ «Thus, the index rule control is which is equivalent to if Æ ¾ ¾ Þ Æ ¾ ¾ ¼ ¼ if Þ Æ ¾ (59) Ù Ø ÖÑÜÑ Þ (60) Ù Ø Ö ÑÜ Þ Ø Þ Ø Æ ¾ (61) If we assign different goals Æ as well as different values Î to each target, then it is easy to see that the corresponding index functions are Ñ Þ given by Ñ Þ Î «if Æ ¾ ¾ Þ Æ ¾ ¾ ¼ ¼ if Þ Æ ¾ (62) «(63) «4.4 Extension to Higher Dimensions We can extend the previous scalar results to more general situations the information state is a covariance matrix. Thus, let Ü be the target track error covariance for target. Assume that Ü evolves according to Ü if and Û Ü Û Ù (64) Ü otherwise Û Ø is a ¼ random variable and Assume that Ü satisfies ÈÖ Û Ø (65) Ü Ü (66) in the sense of positive definite matrices Ü and Ü. Define the reward function Ê Ü by Ê Ü Î if Ü Æ ¼ otherwise Assume that is also monotonic in the sense that (67) Ü Ü ¼ (68) if Ü Ü ¼ in the sense of positive definite matrices. For example, Ü is monotonic in this sense if it is the trace or maximum eigenvalue of the matrix Ü. Finally, define the integer function Ü to be the minimum Ò such that Ò Ü Æ (69) The index rule for the scheduling problem is given by Ñ Ü Î Ü «Ü is defined as above. ¼ otherwise if Ü ¼ (70) Proof. One can prove this extension by using the same approach as for the simple scalar case e.g., by solving the dynamic programming equation by successive approximation and computing the index rule. A more elegant approach can be obtained by using the insightful interpretation of the index function Ñ Ü given by [7] as the maximization over stopping times That is, È Ò Ñ Ü ÑÜ ¼ ؼ «Ø Ê Ü Ø Ü ¼ Ü Ó Ò È Ø¼ «Ø Ü ¼ Ü Ó (71) for the SRM problem Ê is the transformed reward function. 5 Approximations Based on Index Rule and Examples 5.1 Lookahead Approximation The index rule is optimal only if the state of unsensed targets remains unchanged, which is not a realistic assumption for tracking problems. However, as noted in [4], it is a reasonable approximation to make when the state changes slowly. In our problem formulation, we require the track error covariance to grow slowly over the time scale «(the discount rate defines a soft time horizon for the problem). If the error covariance grows more rapidly over this time horizon, we can use the index rule as a basis for a better approximation of the stochastic dynamic programming problem. For example, we have used the index rule s value function to compute Ò-step lookahead approximations to the dynamic programming equations [1]. The state of the dynamic program is the vector Ü Ø of target tracking error covariances ( Ü Ø is the covariance for target ). The dynamic programming equation has the form Â Ü ÑÜ Ù Ê Ü Â Ü Ù Û Ô Û Û (72)

7 Ê Ü is the immediate reward Ê Ü Ò and Ü Ù Û is the dynamic equation Â Ò Ü ÑÜ Ù Ê Ü (73) Ü Ø Ü Ø Ù Ø Û Ø (74) with noise Û Ø. If ¼ Ü denotes an initial approximation for the value function, then Ê Ü Â Ò Ü Ù Û Ô Û Û (75) defines the Ò-step lookahead approximation for Ò. We will use the index rule to define  ¼ and compute the one- and two-step lookahead approximations numerically from it.  ¼ is computed by assuming that the undetected target covariance Ü don t change. Let Ñ Ü denote the index function of target, and define and Ü as in the previous section. Moreover, let Ü Ñ Ü if Ü ¼ and Ü if Ü ¼. If we re-order the target indices so that ¼ implies that Ü ¼ Ü ¼, we can show that  ¼ Ü «Ò Î Ü (76) Using this expression for  ¼ we can compute Â Ò numerically from (75) for Ò ¾. The following examples illustrates the performance for the lookahead approximations and compares them to the performance of the pure index rule (Ò ¼) approximation. 5.2 Example The one- and two-step lookahead and index rule controls are tested on a simple scenario with ¼ targets, one of which has a high value and others which have low values. Each target moves according to a simple one-dimensional Brownian motion model. The simulation parameters are given in Table 1. Performance is measured for each target in two ways. The first is the time-averaged relative mean square error Ì Ü Ø Ì Ì Æ ØÌ ¾ Ü Ø (77) Ì is the length of the run, Ì is the time from which the system is assumed to be in steady-state, Æ is the error goal, Ü is the true target position, and Ü is the tracker position estimate for a simple Kalman filter. For low-value targets, the MSE is averaged over the targets. The second performance measure is the time averaged frequency that the error goal is attained, Ì Ü Ø Ü Ø Æ (78) Ì Ì Æ ØÌ For low-value targets, the frequency of goal attainment is averaged over the targets. These performance measures are computed for different values of the moving process noise. The results are plotted in Figures 1-4. Note that the one and two-step lookahead controls keep the high-value target errors down better than the index rule for moderate to low values of the process noise. At high values, the two-step lookahead does slightly worse. This is because the twostep lookahead control is devoting some time to low value targets, dramatically decreasing their errors. Parameter Number of Monte Carlo Runs Time of Run (Ì ) Steady-state Start Time (Ì ) Discount Factor («) High Target Value (Î ) Low Target Value (Î ¾¼) High-Value Target Error Variance Goal (Æ ) Low -Value Target Error Variance Goal (Æ ¾¼) Measurement Noise Variance (Õ) Probability of Detection ( ) Initial Position Variance (Ü ¼) Relative MSE References Table 1: Simulation Parameters. High value Target Index Rule Lookahead 1 Step Lookahead 2 Steps Moving Process Noise Figure 1: High-Value Target MSE. Value ¼¼ ¾¼¼ ¼¼ ¼ ¼¼ ¼¾ ¼ [1] D.P. Bertsekas, Dynamic Programming and Optimal Control, Athena Scientific, Belmont, Massachusetts, ¾¼

8 Relative MSE Low value Targets Index Rule Lookahead 1 Step Lookahead 2 Steps Moving Process Noise Goal Attainment Frequency Low value Targets 0.2 Index Rule Lookahead 1 Step Lookahead 2 Steps Moving Process Noise Figure 2: Low -Value Target MSE. Figure 4: Low -Value Target Goal Attainment Frequency. 1 High value Target Trans. on Signal Processing, Vol. 49, , December, Goal Attainment Frequency Index Rule Lookahead 1 Step Lookahead 2 Steps Moving Process Noise Figure 3: High-Value Target Goal Attainment Frequency. [2] J.C. Gittins, Bandit processes and dynamic allocation indices, Journal of the Royal Statistical Society,B41, , [3] M.N. Katehakis and A.F. Veinott, Jr., The multiarmed bandit problem: decomposition and computation, Mathematics of Operations Research, Vol. 12, No. 2, pp , [5] J. Niño-Mora, Restless bandits, partial conservation laws and indexability, Dept. of Economics and Business, Universitat Pompeu Fabra, Barcelona, Spain, December 13, [6] J.N. Tsitsiklis, A Lemma on the Multiarmed Bandit Problem, IEEE Transactions on Automatic Control, AC-31, , [7] P.P. Varaiya, J.C. Walrand, and C. Buyukkov, Extensions of the Multiarmed Bandit Problem: The Discounted Case, IEEE Transactions on Automatic Control, AC-30, , [8] R. Washburn, A. Chao, D. Castanon, and R. Malhotra, Stochastic Dynamic Programming for Far-Sighted Sensor Management, Proc. of the IRIS National Symposium on Sensor and Data Fusion, Vol. 2, , [9] P. Whittle, Restless Bandits: Activity Allocation in a Changing World, in J. Gani (editor), A Celebration of Applied Probability, Journal of Applied Probability, 25A, , [4] Vikram Krishnamurthy and Robin J. Evans, Hidden Markov Model Multiarm Bandits: A Methodology for Beam Scheduling in Multitarget Tracking, IEEE