Agent Behavior Monitoring using Optimal Action Selection and Twin Gaussian Processes

Transcription

1 5th Arentine Symposium on Articial Intellience, ASAI 204 Aent Behavior Monitorin usin Optimal Action Selection and win Gaussian Processes Luis Avila, Ernesto Martinez INGAR (CONICE-UN), Avellaneda 3657, Santa Fe S3002 GJC, Arentina {avilaol, Abstract. he increasin trend towards deleatin complex tasks to autonomous artificial aents in safety-critical socio-technical systems makes aent behavior monitorin of paramount importance. In this work, a probabilistic approach for on-line monitorin usin optimal action selection and twin Gaussian processes (GP) is proposed. A Kullback-Leibler (KL) based metric is proposed to characterize the deviation of an aent behavior (modeled as a controlled stochastic process) to its specification. he optimal behavior specification is obtained usin Linearly Solvable Markov Decision Processes (LSMDP) whereby the Bellman equation is made linear throuh an exponential transformation such that the optimal control policy is obtained in an explicit form. Keywords: Aent monitorin, Gaussian processes, optimal action selection. Introduction Safety-critical systems increasinly interate autonomous software aents in an increasin number of applications which requires new monitorin tools. As an example, consider the case of collision avoidance in drivin systems [] where the monitorin task involves a number of autonomous vehicles interactin with each other in a hihspeed hihway. Any monitorin system aimed to warn or prevent collisions and danerous circumstances must contemplate the expected behavior of nearby cars to detect quickly a collision scenario. Also, automatin the task of azin in video surveillance to find suspicious behaviors [2] hihlihts the importance of aent behavior monitorin. Detectin danerous objects and intruders is essential for safety in crowded environments, but monitorin human behaviors and reportin about potential threats is a complex task to be automated. he novelty and relevance of information contained in a data stream, can be correlated with the effect such data has on the observer (monitor) [3]. he amount of information can be measured in a natural way by the Kullback-Leibler (KL) distance -or relative entropy- between the prior and posterior distributions in the monitor beliefs, i.e. reardin the available space of hypotheses about the state of a controlled system. In this work, the novelty of information in a data stream reardin deviations from the specified aent behavior is measured usin twin Gaussian Processes [4]. For behavior 43 JAIIO - ASAI ISSN: Páina

2 5th Arentine Symposium on Articial Intellience, ASAI 204 specification, optimal choice of actions under uncertainty is used here to characterize the desired behavior of an intellient aent. 2 Optimal Action Selection o characterize the expected behavior of an aent, the novel approach of LSMDP is used [5]. his is a class of optimal control problems in which the Bellman s equation can be converted into a linear equation by an exponential transformation of the state nx value function. Consider an aent that by perceivin an environmental state x R, chooses the scalar action u R u, which causes the system to evolve to the state x k + and receives an immediate cost l ( x,u). he state transition function obey to a controlled Ito s diffusion process of the form u dx = a( x) dt + B( x )( udt + σ dω) () where ω R (same space as actions) and σ denote Brownian noise and its scalin parameter, respectively. he term a( x) describes the so called passive dynamics and B( x ) is the input-ain matrix. he passive dynamics represents the behavior of the stochastic dynamics in absence of control actions, it is defined as a diffusion process in continuous domains [5]. o put it in a more convenient form, the h-step transition probability for the uncontrolled dynamics is expressed as ( h x hb x σ x ) 0 p ( k + k ) = k + k + + h B B x x N x x a( ) ( ), ( ) ( x) (2) In turn, the controlled diffusion is approximated as a deterministic function expressed as a Gaussian distribution with mean and covariance iven as ( h( x hb x ) σ x B ) uk p ( xk + xk ) = N xk + xk + a( ) + ( ) u, h B( ) ( x) (3) he controller shifts the probability mass from one reion of the state space to another by actin on the system dynamics. A control policy π( x ) is thus defined as a probability of selectin the action u k at state x k. For any optimal control application, the main objective is to find an optimal control policy π ( x ) which minimizes the expected cumulative cost function v( x ) as v { ~ p ( i ) } ( x) min ( x, π ( x) ) E [ ( ) u v x' ] = l + x' x (4) u where x' denotes the next state for a iven action u. he minimum cumulative cost for startin at state x and actin optimally thereafter enables reedy computation of optimal actions. Eq. (4) is fundamental to optimal control theory and is called the Bellman fundamental equation. he Bellman equation can be simplified by assumin the immediate cost function is iven as 43 JAIIO - ASAI ISSN: Páina 2

3 5th Arentine Symposium on Articial Intellience, ASAI 204 u 0 ( x, u) = hq ( x) + KL( p ( x' x) p ( x' x) ) l (5) he state cost q( x ) is an arbitrary function encodin how (un)desirable different states are, and KL is the Kullback Leibler diverence that measures the distance from the optimally-controlled dynamics to the passive one. he distance can be understood as the price to pay for the optimal shift of the passive dynamics by action u. he KL 2 2 diverence between the above Gaussians can be proven to be h 2σ u which is the quadratic enery cost accumulated over interval h. Definin a desirability function as z ( ) ( ) = exp v ( ) the Bellman equation can be now expressed linearly as where [ z]( x) x x (6) ( x) = exp( ( x) ) [ ]( x) z hq G z (7) G is an interal operator iven by [ ]( ) = 0 ( ) ( ) G f x p x' x f x' dx' (8) he optimal control policy is computed analytically and expressed as ( ) σ ( ) ( ) 2 u B v x = x x x (9) Case study: Car-on-a-hill he car-on-a-hill continuous control problem illustrated in Fi. and details follows. he admissible rane of forces is not sufficient to drive up the car reedily from the initial state. he state vector is x = [ x, x2 ], where x and x 2 denote horizontal position and the tanential velocity of the car, respectively. he dynamics is iven by ( ( )) sn( )sin ( atan ( '( ))) dx = x cos atan s '( x ) dt 2 dx = x x s x dt β x dt + udt + dω = is the slope over the horizontal plane, = 9.8 is the ravitational constant and β = 0.5 is the dampin coefficient. he oal states for the drivin aent are all states such that x 2.5 < 0.2 and x 2 < 0.5. he cost model thus encodes the task of parkin the car at the horizontal position 2.5 in minimal time and with minimal control enery. his is a first-exit settin, since costs are accumulated from time 0 to infinity but accumulation stops when the system reaches a terminal state. Error tolerance is needed because the dynamics is stochastic. As the continuous problem is in the form iven in Eq. () it can be approximated where s '( x ) 2x exp( x 2 2) with a LSMDP. he approximation is constructed by choosin a set of states { x n } and adjustin the matrix P k, k + of transition probabilities from x k to x k+ iven by the (0) 43 JAIIO - ASAI ISSN: Páina 3

4 5th Arentine Symposium on Articial Intellience, ASAI 204 passive dynamics described in Eq. (2). Definin the vector z with elements z ( x n ) and the matrix Q with elements exp hq x on its main diaonal Eq. (7)) becomes and the matrix Q with elements ( ( n )) λ z = QPz () where λ is an eienvalue; Eq. () is solved by an iteration method in exponentiated form. he approximation uses a state space discretization: a 0-by-0 rid spannin x [ 3, +3], x2 [ 9, +9]. he passive dynamics is constructed by discretizin the time e axis (with time step h = 0.05) and definin probabilistic transitions amon dis- crete states so that the mean and variance of the continuous state dynamic are pre- served. he noise distribution is discretized at 9 points spannin ±3 standard devia- tions in the x 2 direction and usin a noise scale parameter σ=0.. his parameteriza- tion corresponds to the aent dynamics used as the specification, i.e.. the manner the aent is expected to behave when it is optimally controlled. Once the desirability function was optimized usin the estimated passive dynamics, the control policy was subsequently derived from the obtained desirability function. he results are shown in Fi., where b) corresponds to the state cost function q( x ) of the first exit cost formulation; c) is the optimal cost-to-o function obtained. Solid lines show stochastic trajectories resultin from the optimal behavior with dif- ferent scales of noise and initial states; d) is the obtained optimal control policy used as the specification for aent behavior monitorin. In all plots blue corre- spond to smaller values and red to larer values. Fi.. he car moves alon a curved road in the presence of ravity. 43 JAIIO - ASAI ISSN: Páina 4

5 5th Arentine Symposium on Articial Intellience, ASAI Behavior Monitorin 3. Learnin ransition Probabilities A Gaussian process (GP) is a collection of random variables, any finite number of which has a joint Gaussian distribution. For GP reression (GPR), random variables represent the value of the function f ( x ) for inputs x. GPR assumes f ( x ) is a zero mean stationary GP with covariance function k ( xi, x j ), encodin correlations between pairs of random variables 2 ( ) ( γ ) k x, x = exp x-x + λ δ (2) i j r i j r ij with γr 0 the kernel width parameter, λr 0 the noise variance and δ ij the Kronecker delta function, which is if i= j, and 0 otherwise. his prior for the kernel function constrains input samples that are nearby to have hihly correlated outputs. Short-term transition dynamics are modeled based on interactions between the aent and its environment [6]. Given a state vector x a separate GP model is trained for each state dimension x, in such a way the effect of uncertainty about its chane due to a control action is modeled statistically as x ~ GP( m, k) (3) k where m is the mean function and k is the covariance function. he trainin inputs to the Gaussian model GP are the states, whereas the tarets are the differences between the successor state and the state in which the action is applied. For an input x k, the predictive distribution p ( xk+ x k ) is Gaussian distributed. We can build the optimal transition probability p ( xk + xk ) as a GP model which describes the stochastic specification of the aent behavior. he superscript indicates the transition probability is shifted by the optimal control policy π ( x ). On the other hand, GP describes the observed aent behavior modeled as the transition probability p ( xk + x k ) which may deviate from optimal action selection. Noteworthy, the GP model that characterize the current system functionin requires to be updated on-line to accommodate the arrivin data stream. 3.2 win Gaussian Processes Aent behavior monitorin must quantify how observin new data affects the internal beliefs that the aent may have over a set of hypotheses or models M of the world. Since the aent acts over an uncertain environment, the monitorin approach should be probabilistic usin distributions to capture subjective expectations or beliefs over the current space. Aent s beliefs must be updated on-line, as data is acquired, transformin prior belief distributions P( M ) into posterior ones P( M D ) and computin the distance between them, which is best done usin the KL diverence 43 JAIIO - ASAI ISSN: Páina 5

6 5th Arentine Symposium on Articial Intellience, ASAI 204 ( ) ( ) KL( P ( M D) P ( M )) = P ( M D) lo P M D dm (4) M P M o provide a sensitive on-line estimation of the diverence from optimal behavior with small samples GP are used. GP can be described as a structured prediction method that employs GPs to estimate outputs, by minimizin the KL distance between a iven system implementation and its specification. Both processes are modeled as normal distributions over a finite set of L trainin examples [4]. hrouh GP, we obtain a powerful monitorin tool by comparin the two GPs. Suppose a particular set containin a sequence of the last W state differences X = { x } k k k W for the stochastic process that results from applyin an arbitrary control policy π. hen, the joint distribution of observed state differences can be modeled usin a zero mean multivariate Gaussian distribution as ( X ) ~ N0, R ij i r ri r K is iven by the kernel matrix ( ) whose covariance R ij = k xi, x j, the kernel vector k ( r i = xi, xk ) and the kernel value r = k ( xk, xk ). For the iven Gaussian dynamics ( N 0, K ) of a sampled sequence of state transitions, the offset or distance to a specified distribution ( N 0, K ) is key to calculate a robust measure of the twinned Kullback-Leibler diverence ( KL ). Considerin the KL diverence as a measure of the alinment between two kernels, the diverence between Gaussian (stochastic) processes is defined as KL { } N ( ) ( ) (5) N N = lo K + r K K + lo K (6) he KL diverence is non-neative and zero if and only if the two multivariate Gaussian distributions have the same covariance. In Fi. 2, KL is used to compute the p performance of the controlled dynamics GP aainst the specified dynamics GP for the optimal aent behavior. Notice that instead of computin the KL distance Fi. 2. he data ordinates are x and GP are the GP distributions for the implemented and the specification. he horizontal black lines indicate fully-connected sets. 43 JAIIO - ASAI ISSN: Páina 6

7 5th Arentine Symposium on Articial Intellience, ASAI 204 pointwise (reen rectanles) for each new estimation xk, GP uses a sequence of observed state transitions { x } k k k W over a finite horizon (red rectanle),, which ives a more robust description of a deviant behavior. Case study (Cont d). It is clear the importance of describin any chane in the system dynamics usin a reduced and relevant data set for modelin the correspondin GPs. Hence, a data set containin the last L=50 state-transition pairs is used to train the GP in order to capis computed usin a movin window stratey over the last W=30 estimations of state transitions. he number of estimations used is a tradeoff between the speed of detection of any event or disturb- ance and the proper characterization of the deradation in the aent behavior. KL is ture any deviant behavior from the specification. KL measured for the implemented dynamics GP to its specification GP in Fi. 3. his measure emphasizes the fact that similar states should produce similar estimates of a) b) Fi. 3. a) Sample trajectories are enerated usin different values of the noise scale parameter σ. he performance deradation in the aent behavior is clearly revealed below by KL. b) Simu- lation outcome for random actions between samples 00 to JAIIO - ASAI ISSN: Páina 7

8 5th Arentine Symposium on Articial Intellience, ASAI 204 both covariates and responses. In a), the noise scale parameter σ is increased from 0. to 0.5 and.0, to simulate different derees of variability. It is quite clear that an increase of parameter σ certainly chane the aent behavior. In b), a failure in the car actuator is simulated. Durin samples the car dynamics is overned by a random policy with u [, +]. Discussion his paper presents a probabilistic approach for on-line to monitorin of an aent behavior under uncertainty based on Bayesian surprise and optimal action selection. he desired behavior is modeled by a prior Gaussian distribution for state transitions, in order to assess if an observed control policy respects its specification. An analytical specification of the desired optimal behavior is obtained usin a class of Markov decision processes which are linearly solvable. win Gaussian processes are used to compare on-line observe state transitions due to the actual aent behavior with the stochastic specification. References. Broi, A., Medici, P., Zani, P., Coati, A., Panciroli, M.: Autonomous vehicles control in the VisLab intercontinental autonomous challene. Annu. Rev. Control. 36, 6 7 (202). 2. Fernández-Caballero, A., Castillo, J.C., Rodríuez-Sánchez, J.M.: Human activity monitorin by local and lobal finite state machines. Expert Syst. Appl. 39, (202). 3. Hasanbelliu, E., Kampa, K., Principe, J.C., Cobb, J..: Online learnin usin a Bayesian surprise metric. Neural Networks (IJCNN), he 202 International Joint Conference on. pp. 8. IEEE (202). 4. Bo, L., Sminchisescu, C.: win aussian processes for structured prediction. Int. J. Comput. Vis. 87, (200). 5. odorov, E.: Efficient computation of optimal actions. Proc. Natl. Acad. Sci. U. S. A. 06, (2009). 6. Deisenroth, M.P., Rasmussen, C.E., Peters, J.: Gaussian process dynamic prorammin. Neurocomputin. 72, (2009). 43 JAIIO - ASAI ISSN: Páina 8