An Online Learning-based Framework for Tracking

Transcription

1 An Online Learning-based Framework for Tracking Kamalika Chaudhuri Compuer Science and Engineering Universiy of California, San Diego La Jolla, CA 9293 Yoav Freund Compuer Science and Engineering Universiy of California, San Diego La Jolla, CA 9293 Daniel Hsu Compuer Science and Engineering Universiy of California, San Diego La Jolla, CA 9293 Absrac We sudy he racking problem, namely, esimaing he hidden sae of an objec over ime, from unreliable and noisy measuremens. The sandard framework for he racking problem is he generaive framework, which is he basis of soluions such as he ian algorihm and is approximaion, he paricle filers. However, hese soluions can be very sensiive o model mismaches. In his paper, moivaed by online learning, we inroduce a new framework for racking. We provide an efficien racking algorihm for his framework. We provide experimenal resuls comparing our algorihm o he ian algorihm on simulaed daa. Our experimens show ha when here are sligh model mismaches, our algorihm ouperforms he ian algorihm. 1 INTRODUCTION We sudy he racking problem, which has numerous applicaions in AI, conrol and finance. In racking, we are given noisy measuremens over ime, and he problem is o esimae he hidden sae of an objec. The challenge is o do his reliably, by combining measuremens from muliple ime seps and prior knowledge abou he sae dynamics, and he goal of racking is o produce esimaes ha are as close o he rue saes as possible. The mos popular soluions o he racking problem are he Kalman filer (Kalman, 196), he paricle filer (Douce e al., 21), and heir numerous exensions and variaions (e.g. (Isard & Blake, 1998; van der Merwe e al., 2)), which are based on a generaive framework for he racking problem. Suppose we wan o rack he sae x of an objec a ime, given only measuremen vecors M(, ) for imes. In he generaive approach, we hink of he sae X() and measuremens M(, ) as random variables. We represen our knowledge regarding he dynamics of he saes using he ransiion process Pr(X() X( 1)) and our knowledge regarding he (noisy) relaionship beween he saes and he observaions by he measuremen process Pr(M(, ) X()). Then, given only he observaions, he goal of racking is o esimae he hidden sae sequence (x 1, x 2,...). This is done by calculaing he likelihood of each sae sequence and hen using as he esimae eiher he sequence wih he highes poserior probabiliy (maximum a poseriori) or he expeced value of he sae wih respec o he poserior disribuion (he ian algorihm). In pracice, one uses paricle filers, which are an approximaion o he ian algorihm. The problem wih he generaive framework is ha in pracice, i is very difficul o precisely deermine he disribuions of he measuremens. The ian algorihm can be sensiive o model mismaches, so using a model which is differen from ha generaing he measuremens can lead o a large divergence beween he esimaed saes and he rue saes. To address his, we inroduce an online learning-based framework for racking. In our framework, we are given a se of sae sequences or pahs in he sae space; bu insead of assuming ha he observaions are generaed by a measuremen model from a pah in his se, we hink of each pah as a mechanism for explaining he observaions. We emphasize ha his is done regardless of how he observaions are generaed. Suppose a pah (x 1, x 2,...) is proposed as an explanaion of he observaions (M(, 1), M(, 2),...). We measure he qualiy of his pah using a predefined loss funcion, which depends only on he measuremens (and no on he hidden rue sae). The racking algorihm selecs is own pah by aking a weighed average of he bes pahs according he pas observaions. The heoreical guaranee we provide is ha he loss of he pah seleced by he algorihm in

2 his online way by he racking algorihm is close o ha of he pah wih he minimum loss; here, he loss is measured according o he loss funcion supplied o he algorihm. Such guaranees are analogous o compeiive analysis used in online learning (Cesa-Bianchi & Lugosi, 26; Freund & Schapire, 1997; Lilesone & Warmuh, 1994), and i is imporan o noe ha such guaranees hold uniformly for any sequence of observaions, regardless of any probabilisic assumpions. Our nex conribuion is o provide an online learningbased algorihm for racking in his framework. Our algorihm is based on NormalHedge (Chaudhuri e al., 29), which is a general online learning algorihm. NormalHedge can be insaniaed wih any loss funcion. When supplied wih a bounded loss funcion, i is guaraneed o produce a pah wih loss close o ha of he pah wih he minimum loss, from a se of candidae pahs. As i is compuaionally inefficien o direcly apply NormalHedge o racking, we derive a Sequenial Mone Carlo approximaion o Normal- Hedge, and we show ha his approximaion is efficien. To demonsrae he robusness of our racking algorihm, we perform simulaions on a simple onedimensional racking problem. We evaluae racking performance by measuring he average disance beween he saes esimaed by he algorihms, and he rue hidden saes. We insaniae our algorihm wih a simple clipped loss funcion. Our simulaions show ha our algorihm consisenly ouperforms he ian algorihm, under high measuremen noise, and a wide range of levels of model mismach. We finally noe ha ian algorihm can also be inerpreed in an online learning-based framework. In paricular, if he loss of a pah is he negaive log-likelihood (he log-loss) under some measuremen model, hen, he ian algorihm can be shown o produce a pah wih log-loss close o ha of he pah wih he minimum log-loss. One may be emped o hink ha our racking soluion follows he same approach; however, he poin of our paper is ha one can use loss funcions ha are differen from log-loss, and in paricular, we show a scenario in which using oher loss funcions produces beer racking performance han he ian algorihm (or is approximaions). The res of he paper is organized as follows. In Secion 2, we explain in deail our explanaory model for racking. In Secion 3, we presen NormalHedge, on which our racking algorihm is based. In Secion 4, we provide our racking algorihm. Secion 5 presens he experimenal comparison of our algorihm wih he ian algorihm on simulaed daa. Finally, we discuss relaed work in Secion 6. 2 AN ONLINE-LEARNING FRAMEWORK FOR TRACKING In his secion, we describe in more deail he seup of he racking problem, and our online learning-based framework for racking. In racking, a each ime, we are given as inpu, measuremens (or observaions) M(, ), and he goal is o esimae he hidden sae of an objec using hese measuremens, and our knowledge abou he sae dynamics. In our online learning framework for racking, we are given a se P of pahs (sequences) over he sae space X R n. A each ime, we assign o each pah in P a loss funcion l. The loss funcion has wo pars: a dynamics loss l d and an observaion loss l o. The dynamics loss l d capures our knowledge abou he sae dynamics. For simpliciy, we use a dynamics loss l d ha can be wrien as l d (p) = l d (x, x 1 ) for a pah p = (x 1, x 2,...). In oher words, he dynamics loss a ime depends only on he saes a ime and 1. A common way o express our knowledge abou he dynamics is in erms of a dynamics funcion F, defined so ha pahs wih x F (x 1 ) will have small dynamics loss. For example, consider an objec moving wih a consan velociy. Here, if he sae x = (p, v), where p is he posiion and v is he velociy, hen we would be ineresed in pahs in which x x 1 + (v, ). In hese cases, he dynamics loss l d (x, x 1 ) is ypically a growing funcion of he disance from x o F (x 1 ). The second componen of he loss funcion is an observaion loss l o. Given a pah p = (x 1, x 2,...), and measuremens M = (M(, 1), M(, 2),...), he observaion loss funcion l o (p, M) quanifies how well he pah p explains he measuremens. Again, for simpliciy, we resric ourselves o loss funcions l o ha can be wrien as: l o (p, M) = l o (x, M(, )). In oher words, he observaion loss of a pah a ime depends only on is sae a ime and he measuremens a ime. The oal loss of a pah p is he sum of is dynamics and observaion losses. We noe ha he loss of a pah depends only on ha paricular pah and he measuremens, and no on he rue hidden sae. As a resul, he loss of a pah can always be compued by an algorihm a any given ime. The algorihmic framework we consider in his model is analogous o, and moivaed by he decisionheoreic framework for online learning (Freund &

3 Schapire, 1997; Cesa-Bianchi & Lugosi, 26). A ime, our algorihm assigns a weigh w p o each pah p in P. The esimaed sae a ime is he weighed mean of he saes, where he weigh of a sae is he oal weigh of all pahs in his sae. The loss of he algorihm a ime is he weighed loss of all pahs in P. The heoreical guaranee we look for is ha he loss of he algorihm is close o he loss of he bes pah in P in hindsigh (or, close o he loss of he op ɛ-quanile pah in P in hindsigh). Thus, if P has a small fracion of pahs wih low loss, and if he loss funcions successfully capure he racking performance, hen, he sequence of saes esimaed by he algorihm will have good racking performance. Algorihm 1 NormalHedge iniialize R i, =, w i,1 = 1/N i for = 1, 2,... do Each acion i incurs loss l i,. Learner incurs loss l A, = N i=1 w i,l i,. Updae cumulaive regres: R i, = R i, 1 +(l A, l i, ) i. Find c > saisfying ( ) 1 N N i=1 exp ([Ri,] +) 2 2c = e. Updae disribuion: ( ) w i,+1 [Ri,]+ ([Ri,] c exp +) 2 2c i. end for 3 NORMALHEDGE In his secion, we describe he NormalHedge algorihm, which forms he basis of our racking algorihm. To presen NormalHedge in full generaliy, we firs need o describe he decision-heoreic framework for online learning. The problem of decision-heoreic online learning is as follows. A each round, a learner has access o a se of N acions; for our purposes, an acion is any mehod ha provides a predicion in each round. The learner mainains a disribuion w i, over he acions a ime. A each ime period, each acion i suffers a loss l i, which lies in a bounded range, and he loss of he learner is i w i,l i,. We noe ha his framework is very general no assumpion is made abou he naure of he acions and he disribuion of he losses. The goal of he learner is o mainain a disribuion over he acions, such ha is cumulaive loss over ime is low, compared o he cumulaive loss of he acion wih he lowes cumulaive loss. In some cases, paricularly, when he number of acions is very large, we are ineresed in achieving a low cumulaive loss compared o he op ɛ-quanile of acions. Here, for any ɛ, he op ɛ-quanile of acions are he ɛ fracion of acions which have he lowes cumulaive loss. Saring wih he seminal work of Lilesone and Warmuh (1994), he problem of decision-heoreic online learning has been well-sudied in he lieraure (Freund & Schapire, 1997; Cesa-Bianchi e al., 1993; Cesa- Bianchi & Lugosi, 26). The mos common algorihm for his problem is Hedge or Exponenial Weighs (Freund & Schapire, 1997), which assigns o each acion a weigh exponenially small in is oal loss. In his paper however, we consider a differen algorihm NormalHedge for his problem (Chaudhuri e al., 29), and i is his algorihm ha forms he basis of our racking algorihm. While he ian averaging algorihm can be shown o be a varian of Hedge when he loss funcion is he log-loss, such is no he case for NormalHedge, which uses a very differen weighing funcion. In he NormalHedge algorihm, for each acion i and ime, we use w i, o denoe he NormalHedge weigh assigned o acion i a ime. A any ime, we define he regre R i, of our algorihm o an acion i as he difference beween he cumulaive loss of our algorihm and he cumulaive loss of his acion. Also, for any real number x, we use he noaion [x] + o denoe max(, x). The NormalHedge algorihm is presened below. The performance guaranees for he Normal- Hedge algorihm can be saed as follows. Theorem 1 (Chaudhuri e al., 29). If NormalHedge has access o N acions, hen for all loss sequences, for all, for all < ɛ 1, he regre of he algorihm o he op ɛ-quanile of he acions is O( ln(1/ɛ)+ln 2 N). A significan advanage of using NormalHedge over oher online learning algorihms is ha i has no parameers o une, ye achieves performance comparable o he bes performance of previous online learning algorihms wih opimally uned parameers. For more discussion, see Secion 3 of (Chaudhuri e al., 29). Anoher advanage is ha he acions which have oal loss greaer han he oal loss of he algorihm, are assigned zero weigh. Since he algorihm performs almos as well as he bes acion, in a scenario where a few acions are significanly beer han he res, he algorihm will assign zero weigh o mos acions. In oher words, he suppor of he NormalHedge weighs may be a very small se, and his propery makes i easier o design an approximaion o i.

4 4 TRACKING USING NORMALHEDGE To apply NormalHedge direcly o racking, we se each acion o be a pah in he sae space, and he loss of each acion a ime o be he loss of he corresponding pah a ime. To make NormalHedge more robus in a pracical seing, we make a small change o he algorihm: insead of using cumulaive loss, we use a discouned cumulaive loss. For a discoun parameer < α < 1, he discouned cumulaive loss of an acion i a ime T is T =1 (1 α)t l i,. Such discouned losses have been used in reinforcemen learning (Kaelbling e al., 1996) as well as online learning (Hazan & Seshadhri, 29); inuiively, i makes he racking algorihm more flexible, and allows i o more easily recover from pas misakes. However, a direc applicaion of NormalHedge is prohibiively expensive in erms of compuaion. If we consider pahs over a discreizaion of he sae space of cardinaliy S, hen, a ime T, here are S T acions. One can ake advanage of he srucure of he loss funcion o formulae he weigh updaes as a dynamic program; however, his is sill expensive as each updae akes S 2 ime. Therefore, in he sequel, we show how o derive a Sequenial Mone-Carlo based approximaion o NormalHedge, and we use his approximaion in our experimens. The key observaion behind our approximaion is ha he weighs on acions generaed by he NormalHedge algorihm induce a disribuion over he saes a each ime. We herefore use a random sample of saes in each round o approximae his disribuion. Thus, jus as paricle filers approximae he poserior densiy on he saes induced by he ian algorihm, our racking algorihm approximaes he disribuion induced on he saes by NormalHedge for racking. The main difference beween NormalHedge and our approximaion is ha while NormalHedge always mainains he weighs for all he acions, we delee an acion from our acion lis when is weigh falls o. We hen replace his acion by our re-sampling procedure, which chooses anoher acion which is currenly in a region of he sae space where he acions have low losses. Thus, we do no spend resources mainaining and updaing weighs for acions which do no perform well. Anoher difference beween NormalHedge and our racking algorihm is ha in our approximaion, we do no explicily impose a dynamics loss on he acions. Insead, we use a re-sampling procedure ha only considers acions wih low dynamics loss. This also avoids spending resources on acions which have high dynamics loss anyway. We noe ha because of hese changes, our racking algorihm does no have Algorihm 2 Tracking algorihm inpu N (number of acions), α (discoun facor), Σ (re-sampling parameer), F (dynamics funcion) A := {x 1,1,..., x N,1 } wih x i,1 randomly drawn from X ; R i, := ; w i, := 1/N i for = 1, 2,... do Obain losses l i, = l o (x i, ) for each acion i and updae regres: R i, := (1 α)r i, 1 + (l A, l i, ) where l A, = N i=1 w i, 1l i,. Delee poor acions: le X = {i : R i, }, se A := A \ X. Re-sample acions: A := A Resample(X, Σ, ). Compue ( weigh of ) each acion i: w i, [R i,] + ([Ri,] c exp +) 2 2c where c is he soluion o he equaion ( ) N i=1 exp ([Ri,] +) 2 = e. 1 N Esimae: x A, := N i=1 w i,x i,. Updae saes: x i,+1 := F (x i, ) i. end for 2c he wors-case heoreical guaranees in Theorem 1; however, we sill expec i o have good performance when racking a slowly moving objec. Our racking algorihm is specified in Algorihm 2. Each acion i in our algorihm is a pah (x i,1, x i,2,...) in he sae space X R n. However, we do no mainain his enire pah explicily for each acion; raher, he sae updae sep of he algorihm compues x i,+1 from x i, using he dynamics funcion F, so we only need o mainain he curren sae of each acion. Recall, applying he dynamics funcion F should ensure ha he pah incurs no or lile dynamics loss (see Secion 2). We sar wih a se of acions A iniially posiioned a saes uniformly disribued over he X, and a uniform weighing over hese acions. In each round, like NormalHedge, each acion incurs a loss deermined by is curren sae, and he racker incurs he expeced loss deermined by he curren weighing over acions. Using hese losses, we updae he cumulaive (discouned) regres o each acion. However, unlike NormalHedge, we hen delee all acions wih zero or negaive regre, and replace hem using a re-sampling procedure. This procedure replaces poorly performing acions wih acions currenly a high densiy regions of X, hereby providing a beer approximaion o he NormalHedge weighs. The re-sampling procedure is explained in deail in Algorihm 3. The main idea is o sample from he regions of he sae space wih high weigh. This is done

5 Algorihm 3 Re-sampling algorihm inpu X (acions o be re-sampled), Σ (re-sampling parameer), (curren ime) for j X do Se X := {i : R i, > }. If X = : se pi = 1/N i. Else: se p i w i, 1 i X and p i = i / X. Draw i Mulinomial( p 1,..., p N ). Draw x j, N (x i,, Σ ), and se R j, := (1 α)r i, 1 + (l A, l o (x j, )). end for 2W Blue: H(x, z ), Red: M(x, ). by sampling an acion proporional o is weigh in he previous round. We hen choose a sae randomly (roughly) from an ellipsoid {x : (x x ) Σ 1 (x x ) 1} around he curren sae x of he seleced acion; he new acion inheris he hisory of he seleced acion, bu has a curren sae which is differen from (bu close o) he seleced acion. This laer sep makes he new sae disribuion smooher han he one in he previous round, which may be suppored on jus a few saes if only a few acions have low losses. We noe ha Σ can be se so ha he resampling procedure only samples acions wih low dynamics loss (and he sae updae sep of he algorihm ensures ha he remaining acions in he se A do no incur any dynamics loss); hus, our algorihm does no explicily compue a dynamics loss for each acion. 5 SIMULATIONS For our simulaions, we consider he ask of racking an objec in a simple, one-dimensional sae space. To evaluae our algorihm, we measure he disance beween he esimaed saes, and he rue saes of he objec. Our experimenal seup is inspired by he applicaion of racking faces in videos, using a sandard face deecor (Viola & Jones, 21). In his case, he sae is he locaion of a face, and each measuremen corresponds o a score oupu by he face deecor for a region in he curren video frame. The goal is o predic he locaion of he face across several video frames, using hese scores produced by he deecor. The deecor ypically reurns high scores for several regions around he rue locaion of a face, bu i may also erroneously produce high scores elsewhere. And hough in some cases he deecion score may have a probabilisic inerpreaion, i is ofen difficul o accuraely characerize he disribuion of he noise. The precise seup of our simulaions is as follows. The objec o be racked remains saionary or moves wih velociy a mos 1 in he inerval [, 5]. A ime 2σ o Blue: ρ =, Red: ρ =.2. Figure 1: Plos of he measuremens (as a funcion of x) for ρ = and σ o = 1 (op) and he densiy of he noise n (x) (boom)., he rue sae is he posiion z ; he measuremens correspond o a 11-dimensional vecor M() = [M(, ), M( 499, ),..., M(499, ), M(5, )] for locaions in a grid G = {, 499,..., 499, 5}, generaed by an addiive noise process M(x, ) = H(x, z ) + n (x). Here, H(x, z ) is he square pulse funcion of widh 2W around he rue sae z : H(x, x ) = 1 if x z W and oherwise (see Figure 1, op). The addiive noise n (x) is randomly generaed independenly for each and each x G, using he mixure disribuion (1 ρ) N (, σ 2 o) + ρ N (, (1σ o ) 2 ) (see Figure 1, boom). The parameer σ o represens how noisy he measuremens are relaive o he signal, and he parameer ρ represens he fracion of ouliers. In our experimens, we fix W = 5 and vary σ o and ρ. The oal number of ime seps we rack for is T = 2. In he generaive framework, he dynamics of he objec is represened by he ransiion model x +1 N (x, σd 2 ), and he observaions are represened by he measuremen process M(x, ) N (H(x, x ), σo). 2 Thus, when ρ =, he observaions are generaed according o he measuremen process supplied o he

6 generaive framework; for ρ >, a ρ fracion of he observaions are ouliers. In he online learning-based framework, he expeced sae dynamics funcion F is he ideniy funcion, and he observaion loss of a pah p = (x 1, x 2,...) a ime is given by l o (x, M(, )) = x [x W,x +W ] G q(m(x, )) where q(y) = min(1 + σ o, max(y, σ o )) clips he measuremens o he range [ σ o, 1 + σ o ]. Tha is, he observaion loss for x wih respec o M(, ) is he negaive sum of hresholded measuremen values q(m(x, )) for x in an inerval of widh 2W around x. Given only he observaion vecors M, we use hree differen mehods o esimae he rue underlying sae sequence (z 1, z 2,...). The firs is he ian algorihm, which recursively applies rule o updae a poserior disribuion using he ransiion and observaion model. The poserior disribuion is mainained a each locaion in he discreizaion G. For he ian algorihm, we se σ o o he acual value of σ o used o generae he observaions, and we se σ d = 2. The value of σ d was obained by uning on measuremen vecors generaed wih he same rue sae sequence, bu wih independenly generaed noise values. The prior disribuion over saes assigns probabiliy one o he rue value of z 1 (which is in our seup) and zero elsewhere. The second algorihm is our algorihm () described in Secion 4. For our algorihm, we use he parameers Σ = 4 and α =.2. These parameers were also obained by uning over a range of values for Σ and α. We also compare our algorihm wih he paricle filer (PF), which uses he same parameers as wih he ian algorihm, and predics using he expeced sae under he (approximae) poserior disribuion. For our algorihm, we use N = 1 acions, and for he paricle filer, we use N = 1 paricles. For our experimens, we use an implemenaion of he paricle filer due o (de Freias, 2). Figures 2 and 3 show he rue sae and he saes prediced by our algorihm (Blue) and he ian algorihm (Red) for wo differen values of σ o for 5 independen simulaions. Table 1 summarizes he performance of hese algorihms for differen values of he parameer ρ, for wo differen values of he noise parameer σ o. We repor he average and sandard deviaion of he RMSE (roo-mean-squared-error) beween he rue sae and he prediced sae. The RMSE is compued over he T = 2 sae predicions for a single simulaion, and hese RMSE values are averaged over 1 independen simulaions. Our experimens show ha he ian algorihm performs well when ρ =, ha is, i is supplied wih he correc noise model; however, is performance degrades rapidly as ρ increases, and becomes very poor even a ρ =.2. On he oher hand, he performance of our algorihm does no suffer appreciably when ρ increases. The degradaion of performance of he ian algorihm is even more pronounced, when he noise is high wih respec o he signal (σ o = 8). The paricle filer suffers a even higher degradaion in performance, and has poor performance even when ρ =.1 (ha is, when 99% of he observaions are generaed from he correc likelihood disribuion supplied o he paricle filer). Our resuls indicae ha he ian algorihm is very sensiive o model mismaches. On he oher hand, our algorihm, when equipped wih a clipped-loss funcion, is robus o model mismaches. In paricular, our algorihm provides a RMSE value of 19.6 even under high noise (σ o = 8), when ρ is as high as.4. We noe ha he degradaion in he performance of he ian algorihm is solely due o a model mismach: when he same experimens are repeaed, wih he ian algorihm being supplied wih he correc likelihood model, i performs a leas as well as, or beer han our algorihm. Moreover, if he ian algorihm is supplied wih a likelihood model wih he correc disribuion (a mixure of wo Gaussians), he correc fracion of ouliers ρ, bu a differen oulier variance (e.g., 2σ o insead of 1σ o ), he performance of he ian algorihm improves significanly over having he incorrec disribuion (alhough i sill performs worse han our algorihm). We performed some addiional experimens wih our algorihm o undersand he effec of varying he parameers Σ and α; he deails are omied due o lack of space. The resuls indicae ha he performance of our algorihm depends on he value of Σ ; if Σ is se oo high, here are no enough acions drawn from he regions of he sae-space where he objec o be racked is, which makes i difficul o rack wih a fine granulariy. If Σ is se oo low, hen he acions are very concenraed a where he curren acions were, and do no explore enough of he sae space. On he oher hand, for a low level of ouliers (abou 2 percen or less), our algorihm appears o have more or less he same performance over a wide range of α values. 6 RELATED WORK The generaive approach o racking has roos in conrol and esimaion heory, saring wih he seminal work of Kalman (Kalman, 196). The mos popular generaive mehod used in racking is he paricle fil-

7 Table 1: Experimenal Resuls. Roo-mean-squarederrors of he prediced posiions over T = 2 ime seps for our algorihm (), he ian algorihm, and he paricle filer (PF). The repored values are he averages and sandard deviaions over 1 simulaions. Low Noise (σ o = 1) ρ PF ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± 1.74 High Noise (σ o = 8) ρ PF ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± 1.74 er (Douce e al., 21), and is numerous varians. The lieraure here is vas, and here have been many exciing developmens in recen years (e.g. (van der Merwe e al., 2; Klaas e al., 25)); we refer he reader o (Douce & Johansen, 28) for a deailed survey of he resuls. The sub-opimaliy of he ian algorihm under model mismach has been invesigaed in oher conexs such as classificaion (Domingos, 2; Grünwald & Langford, 27). The view of he ian algorihm as an online learning algorihm for log-loss is well-known in various communiies, including informaion heory / MDL (Merhav & Feder, 1993; Grünwald, 27) and compuaional learning heory (Freund, 1996; Kakade & Ng, 24). In our work, we look beyond he ian algorihm and log-loss o consider oher loss funcions and algorihms ha are more appropriae for our ask. There has also been some work on racking in he online learning lieraure (see, for example, (Herbser & Warmuh, 1998; Koolen & de Rooij, 28)). Our mehod uses a differen class of acions han he racking framework in hese works. The algorihm of (Herbser & Warmuh, 1998) considers, as heir class of acions, pahs ha swich saes a mos a fixed number Posiion Posiion Posiion σ o = 1, ρ = σ o = 1, ρ =.1 σ o = 1, ρ =.2 Figure 2: Prediced pahs in five simulaions (low noise cases). The blue lines correspond o our algorihm, he red lines correspond o he ian algorihm, and he dashed black line represens he rue saes. of imes. Moreover, heir algorihm reas all swiches beween saes equally, and herefore fails o ake advanage of prior knowledge abou he sae dynamics. In conras, we ake ino accoun such prior knowledge using a dynamics loss so ha sequences approximaely following he expeced dynamics end up wih lower loss han hose ha do no, and as a resul, we can work wih more sophisicaed sae ransiion dynamics mehods. 7 CONCLUSIONS In his paper, we inroduce a new framework for racking based on online learning. We propose a new algorihm for racking in his framework ha deviaes significanly from he ian approach. Experimenal resuls show ha our algorihm significanly ouperforms he ian algorihm, when he observaions are generaed by a disribuion deviaing slighly from he model supplied o he ian algorihm. Our work reveals an ineresing connecion beween decision heoreic online learning and ian filering.

8 Posiion Posiion Posiion σ o = 8, ρ = σ o = 8, ρ =.1 σ o = 8, ρ =.2 Figure 3: Prediced pahs in five simulaions (high noise cases). 8 ACKNOWLEDGEMENTS The auhors would like o hank Nando de Freias for helpful commens. Par of his work was done when KC was par of he ITA Cener a UCSD. Research suppor for his work was provided by NSF, under grans IIS-71354, and IIS KC would like o hank CALIT-2 for suppor. Some of he experimenal resuls were made possible by suppor from he UCSD FWGrid Projec, NSF Research Infrasrucure Gran Number EIA References Cesa-Bianchi, N., Freund, Y., Helmbold, D. P., Haussler, D., Schapire, R. E., & Warmuh, M. K. (1993). How o use exper advice. STOC (pp ). Cesa-Bianchi, N., & Lugosi, G. (26). Predicion, learning and games. Cambridge Universiy Press. Chaudhuri, K., Freund, Y., & Hsu, D. (29). A parameer-free hedging algorihm. Neural Inf. Proc. Sysems. de Freias, N. (2). Malab codes for paricle filering. nando/sofware/upf demos.ar.gz. Douce, A., de Freias, N., & Gordon, N. J. (21). Sequenial mone carlo mehods in pracice. Springer-Verlag. Douce, A., & Johansen, A. M. (28). A uorial on paricle filering and smoohing: Fifeen years laer. In Handbook of nonlinear filering. Oxford Universiy Press. Freund, Y. (1996). Predicing a binary sequence almos as well as he opimal biased coin. COLT (pp ). Freund, Y., & Schapire, R. E. (1997). A decision-heoreic generalizaion of on-line learning and an applicaion o boosing. Journal of Compuer and Sysem Sciences, 55, Grünwald, P., & Langford, J. (27). Subopimal behavior of bayes and mdl in classificaion under misspecificaion. Machine Learning, 66, Grünwald, P. D. (27). The minimum descripion lengh principle. MIT Press. Hazan, E., & Seshadhri, C. (29). Efficien learning algorihms for changing environmens. ICML (p. 5). Herbser, M., & Warmuh, M. (1998). Tracking he bes exper. Machine Learning, 32, Isard, M., & Blake, A. (1998). Condensaion condiional densiy propagaion for visual racking. Inernaional Journal on Compuer Vision, 28, Kaelbling, L. P., Liman, M. L., & Moore, A. P. (1996). Reinforcemen learning: A survey. J. Arif. Inell. Res. (JAIR), 4, Kakade, S. M., & Ng, A. Y. (24). bayesian algorihms. NIPS. Online bounds for Kalman, R. E. (196). A new approach o linear filering and predicion problems. Transacions of he ASME Journal of Basic Engineering, 82, Klaas, M., de Freias, N., & Douce, A. (25). Toward pracical n 2 mone carlo: The marginal paricle filer. UAI. Koolen, W., & de Rooij, S. (28). Combining exper advice efficienly. COLT. Lilesone, N., & Warmuh, M. (1994). The weighed majoriy algorihm. Informaion and Compuaion, 18, Merhav, N., & Feder, M. (1993). Universal predicion. IEEE Transacions on Informaion Theory, 39, van der Merwe, R., Douce, A., de Freias, N., & Wan, E. (2). The unscened paricle filer. Advances in Neural Informaion Processing Sysems. Viola, P., & Jones, M. (21). Rapid objec deecion using a boosed cascade of simple feaures. Conference on Compuer Vision and Paern Recogniion. Domingos, P. (2). ian averaging of classifiers and he overfiing problem. ICML.