Particle Filtering for Multiple Object Tracking in Dynamic Fluorescence Microscopy Images: Application to Microtubule Growth Analysis

Transcription

1 Chaper Three Paricle Filering for Muliple Objec Tracking in Dynamic Fluorescence Microscopy Images: Applicaion o Microubule Growh Analysis I is remarkable ha a science which began wih he consideraion of games of chance should have become he mos imporan objec of human knowledge. Pierre-Simon, marquis de Laplace Théorie Analyique des Probabiliés (8) Absrac Quaniaive analysis of dynamic processes in living cells by means of fluorescence microscopy imaging requires racking of hundreds of brigh spos in noisy image sequences. Deerminisic approaches, which use objec deecion prior o racking, perform poorly in he case of noisy image daa. We propose an improved, compleely auomaic racker, buil wihin a Bayesian probabilisic framework. I beer explois spaioemporal informaion and prior knowledge han common approaches, yielding more robus racking also in cases of phoobleaching and objec ineracion. The racking mehod was evaluaed using simulaed bu realisic image sequences, for which ground ruh was available. The resuls of hese experimens show ha he mehod is more accurae and robus han popular racking mehods. In addiion, validaion experimens were conduced wih real fluorescence microscopy image daa acquired for microubule growh analysis. These demonsrae ha he mehod yields resuls ha are in good agreemen wih manual racking performed by exper cell biologiss. Our findings sugges ha he mehod may replace laborious manual procedures. Based upon: I. Smal, K. Draegesein, N. Galjar, W. Niessen, E. Meijering, Paricle Filering for Muliple Objec Tracking in Dynamic Fluorescence Microscopy Images: Applicaion o Microubule Growh Analysis, IEEE Transacions on Medical Imaging, vol. 7, no. 6, pp , 008.

2 54 3 Paricle Filering for Muliple Objec Tracking 3. Inroducion In he pas decade, advances in molecular cell biology have riggered he developmen of highly sophisicaed live cell fluorescence microscopy sysems capable of in vivo mulidimensional imaging of subcellular dynamic processes. Analysis of imelapse image daa has redefined he undersanding of many biological processes, which in he pas had been sudied using fixed maerial. Moion analysis of nanoscale objecs such as proeins or vesicles, or subcellular srucures such as microubules (Fig. 3.), commonly agged wih green fluorescen proein (GFP), requires racking of large and ime-varying numbers of spos in noisy image sequences [54, 95, 3, 35, 60, 6, 66]. Nowadays, high-hroughpu experimens generae vas amouns of dynamic image daa, which canno be analyzed manually wih sufficien speed, accuracy and reproducibiliy. Consequenly, many biologically relevan quesions are eiher lef unaddressed, or answered wih grea uncerainy. Hence, he developmen of auomaed racking mehods which replace edious manual procedures and eliminae he bias and variabiliy in human judgmens, is of grea imporance. Convenional approaches o racking in molecular cell biology ypically consis of wo subsequen seps. In he firs sep, objecs of ineres are deeced separaely in each image frame and heir posiions are esimaed based on, for insance, inensiy hresholding [9], muliscale analysis using he wavele ransform [5], or model fiing [6]. The second sep solves he correspondence problem beween ses of esimaed posiions. This is usually done in a frame-by-frame fashion, based on neares-neighbor or smooh-moion crieria [33,7]. Such approaches are applicable o image daa showing limied numbers of clearly disinguishable spos agains relaively uniform backgrounds, bu fail o yield reliable resuls in he case of poor imaging condiions [6,3]. Tracking mehods based on opic flow [3,67] are no suiable because he underlying assumpion of brighness preservaion over ime is no saisfied in fluorescence microscopy, due o phoobleaching. Mehods based on spaioemporal segmenaion by minimal cos pah searching have also been proposed [7, 8]. Unil presen, however, hese have been demonsraed o work well only for he racking of a single objec [8], or a very limied number of well-separaed objecs [7]. As has been observed [7], such mehods fail when eiher he number of objecs is larger han a few dozen, or when he objec rajecories cross each oher, which make hem unsuiable for our applicaions. As a consequence of he limied performance of exising approaches, racking is sill performed manually in many laboraories worldwide. I has been argued [95] ha in order o reach similar superior performance as exper human observers in emporal daa associaion, while a he same ime achieving a higher level of sensiiviy and accuracy, i is necessary o make beer use of emporal informaion and (applicaion specific) prior knowledge abou he morphodynamics of he objecs being sudied. The human visual sysem inegraes o a high degree spaial, emporal and prior informaion [3] o resolve ambiguous siuaions in esimaing moion flows in image sequences. Here we explore he power of a Bayesian generalizaion of he sandard Kalman filering approach in emulaing his process. I addresses he problem of esimaing he hidden sae of a dynamic sysem by consrucing he poserior probabiliy densiy funcion (pdf) of he sae based on all available informaion, including

3 3. Inroducion 55 prior knowledge and he (noisy) measuremens. Since his pdf embodies all available saisical informaion, i can be ermed a complee soluion o he esimaion problem. Bayesian filering is a concepual approach, which yields analyical soluions, in closed form, only in he case of linear sysems and Gaussian saisics. In he case of non-lineariy and non-gaussian saisics, numerical soluions can be obained by applying sequenial Mone Carlo (SMC) mehods [39], in paricular paricle filering (PF) [9]. In he filering process, racking is performed by using a predefined model of he expeced dynamics o predic he objec saes, and by using he (noisy) measuremens (possibly from differen ypes of sensors) o obain he poserior probabiliy of hese saes. In he case of muliple arge racking, he main ask is o perform efficien measuremen-o-arge associaion, on he basis of hresholded measuremens [5]. The classical daa associaion mehods in muliple arge racking can be divided ino wo main classes: unique-neighbor daa associaion mehods, as in he muliple hypohesis racker (MHT), which associae each measuremen wih one of he previously esablished racks, and all-neighbors daa associaion mehods, such as join probabilisic daa associaion (JPDA), which use all measuremens for updaing all rack esimaes [5]. The racking performance of hese mehods is known o be limied by he lineariy of he daa models. By conras, SMC mehods ha propagae he poserior pdf, or mehods ha propagae he firs-order saisical momen (he probabiliy hypohesis densiy) of he muliarge pdf [90], have been shown o be successful in solving he muliple arge racking and daa associaion problems when he daa models are nonlinear and non-gaussian [68,04]. Previous applicaions of PF-based moion esimaion include radar- and sonarbased racking [04, 75], mobile robo localizaion [39, 84], eleconferencing or video surveillance [5], and oher human moion applicaions [3, 0, 86]. In mos compuer vision applicaions, racking is limied o a few objecs only [70,89]. Mos biological applicaions, on he oher hand, require he racking of large and ime-varying numbers of objecs. Recenly, he use of PF in combinaion wih level-ses [83] and acive conours [39] has been repored for biological cell racking. These mehods ouperform deerminisic mehods, bu hey are sraighforward applicaions of he original algorihm [70] for single arge racking, and canno be direcly applied o he simulaneous racking of many inracellular objecs. A PF-like mehod for he racking of proeins has also been suggesed [83], bu i sill uses emplae maching for he linking sage, i requires manual iniializaion, and racks only a single objec. In his chaper, we exend our earlier conference repors [43,44], and develop a fully auomaed PF-based mehod for robus and accurae racking of muliple nanoscale objecs in wo-dimensional (D) and hree-dimensional (3D) dynamic fluorescence microscopy images. Is performance is demonsraed for a paricular biological applicaion of ineres: microubule growh analysis. The chaper is organized as follows. In Secion 3. we give more in-deph informaion on he biological applicaion considered in his chaper, providing furher biological moivaion for our work. In Secion 3.3 we presen he general racking framework and is exension o allow racking of muliple objecs. Nex, in Secion 3.4, we describe he necessary improvemens and adapaions o ailor he framework o he applicaion. These include a new dynamic model which allows dealing wih objec

4 56 3 Paricle Filering for Muliple Objec Tracking (a) (b) 5 µm 5 µm 5 µm 5 µm (c) 5 µm 5 µm (d) 5 µm 5 µm (e) 5 µm (f) 5 µm 5 µm 5 µm Figure 3.. Examples of microubules agged wih GFP-labeled plus end racking proeins (brigh spos), imaged using fluorescence confocal microscopy. The images are single frames from six D ime-lapse sudies, conduced wih differen experimenal and imaging condiions. The qualiy of such images ypically ranges from SNR 5 6 (a-c) o he exremely low SNR 3 (d-f).

5 3. Microubule Growh Analysis 57 ineracion and phoobleaching effecs. In addiion, we improve he robusness and reproducibiliy of he algorihm by inroducing a new imporance funcion for daadependen sampling (he choice of he imporance densiy is one of he mos criical issues in he design of a PF mehod). We also propose a new, compleely auomaic rack iniiaion procedure. In Secion 3.5, we presen experimenal resuls of applying our PF mehod o synheic image sequences, for which ground ruh was available, as well as o real fluorescence microscopy image daa of microubule growh. A concluding discussion of he main findings and heir poenial implicaions is given in Secion Microubule Growh Analysis Microubules (MTs) are polarized ubular filamens (diameer 5 nm) composed of α/β-ubulin heerodimers. In mos cell ypes, one end of a MT (he minus-end) is embedded in he so-called MT organizing cener (MTOC), while he oher end (he plus-end) is exposed o he cyoplasm. MT polymerizaion involves he addiion of α/β-ubulin subunis o he plus end. During MT disassembly, hese subunis are los. MTs frequenly swich beween growh and shrinkage, a feaure called dynamic insabiliy [37]. The conversion of growh o shrinkage is called caasrophe, while he swich from shrinkage o growh is called rescue. The dynamic behavior of MTs is described by MT growh and shrinkage raes, and caasrophe and rescue frequencies. MTs are fairly rigid srucures having nearly consan velociy while growing or shrinking [48]. MT dynamics is highly regulaed, as a properly organized MT nework is essenial for many cellular processes, including miosis, cell polariy, ranspor of vesicles, and he migraion and differeniaion of cells. For example, when cells ener miosis, he cdc kinase conrols MT dynamics such ha he seady-sae lengh of MTs decreases considerably. This is imporan for spindle formaion and posiioning [73]. I has been shown ha an increase in caasrophe frequency is largely responsible for his change in MT lengh [7]. Plus-end-racking proeins, or +TIPs [37], specifically bind o MT plus-ends and have been linked o MT-arge ineracions and MT dynamics [4, 67, 80]. Plus-endracking was firs described for overexpressed GFP-CLIP70 in culured mammalian cells [4]. In ime-lapse movies, ypical fluorescen come-like dashes were observed, which represened GFP-CLIP70 bound o he ends of growing MTs. As plusend racking is inimaely associaed wih MT growh, fluorescenly labeled +TIPs are now widely used o measure MT growh raes in living cells, and hey are also he objecs of ineres considered in he presen work. Wih fluorescen +TIPs, all growing MTs can be discerned. Alernaively, he advanage of using fluorescen ubulin is ha all parameers of MT dynamics can be measured. However, in regions where he MT nework is dense, he fluorescen MT nework obscures MT ends, making i very difficul o examine MT dynamics. Hence, in many sudies based on fluorescen ubulin [6, 7, 87], analysis is resriced o areas wihin he cells where he MT nework is sparse. Ideally, one should use boh mehods o acquire all possible knowledge regarding MT dynamics, and his will be addressed in fuure work. +TIPs are well posiioned o perform heir regulaory asks. A nework of iner-

6 58 3 Paricle Filering for Muliple Objec Tracking acing proeins, including +TIPs, may govern he changes in MT dynamics ha occur during he cell cycle [06]. Since +TIPs are so imporan and display such a fascinaing behavior, he mechanisms by which +TIPs recognize MT ends have araced much aenion. In one view, +TIPs binds o newly synhesized MT ends wih high affiniy and deach seconds laer from he MT laice, eiher in a regulaed manner or sochasically [4]. However, oher mechanisms have also been proposed [4, 7, 67]. Measuring he disribuion and displacemen of a fluorescen +TIP in ime may shed ligh on he mechanism of MT end binding. However, his is a labor inensive procedure if fluorescen racks have o be delineaed by hand, and very likely leads o user bias and loss of imporan informaion. By developing a reliable racking algorihm we obain informaion on he behavior of all growing MTs wihin a cell, which reveals he spaioemporal disribuion and regulaion of growing MTs. Imporanly, his informaion can be linked o he spaioemporal fluorescen disribuion of +TIPs. This is exremely imporan, since he localizaion of +TIPs repors on he dynamic sae of MTs and he cell. 3.3 Tracking Framework Before describing he deails of our racking approach, we firs recap he basic principles of nonlinear Bayesian racking in general (Secion 3.3.), and PF in paricular (Secion 3.3.), as well as he exension ha has been proposed in he lieraure o allow racking of muliple objecs wihin his framework (Secion 3.3.3) Nonlinear Bayesian Tracking The Bayesian racking approach deals wih he problem of inferring knowledge abou he unobserved sae of a dynamic sysem, which changes over ime, using a sequence of noisy measuremens. In a sae-space approach o dynamic sae esimaion, he sae vecor x of a sysem conains all relevan informaion required o describe he sysem under invesigaion. Bayesian esimaion in his case is used o recursively esimae a ime evolving poserior disribuion (or filering disribuion) p(x z : ), which describes he objec sae x given all observaions z : up o ime. The exac soluion o his problem can be consruced by specifying he Markovian probabilisic model of he sae evoluion, D(x x ), and he likelihood L(z x ), which relaes he noisy measuremens o any sae. The required probabiliy densiy funcion p(x z : ) may be obained, recursively, in wo sages: predicion and updae. I is assumed ha he iniial pdf, p(x 0 z 0 ) p(x 0 ), also known as he prior, is available (z :0 = z 0 being he se of no measuremens). The predicion sage involves using he sysem model and pdf p(x z : ) o obain he prior pdf of he sae a ime via he Chapman-Kolmogorov equaion: p(x z : ) = D(x x )p(x z : )dx. (3.) In he updae sage, when a measuremen z becomes available, Bayes rule is used o modify he prior densiy and obain he required poserior densiy of he curren

7 3.3 Tracking Framework 59 sae: p(x z : ) L(z x )p(x z : ). (3.) This recursive esimaion of he filering disribuion can be processed sequenially raher han as a bach, so ha i is no necessary o sore he complee daa se nor o reprocess exising daa if a new measuremen becomes available [9]. The filering disribuion embodies all available saisical informaion and an opimal esimae of he sae can heoreically be found wih respec o any sensible crierion Paricle Filering Mehods The opimal Bayesian soluion, defined by he recurrence relaions (3.) and (3.), is analyically racable in a resricive se of cases, including he Kalman filer, which provides an opimal soluion in case of linear dynamic sysems wih Gaussian noise, and grid based filers [9]. For mos pracical models of ineres, SMC mehods (also known as boosrap filering, paricle filering, and he condensaion algorihm [70]) are used as an efficien numerical approximaion. The basic idea here is o represen he required poserior densiy funcion p(x z : ) wih a se of N s random samples, or paricles, and associaed weighs {x (i),w (i) } Ns i=. Thus, he filering disribuion can be approximaed as N s p(x z : ) i= w (i) δ(x x (i) ), where δ( ) is he Dirac dela funcion and he weighs are normalized such ha Ns i= w(i) =. These samples and weighs are hen propagaed hrough ime o give an approximaion of he filering disribuion a subsequen ime seps. The weighs in his represenaion are chosen using a sequenial version of imporance sampling (SIS) [5]. I applies when auxiliary knowledge is available in he form of an imporance funcion q(x x,z ) describing which areas of he saespace conain mos informaion abou he poserior. The idea is hen o sample he paricles in hose areas of he sae-space where he imporance funcion is large and o avoid as much as possible generaing samples wih low weighs, since hey provide a negligible conribuion o he poserior. Thus, we would like o generae a se of new paricles from an appropriaely seleced proposal funcion, i.e., x (i) q(x x (i),z ), i = {,...,N s }. (3.3) A deailed formulaion of q( ) is given in Secion Wih he se of sae paricles obained from (3.3), he imporance weighs w (i) may be recursively updaed as follows: w (i) L(z x (i) )D(x (i) x (i) q(x (i) x (i),z ) ) w (i). (3.4) Generally, any imporance funcion can be chosen, subjec o some weak consrains [40, 6]. The only requiremens are he possibiliy o easily draw samples from i

8 60 3 Paricle Filering for Muliple Objec Tracking and evaluae he likelihood and dynamic models. For very large numbers of samples, his MC characerizaion becomes equivalen o he usual funcional descripion of he poserior pdf. By using his represenaion, saisical inferences, such as expecaion, maximum a poseriori (MAP), and minimum mean square error (MMSE) esimaors (he laer is used for he objec posiion esimaion in he approach proposed in his chaper), can easily be approximaed. For example, ˆx MMSE = E p [x ] = N s x p(x z : )dx i= x (i) w (i). (3.5) A common problem wih he SIS paricle filer is he degeneracy phenomenon, where afer a few ieraions, all bu a few paricles will have negligible weigh. The variance of he imporance weighs can only increase (sochasically) over ime [40]. The effec of he degeneracy can be reduced by a good choice of imporance densiy and he use of resampling [9,40,5] o eliminae paricles ha have small weighs and concenrae on paricles wih large weighs (see [40] for more deails on degeneracy and resampling procedures) Muli-Modaliy and Mixure Tracking I is sraighforward o generalize he Bayesian formulaion o he problem of muliobjec racking. However, due o he increase in dimensionaliy, his formulaion gives an exponenial explosion of compuaional demands. The primary goal in a muliobjec racking applicaion is o deermine he poserior disribuion, which is mulimodal in his case, over he curren join configuraion of he objecs a he curren ime sep, given all observaions up o ha ime sep. Muliple modes are caused eiher by ambiguiy abou he objec sae due o insufficien measuremens, which is supposed o be resolved during racking, or by measuremens coming from muliple objecs being racked. Generally, MC mehods are poor a consisenly mainaining he muli-modaliy in he filering disribuion. In pracice i frequenly occurs ha all he paricles quickly migrae o one of he modes, subsequenly discarding oher modes. To capure and mainain he muli-modal naure, which is inheren o many applicaions in which racking of muliple objecs is required, he filering disribuion is explicily represened by an M-componen mixure model [74]: p(x z : ) = M π m, p m (x z : ), (3.6) m= wih M m= π m, = and a non-parameric model is assumed for he individual mixure componens. In his case, he paricle represenaion of he filering disribuion, {x (i),w (i) } N i= wih N = MN s paricles, is augmened wih a se of componen indicaors, {c (i) } N i=, wih c(i) = m if paricle i belongs o mixure componen m. For he mixure componen m we also use he equivalen noaion {x (l) m,,w (l) m,} Ns l= =

9 3.4 Tailoring he Framework 6 {x (i),w (i) : c (i) = m} N i=. The represenaion (3.6) can be updaed in he same fashion as he wo-sep approach for sandard Bayesian sequenial esimaion [74]. 3.4 Tailoring he Framework Having presened he general framework for PF-based muliple objec racking, we now ailor i o our applicaion: he sudy of MT dynamics. This requires making choices regarding he models involved as well as a number of compuaional and pracical issues. Specifically, we propose a new dynamic model, which does no only cover spaioemporal behavior bu also allows dealing wih phoobleaching effecs (Secion 3.4.) and objec ineracion (Secion 3.4.). In addiion, we propose a new observaion model and corresponding likelihood funcion (Secion 3.4.3), ailored o objecs ha are elongaed in heir direcion of moion. The robusness and compuaional efficiency of he algorihm are improved by using wo-sep hierarchical searching (Secion 3.4.4), measuremen gaing (Secion 3.4.5) and a new imporance funcion for daa-dependen sampling (Secion 3.4.6). Finally, we propose pracical procedures for paricle reclusering (Secion 3.4.7) and auomaic rack iniiaion (Secion 3.4.8) Sae-Space and Dynamic Model In order o model he dynamic behavior of he visible ends of MTs in our algorihm, we represen he objec sae wih he sae vecor x = (x,ẋ,y,ẏ,z,ż,σ max,,σ min,, σ z,,i ) T, where (σ max,,σ min,,σ z, ) T s is he objec shape feaure vecor (see Secion 3.4.3), (x,y,z ) T r is he radius vecor, ṙ v is velociy, and I objec inensiy. The sae evoluion model D(x x ) can be facorized as D(x x ) = D y (y y )D s (s s )D I (I I ), (3.7) where y = (x,ẋ,y,ẏ,z,ż ). Here, D y (y y ) is modeled using a linear Gaussian model [40], which can easily be evaluaed poinwise in (3.4), and is given by ( D y (y y ) exp ) (y Fy ) T Q (y Fy ), (3.8) wih he process ransiion marix F = diag[f,f,f ] and covariance marix Q = diag[q,q,q ] given by F = ( T 0 ) ( q q and Q = q q where T is he sampling inerval. Depending on he parameers q, q, q he model (3.8) describes a variey of moion paerns, ranging from random walk ( v = 0, q 0, q = 0, q = 0) o nearly consan velociy ( v 0, q 0, q 0, q 0) [], [84]. In our applicaion, he parameers are fixed o q = q 3 T 3, q = q T, q = q T, where q conrols he noise level. In his case, model (3.8) corresponds o he coninuous-ime model ṙ() = w() 0, where w() is whie ),

10 6 3 Paricle Filering for Muliple Objec Tracking noise ha corresponds o noisy acceleraions []. We also make he realisic assumpion ha objec velociies are bounded. This prior informaion is objec dependen and will be used for sae iniializaion (see Secion 3.4.8). Small changes in frameo-frame MT appearance (shape) are modeled using he Gaussian ransiion prior D s (s s ) = N(s s,tq I), where N( µ, Σ) indicaes he normal disribuion wih mean µ and covariance marix Σ, I is he ideniy marix, and q represens he noise level in objec appearance. In pracice, he analysis of ime-lapse fluorescence microscopy images is complicaed by phoobleaching, a dynamic process by which he fluorescen proeins undergo phooinduced chemical desrucion upon exposure o exciaion ligh and hus lose heir abiliy o fluoresce. Alhough he mechanisms of phoobleaching are no ye well undersood, wo commonly used (and pracically similar) approximaions of fluorescence inensiy over ime are given by I() = Ae a + B (3.9) and ( ) ) k I() = I 0 ( +, (3.0) L where A, B, a, I 0, L, and k are experimenally deermined consans (see [4,48] for more deails on he validiy and sensiiviy of hese models). The rae of phoobleaching is a funcion of he exciaion inensiy. Wih a laser as an exciaion source, phoobleaching is observed on he ime scale of microseconds o seconds. The high numerical aperure objecives currenly in use, which maximize spaial resoluion and improve he limis of deecion, furher accelerae he phoobleaching process. Commonly, phoobleaching is ignored by sandard racking mehods, bu in many pracical cases i is necessary o model his process so as o be less sensiive o changing experimenal condiions. Following he common approximaion (3.9), we model objec inensiy in our image daa by he sum of a ime-dependen, a ime-independen, and a random componen: I + I c + u = I 0Â Â + ˆB e ˆα + I ˆB 0 Â + ˆB + u, (3.) where u is zero-mean Gaussian process noise and I 0 is he iniial objec inensiy, obained by he iniializaion procedure (see Secion 3.4.8). The parameers Â, ˆB, and ˆα are esimaed using he Levenberg-Marquard algorihm for nonlinear fiing of (3.9) o he average background inensiy over ime, b (see Secion 3.4.3). In order o convenienly incorporae he phoobleaching effec conained in (3.) ino our framework, we approximae i as a firs-order Gauss-Markov process, I = ( ˆα)I +u, which models he exponenial inensiy decay in he discree-ime domain. In his case, he corresponding sae prior D I (I I ) = N(I ( ˆα)I,q 3 T), where q 3 = T σ u and σ u is he variance of u. The phoobleaching effec could alernaively be accommodaed in our framework by assuming a consan inensiy model (ˆα = 0) for D I (I I ), bu wih a very high variance for he process noise, σ u. However, in pracice, because of he limied number

11 3.4 Tailoring he Framework 63 of MC samples, he variance of he esimaion would rapidly grow, and many samples would be used inefficienly, causing problems especially in he case of a highly peaked likelihood L(z x ) (see Secion 3.4.3). By using (3.), we follow a leas he rend of he inensiy changes, and bring he esimaion closer o he opimal soluion. This way, we reduce he esimaion variance and, consequenly, he number of MC samples needed for he same accuracy as in he case of he consan inensiy model. In summary, he proposed model (3.7) correcly approximaes small acceleraions in objec moion and flucuaions in objec inensiy, and herefore is very suiable for racking growing MTs, as heir dynamics can be well modeled by consan velociy plus small random diffusion [48]. The model (3.8) can also be successfully used for racking oher subcellular srucures, for example vesicles, which are characerized by moion wih higher nonlineariy. In ha case, he process noise level, defined by Q, should be increased Objec Ineracions and Markov Random Field In order o obain a more realisic moion model and avoid rack coalescence in he case of muliple objec racking, we explicily model he ineracion beween objecs using a Markov random field (MRF) [76]. Here we use a pairwise MRF, expressed by means of a Gibbs disribuion where d i,j ψ (x (i),x (j) ) exp ( d i,j ), i,j {,...,N}, c (i) c (j), (3.) and c (j) is maximal when wo objecs coincide is a penaly funcion which penalizes he saes of wo objecs c (i) ha are closely spaced a ime. Tha is, d i,j and gradually falls off as hey move apar. This simple pairwise represenaion is easy o implemen ye can be made quie sophisicaed. Using his form, we can sill reain he predicive moion model of each individual arge. To his end, we sample N s imes he pairs (x (l) m,,x(l) m,) (M such pairs a a ime, m = {,...,M}), from p m (x z : ) and q(x x (l) m,,z ), respecively, l = {,...,N s }. Taking ino accoun (3.), he weighs (3.4) in his case are given by m, L(z x (l) m,)d(x m, x (l) (l) q(x (l) m, x (l) m,,z ) w (l) m, ) The mixure represenaion {{x (l) m,,w (l) m,} M m=} Ns l= is hen sraighforwardly ransformed o {x (i),w (i),c (i) } N i= M k=,k m ψ (x (l) m,,x (l) k, ). (3.3). In our applicaion we have found ha an ineracion poenial based only on objec posiions is sufficien o avoid mos racking failures. The use of a MRF approach is especially relevan and efficien in he case of 3D+ daa analysis, because objec merging is no possible in our applicaion Observaion Model and Likelihood The measuremens in our applicaion are represened by a sequence of D or 3D images showing he moion of fluorescen proeins. The individual images (also called

12 64 3 Paricle Filering for Muliple Objec Tracking frames) are recorded a discree insans, wih a sampling inerval T, wih each image consising of N x N y N z pixels (N z = in D). A each pixel (i,j,k), which corresponds o a recangular volume of dimensions x y z nm 3, he measured inensiy is denoed as z (i,j,k). The complee measuremen recorded a ime is an N x N y N z marix denoed as z = {z (i,j,k) : i = 0,...,N x,j = 0,...,N y,k = 0,...,N z }. For simpliciy we assume ha he origins and axis orienaions of he (x,y,z) reference sysem and he (i,j,k) sysem coincide. Le z (r) denoe a firs-order inerpolaion of z ( x i, y j, z k). The image formaion process in a microscope can be modeled as a convoluion of he rue ligh disribuion coming from he specimen, wih a poin-spread funcion (PSF), which is he oupu of he opical sysem for an inpu poin ligh source. The heoreical diffracion-limied PSF in he case of paraxial and non-paraxial imaging can be expressed by he scalar Debye diffracion inegral [90]. In pracice, however, a 3D Gaussian approximaion of he PSF [6] is commonly favored over he more complicaed PSF models (such as he Gibson-Lanni model [55]). This choice is mainly moivaed by compuaional consideraions, bu a Gaussian approximaion of he physical PSF is fairly accurae for reasonably large pinhole sizes (relaive squared error (RSE) < 9%) and nearly perfec for ypical pinhole sizes (RSE < %) [90]. In mos microscopes currenly used, he PSF limis he spaial resoluion o 00 nm in-plane and 600 nm in he direcion of he opical axis, as a consequence of which subcellular srucures (ypically of size < 0 nm) are imaged as blurred spos. We adop he common assumpion ha all blurring processes are due o a linear and spaially invarian PSF. The PF framework accommodaes any PSF ha can be calculaed poinwise. To model he imaged inensiy profile of he objec wih some shape, one would have o use he convoluion wih he PSF for every sae x (i). In order o overcome his compuaional overload, we propose o model he PSF and objec shape a he same ime using he 3D Gaussian approximaion. To model he manifes elongaion in he inensiy profile of MTs, we uilize he velociy componens from he sae vecor x as parameers in he PSF. In his case, for an objec of inensiy I a posiion r, he inensiy conribuion o pixel (i,j,k) is approximaed as h (i,j,k;x ) = b + (I + I c ) ( exp ) mt R T Σ Rm exp ( (k z z m an θ) ), (3.4) where b is he background inensiy, σ z ( 35 nm) models he axial blurring, R = R(φ) is a roaion marix ( ) ( ) cos φ sinφ σ R(φ) =, Σ = m (θ) 0 sin φ cos φ 0 σmin, ( i x x m = j y y σ z ), σ m (θ) = σ min (σ min σ max )cos θ,

13 3.4 Tailoring he Framework 65 an θ = z x + y, an φ = y, π < φ,θ π. x The parameers σ max and σ min represen he amoun of blurring and, a he same ime, model he elongaion of he objec along he direcion of moion. For subresoluion srucures such as vesicles, σ min = σ max 80 nm, and for he elongaed MTs σ min 00 nm and σ max 300 nm. For background level esimaion we use he fac ha he conribuion of objec inensiy values o he oal image inensiy (mainly formed by background srucures wih lower inensiy) is negligible, especially in he case of low SNRs. We have found ha in a ypical D image of size pixels conaining a housand objecs, he number of objec pixels is only abou %. Even if he objec inensiies would be 0 imes as large as he background level (very high SNR), heir conribuion o he oal image inensiy would be less han 0%. In ha case, he normalized hisogram of he image z can be approximaed by a Gaussian disribuion wih mean ˆb and variance σ b. The esimaed background b = ˆb is hen calculaed according o b = N x N y N z N x i=0 N y j=0 N z k=0 z (i,j,k). (3.5) In he case of a skewed hisogram of image inensiy, he median of he disribuion can be aken as an esimae of he background level. The laer is preferable because i reas objec pixels as ouliers for he background disribuion. Since an objec will affec only he pixels in he viciniy of is locaion, r, we define he likelihood funcion as L G (z x ) (i,j,k) C(x ) where C(x ) = {(i,j,k) Z 3 : h (i,j,k;x ) b > 0.I }, p h (z (i,j,k) x ) p h (z (i,j,k) x ) p b (z (i,j,k) b ), (3.6) ( ) σ h (i,j,k) exp (z (i,j,k) h (i,j,k;x )) σh (i,j,k), (3.7) and p b (z (i,j,k) b ) exp ( (z (i,j,k) b ) σ b ), (3.8) wih σh (i,j,k) and σ b he variances of he measuremen noise for he objec + background and background, respecively, which are assumed o be independen from pixel o pixel and from frame o frame. Poisson noise, which can be used o model he effec of he quanum naure of ligh on he measured daa, is one of he main sources of noise in fluorescence microscopy imaging. The recursive Bayesian soluion is applicable as long as he saisics of he measuremen noise is known for each pixel. In his chaper we use a valid approximaion of Poisson noise, wih σh (i,j,k) = h (i,j,k;x ) and σb = b, by scaling he image inensiies in order o saisfy he condiion σb = b [6].

14 66 3 Paricle Filering for Muliple Objec Tracking Hierarchical Searching Generally, he likelihood L G (z x ) is very peaked (even when he region C(x ) is small) and may lead o severe sample impoverishmen and divergence of he filer. Theoreically i is impossible o avoid he degeneracy phenomenon, where, afer a few ieraions of he algorihm, all bu one of he normalized imporance weighs are very close o zero [40]. Consequenly, he accuracy of he esimaor also degrades enormously [5]. A commonly used measure of degeneracy is he esimaed effecive sample size [40], given by ( Ns ) N eff () = (w (i) ), (3.9) i= which inuiively corresponds o he number of useful paricles. Degeneracy is usually srong for image daa wih low SNR, bu he filer also performs poorly when he noise level is oo small [39]. This suggess ha MC esimaion wih accurae sensors may perform worse han wih inaccurae sensors. The problem can be parially fixed by using an observaion model which overesimaes he measuremen noise. While he performance is beer, his is no a principled way of fixing he problem; he observaion model is arificially inaccurae and he resuling esimaion is no longer a poserior, even if infiniely many samples were used. Oher mehods ha ry o improve he performance of PF include pariioned sampling [89], he auxiliary paricle filer (APF) [9], [6] and he regularized paricle filers (RPF) [39,6]. Because of he highly nonlinear observaion model and dynamic model wih a high noise level, he menioned mehods are inefficien for our applicaion. Pariioned sampling requires he possibiliy o pariion he sae space and o decouple he observaion model for each of he pariions, which canno be done for our applicaion. Applicaion of he APF is beneficial only when he dynamic model is correcly specified wih a small amoun of process noise. The racking of highly dynamic srucures wih linear models requires increasing he process noise in order o capure he ypical moion paerns. To overcome hese problems, we use a differen approach, based on RPF, and mainly on progressive correcion [39]. Firs, we propose a second observaion model: (( L S (z x ) σ B S z σ S (x ) exp (x ) S(x b ) ) ( S z (x ) S h (x ) ) ) σs (x, (3.0) ) where and S z (x ) = S h (x ) = σ B (i,j,k) C(x ) (i,j,k) C(x ) z (i,j,k), h (i,j,k;x ), S b = b C(x ), where denoes he se size operaor, and he variances σ S and σ B are aken o approximae he Poisson disribuion: σ S = So and σ B = Sb. The likelihood L S (z x ) is less peaked bu gives an error of he same order as L G (z x ). Anoher advanage is ha L S (z x ) can be used for objecs wihou a predefined

15 3.4 Tailoring he Framework 67 shape; only he region C(x ), which presumably conains he objec, and he oal objec inensiy in C(x ) need o be specified. Subsequenly, we propose a modified hierarchical search sraegy, which uses boh models, L S and L G. To his end, we calculae an inermediae sae a ime, beween ime poins and, by propagaing and updaing he samples using he likelihood L S according o p(x z : ) L S (z x )D(x x )p(x z : ) (3.) where z = z. Afer his sep, N eff is sill raher high, because he likelihood L S is less peaked han L G. In a nex sep, paricles wih high weighs a ime are diversified and pu ino regions where he likelihood L G is high, giving a much beer approximaion of he poserior: p(x z : ) L G (z x )N(x µ,σ ) p(x z : ), (3.) where he expecaion and he variance are given by µ = E p [x ], Σ = E p [(x µ )(x µ ) T ]. (3.3) The described hierarchical search sraegy is furher denoed as L SG. I keeps he number N eff quie large and, in pracice, provides filers ha are more sable in ime, wih lower variance in he posiion esimaion Measuremen Gaing Muliple objec racking requires gaing, or measuremen selecion. The purpose of gaing is o reduce compuaional expense by eliminaing measuremens which are far from he prediced measuremen locaion. Gaing is performed for each rack a each ime sep by defining a subvolume of he image space, called he gae. All measuremens posiioned wihin he gae are seleced and used for he rack updae sep, (3.), while measuremens ouside he gae are ignored in hese compuaions. In sandard approaches o racking, using he Kalman filer or exended Kalman filer, measuremen gaing is accomplished by using he prediced measuremen covariance for each objec and hen updaing he prediced sae using join probabilisic daa associaion [79]. In he PF approach, which is able o cope wih nonlinear and non- Gaussian models, he analog of he prediced measuremen covariance is no available and can be consruced only by aking, for example, a Gaussian approximaion of he curren paricle cloud and using i o perform gaing. Generally, his approximaion is unsaisfacory, since he advanages gained from having a represenaion of a non- Gaussian pdf are los. In he proposed framework, however, his approximaion is jusified by using he highly peaked likelihood funcions and he reclusering procedure (described in Secion 3.4.7), which keep he mixure componens unimodal. Having he measuremens z (r ), we define he gae for each of he racks as follows: C m, = {r R 3 : (r r m, ) T Σ m,(r r m, ) C 0 }, (3.4) where he parameer C 0 specifies he size of he gae, which is proporional o he probabiliy ha he objec falls wihin he gae. Generally, since he volume of he

16 68 3 Paricle Filering for Muliple Objec Tracking gae is dependen on he racking accuracy, i varies from scan o scan and from rack o rack. In our experimens, C 0 = 9 (a 3-sandard-deviaion level gae). The gae C m, is cenered a he posiion prediced from he paricle represenaion of p m (x z : ): where he r (i) r m, = E pm [r ] = r p m (x z : )dx are he posiion elemens of he sae vecor N i=, c (i) =m r (i) w (i), (3.5) x (i) D(x x (i) ), i = {,...,N}. Similarly, he covariance marix is calculaed as Daa-Dependen Sampling Σ m, = E pm [(r r m, )(r r m, ) T ]. (3.6) Basic paricle filers [9,70,43], which use he proposal disribuion q(x x,z ) = D(x x ) usually perform poorly because oo few samples are generaed in regions where he desired poserior p(x z : ) is large. In order o consruc a proposal disribuion which alleviaes his problem and akes ino accoun he mos recen measuremens z, we propose o ransform he image sequence ino probabiliy disribuions. True spos are characerized by a combinaion of convex inensiy disribuions and a relaively high inensiy. Noise-induced local maxima ypically exhibi a random disribuion of inensiy changes in all direcions, leading o a low local curvaure [6]. These wo discriminaive feaures (inensiy and curvaure) are used o consruc an approximaion of he likelihood L(z x ), using he image daa available a ime. For each objec we use he ransformaion p m (r z ) = (G σ z (r ) b ) r κ s (r ) C m, (G σ z (r ) b ) r κ s (r )dxdydz, (3.7) r C m,, where G σ is he Gaussian kernel wih sandard deviaion (scale) σ, he curvaure κ (r ) is given by he deerminan of he Hessian marix H of he inensiy z (r ): κ (r ) = de(h(r )), H(r ) = T z (r ), (3.8) and he exponens r > 0 and s > 0 weigh each of he feaures and deermine he peakedness of he likelihood. Using his ransformaion, we define he new daa dependen proposal disribuion for objec m as q m (x x,z ) = p m (r z )N(I z (r ) b,q 3 T) N(s s MMSE m,,tq I)N(v r ˆr MMSE m,,tq I), (3.9)

17 3.4 Tailoring he Framework 69 Conrary o he original proposal disribuion, which fails if he likelihood is oo peaked, he disribuion (3.9) generaes samples ha are highly consisen wih he mos recen measuremens in he prediced (using he informaion from he previous ime sep) gaes. A combinaion of boh proposal disribuions gives excellen resuls: q m (x x,z ) = γd(x x ) + ( γ) q m (x x,z ), where 0 < γ <. Comparison shows ha he proposal disribuion q m (x x,z ) is uniformly superior o he regular one (γ = ) and scales much beer o smaller sample sizes Clusering and Track Managemen The represenaion of he filering disribuion p(x z : ) as he mixure model (3.6) allows for a deerminisic spaial reclusering procedure ({c (i) },M ) = F({x (i) }, {c (i) }, M) [74]. The funcion F can be implemened in any convenien way. I calculaes a new mixure represenaion (wih possibly a differen number of mixure componens) aking as inpu he curren mixure represenaion. This allows modeling and capuring merging and spliing evens, which also have a direc analogy wih biological phenomena. In our implemenaion, a each ieraion he mixure represenaion is recalculaed by applying K-means clusering algorihm. The reclusering is based on spaial informaion (objec posiions) only and is iniialized wih he esimaes (3.5). Taking ino accoun our applicaion, wo objecs are no allowed o merge when heir saes become similar. Whenever objecs pass close o one anoher, he objec wih he bes likelihood score ypically hijacks he paricles of he nearby mixure componens. As menioned above, his problem is parly solved by using he MRF model for objec ineracions. The MRF model significanly improves he racking performance in 3D+. For D+ daa ses, however, he observed moion is a projecion of he real 3D moion ono he D plane. In his case, when one objec passes above or beneah anoher (in 3D), we perceive he moion as peneraion or merging. These siuaions are in principle ambiguous and frequenly canno be resolved uniquely, neiher by an auomaic racking mehod nor by a human observer. We deec possible objec inersecions during racking by checking wheher he gaes C m, inersec each oher. For example, for wo rajecories, he inersecion is capured if C i, C j,, i,j {,...,M}. In general, he measuremen space C = M m=c m, is pariioned ino a se of disjoin regions C = {C,,...,C K, }, where Ck, is eiher he union of conneced gaes or he gae iself. For each C k,, we define a se of indices J k,, which indicae which of he gaes C i, belong o i: J k, = {i {,...,M} : C i, C k,} (3.30) For he gaes Ck, wih J k, =, he updae of he MC weighs w (i) m, is done according o (3.4). For all oher gaes Ck,, which correspond o objec ineracion, we follow he procedure similar o he one described in Secion For each Ck, for which J k,, he se of saes {x (l) j, }, j J k,, is sampled from he proposal disribuion (for every l = {,...,N s }), and a se of hypoheses Θ (l) k, = {θ(l),...,θ(l) S }, S = Jk,,

18 70 3 Paricle Filering for Muliple Objec Tracking is formed. Each θ (l) i is a se of binary associaions, {a (l) i,j }, j J k,, where a (l) i,j = if = 0 if he objec dies or leaves jus objec j exiss during he ineracion, and a (l) i,j before or during he ineracion and gives no measuremens a ime. The hypohesis ha maximizes he likelihood is seleced as ˆθ (l) k = argmax L(z x ), (3.3) θ (l) i Θ (l) k, where he likelihood L(z x ) can be eiher L G (z x ) or L S (z x ), bu he region C(x ) is defined as C(x ) = j Jk, C(x (l) j, ), and h (.;x ) is subsiued in (3.6) and (3.0) for each θ (l) i wih j J k, a (l) i,j h (.;x (l) j, he region C(x ) = C(x (l) j, ) and h (.;x ) = j J k, â (l) ). For he updae of he MC weighs w(l) j, j h (.;x (l) j, ) are used in (3.6) and (3.0), wih he â (l) j denoing he a (l) (l) i,j corresponding o ˆθ k. Addiionally, in such cases, we do no perform reclusering, bu keep he labels for he curren ieraion as hey were before. If he componen represenaion in he nex few frames afer he ineracion even becomes oo diffuse, and here is more han one significan mode, spliing is performed and a new rack is iniiaed (see Secion for more deails). Finally, for he erminaion of an exising rack, he mehods commonly used for small arge racking [68, 04] canno be applied sraighforwardly. These mehods assume ha, due o imperfec sensors, he probabiliy of deecing an objec is less han one, and hey ry o follow he objec afer disappearance for 4-5 frames, predicing is posiion in ime and hoping o cach i again. In our case, when he densiy of objecs in he images is high, such monioring would definiely resul in confirming measuremens afer 3-5 frames of predicion, bu hese measuremens would very likely originae from anoher objec. In our algorihm in order o erminae he rack we define he hresholds σ max, σ min, σ z ha describe he bigges objecs ha we are going o rack. Then we sample he paricles in he prediced gaes C m, using he daa-dependen sampling (3.7) wih s = 0. If he deerminan of he covariance marix compued for hose MC samples is graer han σ max σ min σ zr 3 he rack is erminaed. If he gae C m, does no conain a real objec he deerminan value will be much higher han he proposed hreshold, which is nicely separae he objecs from he background srucures Iniializaion and Track Iniiaion The prior disribuion p(x 0 ) is specified based on informaion available in he firs frame. One way o iniialize he sae vecor x 0 would be o poin on he desired brigh spos in he image or o selec regions of ineres. In he laer case, he sae vecor is iniialized by a uniform disribuion over he sae space, in predefined inervals for velociy and inensiy, and he expeced number of objecs should be specified. During filering and reclusering, afer a burn-off period of -3 frames, only he rue objecs will remain. For compleely auomaic iniiaion of objec racks in he firs frame, and also for he deecion of poenial objecs for racking in subsequen frames, we use he following procedure. Firs, he image space is divided ino N I = N X N Y N Z

19 3.5 Experimenal Resuls 7 recangular 3D cells of dimensions c c a, wih c = 6σ max and a = 6σ z. Nex, for each ime sep, he image is convered o a probabiliy map according o (3.7), and N = MN s paricles x (i) are sampled wih equal weighs. The number of paricles in each cell represens he degree of belief in objec birh. To discriminae poenial objecs from background srucures or noise, we esimae for each cell he cener of mass ˆr k (k = {,...,N I }) by MC inegraion over ha cell and calculae he number of MC samples n k, in he ellipsoidal regions S k, (r ) cenered a ˆr k (wih semi-axes of lenghs c /, c /, a /). In order o iniiae a new objec, wo condiions have o be saisfied. The firs condiion is ha n k, should be greaer han N S k, z = Nπ(6N I ). The hreshold represens he expeced number of paricles if he sampling was done from he image region wih uniform background inensiy. The second condiion is similar o he one for rack erminaion (see Secion 3.4.7): he deerminan of he covariance marix should be smaller han σ max σ min σ zr 3. Each objec d (ou of M d newly deeced a ime ) is iniialized wih mixure weigh π d, = (M + M d ) and objec posiion r d, (he cener of mass calculaed by MC inegraion over he region S d, (r )). The velociy is uniformly disribued in a predefined range and he inensiy is obained from he image daa for ha frame and posiion. In cases where he samples from an undeeced objec are spli beween four cells (in he unlikely even when he objec is posiioned exacly on he inersecion of he cell borders), he objec will mos probably be deeced in he nex ime frame. 3.5 Experimenal Resuls The performance of he described PF-based racking mehod was evaluaed using boh compuer generaed image daa (Secion 3.5.) and real fluorescence microscopy image daa from MT dynamics sudies (Secion 3.5.). The former allowed us o es he accuracy and robusness o noise and objec ineracion of our algorihm compared o wo oher commonly used racking ools. The experimens on real daa enabled us o compare our algorihm o exper human observers Evaluaion on Synheic Daa Simulaion Seup The algorihm was evaluaed using synheic bu realisic D image sequences (0 ime frames of 5 5 pixels, x = y = 50 nm, T = sec) of moving MT-like objecs (a fixed number of 0, 0, or 40 objecs per sequence, yielding daa ses of differen objec densiies), generaed according o (3.8) and (3.4), for differen levels of Poisson noise (see Fig. 3.) in he range SNR= 7, since SNR=4 has been idenified by previous sudies [6, 3] as a criical level a which several popular racking mehods break down. In addiion, he algorihm was esed using 3D synheic image sequences (0 ime frames of 5 5 pixels 0 opical slices, x = y = 50 nm, z = 00 nm, T = sec, wih 0 40 objecs per sequence), also for differen noise levels in he range of SNR= 7. Here, SNR is defined as he difference in inensiy beween he objec and he background, divided by he sandard deviaion of he objec noise [3]. The

20 7 3 Paricle Filering for Muliple Objec Tracking µm µm SNR= SNR=3 SNR=5 SNR=7 Figure 3.. Examples of synheic images used in he experimens. The lef image is a single frame from one of he sequences, a SNR=, giving an impression of objec appearance. The inses show zooms of objecs a differen SNRs. The righ image is a frame from anoher sequence, a SNR=7, wih he rajecories of he 0 moving objecs superimposed (whie dos), illusraing he moion paerns allowed by he linear sae evoluion model (3.8). velociies of he objecs ranged from 00 o 700 nm/sec, represenaive of published daa [55]. Having he ground ruh for he synheic daa, we evaluaed he accuracy of racking by using a radiional quaniaive performance measure: he roo mean square error (RMSE), in K independen runs (we used K = 3) [04]: wih RMSE k = M RMSE = K K RMSE k, (3.3) i= { } M r m, ˆr k T m m,, (3.33) T m m= where r m, defines he rue posiion of objec m a ime, ˆr k m, is a poserior mean esimae of r m, for he kh run, and T m is he se of ime poins a which objec m exiss Experimens wih Hierarchical Searching In order o show he advanage of using he proposed hierarchical search sraegy (see Secion 3.4.4), we calculaed he localizaion error a differen SNRs for objecs moving along horizonal sraigh lines a a consan speed of 400 nm/sec (similar

21 3.5 Experimenal Resuls 73 RMSE [nm] L G L S L SG RMSE [nm] L G L S LSG SNR SNR Figure 3.3. The RMSE in objec posiion esimaion as a funcion of SNR for round (lef) and elongaed (righ) objecs using he hree differen observaion models, L G, L S, and L SG. o [3]). The racking was done for wo ypes of objecs: round (σ max = σ min = 00 nm) and elongaed (σ max = 300 nm, σ min = 00 nm) using he likelihoods L S, L G, and he combined wo-sep approach L SG. The filering was performed wih 500 MC samples. The RMSE for all hree models is shown in Fig The localizaion error of he hierarchical search is lower and he effecive sample size N eff is higher han in he case of using only L G. For comparison, for he likelihoods L S, L G, and L SG, he raios beween he effecive sample size N eff and N s are less han 0.5, 0.005, and 0.05, respecively Comparison wih Convenional Two-Sage Tracking Mehods The proposed PF-based racking mehod was compared o convenional wo-sage (compleely separaed deecion and linking) racking approaches commonly found in he lieraure. To maximize he credibiliy of hese experimens, we chose o use wo exising, sae-of-he-ar muliarge racking sofware ools based on his principle, raher han making our own (possibly biased) implemenaion of described mehods. The firs is Volociy (Improvision, Covenry, UK), which is a commercial sofware package, and he second is ParicleTracker [3], which is freely available as a plugin o he public-domain image analysis ool ImageJ [] (Naional Insiues of Healh, Behesda, MD, USA). Wih Volociy, he user has o specify hresholds for he objec inensiy and he approximae objec size in order o discriminae objecs from he background, in he deecion sage. These hresholds are se globally, for he enire image sequence. Following he exracion of all objecs in each frame, linking is performed on he basis of finding neares neighbors in subsequen image frames. This associaion of neares neighbors also akes ino accoun wheher he moion is smooh or erraic. Wih ParicleTracker, he deecion par also requires seing inensiy and

22 74 3 Paricle Filering for Muliple Objec Tracking Figure 3.4. Example (SNR=3) showing he abiliy of our PF mehod o deal wih one-frame occlusion scenarios (op sequence), using he proposed reclusering procedure, while ParicleTracker (and similarly Volociy) fails (boom sequence). Figure 3.5. Typical example (SNR=3) showing he abiliy of our PF mehod o resolve objec crossing correcly (op sequence), by using he informaion abou he objec shape during he measuremen-o-rack associaion process, while Paricle- Tracker (and similarly Volociy) fails (boom sequence). objec size hresholds. The linking, however, is based on finding he global opimal soluion for he correspondence problem in a given number of successive frames. The soluion is obained using graph heory and global energy minimizaion [3]. The linking also uilizes he zeroh- and second-order inensiy momens of he objec inensiies. This beer resolves inersecion problems and improves he linking resul. For boh ools, he parameers were opimized manually during each sage, unil all objecs in he scene were deeced. Our PF-based mehod was iniialized using he auomaic iniializaion procedure described in Secion The user-definable algorihm parameers were fixed o he following values: σ max = 50 nm, σ min = 0 nm,

23 3.5 Experimenal Resuls Figure 3.6. Example (SNR=3) where our PF mehod as well as ParicleTracker and Volociy failed (only he rue racks are shown in he sequence), because hree objecs inerac a one locaion and he occlusion lass for more han one frame Table 3.. Comparison of he abiliy of he hree mehods o rack objecs correcly in cases of objec appearance, disappearance, and ineracions. Volociy ParicleTracker Paricle Filer SNR r 0 r r 0 r r 0 r N r = N r = N r = q = 7500 nm /sec 3, q = 5 nm/sec, q 3 = 0., and 0 3 MC samples were used per objec. To enable comparisons wih manual racking, five independen, exper observers also racked he D synheic image sequences, using he freely available sofware ool MTrackJ [94] Tracking Resuls Firs, using he D synheic image sequences, we compared he abiliy of our algorihm, Volociy, and ParicleTracker o rack objecs correcly, despie possible

24 76 3 Paricle Filering for Muliple Objec Tracking objec appearances, disappearances, and ineracions or crossings. The resuls of his comparison are presened in Table 3.. Two performance measures are lised: r 0, which is he raio beween he number of racks produced by he algorihm and he rue number of racks presen in he daa (N r ), and r, which is he raio beween he number of correcly deeced racks and he rue number of racks. Ideally, he values for boh raios should be equal o. A value of r 0 > indicaes ha he mehod produced broken racks. The main cause of his is he inabiliy o resolve rack inersecions in some cases (see Fig. 3.4 for an example). In such siuaions he mehod eiher iniiaes new racks afer he objec ineracion even (because during he deecion sage only one objec was deeced a ha locaion, see Fig. 3.4), increasing he raio r 0, or i incorrecly inerchanges he racks before and afer he ineracion (see Fig. 3.5 for an example), lowering he raio r. From he resuls in Table 3. and he examples in Figs. 3.4 and 3.5, i clearly follows ha our PF mehod is much more robus in dealing wih objec ineracions. The scenario in he laer example causes no problems for he PF, as, conrary o wo oher mehods, i explois informaion abou objec appearance. During he measuremen-o-rack associaion, he PF favors measuremens ha are close o he prediced locaion and ha have an elongaion in he prediced direcion of moion. In some cases (see Fig. 3.6 for an example), all hree mehods fail, which generally occurs when he ineracion is oo complicaed o resolve even for exper biologiss. Using he same daa ses and racking resuls, we calculaed he RMSE in objec posiion esimaion, as a funcion of SNR. To make a fair comparison, only he resuls of correcly deeced racks were included in hese calculaions. The resuls are shown in Fig The localizaion error of our algorihm is in he range of 0 50 nm, depending on he SNR, which is approximaely 3 imes smaller han for manual racking. The error bars represen he inerobserver variabiliy for manual racking, which, ogeher wih he average errors, indicae ha he performance of manual racking degrades significanly for low SNRs, as expeced. The errors of he hree auomaed mehods show he same rend, wih our mehod being consisenly more accurae han he oher wo. This may be explained by he fac ha, in addiion o objec localizaion by cener-of-mass esimaion, our hierarchical search performs furher localizaion refinemen during he second sep (3.). The RMSE in Fig. 3.7 is larger han in Fig. 3.3, because, even hough only correc racks were included, he accuracy of objec localizaion during muliple objec racking is unfavorably influenced a places where objec ineracion occurs. Our algorihm was also esed on he 3D synheic image sequences as described, using 0 MC simulaions. The RMSEs for he observaion model L SG ranged from 30 nm (SNR = 7) o 70 nm (SNR = ). These errors were comparable o he errors produced by Volociy (in his es, ParicleTracker was excluded, as i is limied o racking in D+). Despie he fac ha he axial resoluion of he imaging sysem is approximaely hree imes lower, he localizaion error was no affeced dramaically relaive o he D+ case. The reason for his is ha in 3D+ daa, we have a larger number of informaive image elemens (voxels). As a resul, he difference in he RMSEs produced by he esimaors employed in our algorihm and in Volociy is less compared o Fig. 3.7.

25 3.5 Experimenal Resuls 77 Roo Mean Square Error [nm] Manual Tracking ParicleTracker Volociy Paricle Filer Sandard Deviaion SNR Figure 3.7. The RMSE in objec posiion esimaion as a funcion of SNR for our algorihm (Paricle Filer) versus he wo oher auomaic mehods (Volociy and ParicleTracker) and manual racking (five observers) based on synheic image daa Evaluaion on Real Daa Image Acquisiion In addiion o he compuer generaed image daa, real D fluorescence microscopy image sequences of MT dynamics were acquired. COS- cells were culured and ransfeced wih GFP-agged proeins as described [5,55]. Cells were analyzed a 37 o C on a Zeiss 50 confocal laser scanning microscope (LSM-50). In mos experimens he opical slice separaion (in he z-dimension) was se o µm. Images of GFP+TIP movemens in ransfeced cells were acquired every 3.5 seconds. For differen imaging seups, he pixel size ranged from nm o 0 0 nm. Image sequences of frames were recorded and movies assembled using LSM-50 sofware. Six represenaive daa ses (30 frames of size 5 5 pixels), examples of which are shown in Fig. 3., were preseleced from larger volumes by manually choosing he regions of ineres. GFP+TIP dashes were racked in differen cell areas. Insananeous velociies of dashes were calculaed simply by dividing measured or racked disances beween frames by he emporal sampling inerval Comparison wih Manual Tracking Lacking ground ruh for he real daa, we evaluaed he performance of our algorihm by visual comparison wih manual racking resuls. In his case, he laer were obained from wo exper cell biologiss, each of which racked 0 moving MTs of

26 78 3 Paricle Filering for Muliple Objec Tracking Relaive Frequency PF Tracking Manual Tracking Velociy [nm/sec] Relaive Frequency PF Tracking Manual Tracking Velociy [nm/sec] Figure 3.8. Examples of velociy disribuions obained wih our auomaic racking algorihm versus manual racking applied o real fluorescence microscopy image sequences of growing MTs. Resuls are shown for he daa ses in Fig. 3.(a) (op) and Fig. 3.(f) (boom). ineres by using he aforemenioned sofware ool MTrackJ. The selecion of arge MTs o be racked was made independenly by he wo observers. Also, he decision of which feaure o rack (he ip, he cener, or he brighes poin) was lef o he observers. When done consisenly, his does no influence velociy esimaions, which is wha we focused on in hese experimens. The parameers of our algorihm (run wih he model L SG ) were fixed o he same values as in he case of he evaluaion on synheic daa Tracking Resuls Disribuions of insan velociies esimaed using our algorihm versus manual racking are presened in Fig The graphs show he resuls for he daa ses of Fig. 3.(a) and (f), for which SNR 5 and SNR, respecively. A visual comparison of he esimaed velociies per rack, for each of he 0 racks (he average rack lengh was 3 ime seps), is presened in Fig. 3.9, wih more deails for wo

27 3.5 Experimenal Resuls 79 Velociy [nm/sec] PF Tracking Sandard Deviaion Manual Tracking Track Number Velociy [nm/sec] PF Tracking Sandard Deviaion Manual Tracking Track Number Figure 3.9. Resuls of velociy esimaion for 0 represenaive MT objecs in real fluorescence microscopy image sequences using our auomaic racking algorihm versus manual racking for he daa ses in Fig. 3.(a) (op) and Fig. 3.(f) (boom). Shown are he mean values (black or whie squares) and ± sandard deviaion (bars) of he esimaes. represenaive racks shown in Fig Applicaion of a paired Suden -es per rack revealed no saisically significan difference beween he resuls of our algorihm and ha of manual racking, for boh exper human observers (p 0.05 in all cases). Ofen, biologiss are ineresed in average velociies over ses of racks. In he described experimens, he difference in average velociy (per 0 racks) beween auomaic and manual racking was less han %, for boh observers. Our velociy esimaes are also comparable o hose repored previously based on manual racking in he same ype of image daa [55]. Finally, we presen wo differen example visualizaions of real daa ogeher wih he resuls of racking using our algorihm. Fig. 3. shows he resuls of racking in he presence of phoobleaching, which clearly illusraes he capabiliy of our algorihm o iniiae new racks for appearing objecs, o erminae racks for disappearing objecs, and o deal wih closely passing objecs. The rendering in Fig. 3. gives a

28 80 3 Paricle Filering for Muliple Objec Tracking Velociy [nm/sec] PF Tracking Manual Tracking Time Sep Velociy [nm/sec] PF Tracking Manual Tracking Time Sep Figure 3.0. Velociy esimaes per ime sep for our auomaic racking algorihm versus manual racking. Resuls are shown for rack numbers 4 (op) and 0 (boom) in Fig. 3.9 (also from he op and boom graphs, respecively). visual impression of he full racking resuls for a few ime frames of one of he real daa ses used in he experimens. 3.6 Discussion and Conclusions In his chaper have demonsraed he applicabiliy of paricle filering for quaniaive analysis of subcellular dynamics. Compared o exising approaches in his field, our approach is a subsanial improvemen for deecion and racking of large numbers of spos in image daa wih low SNR. Convenional mehods, which perform objec deecion prior o he linking sage, use non-bayesian maximum likelihood or leas squares esimaors. The variance of hose esimaors is larger han he variance of he MMSE esimaor [], for which some prior informaion abou he esimaed parameers is assumed o be known. In our case, his informaion is he predicion of he objec posiion according o he moion model. This sep, which opimally explois available emporal informaion, makes our probabilisic racking approach

29 3.6 Discussion and Conclusions µm µm Figure 3.. Resuls (six racks) of auomaically racking MTs (brigh spos) in he presence of phoobleaching, illusraing he capabiliy of our algorihm o capure newly appearing objecs (racks 5 and 6) and o deec objec disappearance (for example rack 4). I also shows he robusness of he algorihm in he case of closely passing objecs (racks and 5). perform superior in he presence of severe noise in comparison wih exising frameby-frame approaches, which break down a SNR < 4 5 [6, 3]. As he experimens show, conrary o wo oher popular racking ools, our algorihm sill yields reliable racking resuls even in daa wih SNR as low as (which is no uncommon in pracice). We noe ha he comparison wih hese wo-sage racking approaches mainly evaluaed he linking pars of he algorihms, as he deecion par is based on hresholding, and he parameers for ha sage were opimized manually unil all he desired objecs were localized. In pracice, since hese algorihms were no designed specifically o deal wih phoobleaching effecs, hey can be expeced o perform worse han repored here. The resuls of he experimens on synheic image daa sugges ha our algorihm is poenially more accurae han manual racking by exper human observers. The experimens on real fluorescence microscopy image sequences from MT dynamics sudies showed comparable performance. This is explained by he fac ha in he laer experimens, we were limied o comparing disribuions and averages (Figs. 3.8 and 3.9), which may conceal small local discrepancies, especially when he objecs velociies vary over ime. Insan velociies were also analyzed per rack (Fig. 3.0) bu could no be quaniaively validaed due o he lack of ground ruh. Neverheless, he resuls indicae ha our algorihm may replace laborious manual procedures. Currenly we are evaluaing he mehod also for oher biological applicaions o furher demonsrae is advanages over curren means of manual and auomaed racking and quanificaion of subcellular dynamics. Our findings encourage use of he mehod o analyze complex biological image sequences no only for obaining saisical esimaes of average velociy and life span, bu also for deailed analyses of complee life hisories. The algorihm was implemened in he Java programming language (Sun Microsysems Inc., Sana Clara, CA) as a plugin for ImageJ (Naional Insiues of Healh, Behesda, MD []), a public domain and plaform independen image pro-

30 8 3 Paricle Filering for Muliple Objec Tracking Figure 3.. Visualizaion of racking resuls (80 racks) produced by our algorihm in he case of he real fluorescence microscopy image sequence of Fig. 3.(a). Lef: Trajecories projeced on op of one of he frames, giving an impression of he MT dynamics in his image sequence. Righ: Five frames from he sequence (ime is increasing from boom o op) wih he rajecories rendered as small ubes connecing he frames. The rendering was accomplished using a scrip developed in-house based on he Visualizaion Toolki [36].