Kernel-based Tracking in Ultrasound Sequences of Liver

Transcription

1 Kernel-based Tracking in Ultrasound Sequences of Liver Tobias Benz 1, Markus Kowarscik 2,1 and Nassir Navab 1 1 Computer Aided Medical Procedures, Tecnisce Universität Müncen, Germany 2 Angiograpy & Interventional X-Ray Systems, Siemens AG, Healtcare Sector, Forceim, Germany Abstract. Objective: Object tracking in 2D ultrasound sequences of liver to infer real-time respiratory organ movement and offer motion compensation in image-guided abdominal interventions. Metods: A kernel-based tracking algoritm tat is adaptive to scale and orientation canges of te tracking target is applied to 54 vessel targets in 21 ultrasound sequences acquired from volunteers under free breating. Tracking performance is evaluated based on manually annotated ground trut information. Results: Tracking results sow tat te algoritm is able to track te assessed targets in a precise and robust manner in real-time performance. Te overall mean tracking error is 1.43 ± 1.22 mm. 1 Introduction Object tracking in ultrasound (US) sequences of liver under respiratory motion is a callenging task wit several applications in, for instance, motion compensation in abdominal interventions like needle biopsies, radio frequency ablations, and radiation terapy. In tis work, we present te application of a scale and orientation adaptive mean sift procedure to track vessel targets in long 2D US sequences acquired under free breating. 3 Te mean sift procedure was first introduced by Fukunaga et al. [7] for data clustering. Ceng et al. [1] and Comaniciu et al. [3] later applied it to te task of visual object tracking. Recently, Ning et al. proposed modifications to make te mean sift tracker adaptive to orientation and scale [9]. In medical image processing, te mean sift algoritm was used for vessel segmentation in CT data [11] and for blood cell segmentation in images of blood smear [2]. In regard to tracking in US sequences, te mean sift was employed to myocardial border tracking [5]. An application to vessel tracking in US series of liver as, to our knowledge, not been presented before. 3 Te US image data was obtained from te CLUST 2014 MICCAI Callenge on Liver Ultrasound Tracking (ttp://clust14.etz.c/).

2 2 Metods Tis section gives an introduction to kernel density estimation, te general mean sift algoritm for visual tracking, te adaptations to tis algoritm as recently proposed by Ning et al. [9], and details on our implementation and modifications. 2.1 Kernel Density Estimation and te Mean Sift Given a set of samples assumed to be drawn from some probability distribution, kernel density estimation (KDE) is a metod to obtain a non-parametric estimate of te underlying probability density function. Te kernel density estimator ˆf (x) at location x R D of a function f is ˆf (x) = 1 N D N ( ) x xi K, (1) were N is te number of samples x i R D witin te kernel K wit window size. Using te kernel profile k of te radially symmetric kernel K wic satisfies K(x) = c k k( x 2 ) (c k is a normalization factor), Equation (1) can be rewritten into ˆf (x) = c k N D ( N x x i 2) k. (2) Te output of KDE is a function tat is a smooted representation of te given sample distribution (cf. Figure 1a) and can intuitively be understood as a generalization of weigted istograms (cf. Figure 1b). To find modes in a given KDE, mean sift procedures can be applied [3,7]. Te mean sift is a gradient ascent on te gradient of te density estimate ˆf (x) = 2 g(x)= k (x) = 2 = 2 c k N D+2 c k N D+2 c k N D+2 ( N (x x i )k x x i 2), (3) ( N x x i 2) ( x i g c k N x x i 2 N D+2 xg (4) ( N x x i ) ( 2 N g x x x ig i 2) ( N g x x i 2) x. }{{} =m K (x) (5) 2 ) Te second term in Equation (5) is also referred to as te generalized mean sift vector m K (x). We can find modes in te gradient of te density estimate

3 0.25 Kernel Density Estimate 20 Histogram (a) Gaussian KDE (blue) of 100 samples (black) drawn from two sifted normal distributions. Kernel weigts for eac sample point are indicated in red. (b) A istogram of te same set of samples as in (a). Fig. 1: KDEs are a generalization of weigted istograms. obtained by kernel K by iteratively sifting te center of K from an initial location by m K (x). Wen mode-seeking is applied to images, were pixels form a regular grid te generalized mean sift as to be extended to introduce te notion of pixel density. Tis can be acieved by employing a weigted mean sift, were eac pixel location x i is assigned a weigt w i derived from, for instance, te pixel s intensity. 2.2 Mean Sift for Tracking in Ultrasound Sequences To apply te mean sift mode-seeking procedure to visual tracking te tracking target is selected in te first frame and represented in a suitable feature space to obtain a model of te target. For eac subsequent frame a weigt image is computed by assigning a weigt to eac pixel wic depends on te probability of te pixel belonging to te target. On tis weigt image, wic is also referred to as a target confidence map, te mean sift algoritm is initialized wit te target location in te previous frame and an appropriate kernel size. After mean sift convergence te found mode is taken as te target location in te current frame. For mean sift tracking in US sequences we used normalized weigted intensity istograms to represent te target model q = {q u } u=1...m and te target candidate model p(y) = {p u } u=1...m at location y, were m is te number of bins. Te weigts tat determine te contributions of eac pixel to a istogram bin u are based on a radially symmetric kernel K. For te target model location

4 y = (0, 0) and kernel size = 1 is assumed by using normalized pixel locations x i : q u = 1 C N k( x i 2 ) δ[b(x i ) u], (6) }{{} =1 I t(x i)=u were δ is te Kronecker delta function, k te kernel profile of K and b(x) a function tat maps te image intensity at location x to a bin number. C is te sum of te kernel weigts at all locations suc tat te sum of all q u is 1. For te target candidate model in te current frame te same model representation is used, but wit te kernel of size sifted to te current target location y: p u (y) = 1 C ( N y x i 2) k δ[b(x i ) u], (7) were C is te sum of te kernel weigts of all locations on te regular pixel lattice witin te kernel wit window size. Intuitively, te istograms q and p give te probability of a pixel s intensity belonging to te target and te target candidate model, respectively. Te kernel assigns smaller weigts to pixel locations farter away from te center. Tis increases robustness since pixels closer to te center are also closer to te target center and pixel locations close to te target center offer more reliable features due to, for instance, canges in te appearance of te target propagating from its boundaries towards te center. Since te aim in te current frame is to find te target candidate model p(y) tat best matces te target model q, a similarity metric is introduced next. We follow Comaniciu et al. [4] and use te discrete Battacaryya coefficient [8] ρ(y) to compare te target model q and te candidate model p(y) at location y: ρ(y) = ρ [p(y), q] = m pu (y)q u. (8) Intuitively, te Battacaryya coefficient is a measure for te amount of overlap between two sample distributions. Based on ρ(y), a distance metric d between p(y) and q can be defined as d(y) = 1 ρ(y). u=1 d(y) = 1 ρ(y). (9) Practically, d(y) is minimized by maximizing ρ(y). In eac frame t, te procedure to find te location ŷ tat maximizes ρ(y) is started at location ŷ 0, wic in te beginning is set to te position of te target in te previous frame ŷ t 1. By linearization troug a Taylor series expansion around y 0, ρ(y) can be approximated as

5 Fig. 2: Illustration of te set-up for model representation and comparison. Te target model q at location 0 is computed in te first frame. In frame t, te target candidate model p(y) at location y is computed. Te m-bin istograms are compared using te Battacaryya coefficient ρ. Te mean sift procedure is used to find a ŷ t tat maximizes ρ. ρ(y o ) 1 2 ρ [p(y 0), q] (7) = 1 2 ρ [p(y 0), q] = 1 2 ρ [p(y 0), q] }{{} independent of y + m qu p u (y) p u (y 0 ) ( m 1 N k u=1 y x i 2) C u=1 δ[b(x i ) u] }{{} p u(y) ( 1 N y x i 2) w i k 2C, }{{} KDE obtained wit kernel K at location y q p(y 0 ) (10) (11) (12) were w i = m u=1 qu δ [b(x i ) u] p u (y). (13) Te first term in Eq. (12) is independent of y, wereas te second term is a KDE obtained using te kernel K at location y and weigts w i, wic can be maximized using te mean sift algoritm (cf. Section 2.1). Maximizing tis KDE means maximizing te Battacaryya coefficient, wic finally leads to te minimization of te distance between p(y) and q.

6 Te mean sift iteration step to move te kernel center position from ŷ 0 to te new position ŷ 1 is ŷ 1 = ( x ŷ iw i g 0 x i 2) ( n w ŷ ig 0 x i 2). (14) n A favorable coice for te kernel K is te Epanecnikov kernel [6] { 1 D+2 K E (x) = 2 c D (1 x 2 ) if x < 1, (15) 0 else were x R D and c D is te volume of te D dimensional unit spere, e.g. c d = 2 for D = 1 and c d = π for D = 2. Te Epanecnikov kernel minimizes te mean integrated squared error between te KDE and te true density [10] and, since te profile of K E is alf-triangular { 1 D+2 k E (x) = 2 c D (1 x) if x < 1, (16) 0 else we see tat g(x) = k (x) = 1 and Equation (14) can be reduced to ŷ 1 = n x iw i n w. (17) i After eac mean sift iteration, convergence is cecked based on maximum number of iterations and minimum lengt of te mean sift vector. In case of convergence ŷ t is set to ŷ 1, oterwise ŷ 0 is set to ŷ 1 and te mean sift procedure is repeated. 1.2 KE (Epanecnikov kernel) ke (Epanecnikov kernel profile) k E (Derivative of Epanecnikov kernel profile) Fig. 3: Te derivative of te Epanecnikov kernel is uniform.

7 2.3 Scale and Orientation Adaptive Mean Sift Tracking In te original mean sift tracking algoritm te kernel window size and orientation remains fixed. Tis is unfavorable wen tracking a target tat canges its size and orientation over te course of te image sequence. We follow modifications proposed by Ning et al. [9] to make te procedure adaptive to scale and orientation. To tis end, first te target s scale is estimated, wic is te area in te target searc region occupied by te target. Te estimated area is ten used to adjust an ellipsoid target descriptor to matc te current widt, eigt, and orientation of te tracking target. Estimating te Target s Scale In eac frame t te kernel center position is initialized wit te target s position in te previous frame ŷ t 1. Also te kernel is sligtly enlarged by a factor d, enabling te algoritm to capture a tracking target tat increased in size since te last frame. Since te weigt w i for eac pixel witin te increased searc region (cf. Equation (13)) gives te likeliood of te pixel being part of te target, te sum of all weigts (i.e. te 0 t order image moment of te searc region in te weigt image or target confidence map) M 00 = N w i (18) is a good initial approximation of te area of te searc region covered by te tracking target. However, if background features are present in te target searc region, te weigts of pixels witin te searc region tat belong to te target are amplified. Tis is because te probability of target features in te target searc area is decreased in te presence of background pixels, wic, as per Equation (13) (te target candidate intensity distribution p(y) is in te denominator), increases te weigts of target pixels. Terefore, te 0 t order moment overestimates te size of te tracking target in case background features are present. On te oter and, te Battacaryya coefficient between te target model q and te target candidate model p(y) is a measure for ow many target and background features are in te current searc region. Terefore Ning et al. [9] proposed to use te Battacaryya coefficient to adjust te 0 t order moment approximation for te target scale. Te estimated area is computed as ( ρ ) Â = exp M 00, (19) σ were σ is a parameter tat governs te magnitude of adjustment of te M 00 estimate given a certain Battacaryya value. In te experiments described below σ was empirically set to 0.2.

8 Fig. 4: Ellipsoid target descriptor. Estimating te Target s Orientation For estimating te target s orientation an ellipsoid image descriptor (cf. Figure 4) is introduced wic is defined by a covariance matrix based on te first and second order central image moments ( ) µ Cov = 20 µ 11 µ 11 µ, 02 (20) were N µ pq = i,1 x 1 ) p (x i,2 x 2 ) q w i N w, i (21) were ( x 1, x 2 ) is te kernel center position. An ortogonal decomposition of Cov [ ] [ ] [ ] Cov = U S U T u11 u = 12 λ 2 T 1 0 u11 u u 21 u 22 0 λ 2 12 (22) 2 u 21 u 22 yields te semi-minor and semi-major axis of te target descriptor as column vectors in U and te aspect ratio a b = λ1 λ 2 troug te singular values in S. Subsequently, a scaling factor k can be introduced suc tat a = kλ 1 (23) b = kλ 2. (24) Introducing te previously estimated target area Â (cf. Section 2.3) and te general area formula for an ellipse, we can furter derive Â = πab (25) = π(kλ 1 )(kλ 2 ) (26) k = Â, πλ 1 λ 2 (27) wic finally allows us to adjust te ellipsoid descriptor based on te estimated target scale: [ ] [ ] Cov = U S U T u11 u Âλ1 [ ] T = 12 πλ 2 0 u11 u 12. (28) u 21 u 22 0 u 21 u 22 Âλ 2 πλ 1

9 2.4 Tracking Failure Recovery Tracking may be lost over te course of te US sequence due to, for instance, drastic cange of appearance of te tracking target, too large target displacements between frames, or erroneous estimation of te target s scale and orientation leading to a disadvantageous searc area. Based on te assumed periodicity in te motion of liver vessels induced by respiration we integrated strategies to detect frames in wic tracking performance is problematical and to recover from tese situations. A first ceck is based on te analysis of te Battacaryya coefficient ρ. If it drops below 0.8 we discard te found target position and instead use te target position from te previous frame. Furtermore, te searc region is reset to te one set by te user in te first frame. If tis ceck triggers twice in two subsequent frames, te target position is reset to te centroid of te searc area selected in te first frame. A second ceck is based on te analysis of te estimated target size. If te target searc area in te current frame is found to be larger tan tree times te initial target s size, te searc area and its position is reset to te one set in te first frame. Tese failure recovery strategies are rater crude but were found to only be triggered in rare situation were tracking would oterwise fail completely. 3 Results Te scale and orientation adaptive mean sift procedure was applied to track 54 vessel targets in 21 2D US sequences of liver acquired from volunteers under free breating. In total, te sequences comprised frames. Te overall mean tracking error (MTE) was 1.43 mm wit a standard deviation (SD) of 1.22 mm and 95t-percentile 3.67 mm. Te minimum tracking error over all frames and tracking targets was 0.01 mm, te maximum tracking error mm. Te algoritm was developed in MATLAB Release 2013b, and te experiments were conducted on a macine equipped wit an Intel i5-3320m processor at 2.6 GHz clock speed and 8 GB RAM. Tracking speed using tis ardware set up was about 20 Hz. Table 1 gives an overview of te data set and te results obtained. Fig. 5 gives a visual impression of te tracking of tree vessels in series MED-02. Fig. 5: Ellipsoid target descriptors of tree tracking targets overlaid on four frames of te US sequence MED-02. 4

10 Sequence information Results Sequence Targets Frames Resolution Probe freq. FPS MTE SD 95% Min Max [mm/px] [Hz] [Hz] [mm] [mm] [mm] [mm] [mm] ETH ETH ETH ETH ETH ETH ETH ETH ETH All ETH sequences MED MED MED MED MED MED MED MED MED MED MED MED All MED sequences All sequences Table 1: Overview of te data set comprising 54 vessel targets in 21 US sequences. Te table gives te sequence identity, te number of targets, te image resolution, te probe frequency, te sampling frequency (FPS), te mean tracking error (MTE), te standard deviation of te tracking error (SD), te minimum tracking error (Min), and te maximum tracking error (Max). 4 Conclusion Tracking of vessel targets in 2D US series of liver under free breating using a scale and orientation adaptive kernel-based tracking algoritm is feasible, fast, robust and precise. For future work te incorporation of a target descriptor tat is adaptive to te outline of te tracking target seems wortwile. By suggestion of one reviewer of an initial draft of tis article, we will also look into making 4 Link to video: ttp://campar.in.tum.de/files/benz/clust2014/med-02.webm

11 te istogram representations adaptive to global illumination canges. Te application to native 3D US is also desirable. Furtermore, oter feature spaces for model representation could be investigated wit a focus on, for instance, gradient-based descriptors and joint-istograms. Finally, a more sopisticated failure recovery strategy based on, for instance, a collection of keyframes or predictive motion regularization could be advantageous. References 1. Ceng, Y.: Mean sift, mode seeking, and clustering. Pattern Analysis and Macine Intelligence, IEEE Transactions on 17(8), (1995) 2. Comaniciu, D., Meer, P.: Cell image segmentation for diagnostic patology. In: Advanced algoritmic approaces to medical image segmentation, pp Springer (2002) 3. Comaniciu, D., Meer, P.: Mean sift: A robust approac toward feature space analysis. Pattern Analysis and Macine Intelligence, IEEE Transactions on 24(5), (2002) 4. Comaniciu, D., Rames, V., Meer, P.: Kernel-based object tracking. Pattern Analysis and Macine Intelligence, IEEE Transactions on 25(5), (2003) 5. Comaniciu, D., Zou, X.S., Krisnan, S.: Robust real-time myocardial border tracking for ecocardiograpy: an information fusion approac. Medical Imaging, IEEE Transactions on 23(7), (2004) 6. Epanecnikov, V.A.: Non-parametric estimation of a multivariate probability density. Teory of Probability & Its Applications 14(1), (1969) 7. Fukunaga, K., Hostetler, L.: Te estimation of te gradient of a density function, wit applications in pattern recognition. Information Teory, IEEE Transactions on 21(1), (1975) 8. Kailat, T.: Te divergence and Battacaryya distance measures in signal selection. Communication Tecnology, IEEE Transactions on 15(1), (1967) 9. Ning, J., Zang, L., Zang, D., Wu, C.: Scale and orientation adaptive mean sift tracking. Computer Vision, IET 6(1), (2012) 10. Scott, D.W.: Multivariate density estimation: teory, practice, and visualization, vol Jon Wiley & Sons (2009) 11. Tek, H., Comaniciu, D., Williams, J.P.: Vessel detection by mean sift based ray propagation. In: Matematical Metods in Biomedical Image Analysis, MM- BIA IEEE Worksop on. pp IEEE (2001)