Using Time Series Analysis to Visualize and Evaluate Background Subtraction Results for Computer Vision Applications

Transcription

1 Using Time Series Analysis to Visualize and Evaluate Background Subtraction Results for Computer Vision Applications Samah Ramadan Computer Science Department University of Maryland College Park, MD ABSTRACT Information visualization has proved its significance and importance for exploring and analysis of complex data. This paper introduces a novel framework for applying time series analysis and visualization methods to evaluate and compare the results of background subtraction algorithms for computer vision applications. Instead of judging the results of background subtraction algorithms by just viewing the resulted binary sequences which might not give enough insights about the results, we apply time series analysis and visualization to provide more details and insights of the results. The objective is to evaluate the results and possibly help to identify the weaknesses and strengthes of background subtraction algorithms which leads to further improvements for the performance of these algorithms. Keywords: Time series, Background Subtraction Visualization,Information Visualization, Autoregression. 1. INTRODUCTION Time series data arise in virtually all fields of science and business. For example, in medicine, it is critically important to visualize and query Electrocardiogram (ECG) data. Another example is stock market data, which requires very accurate visualization and analysis in order to understand market trends and predict future stock market events. In computer vision, time series analysis is important in applications such as activity recognition and motion analysis. Unfortunately, many fields have not benefitted from the wide range of visualization and analysis techniques developed by the information visualization community. There has been very limited interaction between the computer vision community and information visualization community. This paper crosses the gab between the two communities to introduce a novel approach to evaluate and compare different background subtraction algorithms using time series visualization, in an attempt to build a connection between information visualization in its pure meaning and computational intelligence. This paper aims to address two issues. The first issue is to compare and evaluate the performance of the background subtraction algorithm based on its results given some test video sequences. Visualization is used to examine the evolution of a given pixel or an image area. Results of the background subtraction algorithm being evaluated are compared against both the time series of the pixel and another supposedly robust (ground truth) background subtraction algorithm.three levels of visualization are allowed. The first level of visualization is an non-interactive overview visualization that simply demonstrates the time series for the selected pixel or are. The second level of visualization is to give more insights about the data through interactive time series visualization for the area of interest inside the image. The third level of visualization is to compare the results on the pixel granularity and provide more detailed visual analysis about this pixel time series. The second issue this paper addresses is using time series analysis methods to model the background. Autoregression is applied to predict future pixel value form its previous values (history.) During the training phase the system builds a history of each pixel in video which represents a time series. In order to detect moving objects in a new frame, the new pixel intensity is forecasted based on its history and this value is compared to the true intensity of the pixel. If the difference between the expected intensity value and the real one is above a specific threshold then an inference can be made that the pixel being tested is a part of a foreground moving objects. Otherwise, the pixel is another sample of the background and can be used to update the time series pixel history. This paper is organized as follows. Section 2 briefly summarizes the related work in time series visualization and background subtraction. The proposed evaluation and visualization method is discussed in Section 3. In Section 4 we present some evaluation and visualization results. Finally, Section 5 concludes the paper and introduce some directions for future work.

2 2. RELATED WORK 2.1 Time Series Visualization and Analysis Time series visualization provides an efficient and compact means to explore, analyze and compare time series data. The research in time series visualization can generally be categorized into four main areas. Namely, these areas are anomaly detection, time series query, time series classification and time series clustering. In [11], Keogh et al. present a data compression-based similarity measure algorithm which they use in different domains such as anomaly detection and clustering. As an example for anomaly detection, they used a sequence of images extracted from a video clip, showing an actor drawing a gun, aiming it at a target and then returning the gun to its holster, as a normal behavior time sequence. To detect anomalies, the actor was asked to repeated the sequence over and over, until at some point the actor misses the holster and laughs. The algorithm detects missing the holster and laughing as an anomaly. Wavelet transform is used by Shahabi et al. [15] to create what is called Trend and Surprise Tree (TSA-tree.) A TSA-tree is used to analyze and search time series data for anomalies at different levels of abstraction. In the time series clustering area, Kalpakis et al. [8] present an algorithm for clustering Auto-Regressive Integrated Moving Average (ARIMA) time series. The algorithm is based on modelling the cepstral coefficients of the time series using a linear predictive model and computing the Euclidean distance between representations of different signals in order to classify them. Struzik and Siebes [16] introduced a Wavelet Transform based method for finding features of interest at different scales from time series data. An interesting comparison between different similarity measures when applied to stock market data can be found in [5]. From an application perspective, time series visualization has been applied in several fields. Keogh et al. [10] applied time series methods to finding the most unusual subsequence of a much longer electrocardiogram time series. They added two heuristics to the brute force search algorithm to find the most unusual subsequence of a predefined length. In [2], Chan and Mahoney propose an algorithm to model a set of normal time series and a scoring method to compare new time series to the model. They apply the proposed algorithms to NASA valve data set. The data consists of a set of normal valve opening and closing currents another set of abnormal opening and closing sequences. Also for aerospace applications, Chiu et al. [3] developed algorithms for finding repeated subsequences of time series, which they call Motifs. They tested their algorithms on the space shuttle telemetry signals and they showed robust performance. Tools have been developed to interactively visualize time series data and querying them. Timesearcher [14] is a tool that enables the user to visualize and search different time series interactively by the means of Timeboxes [7] and Variable Time Timeboxes [9]. Timesearcher enables the user to search multidimensional time series, filter the data to reduce scope. Also it has zooming capabilities and similarity search features. Another time series data visualization tool is BinX [1]. BinX enables the user to dynamically cluster the time series data as the level of viewing details changes. Lin et al. [13] developed VizTree, a tool to visualize large scale time series in an online fashion. VizTree transforms the time series into a symbolic representation and then encodes this representation in a suffix tree. The frequency of time series patterns are mapped into thickness and colors of branches. The tool also enables the user to conduct subsequence matching, motif discovery and anomaly detection. This tool presents a unique representation for comparing two time series through DiffTree. 2.2 Background Subtraction Background subtraction is a critically important step in many computer vision applications, such as visual surveillance and activity recognition. The main idea behind background subtraction is to subtract an image containing some moving objects from another image representing a model of the background of the scene. Ideally, the result is an image containing only the moving objects. However, due to several sources of noise, such as lighting, this process does not yield ideal results. This motivates researchers to investigate other more sophisticated methods for performing the background subtraction step. There is an abundance of publications in this area. Following, we will describe a few of the sophisticated, yet popular, methods for background subtraction algorithms. Elgammal et al. [4] uses kernel density estimation to compute probability density function for the color of each pixel individually. At subtraction time, a pixel is classified as a foreground pixel if the probability is less than a certain threshold. Stauffer and Grimson [17] took a slightly different modeling approach by modelling the color of each background pixel as a mixture of Gaussian distributions. During the subtraction time, they computed the probability that a pixel is a background pixel and thresholded the result in a similar fashion to [4]. They also proposed a method for dynamically adapting the background model over time. A major drawback of algorithms such as that of Stauffer and Grimson [17] is that they fail when fast light changes occur in the scene. Rather than using only color information in the mixture of Gaussians model, Tian et al. [18] augment the model by including texture and intensity information. Their results show improved performance when fast light changes occur. Kim et al [12] introduced a real time algorithm for background subtraction by building a compressed background model. The model is build using codebook representation for each pixel intensity value.the model built was able to capture the underlying background structure even for dynamic backgrounds. Each pixel in the model is represented

3 by set of codewords. Quantization/ clustering technique is applied to extract the codeword for building the codebook. 3. PROPOSED METHOD The best way to measure the accuracy and robustness of a background subtraction algorithm is to apply it to video sequences and see how it will perform. However, if the video sequence has lots of details, it might be difficult to perceptually judge the performance for small areas that do not show well on the regular computer monitor. For this reason, it is intuitive to go to the pixel granularity to evaluate and compare different background subtraction results. Since video sequences are essentially sequences of frames with a specific frame rate, the intensity of one or more pixels as it evolves from one frame to the next can be thought of as a time series. Visualizing the time series of pixel intensity and compare between time series of corresponding pixels in two different background subtraction algorithms provides an easy, yet effective, way of evaluating and comparing the results of different background algorithms when put together under the same evaluation metrics. In this paper different visualization tools are employed to evaluate and compare the results of different background subtraction algorithm. Three different strategies are adopted in order to assess the performance of a given background subtraction algorithms. For simple mismatches, a visualization overview is provided by just plotting the time series with no further interaction. If the user is unsure about the results of the background subtraction, then we use TimeSearcher [7] for more accurate visualization and analysis of the time series. TimeSearcher allows for more interactive visualization and further investigations such as trends, similarities and anomaly spotting. In order to interface with TimeSearcher, our software automatically exports the time series data of the original pixel intensity as well as the output of the two background subtraction algorithm being compared to TimeSearcher by reformatting the data into the appropriate format and modifying the initialization file of TimeSearcher to use the exported data. On the other hand, in order to detect motifs in the original pixel time series, VizTree [13] is used. Motifs in the original pixel data simply indicate the existence of a foreground object. VizTree is also utilized to compare between the results of two background subtraction results to measure how different or similar the results are. Our approach for assessing the performance of a given background subtraction algorithm is to apply the algorithm to a set of video sequences and compare the results against those of another algorithm that is known to perform robustly on the test set (i.e. ground truth results.) As a reference background subtraction algorithm, we use the codebook-based background subtraction algorithm developed by Kim et al. [12]. On the other hand, as an example of a background subtraction algorithm being assessed, we developed a simple algorithm based on autoregression. Let I t(x, y) be the intensity of an image pixel (x, y) at time t. The intensity at time t can be predicted using a linear combination of previous intensity values of the pixel as shown by Equation 1, I p t (x, y) = N a i I t i (x, y) (1) i=1 where N is the order of the autoregressive model and I t i(x, y) is the intensity at time t i. Given a sequence of pixels, the linear combination coefficients a i can be estimated by solving a system of linear equation. The details of computing a i are beyond the scope of this paper, however, for more information the reader is referred to [6]. The estimation process is repeated for every pixel in the training sequence. Since the training is a computationally expensive process, training is performed off-line for both the ground truth algorithm and the algorithm being tested. At testing time, the predicted intensity I p t (x, y) is compared to the true intensity I t(x, y) as shown by Equation 2 I t (x, y) I p t (x, y) > τ (2) where τ is a user-defined threshold. If the inequality of Equation 2 is not satisfied, this means that the predicted intensity is similar to the true intensity, and therefore no foreground objects exist. Otherwise, if the inequality holds true, this means the existence of a foreground object that is different in intensity than the background. 4. EXPERIMENTAL RESULTS The results of applying the two background subtraction algorithms are provided for a number of video sequences. The users can select the video sequence on which they want to evaluate the background subtraction algorithm. The users then can browse though the binary images representation the output of the two algorithms. This gives the users an overview of the performance of the background subtraction results. Since a high resolution video sequence will contain an enormous number of pixels ( pixels for a 720x480 grayscale video,) visualizing the time series of each pixel will be useless for the users. Therefore, we give the users the options to either visualize the time series of a selected pixel or an area around a selected pixel. So selecting the area of interest will perform as a filtering method to reduce the dimensionality of the pixel space. As shown in Figure 1 the users can select the video sequence that they want to evaluate the two background subtraction algorithms against. The users also select the frame range of the video. As mentioned earlier, the set of video sequences are preprocessed off-line. Figure 2 shows an example screen for comparing the results of the two background subtraction algorithms. The first step in comparing the two algorithm is for the users to browse the video sequence for an overview of the performance on the granularity of frames. As shown form the Figure, the users can preview the original video along with the results of the two background subtraction algorithm in order to obtain a perceptual assessment of the performance of the two algorithms. To focus the analysis on a particular pixel or area, the next step would be selecting a pixel or an area sur-

4 Figure 1: Main window of the background subtraction evaluation software. The users select one of the preprocessed video sequences and the range of frame on which they want to evaluate the algorithm Figure 2: The users can play the selected range of frames in order to make sure this is the desired range and/or video

5 Figure 3: The users can play the original video along with the videos for the background subtraction algorithm (top three controls, original video sequence, first background subtraction binary video results, second background subtraction algorithm binary video results). The users can view an non interactive overview of the time series of a selected pixel/area. Three time series shown (top: original video time series selected pixel(s), first background subtraction algorithm result time series selected pixel(s), and second background algorithm result time series for selected pixel(s). In this window, the users can also choose to visualize the data in TimeSearcher or VizTree for further detailed interactive visualizations. rounding a pixel to compare their time series visually. An interactive widget is provided to allow the users to first select a pixel of interest, then select the size of the neighborhood area around the chosen pixel. The area selection is also provided by another widget to choose from the range of single pixel neighborhood to whole image. After selecting the area of interest, the users can have a non interactive overview of the times series of the select area in original video along with the times series of the selected area in the two background subtraction results. This allows the users to conduct simple comparisons between the two time series. Figure 3 shows the non-interactive overview of the three time series: original video time series for intensity values of selected pixel(s), first background subtraction algorithm result time series for binary values of selected pixel(s), and second background algorithm result time series for binary values of selected pixel(s). For image areas that need more in detail analysis, the next phase is to allow the users to view an interactive visualization of the results using TimeSearcher 2.4[14]. By opting to visualize using TimeSearcher, the software automatically exports the intensities of the selected pixel/area of the original image and the background subtraction results to a TimeSearcher-specific data format. TimeSearcher is invoked automatically from the software and the initialization file is modified in order to load the data into TimeSearcher without any more user intervention. TimeSearcher is a very powerful tool for visualizing time series. It has many capabilities for previewing, zooming, filtering, dynamic queries and similarity search. As shown in Figure 4 the time series for the neighborhood of the chosen pixel can be visualized in original video and in the two binary results of the two background algorithms. As shown in the Figure, by comparing the three time series we can infer that a false alarm foreground was detected by the second background subtraction algorithm at frame number 52, while it was correctly classified as a background pixel in the first background subtraction algorithm. Figure 5 shows a utilization of some capabilities of Tmesearcher, where zooming and filtering are used. Zooming is used to focus on the frames that has a foreground object. While Filtering is used to reduce the visualized time series to the ones that meet the chosen range. To provide more insights of the results on the pixel granularity, VizTree [13] is used to visualize the detection of a foreground object as shown in Figure 6. The idea of VizTree is to transform the time series into a symbolic representation then encode the symbolic representation in a suffix tree where the frequency of the patterns is encoded is the thickness of the branches of the tree. The foreground object here is detected as an anomaly for the intensity values of the selected pixel. The anomaly was captured by the small thickness of tree branches, while frequent values are represented by thick branches. VizTree is also used to compare between two background subtraction results on the granularity of a single pixel comparison buy DiffTree representation. DiffTree provides a very unique representation to compare between two time series by comparing their two suffix trees representation. DiffTree encodes the difference in frequencies (which was originally coded by the thickness of tree branches) in colors to indicate over-representation or under-representation of patterns is each time series. Figure 7 shows an example DiffTree for the two background subtraction results for the chosen pixel. The surprising pattern is represented by red color in both the DiffTree representa-

6 Figure 4: TimeSearcher is used for an in-depth and interactive analysis of the time series data for the chosen pixel/area. Time series for the selected area in the three videos (original video, first background subtraction algorithm result video, and second background subtraction algorithm result video)the Y-axis represent the intensity values of the pixels, while the X-axis represents the frame number. Figure 5: Zooming and filtering capabilities in TimeSearcher are utilized to focus on interesting frames (zooming) and pixels (filtering.)

7 Figure 6: VizTree detects a foreground object as an anomaly. This is indicated by the thin branch on the top of the tree, and the associated red marked signal. Figure 7: DiffTree result of the two trees representation for the results of the two background subtraction algorithms. The difference of the two results shown in bright red color, phase difference shown in dark red color

8 tion and in the signal representation. 5. CONCLUSION AND FUTURE WORK In this work, we introduce a novel approach for evaluating and comparing the results of background subtraction algorithms. We employed some information visualization methodologies to solve a fundamental computer vision problem. The new approach allows the user for an easy and intuitive way to visually evaluate and compare the results of background subtraction algorithm. Three different visualization methods have been employed. First, simple plotting of pixel intensities and subtraction results is used to provide an overview of changes of of the foreground/background values over time. TimeSearcher is used to perform in-depth analysis of the results of the subtraction algorithms being compared. Finally, VizTree is used to verify the existence of foreground objects and to compare the results of background subtraction results using tree representation.. We plan to implement the SAX representation and suffix trees internally in our software in order to improve the visualization capabilities of the proposed method. For the sake of comparing a new algorithm to a ground truth one, we developed an autoregression-based background subtraction algorithm. The algorithm depends on modelling the intensity of a given pixel as a time series and estimating a number of autoregression parameters. The difference between the predicted values using the estimated parameters and the true value is used to indicated the existence of a foreground object. For future work we plan to improve the autoregression background subtraction algorithm to provide more robust results that can provide a better guidance in visually evaluating other background subtraction algorithms. Instead of using intensity values, we plan to use red, green and blue data channels to improve the subtraction results. Also, non-linear autoregression and more advanced time series modelling approach will be investigated. 6. REFERENCES [1] L. Berry and T. Munzner. Binx: Dynamic exploration of time series datasets across aggregation levels. In Information Visualization, [2] P. Chan and M. Mahoney. Modeling multiple time series for anomaly detection. In 5th IEEE International Conference on Data Mining, [3] B. Chiu, E. Keogh, and S. Lonardi. Probabilistic discovery of time series motifs. In 9th ACM SIGKDD, [4] A. Elgammal, D. Harwood, and L. Davis. Non-parametric model for background subtraction. In IEEE 7th International Conference on Computer Vision, [5] M. Gavrilov, D. Anguelov, P. Indyk, and R. Motwani. Mining the stock market: Which measure is best? In Proceedings of the 6th ACM SIGKDD, [6] J. D. Hamilton. Time Series Analysis. Princeton University Press, [7] H. Hochheiser and B. Shneiderman. Interactive exploration of time series data. In Proc. 4th International Conference on Discovery Science, [8] K. Kalpakis, D. Gada, and V. Puttagunta. Distance measures for effective clustering of arima time-series. In Proceedings of the IEEE International Conference on Data Mining, [9] E. Keogh, H. Hochheiser, and B. Shneiderman. An augmented visual query mechanism for finding patterns in time series data. In Proc. 5th International Conference on Flexible Query Answering Systems, [10] E. Keogh, J. Lin, A. Fu, and H. V. Herle. Finding the unusual medical time series: Algorithms and applications. IEEE Trans. on Information Technology in Biomedicine, To Appear. [11] E. Keogh, S. Lonardi, and C. Ratanamahatana. Towards parameter-free data mining. In Proceedings of tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, [12] K. Kim, T. H. Chalidabhongse, D. Harwood, and L. Davis. Real-time foreground-background segmentation using codebook model. Real Time Imaging, 11(3), June [13] J. Lin, E. Keogh, S. Lonardi, J. Lankford, and D. N. Nystrom. Vistree: A tool for visually mining and monitoring massive time series databases. In Proc. 30th VLDB Conf., [14] C. P. A. K. P. Buono, A. Aris and B. Shneiderman. Interactive pattern search in time series. In Proc. Conference on Visualization and Data Analysis, VDA, pages , Washington DC, [15] C. Shahabi, X. Tian, and W. Zhao. Tsa-tree: A wavelet-based approach to imporove the efficiency of multi-level surprise and trend queries on time series data. In 12th International Conference on Scientific and Statistical Database Management (SSDBM), [16] Z. Struzik and A. Sibes. Measuring time series similarity through large singular features revealed with wavelet transformation. In Proceedings 10th International Workshop on Database and Expert Systems Applications, [17] C. Stuaffer and W. E. L. Grimson. Adaptive background mixture models for real-time tracking. In IEEE Conference on Computer Vision and Pattern Recognition, [18] Y. Tian, M. Lu, and A. Hampapur. Robust and efficient foreground analysis for real-time video surviellance. In IEEE Conference on Computer Vision and Pattern Recognition, 2005.