Semantic Video Annotation by Mining Association Patterns from Visual and Speech Features

Transcription

1 Semantic Video Annotation by Mining Association Patterns from and Speech Features Vincent. S. Tseng, Ja-Hwung Su, Jhih-Hong Huang and Chih-Jen Chen Department of Computer Science and Information Engineering National Cheng Kung University, Tainan, Taiwan, R.O.C. Abstract. In this paper, we propose a novel approach for semantic video annotation through integrating visual features and speech features. By employing statistics and association patterns, the relations between video shots and human concept can be discovered effectively to conceptualize videos. In other words, the utilization of high-level rules can effectively complement the insufficiency of statistics-based methods in dealing with broad and complex keyword identification in video annotation. Empirical evaluations on NIST TRECVID video datasets reveal that our proposed approach can enhance the annotation accuracy substantially. 1 Introduction Recent advanced multimedia capturing technology records our colorful living. To support great multimedia retrieval, content-based multimedia retrieval has been widely adopted as a method for organizing the huge amount of multimedia data into the repositories. Typically, a video can be divided into several scenes/stories and each scene contains a set of shots that consist of a few time-split/similarity-split image frames. From these sequential frames, a representative image frame will be defined as a key-frame. Due to the relations and rich contents of these sequential images, the annotation method for a video is very different from that for a single image [6]. In past studies, association rules were taken charge to annotate a video but in vain since the generated association rules may be too specialized to fit for a wide range of videos. That is to say, once the rule set is too small, we may not get sufficient matching rules to support video annotation. Hence, annotations by using only the specialized association rules possibly lead to high errors. With more considerations than association rules, the work in [8][9][10] took account of temporal continuity and used event detection to index and explore sequential association rules in the sequential keyframes. However, the results of sequential association rules are also limited in the range of video types. In addition to rule-based solutions above, CRM (Continuous Relevance Model) is the classic statistics-based method developed by Lavrenko et al. [1][4] for annotating videos. It segments each sequential key-frame into several rectangle regions and then extracts the referred visual features from these segmented regions. The annotations of each image are yielded soon after calculating the related probabilities with Gaussian Mixture Function. By exploiting the temporal continuity of video sequences and assuming Markovian property between image frames, DBNs 1

2 (Dynamic Bayesian Networks) proposed by Luo et al. [5] projected low-level features onto high-level concept space. In [6][7], Tseng et al. proposed hybrid methods for video annotation by integrating statistics-based and rule-based methods. In this paper, we present a hybridized solution for semantic video annotation by exploiting multi-contents of videos, such as visual features and speech features. The major contribution of the proposed method is that visual features, speech features and semantic patterns are considered simultaneously to enhance the accuracy of video annotation. The empirical evaluations reveal that the proposed approach can effectively arrange the relevant keywords to the video shots. The rest of this paper is organized as follows. In section 2, we demonstrate our proposed method for annotating videos in great detail. Experimental evaluations of the proposed methods are illustrated in section 3. Finally, conclusions and future work are stated in section 4. 2 Proposed Method The proposed method is basically extended from the work in [6][7]. As illustrated in Figure 1, the whole procedure contains two types of prediction models: Rule-based model (Model Vseq, Model Vasso and Model Sasso ) and Statistics-based model (Model CRM ). The details are described as the following subsections. Unknown Video Annotated Videos Speech Probability Sequence Association Speech Association M o d M oe dl M Ce o Rd l M eo Model CRM Model Vseq Model Vasso Model Sasso Fusion Annotations M V Fig. 1. Workflow of the proposed approach. se dl 2.1 Operation q Ve Functionally, the preprocessing operation can be viewed as a foundational stage that as l generates the necessary information used in the training phase and prediction phase.. This process is primarily for visual-based so S models. First, we perform shot detection to divide a video and combine several sequential shots to form a scene. Then, the representative key-frame of each shot is determined. as Second, each shot of a video has to be further divided into m*n rectangle so regions. These regions will be the basic elements for Model CRM. Third, scalable color and homogeneous texture are extracted from both the un-segmented shots and the segmented regions. Speech. This is a process for constructing speech-based model. A n n ot at io n 2

3 First, after the scene division, automatic speech recognition (ASR) [2] is triggered to transform audio features into text descriptions shot by shot for each divided scene. Second, IR techniques including Removing stop-words and Stemming words are employed to filter the usable speech words. Third, we utilize JwordNet [3] to project these filtered keywords onto the specific keyword space regulated by NIST. 2.2 Training Phase This phase is primarily concerned with the generation of four major models, namely Model CRM, Model Vasso, Model Vseq and Model Sasso. In this phase, three rule-matching matrices for building Model Vasso, Model Vseq and Model Sasso and a keyframe-matching matrix for building Model CRM are yielded by visual association rules, speech association rules and visual features, respectively. Construction of Model Sasso. The first task in this model [7] is to establish a transaction table for Model Sasso based on a presetting shot window. A shot window contains a sequence of shots, and the window slides along the scene. The target keywords (annotations) of each central shot of each sliding shot window can be used to form a transaction with the speech keywords of each shot of each sliding shot window. Assume that a shot window consists of 2z+1 shots where z 0 and win 0 is the central shot. Then the target keywords annotated in win 0 and the speech keywords to left z shots and right z shots form 2z+1 transactions. Construction of Model Vseq. In this model [6], we first discover the frequent itemsets from the scene-transaction table. These generated frequent itemsets can be viewed as association rules directly since temporal continuities are inherent in them. For example, the sequential association rule (A B) can be derived from frequent sequential itemset {A, B}. Next, each generated frequent itemset is used to seek for its associated keywords and the frequencies of keywords referred each frequent itemset are used to form the rule-matching matrix PL X W. Construction of Model Vasso. As mentioned above, the major difference between Model Vasso and Model Vseq is that Model Vasso ignores the temporal continuities of the frequent patterns [6]. In other words, the duplicate items have to be pruned in each tuple of scene-transaction table. 2.3 Prediction Phase As stated in the training subsection, three visual matching matrices are derived from three visual-based models that can represent the relations between key-frames and keywords and those between rules and keywords. Besides, speech association rules can reflect the relatedness between speeches and keywords. These derived matrices and rules, actually, can offer video annotation a great support. Prediction by Model CRM. This model is mainly based on the CRM method [1][4]. Prediction by Model Vseq. As soon as the scenes containing a sequence of unknown shots are sequentially received in our method, each shot within a scene has to be encoded first by computing the similarity (Eucilid Distance) between the shot and the clusters generated in the training phase. The prediction algorithm is discussed in [6]. Prediction by Model Vasso. In some cases, the temporal continuity of shots is not an essential factor for video annotation since we can get the better results without 3

4 considering the temporal continuity. Moreover, due to the temporal continuity is skipped, the related rule-matching matrix derived from Model Vasso differs from that derived from Model Vseq. The results are accordingly changing [6]. Prediction by Model Sasso. In our method, Model Sasso can really convey more important information than those of the other models since embedded speeches are always stably related to the referred shots. In this prediction [7], each shot is first preprocessed to generate its own speech keyword set. Next, these shots are sequentially predicted by looking for the relevant rules which left-hand itemsets are matched with the speech keywords within a specified sliding window. Finally, the average confidence of each annotation for each shot is generated. Prediction by Fusion Models. To integrate different viewpoints on four special prediction models, we design multiple fusion approaches to examine the annotation accuracy. Basically, the design of each fusion model is to take Model CRM as the foundational model and the others as the auxiliary models. Due to the high variations of videos, it is hard to represent all kinds of video just by the finite rules. Hence, the primary aim of this design is to avoid the missing-rule problem in rule-based models. In other words, by employing Model CRM, we can annotate any shot with at least one keyword whether joining with the rule-based models or not. Finally, the derived result of each prediction model is on the basis of its normalized Z-probability. The fusion models are defined as follows: Fusion 1 = Model CRM + Model Vseq Fusion 2 = Model CRM + Model Vasso Fusion 3 = Model CRM + Model Vseq + Model Vasso Fusion 4 = Model CRM + Model Vseq + Model Sasso Fusion 5 = Model CRM + Model Vasso + Model Sasso Fusion 6 = Model CRM + Model Vseq + Model Vasso + Model Sasso 3 Empirical Evaluation The experimental data came from the collection of TREC Video Retrieval Evaluation (TRECVID) provided by the National Institute of Standards and Technology (NIST). From the TREC videos, we chose four CNN and four ABC news videos as our experimental data. The total duration of the experimental data is around 233 minutes and the data size is about 3158MB. Moreover, there are 161 scenes and 1414 shots split in this experimental data set. The evaluation was investigated in terms of precision. In our experiments, we adopted the 8-fold approach to carry out the evaluations. That is, seven videos took turns as a testing video and the others were taken as training videos. Figure 2 shows that all of rule-based models, Model Vseq Model Vasso and Model Sasso, outperform CRM in terms of the precision, and Model Sasso performs better than any other model on average. This indicates that the speech rulebased models can effectively capture the intra-relations or inter-relations among the shots as we expect. In contrast, annotations by using only visual features encounter higher difficulty in dealing with high variations of visual features in the videos. Figure 3 reveals that the precisions for fusing visual features and speech features are significantly better than those of the models that consider only visual features or considering speech features individually. In other words, higher precision relies on the integration of all involved individual models. On average, our proposed fusion 4

5 precision precision method, Fusion 6, exhibits the improvements over CRM for about 335% on precision. Figure 4 reveals that Fusion 6 performs stably and outperforms the other hybrid fusion models under different Z thresholds. This delivers that correct answers are adequately strengthened by the integration of all individual models Sasso Vseq Vasso CRM CNN0202 CNN0204 CNN0205 CNN0207 ABC0218 ABC0319 ABC0324 ABC0329 Fig. 2. The precisions of CRM, Model Vseq, Model Vasso and Model Sasso CNN0202 CNN0204 CNN0205 CNN0207 ABC0218 ABC0319 ABC0324 ABC0329 Fusion1 Fusion2 Fusion3 Fusion4 Fusion5 Fusion6 Sasso CRM Fig. 3. The precisions of CRM and all fusion models. average precision threshold Fusion4 Fusion5 Fusion6 Fig. 4. The average precisions of Fusion4, Fusion5 and Fusion6 under different thresholds. 4 Conclusions and Future Work In this paper, we propose a novel method to exploit visual features and speech 5

6 features for video annotation by integrating statistics and association patterns. The utilization of high-level patterns can effectively complement the insufficiency of visual-based methods in dealing with complex and compound videos. As a result of the experiments, the proposed approach is shown to be very promising for video annotation through the integration of visual features and speech features. In the future, we will further investigate an adaptive fusion method by tuning the weight of each model. Acknowledgement This research was supported by Ministry of Economic Affairs, R.O.C., under grant no. 95-EC-17-A , and by National Science Council, R.O.C., under grant no. NSC H and NSC E References [1] S. L. Feng, R. Manmatha, and V. Lavrenko, Multiple Bernoulli Relevance Models for Image and Video Annotation, IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Vol. 2, pp , [2] K. Hacioglu and B. Pellom. A Distributed Architecture for Robust Automatic Speech Recognition. in Proc. of IEEE International Conference on Acoustics, Speech, and Signal Processing, (ICASSP '03) Vol. 1,pp , April [3] K. Johar and R. Simha, The George Washington University JWord 3.0, Avaliable at: [4] V. Lavrenko, S. L. Feng, and R. Manmatha, Statistical Models for Automatic Video Annotation and Retrieval, in Proc. of the International Conference on Acoustics, Speech and Signal Processing, May [5] Ying Luo, and Jenq-Neng Hwang, Video Sequence Modeling by Dynamic Bayesian Networks: A Systematic Application from Coarse-to-Fine Grains, in Proc. of IEEE International Conference on Image Processing, September [6] Vincent. S. Tseng, Ja-Hwung Su and Jhih-Hong Huang, A Novel Video Annotation Method by Integrating Features and Frequent Patterns, in Proc. of 7 th International Workshop on Multimedia Data Mining (held with KDD 06), Philadelphia, Pennsylvania, USA, August, [7] Vincent. S. Tseng, Ja-Hwung Su, and Chih-Jen Chen, Effective Video Annotation by Mining Features and Speech Features, in Proc. of the third International Conference on Intelligent Information Hiding and Multimedia Signal Processing, Kaohsiung, Taiwan, November 26-28, [8] Xingquan Zhu, and Xindong Wu, Sequential Association Mining for Video Summarization, in Proc. of the 4 th IEEE International Conference on Multimedia and Expo, Baltimore, USA, July [9] Xingquan Zhu, and Xindong Wu, Mining Video Associations for Efficient Database Management, in Proc. of 18 th the Internal Joint Conference on Artificial Intelligence, pp , August [10] Xingquan Zhu, Xindong Wu, Ahmed K. Elmagarmid, Zhe Feng, and Lide Wu, Video Data Mining: Semantic Indexing and Event Detection from the Association Perspective, IEEE Transactions on Knowledge and Data Engineering, Vol. 17, No. 5, May