Ontology Based Video Annotation and Retrieval System

Transcription

1 Ontology Based Video Annotation and Retrieval System Liya Thomas 1, Syama R 2 1 M.Tech student, 2 Asst.Professor, Department of Computer Science and Engineering, SCT college of Engineering Trivandrum Abstract As the multimedia content over the internet is increasing day by day, efficient methods for retrieval of these huge amount of data is required. Video annotation is one of the widely used methods to analyze and retrieve these huge video data. The process of video annotation is complicated as it requires a large amount of processing to analyze the contents in the video. This paper introduces a video annotation system based on ontology with HoG features given for training the classifiers. The SIFT and HoG features of the images are extracted and used for training the classifiers for doing a comparison on performance of the classifiers. An analysis of the results is done to find which feature is better to train the classifier for getting more prominent annotated video database. Retrieval of the videos based on objects from the annotated video database is also done. Keywords- Key frame extraction, Feature extraction, Video annotation. I. INTRODUCTION The World Wide Web is the most important source of information now a days. The knowledge and information which we obtain from internet is in the form of textual data, images, videos etc. As the amount of information available in internet is increasing day by day, these huge databases of information are needed to be analyzed properly in order to get the required information. During the analysis of video database tremendous effort is required for the correct analysis and retrieval of the videos. Efficient browsing of the video database depends on the correctly retrieved or indexed videos from the database. Content based video search and retrieval is becoming more important as it helps for retrieval of videos based on the semantics in the video but it is a challenging problem as it needs to represent the semantics of the video. The content based video retrieval systems suffered from the problem of semantic gap-a gap that exists between low level features and high level features [1]. In order to bridge the gap, the solution is to define a large number of semantic detectors which detect the semantic concepts. These detectors try to learn the mapping between the low level visual features with the high level concepts from the video. The detection of the semantic concepts is still very difficult, so their detection in videos is a challenging problem. Annotation of the video can be used for analysis and retrieval of the video database. Annotation is an explanatory note or body of notes added to a text, diagram, document, image or video. Annotation files are generally extensible markup language (XML). These can be referred for the retrieval of desired image or video. Video annotation aims to assign several semantic & visual features to the contents of video. It is done to help semantic retrieval of videos from a large video database. Here an approach for video annotation based on ontology and machine learning has been introduced to perform more complete semantic annotation. Ontology is a formal representation of a set of concepts within a domain and the relationships between those concepts Most of Ontology-based information retrieval systems use the concepts mapping. Ontology helps in establishing links to other concepts and disambiguating the results of classification. Ontology based techniques are fast emerging and give better results than other techniques [2]. In this proposed approach the SIFT and HoG features from the frames are extracted and the classifiers are trained based on these features separately and the performance of these trained classifiers are analyzed. The key frames from the video are passed to the set of classifiers with highest performance. Then the annotation of the keyframes is done based on the ontology given. Considering that the videos have annotation, to access it only the annotation file have to be consulted in-order to determine whether the video is pertinent to the search, reducing overall access time. The rest of the paper is organized as follows: In section 2, the related works are discussed. The proposed work is explained in section 3. The results are discussed in section 4 and the conclusion is summarized in section 5. II. RELATED WORKS Video content annotation is gaining importance now a days for easy retrieval of required image/video from large number of available resources. At the early stages annotation of image/video content was done manually by user by navigating through various frames of videos. The need for automatic video annotation technique arose, as purely manual annotation requires intensive labor cost,. D. Chen, J. Odobez, H. Bourlard [3] proposed a segmentation method based on Markov random field to extract more accurate text characters. This methodology allows handling background gray-scale multimodality and unknown text gray-scale values. Support vector machine (SVM) is used for text verification followed by traditional OCR algorithm. 617

2 J. LU, Y. Tian, Y. Li, et al [4] proposed a framework for semantic video event annotation, which exploits global feature, local feature and motion feature. Jin-Woo Jeong, Hyun-Ki Hong, and Dong-Ho Lee [5], have presented an automatic video annotation technique which employ ontology to facilitate video retrieval and sharing process in smart TV environment. Multimedia contents are represented through a well-structured ontology. The visual features are extracted from MPEG-7 visual descriptors. These features are mapped to semi concept values. Semantic inference rules and support vector machine are used to detect the high level concepts. B. Vrusias, D. Makris, J. Renno [6] proposed a framework for ontology enriched semantic annotation of CCTV video. Visual and text semantics are linked with appropriate keywords provided by domain experts. Video segmentation is done to find moving objects, which are classified as agent, action and recipient. These visual semantics are annotated by keywords of CCTV ontology. Shih-Wei Sun, Yu-Chiang Frank Wang in [7] proposed a robust moving foreground object detection method followed by the integration of features collected from heterogeneous domains. More focus is on annotating rigid moving objects & considers videos with only one foreground object present. III. PROPOSED WORK The system overview and the modules in the proposed work are discussed in this section. A. System Overview Figure 1 shows the system overview. Various modules in the proposed work are discussed here. 1) Training Classifiers for SIFT and HoG features For recognition of objects present in the video we need to train different classifiers for object identification. For this features from the images that are invariant to image formation process and matching only to those features are extracted. SIFT approach transforms the image into an assortment of local feature vectors; these vectors are invariant to translation, scaling, rotation and partially invariant to illumination changes [8]. Another approach which is used here is HoG feature [9] extraction. Such features are similar to SIFT descriptors but take a dense sampling strategy. Machine learning technique is adopted to train the classifiers. i.e., first training the object detectors and then testing. Here five object classifiers are trained for both the SIFT and HoG features. Now SIFT and HoG features for some test images are calculated and are given to the corresponding trained classifiers. The object is then recognized. Support Vector Machine (SVM) is used as the classifier. It is a supervised technique which classifies into two classes. i.e., object present or absent. Now the classifier performance is analyzed for different classifiers with SIFT and HoG features. Figure 2 shows the procedure for performance analysis of trained classifiers [10]. Figure 1. System Overview In extracting the SIFT features, key points and descriptors from the images are found and then histogram of visual words is constructed from a set of images. These features are given as input to the object classifiers for training. Preprocess step consists of creating a vocabulary of visual words. In order to create the vocabulary of visual words a list of images from each of the categories and a set of images from the background image set are given as input. The features from each of these images are extracted and saved. The visual words from the extracted features are constructed by performing k-means algorithm. A k-dimensional tree of visual words is constructed and saved for easy querying. The computed visual words and the k-dimensional tree are saved in the vocabulary.mat file for future reference. Figure 2. Training classifiers In order to find the input to the classifier for training based on SIFT features, the vocabulary of visual words constructed in the pre-processing step is used. 618

3 The input images to be trained along with the vocabulary are used to find the histogram for each image. These histograms are saved as mat file for later use. Also the labels are to be calculated based on the positive and negative images to be given as input to the classifier. These inputs are given to the linear SVM classifier and are trained. The number of classifiers to be trained is based on the number of objects to be detected. Figure 3 shows the training procedure for a single object classifier based on SIFT Feature [10]. Figure 3. Training classifier based on SIFT features In extracting the HoG features the vocabulary of features from each of the training images are extracted. The training images should consist of both positive and negative images. The labels for these images should also be calculated. The vocabulary of features and the labels should be given to train the object classifiers. The training set for each of the classifiers should have positive and negative images in it. The different object classifiers should be trained based on the corresponding training set. Here also an SVM classifier is used. There are several key frame extraction algorithms that can be used. Key frame extraction algorithms select a subset of the most informative frames from videos. An algorithm which was proposed in [2] that uses edge difference to find the similarity between two frames is used to find the key frames. 3) Visual Feature Extraction from keyframes and Object identification The next step is the extraction of visual features from the keyframes in the video.as the video has been reduced to key frames, only the objects present in these frames have to be considered. The HoG features are extracted from the keyframes here. These features are given to the trained classifiers. All the keyframes are given to all the trained classifiers. If the objects are present, then the keyframes are given to the annotation module to perform the annotation of the keyframes. The presence of objects is identified by checking whether it belongs to the positive class or negative class. 4 ) Ontology based annotation and retrieval Ontology is simply a knowledge representation method which provides the way to consolidate the information in a structured manner [11].The transportation domain which is used is shown in Figure 5 [10]. The leaf node consists of the different objects identified by the classifier. For the objects identified by the classifiers, the annotation file for the corresponding keyframe is generated in XML format and it is saved in separate folder. Figure 4. Training classifier based on HoG features The performance analysis of the different classifiers based on SIFT and HoG features is done for a set of testing images and the performance of the classifiers is evaluated. Based on the evaluation the performance of the classifiers trained using HoG features is the best for the five different object classifiers trained in this work. So the classifiers trained in this video annotation system are using HoG features. 2) Keyframe Extraction Videos consist of large amount of frames. Video annotation and analysis based on these large amounts of frames is a complicated task. As analyzing all the frames in the video is a time consuming task it is better to analyze only the frames which represent the best visual features of the scene. So keyframe detection can be used to reduce the frames. Figure 5. Transportation domain ontology For the retrieval of the video, the object to be searched is given. The search for the object is done in the XML files saved. The videos for which the object is present are retrieved. Since the search for the videos is done only in the XML files, the retrieval process of videos is much easier. 619

4 IV. EXPERIMENTAL RESULTS The classifiers for five objects were trained using SIFT and HoG features. These classifiers were tested to see how they predict the objects. For this from each category a set of test images containing positive and negative images were given to each of the classifiers trained using different features. For all the objects, the classifiers which were trained using HoG features predicted more objects correctly. The number of negative images which were classified as positive (False Positive) were less in case of classifiers trained using HoG features when compared to those trained using SIFT features. Also the number of positive images which were predicted as positive (True Positive) were more in case of those classifiers trained with HoG features. Hence the precision and recall values which are calculated based on True Positive (TP) and False Positive (FP) values of each classifier, will be more accurate for the classifiers trained with HoG features. From the experimental analysis we came to the conclusion that the classifiers trained using HoG features were more correct when compared to those trained using SIFT features. The precision and recall for each of the classifiers can be calculated based on the following equations: Precision = TP/ (TP+FP); Recall = TP/ P; Where TP indicates True positive, FP indicate False Positive and P indicates total number of positives Figure 6. Precision values of classifiers SIFT HoG Figure 6 shows the precision values for the different classifiers used in this work. Y axis indicates the precision values. From this graph it is clear that the classifiers trained using HoG features gives better precision than those trained using SIFT features. Figure 7 shows the recall values for different classifiers. Here Y axis indicates recall values. From the graph it is clear that the values of classifiers trained using HoG is better than that of SIFT based classifiers Figure 7. Recall values of classifiers SIFT HoG Based on these results, the system used HoG features to train the classifiers for video annotation. The Keyframes were extracted from the video given and the HoG features for these frames were calculated. These features were given to the classifiers to identify the objects present in the frames. The annotations of the frames were done based on the objects identified. The annotated files were saved in separate folder for retrieval purpose. Based on the objects searched the videos containing the objects were retrieved. V. CONCLUSION Content based video analysis and retrieval has attained more importance as it helps for retrieval of videos based on the semantics of the video, but it is a challenging problem as it needs to consider the semantics of the video. As video annotation aims to assign several semantics & visual features to the contents of video, it has become an important technique now a days. Here a system for video annotation based on ontology which makes use of HoG features for training the SVM classifiers is proposed. Experimental results show that the performance of classifiers trained based on HoG features are best as compared to SIFT features. Since, only the keyframes are analyzed for the object identification, much time is saved when compared to analyzing all the frames in the video. As the annotation files created are based on ontology, the retrieval of the videos from these annotated databases is much easier as compared to the huge database of videos. This system can be expanded into other domains and the number of object detectors can be increased. 620

5 REFERENCES [1] L. Ballan, M. Bertini, A. Bimbo, G. Serra, Video Annotation and Retrieval Using Ontologies and Rule Learning, IEEE Multi Media, Vol. 17, issue 4, [2] K.Khurana, M.B.Chandak, Key Frame Extraction Methodology for Video Annotation, International Journal of Computer Engineering and Technology, Volume 4, issue 2, pp , March- April [3] D. Chen, J. Odobez, H. Bourlard, Text detection and recognition in images and video frames, Pattern Recognition 37, pp , [4] J. LU, Y. Tian, Y. Li, Y. Zhang, Z. Lu, A Framework for Video Event Detection Using Weighted SVM Classifiers, Artificial Intelligence and Computational Intelligence, AICI '09 International Conference, Vol. No.4, pp , [5] J. Jeong, H.Hong, and D. Lee, Ontology-based Automatic Video Annotation Technique In Smart TV Environment, IEEE Transaction on consumer Electronics, Vol. 57, No. 4, November [6] B. Vrusias, D. Makris, J. Renno, A Framework For Ontology Enriched Semantic Annotation of CCTV Video, Eight International workshop on image analysis for multimedia interactive Services, IEEE, [7] Shih-Wei Sun,Yu-Chiang Frank Wang, Automatic annotation of web videos, IEEE [8] D. G. Lowe, Object recognition from local scale-invariant features, In Computer vision. Proceedings of the International Conference on Computer Vision. 2, pp , [9] N. Dalal and B. Triggs. Histograms of oriented gradients for human detection. Computer Vision and Pattern Recognition, 1: vol. 1, June [10] Khushboo Khurana, Dr.M.B.Chandak, Video Annotation Methodology Based on Ontology for Transportation Domain, International Journal of Advanced Research in Computer Science and Software Engineering, Volume 3, Issue 6, June [11] N. Magesh, P. Thangaraj, Semantic Image Retrieval Based on Ontology and SPARQL Query, In proceedings of International Journal of Computer Applications (IJCA) ICACT, Number 1, pp.12-16, August