Collaborative Feature Maps for Interactive Video Search

Transcription

1 Collaborative Feature Maps for Interactive Video Search Klaus Schoeffmann 1, Manfred Jürgen Primus 1, Bernd Muenzer 1, Stefan Petscharnig 1, Christof Karisch 1, Qing Xu 2, and Wolfgang Huerst 3 1 Klagenfurt University, Institute of Information Technology, Klagenfurt, Austria, {ks,mprimus,bernd,spetsch}@itec.aau.at, ckarisch@outlook.com, 2 Tianjin University, School of Computer Science and Technology, Tianjin, China, qingxu@tju.edu.cn 3 Utrecht University, Information and Computer Sciences, Utrecht, The Netherlands, {huerst}@uu.nl Abstract. This extended demo paper summarizes our interface used for the Video Browser Showdown (VBS) 2017 competition, where visual and textual known-item search (KIS) tasks, as well as ad-hoc video search (AVS) tasks in a 600-hours video archive need to be solved interactively. To this end, we propose a very flexible distributed video search system that combines many ideas of related work in a novel and collaborative way, such that several users can work together and explore the video archive in a complementary manner. The main interface is a perspective Feature Map, which shows keyframes of shots arranged according to a selected content similarity feature (e.g., color, motion, semantic concepts, etc.). This Feature Map is accompanied by additional views, which allow users to search and filter according to a particular content feature. For collaboration of several users we provide a cooperative heatmap that shows a synchronized view of inspection actions of all users. Moreover, we use collaborative re-ranking of shots (in specific views) based on retrieved results of other users. Keywords: video retrieval, interactive search, collaboration 1 Introduction & Related Work The Video Browser Showdown (VBS) is an annual live evaluation competition of interactive video search tools. It started in 2012 with visual known-item search (KIS) tasks in single videos, randomly selected from a set of 30 videos that were about one-hour long in duration, and has became increasingly challenging over the years. In 2014 the tasks were selected from 76 videos and extended by textual KIS tasks, where a textual description about the target scene is presented as a query instead of a visual excerpt [7]. The collection to search further increased in 2015 and 2016 to about 100 hours and 250 hours, respectively [2]. VBS 2017 (this year) is particularly challenging, because the data set increased even further, to 4593 video files with about 600 hours of content. In addition to the increase

2 in size, there is also a second type of querying, namely ad-hoc video search (AVS), which is the interactive version of AVS from TRECVID [6]. In contrast to KIS tasks which requests participants to find one particular 20 seconds long segment AVS tasks may have many results across the whole data set. We approach the VBS 2017 challenge with a highly flexible interactive video search system that combines several ideas from previous years and integrates collaborative search features. The main interface is a similarity-based map of keyframes (called Feature Map), which uses hierarchical refinement to provide an overview of keyframes with different levels of granularity, similar to the one used by the winner of VBS 2016 [1]. This Feature Map is presented in 3D perspective to better exploit the screen real estate and show more images at once. The last iterations of the VBS showed that none of the many different interfaces worked well for all of the tasks and session. Particularly text-based KIS tasks are very hard to solve with color-based search only. Similarly, it is often hard to derive the best matching semantic concept from a given visual example. Also, sometimes users need the temporal context of keyframes within the corresponding video sequence/file, or would rather like to temporally browse through the data instead of searching. Therefore, in addition to the Feature Map, our search system provides several other views, which can be used for search and filtering according to some specific content feature, for web-based search by example, or for simple temporal browsing of keyframes. All of these views are designed in a way that allows several users to search simultaneously (and cooperatively, if desired). This paper describes the general architecture of our interface, but omits many details due to the page limit. An elaborate technical paper detailing all the different parts of the interface and the underlying content analyses is under preparation for submission. 2 Proposed Approach As already mentioned in the introduction, we propose a collaborative video search system that uses several different interfaces (called views), where the user can select the most-appropriate one for the current search task and intent. 2.1 Feature Map The main user interface of our tool is the Feature Map (see Fig. 1). It is basically a two-dimensional grid of keyframes which are arranged based on similarity. The underlying user-selectable similarity metric can be any combination of the following four criteria (i.e., in total 15 different map arrangements can be selected): Concept similarity (CNN-Features) Color similarity (Feature Signatures) Texture similarity (Histogram of Oriented Gradients) Motion similarity (Motion Histogram)

3 The Feature Map is shown in a configurable 3D perspective view, which allows a better overview over a larger area than a flat 2D view. The Feature Map is built up hierarchically. The top layer shows approximately frames. Each of the four subjacent layer shows four times the amount of keyframes than its overlying layer. The lowest layer shows all the keyframes of the entire video collection, which are in total. Fig. 1: Feature Map: the main view of our interface arranges keyframes according to a similarity criteria, which can be selected on the left. The current hierarchical layer is visualized by a pyramidal indicator at the bottom left. The minimap below indicates the currently visible section and includes a heatmap that summarizes the current search activity of all users. Filter Views can be selected via a navigation menu at the top. 2.2 Browse & Filter Views The Feature Map is intended to be used as main interface, where users browse keyframes according so some similarity and refine their search over time. However, for some search tasks it might be inappropriate to start with this view. Therefore, we provide several complementary views that display a list of keyframes according to a filtering criterion. Each of these keyframes can then be used as starting point in the Feature Map. Due to space limitations we, however, show only a screenshot of the first view (the Storyboard) and omit others. Storyboard: In this view all videos are shown in a sequential list (Fig. 2). Each video is represented by uniformly sampled frames that are coherently visualized for fast human inspection, as described in [4]. The list is re-ranked

4 Fig. 2: The storyboard shows a coherent visualization of keyframes temporally sampled from the videos at equidistant positions (other browse & views are not shown due to space limitations). according to the search activities of the collaborators. For example, if other users inspect many shots of a video in the Feature Map, it is up-ranked in the storyboard. Similarly, if other searchers are also browsing in the storyboard, already inspected videos are down-ranked (we call this context-sensitive collaborative re-ranking). This is an advancement of our previous approach [3]. Color Filter: Here the user can choose different hue, saturation, and value areas from the HSV color space, to filter for matching keyframes. Concept Filter: This is a text-based search for semantic content classes, detected by convolutional neural networks (CNNs). We use two different CNNs for that purpose: (i) the well-known AlexNet [5], and (ii) a self-trained version of AlexNet using a manually selected large set of images from ImageNet ( images of 77 classes). Web Example: Here, a search engine is provided to gather appropriate images from the web (e.g., Bing or Google). The user can select an image from the result set and directly analyze it on-the-fly through a web service running on our content analysis server. The result of this analysis is used for similarity search in the Feature Map. 2.3 Architecture Our collaborative video search system uses two different servers: a video server and an interaction server. The video server performs several types of video content analysis (see top left in Fig. 3) and stores all the results as well as the videos, and makes them accessible via a web server and web services. It uses a content-sensitive shot detection algorithm that builds on motion flow analysis and comparison of edge histograms and color histograms for selecting the

5 most representative keyframe. The video server provides a web interface with several different views, which can be used simultaneously by several users that are additionally connected to a collaboration and interaction server. The interaction server uses a WebSocket connection among all clients and can actively contact clients for forwarding interaction data of a specific user. Such interaction data could be a notification (e.g., a hint to another user to inspect a specific area or video) or inspection information that provides the basis for collaboration features such as collaborative re-ranking or information sharing. Fig. 3: Collaborative video search system) 2.4 Collaboration Special attention was paid to the support of collaborative features to allow for cooperative work. One element that supports collaboration is a heatmap that is shown in the lower left corner of Fig. 1. Red parts are locations of potentially correct keyframes, marked by other users. The green color highlights areas where users have looked into without finding the correct shot. The heatmap is influenced by the filtering and search actions of all participants. A further feature that is influenced by the work of the users is the ranking of the video list

6 shown in Fig. 2. A video that was already inspected by a different user is shifted down to a rearward place in the list. Additionally, the ranking of the video list is rebuilt based on the filter events of the users. 3 Summary We present a versatile video search system that is novel in several ways. First, it provides a toolbox of several different views (i.e., sub-interfaces) and hence is a flexible tool that supports different types of search. Second, it uses several collaboration features like the collaborative heatmap, which immediately shows which areas were inspected intensively (and which were not), the collaborative re-ranking as well as specific notifications to other users. Finally, it uses a hierarchically refineable Feature Map with a changeable underlying feature for similarity arrangement of keyframes. This enables users to quickly switch from color-based similarity to texture-based similarity, or motion- or concept-based similarity, while always keeping the currently selected keyframe in center view. We expect this distributed interactive video search system to be efficient for textual and visual known-item search tasks, as well as ad-hoc video search tasks, as issued at VBS References 1. K. U. Barthel, N. Hezel, and R. Mackowiak. Navigating a Graph of Scenes for Exploring Large Video Collections, pages Springer International Publishing, Cham, C. Cobârzan, M. Del Fabro, and K. Schoeffmann. Collaborative browsing and search in video archives with mobile clients. In X. He, S. Luo, D. Tao, C. Xu, J. Yang, and M. Hasan, editors, MultiMedia Modeling, volume 8936 of Lecture Notes in Computer Science, pages Springer, M. A. Hudelist, C. Cobârzan, C. Beecks, R. van de Werken, S. Kletz, W. Hürst, and K. Schoeffmann. Collaborative video search combining video retrieval with humanbased visual inspection. In International Conference on Multimedia Modeling, pages Springer, W. Hürst, R. van de Werken, and M. Hoet. A storyboard-based interface for mobile video browsing. In X. He, S. Luo, D. Tao, C. Xu, J. Yang, and M. Hasan, editors, MultiMedia Modeling, volume 8936 of Lecture Notes in Computer Science, pages Springer, A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C. Burges, L. Bottou, and K. Weinberger, editors, Advances in Neural Information Processing Systems 25, pages Curran Associates, Inc., P. Over, G. Awad, M. Michel, J. Fiscus, G. Sanders, B. Shaw, W. Kraaij, A. F. Smeaton, and G. Quénot. Trecvid 2012 an overview of the goals, tasks, data, evaluation mechanisms and metrics. In Proceedings of TRECVID 2012, K. Schoeffmann. A user-centric media retrieval competition: The video browser showdown MultiMedia, IEEE, 21(4):8 13, Oct 2014.