Open issues and research trends in Content-based Image Retrieval

Transcription

1 Open issues and research trends in Content-based Image Retrieval Raimondo Schettini DISCo Universita di Milano Bicocca IEEE Signal Processing Society Transactions on Image Processing EDICS (editors information classification scheme) Amended: 2 July IMAGE/VIDEO CODING AND TRANSMISSION 2 IMAGE/VIDEO PROCESSING 3 IMAGE FORMATION 4 IMAGE SCANNING, DISPLAY, AND PRINTING 5 IMAGE/VIDEO STORAGE, RETRIEVAL, AND MULTIMEDIA IEEE Signal Processing Society Transactions on Image Processing EDICS Why content-based image retrieval is so popular? 5 IMAGE/VIDEO STORAGE, RETRIEVAL, AND MULTIMEDIA 5-DTBS Image and Video Databases 5-SRCH Image Search and Sorting 5-INDX Video Indexing and Editing 5-INTG Integration of Images and Video and Other Media 5-CONT Content-based Multimedia 5-MAPP Multimedia Applications 5-AUTH Authentication and Watermarking 5-OTHE Other Image/Video Storage, Retrieval, and Multimedia -Large collection of image have been -digitized in the hope of -Conservation -Distribution -Better access

2 Applications Art gallery, museum. Photographic archives (journalism). Textile databases. Trademarks databases. E-commerce catalogues. Web-searches. Medical image databases. Facial databases (security, law enforcement). Engineering design databases. Geographic databases. Satellite image databases. Home-photo collections Education and training Image domains Narrow domains: limited and predictable variability of relevant aspects of its appearance Image domains Broad domains: unlimited and unpredictable variability of relevant aspects of its appearance even for the same semantic meaning Possible queries in content-based image retrieval Target search Does a given image exist as is in the database? Does a scaled and/or rotated version of this image exist in the database? Does a region of this image exist in the database? Does an object exist in the database? Search by association Show similar images to a query image Category search Show images belonging to the same category

3 Impossible queries in content-based image retrieval High level semantic queries (in broad domains): Schematic description of the activities of a CBIR system - A girl dancing on a sunny beach - the pope kissing a baby - a beautiful girl/boy -... Feature selection Features will ideally present the following basic properties: perceptual similarity: the feature distance between two images is large only if the images are not "similar"; efficiency: they can be rapidly computed; economy: their dimensions are small in order not to affect retrieval efficiency; scalability: the performance of the system is not influenced by the size of the database; robustness: changes in the imaging conditions of the database images do not affect retrieval. How to describe visual content? General features Color Color Histogram Color Coherence Vectors Correlograms Spatial Chromatic Histogram... Texture Wavelets Grey-Levels Distribution Gabor Filters Tamura descriptors... Shape Deformable Template Invariant Moments Wavelets Description of boundaries... Other Discrete Cosine Transform Regions Layout/Relationship...

4 How to describe visual content? The role of segmentation Evaluating similarity between segmented images Unsupervised segmentation is still very difficult. Quantitative measures for evaluating segmentation results are needed. Evaluating similarity between segmented images Evaluating similarity between segmented images Multidimensional indices (Pala, 2000) Representation of segmented images: Region adjacency graphs Multidimensional indices C. Carson et al., 1999

5 Evaluating similarity between segmented images Evaluating similarity between segmented images How to describe visual content? Weak segmentation How to describe visual content? Domain specific features THESAURUS DEFINITION... Snow Water Ground Sky Vegetation MATCHING /DECISION RULE Weak segmentation is possible (and very often useful) Vegetation, + Water, Sky

6 How to describe visual content? Domain specific features subsampling Hidden annotation: face detection and Input image pyramid recognition extract window (20x21 pixel) Matching preprocessing mask input hidden layer output Face detector How to describe visual content? Features definition Image processing in content-based image retrieval Color distribution Degree of difficulty Texture pattern Shape Objects Partial semantic Real content

7 Image processing in content-based image retrieval Image processing in content-based image retrieval Original scene? Image processing in content-based image retrieval Image processing in content-based image retrieval

8 Image processing in content-based image retrieval Image processing in content-based image retrieval Image processing in content-based image retrieval Image processing in content-based image retrieval Image normalization using Retinex Illuminant change Retinex filtering result Original image 3 2

9 O R I G I N A L Illuminant invariance color indexing Experimental Setup N O R M A L I Z E D R E T I N E X E D Illuminant invariance color indexing Experimental Setup Image processing in content-based image retrieval image Enhancement HSVM HIST CCV SCH Success of Target Search HSVM HIST CCV SCH

10 Feature selection and similarity evaluation Feature selection and similarity evaluation Are these similar? Gap between low-level features and image semantics heavy involvement of the user lack of flexibility A common idea is that similarity may be modeled as a weighted sum of the feature distances. An emerging idea is to see similarity as a probabilistic concept. Feature selection and similarity evaluation Scalability in CBIR Rubner, 1999 Are these similar?

11 Scalability and image semantics Possible solutions: Apply relevance feedback Reduce the DB size by image classification Design and Exploit image annotation tools Exploit all the available information Relevance feedback Positive Examples Negative Examples Agreement between positive examples Agreement between positive and negative examples W h + Weight Estimation W + * h -W h W h * W h User interaction The idea Relevance Feedback Example Relevance feedback: Query Reformulation Relevant Objects The idea Shape Color Query User interaction

12 Shape pianta con lunghi steli sinu pianta con lunghi steli sinuosi e larghe pianta con lunghi steli foglie (" pianta con sinu pianta con lunghi lunghi steli sinu pianta steli sinuosi e larghe con lunghi steli sinuosi e foglie ("albero della larghe foglie ("albero vita"); sul collo, ampia della vita"); sul collo fascia con motivo a albero della vita"); sul treccia; tipologia ("albero collo, ampia fascia con della vita"); motivo a treccia; tipologia pianta ("albero con della lunghi steli vita"); sinu pianta con lunghi steli sinuosi e larghe foglie ("albero della vita"); sul collo, ampia Relevant Objects Query Color pianta con lunghi steli sinu pianta con lunghi steli sinuosi e larghe foglie ("albero della vita"); sul collo, ampia fascia con motivo a treccia; tipologia ("albero della vita"); Positive Examples Agreement between positive examples W h + Weight Estimation W + * h -W h W h Negative Examples Agreement between positive and negative examples W h * pianta con lunghi steli sinu pianta con lunghi steli sinuosi e larghe foglie ("albero della vita"); sul collo, ampia fascia con motivo a treccia; tipologia "arcaica". Dipinto in bruno di manganese e verde rame. osi e larghe foglie ("albe pianta con lunghi steli sinuosi e larghe foglie ("albero della vita"); sul collo, ampia fascia con motivo a treccia; tipologia "arcaica". Dipinto in bruno di manganese e verde rame. ro della vita"); sul collo, ampia fascia con motivo a treccia; tipologia "arcaica". Dipinto in bruno di manganese e verde rame.v pianta con lunghi steli sinuosi e larghe foglie ("albero della vita"); Relevance feedback: Exploit all the available information Relevance Feedback Human-in-the-Loop Query Reformulation Features Weights Pictorial Textual SIMILARITY EVALUATION Ranked Results Database Browsing and visualization Database Organization Clustering Can be used to: Improve retrieval efficiency Provide an overview of database content Rubner, 1999

13 Relevance feedback, Database Filtering and Semantic Associations Impossible queries in content-based image retrieval Combination of - unsupervised learning; - supervised learning; - short term learning (relevance feedback); - long term learning (user profile) Clutter scene... Impossible queries in content-based image retrieval Hidden Annotation High level categorization IMAGE CLASSIFICATION... Close-up Text Graphic Outdoor Indoor Reality vs image reality

14

15 General Purpose Content-based retrieval A few applications... Design a retrieval system able to use both low level and high level descriptions of visual contents automatically extracted. cover a wide range of users requirements. adaptively select the retrieval strategy that is most suited for the users needs. All these with a simple interface!!! The Quicklook 2 system Overview The Quicklook 2 system Queries available Pictorial queries Provide an example of ready-made image Sketch an image with the available tool Browse the database to find one or more relevant images with which to begin Textual queries Filter the database with textual keywords Search images by textual annotation similarity Search images by text similarity Web-Demo at Advanced queries Combination of one or more of the above queries Relevance feedback

16 Journal indexing Photo indexing Photos to classify Digital documents Compound documents Not compound documents Segmented documents Graphics Photographs Texts Pornographic Images Non-Pornographic Images Rejected Images Indoor Outdoor Close-up Biometrics Pattern recognition in cultural heritage archives

17 Pattern recognition in cultural heritage archives Video Indexing Cuts / Fades Detection Key-Frames Video Indexing Abstraction: summarization Classification of videos Scenes classification Structure Levels... Video surveillance Key Frame Levels...

18 Classification of web pages Improving the performance of search engines Challenges 3D Pre-processing 3D Objects Features to use in indexing 3D objects 2D Features 3D Features Invariants Interface to use to submit a query 2D Projections View 3D Examples Matching 2D and 3D data Adult image filtering 3D Object indexing 3D face recognition

19 Cross-media 3D face recognition Open issues and future researches in Content-based Image Retrieval Media 3D objects Web pages Photographs Videos Images Low level Middle level High level Target search Similarity search Semantic retrieval Videos Tridimensional objects Features Matching strategies Open issues and research trends in Content-based Image Retrieval