INSTITUTE OF COMPUTING University of Campinas Filling the Semantic Gap: A Genetic Programming Framework for Content-Based Image Retrieval Ricardo da Silva Torres rtorres@ic.unicamp.br www.ic.unicamp.br/~rtorres
Outline Motivation Genetic Programming (GP) Image retrieval based on GP Data fusion Relevance Feedback Conclusions
Motivation: Semantic Gap
Motivation: Semantic Gap Beach My vacation Summer 2007 Blue sky Sky and sea
Motivation: Semantic Gap Similarity Value Similarity Function Simple Descriptor Feature Vector A Feature Vector B Feature Vector Extraction Algorithm Feature Vector Extraction Algorithm Image A Image B
Motivation Given an image database and a set of image descriptors, how to combine them to support search services? Our hypotheses: GP provides a comparable and effective framework to combine image descriptors
Genetic Programming Solution = GP Individual = GP Program
Genetic Programming Algorithm for evolution 1. Generate a initial population of individuals 2. For N generations do (a) Compute the fitness of each individual (b) Select the individuals for genetic operations (c) Apply reproduction (d) Apply crossover (e) Apply mutation
Genetic Programming Algorithm for evolution 1. Generate a initial population of individuals 2. For N generations do (a) Compute the fitness of each individual (b) Select the individuals for genetic operations (c) Apply reproduction (d) Apply crossover (e) Apply mutation
Genetic Programming Solution = Individual = Program Represented by TREES
Genetic Programming
Genetic Programming Algorithm for evolution 1. Generate a initial population of individuals 2. For N generations do (a) Compute the fitness of each individual (b) Select the individuals for genetic operations (c) Apply reproduction (d) Apply crossover (e) Apply mutation
Genetic Programming Reproduction
Genetic Programming Crossover
Genetic Programming Mutation
A genetic programming framework for content-based image retrieval Ricardo da S. Torres (UNICAMP), Alexandre X. Falcão (UNICAMP), Marcos A. Gonçalves (UFMG),, João P. Papa (UNICAMP), Baoping Zhang (VT), Weiguo Fan (VT), and Edward A. Fox (VT) Pattern Recognition Volume 42, Issue 2, February 2009, Pages 283-292 Learning Semantics from Multimedia Content
Genetic Programming Problem: How to rank database images Solution = similarity function = GP Individual = GP program
Genetic Programming
Image Descriptor Similarity Value Similarity Function Simple Descriptor Feature Vector A Feature Vector B Feature Vector Extraction Algorithm Feature Vector Extraction Algorithm Image A Image B
Intelligent Image Descriptor Combination Similarity Value Similarity Function Similarity Value Similarity Value Similarity Value Simple Descriptor 1 Simple Descriptor 2 Simple Descriptor k Image A Image B
Intelligent Image Descriptor Combination Similarity Value GP Similarity Individual Function Similarity Value Similarity Value Similarity Value Simple Descriptor 1 Simple Descriptor 2 Simple Descriptor k Image A Image B
Fitness Function How well a GP individual (similarity function) rank training set images More relevant images at first positions
Experiments different collections: fish shape collection MPEG-7 collection different samples different fitness functions validation set time
Desciptors
Baselines
Without validation set
With validations set
GP individual: combination function
Without BAS, without validation set
Without BAS, with validation set
Image Retrieval with Relevance Feedback based on Genetic Programming Cristiano Ferreira (UNICAMP), Ricardo Torres (UNICAMP), Marcos Gonçalves (UFMG), Weiguo Fan (VT) XXIII Brazilian Symposium on Databases (SBBD) Campinas, 2008, p. 120-134. Best paper award
Relevance Feedback
Retrieval Process 1. User indication of query image 2. Show the initial set of images 3. While the user is not satisfied do (a) User indication of relevant images (b) Update the query pattern (c) Apply GP to discover the best individuals (d) Rank the database images (e) Show the images
Retrieval Process 1. User indication of query image 2. Show the initial set of images 3. While the user is not satisfied do (a) User indication of relevant images (b) Update the query pattern (c) Apply GP to discover the best individuals (d) Rank the database images (e) Show the images
Image Descriptor Combination
Training set
Fitness computaion
Retrieval Process 1. User indication of query image 2. Show the initial set of images 3. While the user is not satisfied do (a) User indication of relevant images (b) Update the query pattern (c) Apply GP to discover the best individuals (d) Rank the database images (e) Show the images
Voting for best images sorting Images that will be showed
Experiments Three image collections (FISHES, MPEG7, COREL) Evaluation measures with statistical significance tests Comparison with three (five) other methods Color, shape and texture descriptors 10 iterations 20-40 images exhibited per iteration
Experiment 1
Recall x Iterations
Precision x Recall
Experiment 2
Recall x Iterations
Precison x Recall
References (selected) R. da S. Torres, A. X. Falcão, M. A. Goncalves, J. P. Papa, B. Zhang, W. Fan, and E. A. Fox. A Genetic Programming Framework for Content-based Image Retrieval. Pattern Recognition, 42(2): 283-292, February 2009. C. D. Ferreira, R. da S. Torres, M. A. Goncalves, and W. Fan. Image Retrieval with Relevance Feedback based on Genetic Programming. In: XXIII Simpósio Brasileiro de Banco de Dados, 2008, Campinas. SBBD, 2008. J. A. Santos, C. D. Ferreira, and R. da S. Torres. A Genetic Programming Approach for Relevance Feedback in Regionbased Image Retrieval Systems. In: SIBGRAPI 2008, 2008, Campo Grande, MS. XXI Brazilian Symposium on Computer Graphics and Image Processing, 2008. p. 155-162. 51
Summary Motivation Genetic Programming (GP) Image retrieval based on GP Data fusion Relevance Feedback
Acknowledgements: Support CAPES CNPq FAEPEX FAPESP Microsoft,
INSTITUTE OF COMPUTING University of Campinas Filling the Semantic Gap: A Genetic Programming Framework for Content-Based Image Retrieval Ricardo da Silva Torres rtorres@ic.unicamp.br www.ic.unicamp.br/~rtorres LIS Laboratory of Information Systems