Filling the Semantic Gap: A Genetic Programming Framework for Content-Based Image Retrieval

Transcription

1 INSTITUTE OF COMPUTING University of Campinas Filling the Semantic Gap: A Genetic Programming Framework for Content-Based Image Retrieval Ricardo da Silva Torres rtorres@ic.unicamp.br

2 Outline Motivation Genetic Programming (GP) Image retrieval based on GP Data fusion Relevance Feedback Conclusions

3 Motivation: Semantic Gap

4 Motivation: Semantic Gap Beach My vacation Summer 2007 Blue sky Sky and sea

5 Motivation: Semantic Gap Similarity Value Similarity Function Simple Descriptor Feature Vector A Feature Vector B Feature Vector Extraction Algorithm Feature Vector Extraction Algorithm Image A Image B

6 Motivation Given an image database and a set of image descriptors, how to combine them to support search services? Our hypotheses: GP provides a comparable and effective framework to combine image descriptors

7 Genetic Programming Solution = GP Individual = GP Program

8 Genetic Programming Algorithm for evolution 1. Generate a initial population of individuals 2. For N generations do (a) Compute the fitness of each individual (b) Select the individuals for genetic operations (c) Apply reproduction (d) Apply crossover (e) Apply mutation

9 Genetic Programming Algorithm for evolution 1. Generate a initial population of individuals 2. For N generations do (a) Compute the fitness of each individual (b) Select the individuals for genetic operations (c) Apply reproduction (d) Apply crossover (e) Apply mutation

10 Genetic Programming Solution = Individual = Program Represented by TREES

11 Genetic Programming

12 Genetic Programming Algorithm for evolution 1. Generate a initial population of individuals 2. For N generations do (a) Compute the fitness of each individual (b) Select the individuals for genetic operations (c) Apply reproduction (d) Apply crossover (e) Apply mutation

13 Genetic Programming Reproduction

14 Genetic Programming Crossover

15 Genetic Programming Mutation

16 A genetic programming framework for content-based image retrieval Ricardo da S. Torres (UNICAMP), Alexandre X. Falcão (UNICAMP), Marcos A. Gonçalves (UFMG),, João P. Papa (UNICAMP), Baoping Zhang (VT), Weiguo Fan (VT), and Edward A. Fox (VT) Pattern Recognition Volume 42, Issue 2, February 2009, Pages Learning Semantics from Multimedia Content

17 Genetic Programming Problem: How to rank database images Solution = similarity function = GP Individual = GP program

18 Genetic Programming

19 Image Descriptor Similarity Value Similarity Function Simple Descriptor Feature Vector A Feature Vector B Feature Vector Extraction Algorithm Feature Vector Extraction Algorithm Image A Image B

20 Intelligent Image Descriptor Combination Similarity Value Similarity Function Similarity Value Similarity Value Similarity Value Simple Descriptor 1 Simple Descriptor 2 Simple Descriptor k Image A Image B

21 Intelligent Image Descriptor Combination Similarity Value GP Similarity Individual Function Similarity Value Similarity Value Similarity Value Simple Descriptor 1 Simple Descriptor 2 Simple Descriptor k Image A Image B

22 Fitness Function How well a GP individual (similarity function) rank training set images More relevant images at first positions

23 Experiments different collections: fish shape collection MPEG-7 collection different samples different fitness functions validation set time

24 Desciptors

25 Baselines

26 Without validation set

27 With validations set

28 GP individual: combination function

29

30

31 Without BAS, without validation set

32 Without BAS, with validation set

33 Image Retrieval with Relevance Feedback based on Genetic Programming Cristiano Ferreira (UNICAMP), Ricardo Torres (UNICAMP), Marcos Gonçalves (UFMG), Weiguo Fan (VT) XXIII Brazilian Symposium on Databases (SBBD) Campinas, 2008, p Best paper award

34 Relevance Feedback

35 Retrieval Process 1. User indication of query image 2. Show the initial set of images 3. While the user is not satisfied do (a) User indication of relevant images (b) Update the query pattern (c) Apply GP to discover the best individuals (d) Rank the database images (e) Show the images

36 Retrieval Process 1. User indication of query image 2. Show the initial set of images 3. While the user is not satisfied do (a) User indication of relevant images (b) Update the query pattern (c) Apply GP to discover the best individuals (d) Rank the database images (e) Show the images

37 Image Descriptor Combination

38 Training set

39 Fitness computaion

40 Retrieval Process 1. User indication of query image 2. Show the initial set of images 3. While the user is not satisfied do (a) User indication of relevant images (b) Update the query pattern (c) Apply GP to discover the best individuals (d) Rank the database images (e) Show the images

41 Voting for best images sorting Images that will be showed

42 Experiments Three image collections (FISHES, MPEG7, COREL) Evaluation measures with statistical significance tests Comparison with three (five) other methods Color, shape and texture descriptors 10 iterations images exhibited per iteration

43 Experiment 1

44 Recall x Iterations

45 Precision x Recall

46

47 Experiment 2

48 Recall x Iterations

49 Precison x Recall

50

51 References (selected) R. da S. Torres, A. X. Falcão, M. A. Goncalves, J. P. Papa, B. Zhang, W. Fan, and E. A. Fox. A Genetic Programming Framework for Content-based Image Retrieval. Pattern Recognition, 42(2): , February C. D. Ferreira, R. da S. Torres, M. A. Goncalves, and W. Fan. Image Retrieval with Relevance Feedback based on Genetic Programming. In: XXIII Simpósio Brasileiro de Banco de Dados, 2008, Campinas. SBBD, J. A. Santos, C. D. Ferreira, and R. da S. Torres. A Genetic Programming Approach for Relevance Feedback in Regionbased Image Retrieval Systems. In: SIBGRAPI 2008, 2008, Campo Grande, MS. XXI Brazilian Symposium on Computer Graphics and Image Processing, p

52 Summary Motivation Genetic Programming (GP) Image retrieval based on GP Data fusion Relevance Feedback

53 Acknowledgements: Support CAPES CNPq FAEPEX FAPESP Microsoft,

54 INSTITUTE OF COMPUTING University of Campinas Filling the Semantic Gap: A Genetic Programming Framework for Content-Based Image Retrieval Ricardo da Silva Torres rtorres@ic.unicamp.br LIS Laboratory of Information Systems

55