Shiliang Zhang, Qi Tian*, *University of Texas at San Antonio

Transcription

1 Shiliang Zhang, Qi Tian*, Qingming Huang, and Wen Gao *University of Texas at San Antonio 1

2 Shiliang Zhang QiTian Qingming Huang Wen Gao ICT of Chinese Academy of Science Computer Science Department, UTSA Graduate School of Chinese Academy of Science Beijing University 2

3 Introduction ObjectBook Construction Large scale Semantic Image Retrieval Experiments Conclusions 3

4 What is Happening? Explosion of images and videos on the Internet Sep. 2010: 5 billion 3000 photos /minute Sep : 120 million 20 hours uploaded/minute How to efficiently access, analyze, understand, and search these large scale Internet Images? This Work Large scale Semantic Image Retrieval July 2010: 50 billion 2.5 bll billion/monthh facebook hits 500 million users_n.htm 4

5 Image Retrieval Content based Image Retrieval (CBIR) Lots of progress since mid 1990 s Still difficult dff due to the well known Semantic Gap between low level visual features and high level concept Color, texture, and shape vs. concept Recent Focus Near Duplicate Image Retrieval Image Annotation 5

6 Near Duplicate Image Retrieval Clear and well defined Problem Limited applications (landmark, CD cover) No Semantics Well studied in recent years Visual Vocabulary [Nister, CVPR 06] Bundled Feature [Wu, CVPR2009] Contextual Visual Vocabulary [Zhang, ACM MM 2010] Spatial Coding [Zhou, ACM MM 2010] 6

7 Image Annotation Popular topic and different problems Auto Tagging Ranking alex dog speed leave leave tree dog speed 101 alex tree 101 Liu, Li, Statistical Tag ranking, modeling, WWW 09 PAMI 03 Wang, Jeon, Cross media Learning to rank relevance tags, CIVR models, 10 SIGIR 03. Tag Refinement Tag to Region top 101 tour tiger sweet big cloud dog house tree sky ground cloud Wang, Liu, Bi layer Content sparse based coding, annotation ACM MM refinement, 2010 CVPR 06 Liu, Image Multi edge Retagging, graph, ACM MM 2010 Promising for image understanding and semantic image retrieval Complicated models Hard to be scalable Courtesy of Dong Liu, Unified tag analysis via multi edge graph. ACM MM,

8 How to Achieve Scalable Image Annotation and Semantic Retrieval? A Data driven Approach Large scale loosely l labeled bld image set makes it possible ImageNet [Deng, CVPR 09] 12 million images and 17,624 categories One category label per image Simple Annotation by K NN Voting Problems Object is not clean < cluttered backgrounds Label information is too coarse < single label at image level Low computational tti efficiency i < huge data 8

9 Our Idea: ObjectBook Construct ObjectBook from ImageNet Tiger Grass Creek Sky Dog ObjectBook Query Bag of ObjectWords ObjectWords annotated with object category Clean only object related patches are kept Refined tags from image to patch th levell Compact 1.2million images 42K ObjectWords Returned Inverted File Index Images Index millions of images offline 9

10 Our Idea: ObjectBook Construct ObjectBook from ImageNet Tiger Grass Creek Sky Dog ObjectBook ObjectWords annotated with object category Clean Refined tags only object related patches are kept from image to patch level Compact 1.2million images 42K ObjectWords 10

11 Outline Introduction ObjectBook Construction Large scale Semantic Image Retrieval Experiments Conclusions 11

12 ObjectBook Construction Step 1: Image Patch Extraction Step 2: 1 st Layer Dense Patch Group (DPG) Extraction DPGs: visually similar patch groups ObjectWord: top DPGs with ih each category Step 3: 2 nd Layer Dense Patch Groups Extraction top DPGs across categories Step 4: ObjectWord Annotation 12

13 Step 1: Image Patch Extraction Image Patch Extraction Dense sampling with overlap simple, fast, effective Multi Resolution: 60 60, , Feature Extraction SIFT (appearance, shape) [Lowe, IJCV 04] Color Histogram (color) [Stricker, SPIE 95] Local Binary Pattern (texture) [Ahonen, PAMI 06] 13

14 Step 2: 1 st Layer DPG Extraction 1 st layer DPGs: within each category Visually similar patches => Dense Patch Groups (DPG) ObjectWord: Top DPGs for each category Discarded the other noisy patches Cluster 2 K means What we Clustering really want is not is dense optimal! patch groups! Cluster 1 identified clusters with K means date points belonging to three clusters Cluster 3 :desired groups DPG extraction in Tiger category 14

15 Step 2: 1 st Layer DPG Extraction (cont.) Discover the Dense Sub Graph (by GraphShift) Patch Current Subgraph Shrink Expansion Similarity Shrink Color Texture SIFT Densest Subgraph GraphShift: Find DPGs with high efficiency and accuracy 15 Courtesy of Hairong Liu, Robust Graph Mode Seeking by Graph Shift. ICML, 2010.

16 1 st Layer Dense Patch Group Examples (a) (b) Four Four generated groups groups from from the the category common electric newt ray 16

17 Step 3: 2 nd Layer y DPG Extraction 2 nd Layer DPG: visually similar patch groups across categories GraphShift 1 st Layer DPGs for each category Creek Tiger... Tree 2 nd Layer DPGs 17

18 Step 4: ObjectWord Annotation Easy: same source category Difficult: DPGs from different source categories 2 nd layer DPG 1 2 nd layer DPG 2 From: Tiger Tag: Tiger From: From: Creek Tree From: Tiger From: From: Creek From: Tree Tiger Tiger? Tree? Creek? Tiger? Tree? Creek? From: Assumption 1 : Patch groups in the same 2 nd layer DPG will be labeled with the same tags. Tiger Assumption 2: Annotation by Relevance Degree Voting Tag: Tiger 18

19 Step 4(a): Relevance Degree Computing Take the Tree and Tiger as examples From: Tree Relevance: Position and Portion From: Creek From: Tiger Position Relevance Portion Relevance Tree Tiger Tree Tiger Category: Tree Category: Tiger Category: Tree Category: Tiger The positions in source categories The portions in source categories 19

20 Step 4(b): Relevance Degree Voting Creek Tree Tree Tiger Creek Tiger Final Label: Final Label: Tree Tiger Creek Tree Tiger Creek ObjectWord: labeled bld 1 st layer l DPG Mean visual feature Annotated Label 20

21 Results: ObjectBook ObjectBook from ImageNet Tiger Creek Sky Grass Dog 1.2 million images 1000 categories 42K ObjectWords 11 to 80 ObjectWords ObjectBook per category Bag of ObjectWords Inverted File Index Hierarchical ObjectWord Tree Indexed Inverted File Index the image database offline 21

22 Outline Introduction ObjectBook Construction Large scale Semantic Image Retrieval Experiments Conclusions 22

23 Scalable Image Retrieval Inverted File Indexing: Efficient: borrowed from classic Information Retrieval (IR) Easy integration i with ih ObjectBook Semantic Weighted Term Frequency (TF) weighting ObjectWord a Proposed Classic Indexing Framework Image Image Image Image Info Info Info Info ObjectWord b Image ID: I TF weighting: g Semantic weighted TF: Semantic Term Frequency Weighted (TF) Semantic Visual &Visual Rl Relevance 23

24 Semantic Weighting g Semantic Weighted TF Visual relevance by TF Semantic relevance b/w an ObjectWord and an image.: dist b/w their probabilistic labels TF Weighting: Image Label: Semantic Similarity: il it The ObjectWord Tree is more frequent than the Tiger TigerTreeSky Object Word Label: Tiger TreeSky Tiger TreeSky Semantic Weighted TF Weighting: Semantic Similarity il i TF Weighting ihi 24

25 Outline Introduction ObjectBook Construction Large scale Semantic Image Retrieval ti Experimentsi t Conclusions 25

26 Dataset Large scale Visual Recognition Challenge 2010 A subset of ImageNet, 1000 image categories 1.2 million training images for ObjectBook construction 150K test images Compared Algorithms A1: The linear search strategy A2: Traditional visual words with TF IDF weighting. A3: ObjectWord with TF IDF weighting A4: ObjectWord with Semantic Weighted TF+ IDF weighting Performance Measures MAP (mean average precision) Average query time per image 26

27 Large scale Image Retrieval The comparisons of MAP and efficiency MA AP second) Tim me (Milli A1 A2 A3 A4 A1 A2 A3 A4 (a) The comparison of MAP (b) The comparison of efficiency more effective than traditional visual word MAP from (A2) to (A3) and (A3): 12.8% and 15.7% improvement Semantic Weighted TF weighting is more effective than TF weighing From (A3) to (A4), 2.54% improvement Real time response 0.33 seconds (A4) per query time 27

28 Retrieval Examples Queries Returned images before the first false positive 28

29 Outline Introduction ObjectBook Construction Large scale Semantic Image Retrieval ti Experiments Conclusions 29

30 Summary Conclusions ObjectBook: a compact and clean representation of ImageNet ObjectWord: Semantic preserving visual vocabulary Effective and efficient in large scale semantic image retrieval ObjectBook available for public sharing Future Work Contextual information between ObjectWords Refined relevance degree computation Semantic relationships by WordNet On the whole ImageNet 30

31 ObjectBook: bridge the Semantic gap Giraffe Concept & Scene Semantic Gap ObjectBook Refined, visual and semantic image patch feature Descriptive Local Features DVW, DVP, Bundled Feature, Contextual Visual Vocabulary Local Features SIFT, Salient Points, Visual Word, Image Patches Regional Features Region of Interests, Segmentation, Multiple Instances Global Features Color Histogram, Texture, Color Correlogram, edge map Refined Noisy Difficult Coarse 31

32 Thanks! Questions: cn 32