Industrial Challenges for Content-Based Image Retrieval

Transcription

1 Title Slide Industrial Challenges for Content-Based Image Retrieval Chahab Nastar, CEO Vienna, 20 September LTU technologies Page 1

2 Agenda CBIR what is it good for? Technological challenges of CBIR Evaluating CBIR About LTU technologies Q&A LTU technologies Page 2

3 CBIR... what is it good for? Historically, (generic) CBIR is a cool technology... - convergence of information retrieval and computer vision - «user in the loop», link to web search engines - «query by example» approach - feature extraction, fast search, relevance feedback a lot of academic papers... in search of a useful application in the «real» world - worst-case scenario for entrepreneurship: technology preceding market traction LTU technologies Page 3

4 LTU technologies Page 4 Not interesting: Similar sunsets!

5 LTU technologies Page 5 Google does the job: category search

6 CBIR s redemption a powerful paradigm for object/scene recognition finding a series of images of the same scene/object user is in the loop linking the unknown world to the known world LTU technologies Page 6

7 CBIR in the «real» world Unknown world (query) Known world (reference) duplicate images B clone images similar images Image mining LTU technologies Page 7

8 User scenarios Verification: 1:1 - unrelated to information retrieval Identification: 1:N - target search - applications: stolen art, counterfeiting, media monitoring Investigation: q:n - find common ground between images - applications: child exploitation, IT forensics, intelligence Classification: A K LTU technologies Page 8

9 LTU s technology generalized CBIR Input Any digital image format, attachement, video frame, Real-time Analysis Visual Feature Extraction color, shape, texture, arrangements, moments, Image DNA Image Database / Knowledge Base Similarity Retrieval Clone Detection Output Categorization face 100% indoor 80% LTU technologies Page 9

10 Technological challenges of CBIR Application «Clone» detection Visual retrieval Image annotation Problem Matching: A = B? Similarity: A ~ B? Classification: A K? Metric Matching distance Visual/semantic distance Hamming distance where A and B are images, and K is a concept LTU technologies Page 10

11 Technological challenges of CBIR Application «Clone» detection Visual retrieval Image annotation Problem Matching: A = B? Similarity: A ~ B? Classification: A K? Metric Matching distance Visual/semantic distance Hamming distance where A and B are images, and K is a concept LTU technologies Page 11

12 Clone Detection Image clones same visual content, different graphical rendering (typically Photoshop manipulation) original old B&W negatized bad scan cropped inverted titled face erased face erased stretched reencoded LTU technologies Page 12

13 Technological challenges of CBIR Application «Clone» detection Visual retrieval Image annotation Problem Matching: A = B? Similarity: A ~ B? Classification: A K? Metric Matching distance Visual/semantic distance Hamming distance where A and B are images, and K is a concept LTU technologies Page 13

14 Cross-modal retrieval: LTU technologies Page 14

15 Technological challenges of CBIR Application «Clone» detection Visual retrieval Image annotation Problem Matching: A = B? Similarity: A ~ B? Classification: A K? Metric Matching distance Visual/semantic distance Hamming distance where A and B are images, and K is a concept LTU technologies Page 15

16 Annotation by supervised learning Classifier bank Fusion module AA BB CC Keyword DD Finite vocabulary Binary or multi-class classification LTU technologies Page 16

17 LTU technologies Page 17 Annotation by case-based reasoning

18 Evaluation of CBIR depends upon Image Content - Scenes/objects represented - Image Quality - Domain-specific knowledge User scenario - identification benchmarking database? - investigation using a series of images - classification Usage scenario - Search Acceptable max rank of «retrieved images»? precision or recall? - Classification LTU technologies Page 18

19 Evaluation within a customer project Large database with Ground Truth - provided by customer or - put together by LTU User scenario & usage scenario taken into account - in most mission-critical scenarios, maximizing recall is the main goal - precision is unimportant LTU technologies Page 19

20 The LTUtech dataset LTU has collected (mostly over the web) more than 80,000 images of 267 visual categories A visual category is a set of examples of an object - animals, CD covers, gloves, watches etc. An average of 300 images per visual category The total database size is 1.3 GB The images vary in quality, size, format We do not known the copyright of the images The database is shared with the WP3 members of the MUSCLE network LTU technologies Page 20

21 About LTU Technologies Golden Award for Technology The world s top 100 innovators MIT Technology Review Founded: 1999 from our group s research at INRIA (imedia research group) Mission: Bringing image/video indexing/retrieval to the market Europe s Top 100 Companies The Red Herring Headquartered: Paris & Washington D.C. Acquired by company JASTEC in 2005 European Information Society Technologies Prize LTU technologies Page 21

22 Value Prop: Law Enforcement & Intelligence Providing tools to automatically mine information from large, unknown sets of images Data Collection Data Analysis Content Management Seized HD Database of images w/ case information Internet Database of suspect Images P2P LTU technologies evidence Page 22

23 Key-Clients (1) Government, Police, International agencies Investigations Child protection, Stolen art Computer forensics Counterfeiting, ID theft Intelligence and monitoring Computer, Network or Media LTU technologies Page 23

24 Value Prop: Media & Intellectual Property Providing efficient access to images and tools for monitoring their usage Enroll Manage Distribute Internet 3 Keywording 1 Search 2 Monitoring LTU technologies Page 24

25 Key-Clients (2) Media and IP protection Image bank management e-commerce, media Patent and design protection Search, classification of designs Online IP protection Logos, visual assets Partners DAM software, text search engines Content filtering, Antivirus software LTU technologies Page 25

26 Summary Challenges in bringing CBIR to the market - mission-critical applications Evaluation is case-by-case - domain-specific - customer-specific LTU technologies Page 26

27 Questions & Answers Chahab Nastar CEO Paris, France cn AT LTUtech.com LTU technologies Page 27