<is web> Information Systems & Semantic Web University of Koblenz Landau, Germany

Transcription

1 Information Systems University of Koblenz Landau, Germany Exploiting Spatial Context in Images Using Fuzzy Constraint Reasoning Carsten Saathoff &

2 Agenda Semantic Web: Our Context Knowledge Annotation Improving Knowledge Annotation by Fuzzy Constraint Reasoning Overall Process Knowledge Acquisition Modelling of the Constraint Problem Initial Evaluation Understanding, 3 of 45

3 The Semantic Web on One Slide Ontology rdfs:domain cooperateswith Person rdfs:range rdfs:subclass rdfs:subclass Employee rdfs:subclass rdf:type PostDoc Professor rdf:type Metadata <swrc:postdoc rdf:id="person_sha"> <swrc:name>siegfried Handschuh</swrc:name> <swrc:cooperateswith rdf:resource = " #person_sst"/>... </swrc:postdoc> <swrc:professor rdf:id="person_sst"> <swrc:name> </swrc:name>... </swrc:professor> swrc:cooperateswith Web page URL Understanding, 4 of 45

4 No Free Lunch, but Free Knowledge Number of triples on the Semantic Web according to Swoogle ( 2,418,811 documents with Semantic Web content 585,366,088 triples Understanding, 5 of 45

5 Multimedia Challenge Reg1 Seq1 The "ISWeb" at Bad Kreuznach (team web page) Multimedia objects are complex Summer School Semantic Web-05 (video) Compound information objects, fragment identification Semantic annotation Allow for Reuse of Knowledge: Subjective interpretation, context dependent Linked data principle Open to reuse existing knowledge Seq4 Multimedia O Context O COMM - A Core Ontology for Multimedia [ISWC 2007] Domain O Understanding, 6 of 45

6 Semantic Web Issues and Apps Knowledge Structuring Multimedia Ontologies Knowledge Exchange Semantic Web Language (W3C Standards) RDF, OWL Knowledge Reuse/Retrieval Semantic Web Querying SPARQL Resource Retrieval Domain Ontologies Understanding, 7 of 45

8 KAT: K-Space Annotation Tool Goal Efficient annotation of multimedia content Means to create semantically rich annotations KAT provides framework for Executing analysis plugins Providing visualisation plugins Displaying/annotating content Browsing Interfaces with Core Ontology for Multimedia (COMM) Provides the common model Role based messaging to leverage reuse of components Understanding, 9 of 45

9 Efficient Annotation Reduce time required by user for annotating content Semi-automation Integration of Automatic analysis methods Region labeling, object detection Key Frame Extraction, Shot Boundary Detection Automatic Organisation Clustering Inferencing Based on formal domain ontologies Understanding, 10 of 45

10 Semantically Rich Annotations Relational Annotation Express how depicted entities are related Example: Soccer Game Who is tackling whom? Why was the penalty given? Ontologies provide means to express relations KAT aims at providing the means to efficiently create them Event-Based annotation Events are prominent in multimedia Create and manage events Relate events and media Allow for event-based retrieval and exploration Understanding, 11 of 45

11 Importing Content List of imported images Understanding, 12 of 45

12 Importing Content Select Analysis Plugin Understanding, 13 of 45

13 Annotation Queue List of completely analysed content Understanding, 14 of 45

14 Annotation Tool Selection of regions. Annotations Understanding, 15 of 45

15 Manual Refinement Drag & Drop Annotation extended from automatic result Understanding, 16 of 45

16 Browsing Show content annotated with Man Understanding, 17 of 45

17 KAT Status Based on Ontomat Annotizer Java Core Ontology for Multimedia (COMM) Provides Plugin-Infrastructure GUI framework Default components for image/video/audio annotation and browsing Framework finished First plugins being integrated Only simple GUI, will be improved during this year Released as Open Source in March/April Probably LGPL license Framework including Image Annotation Tool Browser Ontology Browser/Editor Coming next month from Understanding, 18 of 45

19 Motivation Core Question for us: Given sparse annotation data how to use relational information between objects? Sky Sea Sea Sand Understanding, 20 of 45

20 Objectives Identify meaningful regions in images Label regions with their contents Resulting labels are useful for Retrieval Inference of higher-level annotations Boosting image classification Sky Sea Sea Sand Understanding, 21 of 45

21 Analysis Framework Constraint Acquisition Hypotheses Generation Spatial Relation Extraction Spatial Reasoning Mining of spatial constraint templates from spatial prototypes (labelled examples). Templates are explicitly represented spatial arrangements of concepts. They comprise the background knowledge. Understanding, 22 of 45

22 Analysis Framework Constraint Acquisition Hypotheses Generation Spatial Relation Extraction Spatial Reasoning Segmentation of the input image Classification of each segment using multiple classifiers. Each classifier produces a degree of confidence for a given concept. The set of all concept-degree pairs comprise the hypotheses set for the given region. Understanding, 23 of 45

23 Analysis Framework Constraint Acquisition Hypotheses Generation Spatial Relation Extraction Spatial Reasoning Spatial Relations between the segments are extracted. Understanding, 24 of 45

24 Analysis Framework Constraint Acquisition Hypotheses Generation Spatial Relation Extraction Spatial Reasoning Creation of a Fuzzy Constraint Satisfaction Problem based on Hypotheses sets Spatial relations Constraint templates The best solution gives the final labelling of the image. Understanding, 25 of 45

25 Hypotheses Set Generation Comprises two steps Image Segmentation Segment Classification Segmentation Sky, 0.8 Sea,0.76 Sand, 0.68 Person, 0.67 Building, 0.54 Classification Understanding, 26 of 45

26 Spatial Relations Extraction Based on the center of the minimal bounding box of a segment. 4 directional spatial relations above-of, below-of, left-of, right-of 2 topological relations contains, adjacent 2 absolute spatial relations above-all, below-all Relative Spatial Relations Absolute Spatial Relations Understanding, 27 of 45

27 Region Labelling as a FCSP Goal: Find an assignment of labels to regions that is spatially consistent. Approach: Transform hypotheses sets and spatial relations into a Fuzzy Constraint Satisfaction Problem. Use background knowledge (i.e. previous observations) about spatial arrangements to define constraints. Understanding, 28 of 45

28 Fuzzy Constraint Satisfaction Problems Model to represent and solve systems of related variables. Set of fuzzy variables, defined on a set of domains Variables related by constraints D(X) x D(Y) -> [0,1] Y D(Z) x D(Y) -> [0,1] Constraints are fuzzy relations X Z Global evaluation function (joint degree of satisfaction) to assess quality of (partial) solutions Search/Global Optimisation Problem Understanding, 29 of 45

29 Fuzzy CSP - Domains Fuzzy Constraint Satisfaction Problem consists of: An ordered set of fuzzy variables V := {v 1,, v k } Each with crisp domain L i = {l 1,, l n } and membership function μ ι : L i [0, 1] μ ι (l), l L i, is called the degree of satisfaction of the variable for the assignment v i := l. Understanding, 30 of 45

30 Fuzzy CSP - Constraints Fuzzy Constraint Satisfaction Problem consists of: Fuzzy constraints C := {c 1,, c m }. Each constraint c j is defined on a set of variables v 1,,v q V, and Interpretation of c j : L q [0, 1] c(l 1,, l 1 ); v i = l i is degree of satisfaction of the variable assignment l 1,, l q for the constraint c. Full satisfaction: c(l 1,,l q ) = 1, Full violation: c(l 1,,l q ) = 0 Understanding, 31 of 45

31 Joint Degree of satisfaction Joint degree of satisfaction per Variable v i : weight Fully Not Fully Instantiated constraints DoS for each DoS for v i fully inst. constraint Overestimated DoS for each partially inst. contraints Understanding, 32 of 45

32 Region Labelling as a FCSP above-of above-of above-of left-of above-of Sand, 0.8 Sea, 0.7 Person,0.5 Hypotheses Domain Sand, 0.8 Sea, 0.7 Person,0.5 Each segment is represented as a Variable The membership function μ i is defined by the segment classifier (e.g. confidence, margins, ) Understanding, 33 of 45

33 Region Labelling as a FCSP above-of above-of above-of left-of above-of Sand, 0.8 Sea, 0.7 Person,0.5 Hypotheses Background Knowledge above-of (Sky, Sea) -> 1.0 (Sea, Sand) -> 1.0 (Sea, Sky) -> 0.0 Domain Sand, 0.8 Sea, 0.7 Person,0.5 (Sky, Sea) -> 1.0 (Sea, Sand) -> 1.0 (Sea, Sky) -> 0.0 Understanding, 34 of 45

34 Relationships between Segments Each relationship between two segments has a type. For each relationship type there exists background knowledge (i.e. a constraint template) defining constraints Relationship type for given labeling seen before at prototype, then dos=1.0 Otherwise dos=0.0 Understanding, 35 of 45

35 Background Knowledge Set of constraint templates acquired from examples (later) Each spatial relation is associated with one template Constraint template Explicitly represented fuzzy relation on the set of labels Use for instantiation of concrete constraints in FCSP Understanding, 36 of 45

36 Region Labelling as a FCSP above-of above-of above-of left-of above-of Sand, 0.8 Sea, 0.7 Person,0.5 Hypotheses Background Knowledge above-of (Sky, Sea) -> 1.0 (Sea, Sand) -> 1.0 (Sea, Sky) -> 0.0 Domain Sand, 0.8 Sea, 0.7 Person,0.5 (Sky, Sea) -> 1.0 (Sea, Sand) -> 1.0 (Sea, Sky) -> 0.0 Understanding, 37 of 45

37 Requirements for solving Region FCSPs Global optimisation problem Evaluation function Components: Primary: membership function according to segment classifier Secondary: spatial constraints Optimization: Maximize minimal degree of satisfaction: max min i { dos(v i ) } varassigns Recursively max min Search algorithms Branch-and-Bound Heuristics to boost efficiency Ordering based on naïve, DoS(v i ), most constraints, Understanding, 38 of 45

38 Evaluation Function above-of above-of above-of left-of above-of Use all available information to (over)estimate a variables degree of confidence. Domain Sand, 0.8 Sea, 0.7 Person,0.5 above-of (Sky, Sea) -> 1.0 (Sea, Sand) -> 1.0 (Sea, Sky) -> 0.0 Understanding, 39 of 45

39 Constraint Acquisition Manual definition of constraints is tedious Use mining strategy to acquire constraint templates Aim: find robust constraint definitions Examples are called Spatial Prototypes Using (support and) confidence for filtering Conf = (l 1, l 2,l n ) observed / (*, l 2,,l n ) observed Understanding, 40 of 45

40 Evaluation Data set of 930 images Ground truth defined on fixed segmentation mask from initial step No evaluation of segmentation performance 10 concepts supported person, boat, sand, building, road, mountain, water, sky, plant, snow Regions labelled with the dominant concept Ignoring regions depicting unsupported concepts regions showing no dominant concept Understanding, 41 of 45

41 Data set Understanding, 42 of 45

42 Experimental setup 151 images used for acquisition of constraints Threshold on confidence 30% Filtering on support did not influence results Spatial relations used Adjacent versions of above, below Merged left-of and right-of into left-right-of above-all and below-all 765 images used for evaluation precision/recall/f-measure on region level Understanding, 43 of 45

43 Preliminary Results person boat sand building road mountain water sky plant snow macro avg precision svm csp svm recall csp F-measure svm Understanding, 44 of 45 csp gain 4% 1% 9% 4% 89% 17% 3% 6% 1% -4% 7%

44 Results Spatial constraints led to improvement in general For some concepts (Snow, boat) it has not been helpful Overall gain comparable to other methods based on Conditional Random Fields Experiments indicate advantage on required training set size Experiments with different spatial relations In general: adding additional spatial relations does not improve results Few, but discriminative ones preferred Influence depends strongly on type of spatial relations Understanding, 45 of 45

45 Conclusions Spatial features improve region labelling performance Selection of discriminative relations fundamental FCSP appropriate for this problem Evaluation indicates that FCSPs require fewer training examples Understanding, 46 of 45

46 Future Work Set up of more rigorous experiment Compare performance with different training set sizes FCSP Graphical model Classification without spatial features Improve acquisition strategy Interactive acquisition Clustered images KAT Understanding, 47 of 45

47 Information Systems University of Koblenz Landau, Germany Thank you!

48 Information Systems SAMT 2008 University of Koblenz Landau, Germany Semantics and Digital Media Technology Koblenz, December 3-5, PC Chairs Lynda Hardman Alex Hauptmann Nadia Magnenat-Thalman General Chairs Dietrich Paulus