Knowledge integration in remote sensing image analysis

Transcription

1 Knowledge integration in remote sensing image analysis Cédric Wemmert ICube - BFO - University of Strasbourg Thematic School Image mining and information extraction from multi-source data Cédric Wemmert Knowledge integration in image analysis 1/36

2 Outline 1 Introduction 2 Skills 3 Contextual information 4 Domain knowledge 5 Some propositions 6 Conclusion Cédric Wemmert Knowledge integration in image analysis 2/36

3 Remote sensing image analysis Context Large and increasing amount of data (very high spatial resolution, time series,... ) Need of automatic or semi-automatic tools to analyze these data Lots of a priori information available from the experts (geography, geology,... ) Remote sensing image interpretation Goal : help an expert to understand the content of the image (urbanism, environment, geology, etc.) The interpretation consist in identifying the objects in the image and to provide a thematic map Cédric Wemmert Knowledge integration in image analysis 3/36

4 Semantical gap Semantical gap between objects of interest and data Information given by the pixel value (spectral information) is not enough One pixel does not correspond to one semantic object Cédric Wemmert Knowledge integration in image analysis 4/36

5 Semantical gap Semantical gap between objects of interest and data Information given by the pixel value (spectral information) is not enough One pixel does not correspond to one semantic object Cédric Wemmert Knowledge integration in image analysis 4/36

6 Semantical gap Semantical gap between objects of interest and data Information given by the pixel value (spectral information) is not enough One pixel does not correspond to one semantic object Cédric Wemmert Knowledge integration in image analysis 4/36

7 Semantical gap Semantical gap between objects of interest and data Information given by the pixel value (spectral information) is not enough One pixel does not correspond to one semantic object Cédric Wemmert Knowledge integration in image analysis 4/36

8 Semantical gap Semantical gap between objects of interest and data Information given by the pixel value (spectral information) is not enough One pixel does not correspond to one semantic object Cédric Wemmert Knowledge integration in image analysis 4/36

9 Semantical gap Semantical gap between objects of interest and data Information given by the pixel value (spectral information) is not enough One pixel does not correspond to one semantic object It is mandatory to use more information or knowledge that only the spectral information given by the pixels. Cédric Wemmert Knowledge integration in image analysis 4/36

10 Knowledge Knowledge in image analysis Context data and information about the data (raw data, parameters,... ) Knowledge Skills ability to solve partly or totally the problem (algorithms, methods,... ) Domain general information about the domain (expert knowledge,... ) Cédric Wemmert Knowledge integration in image analysis 5/36

11 Context Need of new tools Using all available knowledge (skills, contextual information, domain knowledge) for images analysis Cédric Wemmert Knowledge integration in image analysis 6/36

12 Context Need of new tools Using all available knowledge (skills, contextual information, domain knowledge) for images analysis Disclaimer No solution will be given in this talk, only problems will be exposed! Cédric Wemmert Knowledge integration in image analysis 6/36

13 Skills Pixel-based approach Image Classification Cédric Wemmert Knowledge integration in image analysis 7/36

14 Skills Object-oriented approach µ, size, area,... Image Segmentation Characterization Classification Cédric Wemmert Knowledge integration in image analysis 8/36

15 Skills Extractors vegetation Image buildings Classification roads Cédric Wemmert Knowledge integration in image analysis 9/36

16 Skills Multi-strategy techniques OBIA: collaboration between segmentation and classification; collaborative clustering; collaboration of specific extractors; hybrid methods... Cédric Wemmert Knowledge integration in image analysis 10/36

17 Skills Multi-strategy techniques OBIA: collaboration between segmentation and classification; collaborative clustering; collaboration of specific extractors; hybrid methods... How making all these techniques collaborate? How combining results of different types? How dealing with conflicts? Cédric Wemmert Knowledge integration in image analysis 10/36

18 Context Contextual knowledge Contextual knowledge is composed of information on the specific problem you are dealing with: the data; some samples or none; available databases (buildings, roads,... ); number of classes to find; more generally: any parameter of the classification method. Cédric Wemmert Knowledge integration in image analysis 11/36

19 Context Contextual knowledge Contextual knowledge is composed of information on the specific problem you are dealing with: the data; some samples or none; available databases (buildings, roads,... ); number of classes to find; more generally: any parameter of the classification method. Parameters issue Most of the time, parameters are difficult to choose and do not correspond to any a priori knowledge of the expert on his data. The best you can have is the number of classes to find which does not necessarily correspond to the number of classes in the data... Cédric Wemmert Knowledge integration in image analysis 11/36

20 Samples Samples (labeled objects) are a very important information as they describe what the user expect. Cédric Wemmert Knowledge integration in image analysis 12/36

21 Samples Samples (labeled objects) are a very important information as they describe what the user expect. Supervised classification If you have enough samples you can provide a supervised classification. But sometimes you do not have enough samples or no sample for some classes (semi-supervised classification, supervised clustering); Cédric Wemmert Knowledge integration in image analysis 12/36

22 Samples Samples (labeled objects) are a very important information as they describe what the user expect. Supervised classification If you have enough samples you can provide a supervised classification. But sometimes you do not have enough samples or no sample for some classes (semi-supervised classification, supervised clustering); Cédric Wemmert Knowledge integration in image analysis 12/36

23 Samples Samples (labeled objects) are a very important information as they describe what the user expect. Supervised classification If you have enough samples you can provide a supervised classification. But sometimes you do not have enough samples or no sample for some classes (semi-supervised classification, supervised clustering); in the case of image analysis you cannot directly use the samples provided by the expert in the classification process. Cédric Wemmert Knowledge integration in image analysis 12/36

24 Samples Samples (labeled objects) are a very important information as they describe what the user expect. Supervised classification If you have enough samples you can provide a supervised classification. But sometimes you do not have enough samples or no sample for some classes (semi-supervised classification, supervised clustering); in the case of image analysis you cannot directly use the samples provided by the expert in the classification process. pixel based: mean of all pixel values? textured object? Cédric Wemmert Knowledge integration in image analysis 12/36

25 Samples Samples (labeled objects) are a very important information as they describe what the user expect. Supervised classification If you have enough samples you can provide a supervised classification. But sometimes you do not have enough samples or no sample for some classes (semi-supervised classification, supervised clustering); in the case of image analysis you cannot directly use the samples provided by the expert in the classification process. OBIA: the segments do not necessarily correspond to the samples... How to constraint a segmentation with samples? Cédric Wemmert Knowledge integration in image analysis 12/36

26 Databases A lot of databases are available that can bring information about the studied area. For example, free data from the IGN: Cédric Wemmert Knowledge integration in image analysis 13/36

27 Databases A lot of databases are available that can bring information about the studied area. For example, free data from the IGN: topography, altitudes, roads,... How using vector data into a raster image classification process? How evaluating the quality/accuracy/precision of the information? Correspondence in terms of date (is your information up-to-date?) Cédric Wemmert Knowledge integration in image analysis 14/36

28 Domain knowledge Domain knowledge is all the generic information about the objects you are looking for in the image. Cédric Wemmert Knowledge integration in image analysis 15/36

29 Domain knowledge Domain knowledge is all the generic information about the objects you are looking for in the image. How to store all these information? Which are interesting in the field of image analysis? How to fill the semantic gap between domain knowledge and image analysis? Cédric Wemmert Knowledge integration in image analysis 15/36

30 Geographical ontologies Representing expert knowledge into ontologies requires: formalizing the symbolic knowledge of an expert of a specific image object type in an ontology, associating this knowledge with image segments described through annotations that are based on the same ontology. Example Separation between the expert domain knowledge expressed by high-level concepts (e.g., photosynthetic vegetation is characterized by a high leaf chlorophyll content, which is correlated to high values of vegetation index) from the low-level concept features of image objects (e.g., measured NDVI value of an image object). Advances in Geographic Object-Based Image Analysis with ontologies: A review of main contributions and limitations from a remote sensing perspective. Damien Arvor, Laurent Durieux, Samuel Andrés, Marie-Angélique Laporte Cédric Wemmert Knowledge integration in image analysis 16/36

31 Geographical ontologies: an example Cédric Wemmert Knowledge integration in image analysis 17/36

32 Ontology issue General issues in ontological research building ontologies is difficult and time-consuming; involving domain experts in the construction of ontologies. Specific issues for the application in GEOBIA linking real-world concepts to image concepts; handling qualitative and quantitative information; vague and fuzzy geographic concepts; scale and resolution of the image; changement; processing performance. Cédric Wemmert Knowledge integration in image analysis 18/36

33 Ontology issue General issues in ontological research building ontologies is difficult and time-consuming; involving domain experts in the construction of ontologies. Specific issues for the application in GEOBIA linking real-world concepts to image concepts; handling qualitative and quantitative information; vague and fuzzy geographic concepts; scale and resolution of the image; changement; processing performance. Still a lot of work to do! Cédric Wemmert Knowledge integration in image analysis 18/36

34 Knowledge modeling Expert knowledge representation Acquisition and formalization of the knowledge data dictionary knowledge base building (hierarchy of concepts) Geographical object Road Vegetation Building Individual Collective Figure: Hierarchy of concepts Type Attribute Weight Values Spectral Shape min max blue green red near IR NDVI diameter area elongation Table: Example of concept Individual house. Cédric Wemmert Knowledge integration in image analysis 19/36

35 Knowledge integration in the process Knowledge can be used: to automatically interpret the image, to enhance the segmentation, to drive the clustering, to create specific extractors. Cédric Wemmert Knowledge integration in image analysis 20/36

36 Knowledge and interpretation Knowledge can be used to directly interpret the segmentation of the image Attributes of the segments are compared to those of the concepts of the hierarchy Knowledge Image Segmentation Interpretation N. Durand et al. Ontology-based Object Recognition for Remote Sensing Image Interpretation, IEEE International Conference on Tools with Artificial Intelligence, Cédric Wemmert Knowledge integration in image analysis 21/36

37 Knowledge and interpretation (a) Quickbird image of Strasbourg (France) - 900x m/px. (b) Classification obtained directly by identification with the concepts. x x Road Building x Water x Unknown x Vegetation Cédric Wemmert Knowledge integration in image analysis 22/36

38 Knowledge and interpretation (c) Quickbird image of Strasbourg (France) - 900x m/px. (d) Classification obtained directly by identification with the concepts. x x Road Building x Water x Unknown x Vegetation Cédric Wemmert Knowledge integration in image analysis 22/36

39 Knowledge and interpretation (e) Quickbird image of Strasbourg (France) - 900x m/px. (f) Classification obtained directly by identification with the concepts. x x Road Building x Water x Unknown x Vegetation Cédric Wemmert Knowledge integration in image analysis 22/36

40 Knowledge and interpretation Problem No matching between objects of interest and segments Cédric Wemmert Knowledge integration in image analysis 23/36

41 Knowledge and interpretation Problem No matching between objects of interest and segments Cédric Wemmert Knowledge integration in image analysis 23/36

42 Knowledge and interpretation Problem No matching between objects of interest and segments Cédric Wemmert Knowledge integration in image analysis 23/36

43 Knowledge and interpretation Problem No matching between objects of interest and segments Cédric Wemmert Knowledge integration in image analysis 23/36

44 Knowledge and interpretation Problem No matching between objects of interest and segments Cédric Wemmert Knowledge integration in image analysis 23/36

45 Knowledge and interpretation Propositions Using knowledge to drive the segmentation Using knowledge during the collaborative clustering to identify the clusters Knowledge Image Segmentation Clustering Interpretation Cédric Wemmert Knowledge integration in image analysis 24/36

46 Knowledge and segmentation Genetic algorithm parameters optimization by evolutionary algorithm fitness = recognition rate by the knowledge base (g) 37%, (1 st generation) (h) 48%, (7 th generation) (i) 56%, (15 th generation) S. Derivaux et al., Supervised Image Segmentation Using Watershed Transform, Fuzzy Classification and Evolutionary Computation, Pattern Recognition Letters, 2010 Cédric Wemmert Knowledge integration in image analysis 25/36

47 Knowledge and collaborative clustering The collaborative clustering is applied on the segments given by the segmentation Purity index to evaluate the purity of the clusters in terms of classes (concepts) are integrated in the quality criterion Knowledge Image Segmentation Clustering Interpretation G. Forestier et al. Knowledge-based region labeling for remote sensing image interpretation, Computers, Environment and Urban Systems, Cédric Wemmert Knowledge integration in image analysis 26/36

48 Knowledge and extractors Extractors issues An extractor integrates its own knowledge problem of reusability difficult to integrate new knowledge (needs new development) Use knowledge to build the extractors knowledge is formalized on concepts a process builds an extractor for each concept, according to the knowledge Knowledge Extractor 1 Image Building of extractors Extractor 2 Interpretation... Cédric Wemmert Knowledge integration in image analysis 27/36

49 Knowledge and extractors Filters approach Some filters are generated for each concept Each filter is applied successively Three kinds of filters: pixel : remove pixels that do not validate a condition shape : keep only segments compatible with a shape criteria object : consider each conned component as an object and keep only those validating a condition Cédric Wemmert Knowledge integration in image analysis 28/36

50 Introduction Skills Contextual information Domain knowledge Some propositions Conclusion Knowledge and extractors Example on individual house I Description of an individual house: spectral type: mineral shape: rectangular area: between 40 and 200 sq.m C edric Wemmert Knowledge integration in image analysis 29/36

51 Introduction Skills Contextual information Domain knowledge Some propositions Conclusion Knowledge and extractors Example on individual house I Description of an individual house: spectral type: mineral shape: rectangular area: between 40 and 200 sq.m C edric Wemmert Knowledge integration in image analysis 29/36

52 Introduction Skills Contextual information Domain knowledge Some propositions Conclusion Knowledge and extractors Example on individual house I Description of an individual house: spectral type: mineral shape: rectangular area: between 40 and 200 sq.m C edric Wemmert Knowledge integration in image analysis 29/36

53 Spatial relations Not only spectral and shape attributes can be calculated Relations between objects can be computed: neighborhood; proximity; south, north, east, west; surrounding... Cédric Wemmert Knowledge integration in image analysis 30/36

54 Spatial relations Example: coastal area interpretation Cédric Wemmert Knowledge integration in image analysis 31/36

55 Spatial relations Example: coastal area interpretation Cédric Wemmert Knowledge integration in image analysis 31/36

56 Knowledge functions Mineral Vegetation Water Classes Knowledge Building Neighbors: Wooded area, Pond Compact form Slikke Neighbors: Salt meadow, Channel, Sea, Beach Beach Neighbors: Slikke, Sea Linear form Neighbors: Beach Dune Elevation > 5m Altitude variation (±5m) Neighbors: Slikke Distance to the sea: [5; 6]m Salt meadow Distance to a channel: [5; 6]m Elevation < 5m No altitude variation Wooded area Neighbors: Building, Field Surface: [100; 200]m 2 Field Distance to the sea: [15; 20]m Rectangular form Pond Neighbors: Salt meadow, Wooded area, Building Surface: [100; 200]m 2 Sea Neighbors: Slikke, Beach, Channel, Salt meadow Channel Neighbors: Slikke Cédric Wemmert Knowledge integration in image analysis 32/36

57 Knowledge functions The knowledge is formalized as knowledge functions that will be applied after a first classification and identification of the segments: O(r): returns the score associated to the best ontology hypothesis. S(r): knowledge on the shape of a concept and a segment. E(r): knowledge on the elevation of a concept and a segment. N (r): knowledge of the potential concepts in the neighborhood of a segment. D(r): knowledge about distance to other concepts of a segment. Cédric Wemmert Knowledge integration in image analysis 33/36

58 Spatial relations (j) The raw image. (k) The segmentation of the image (border of the region are highlighted in blank). (l) The elevation map (the brighter, the higher) Cédric Wemmert Knowledge integration in image analysis 34/36

59 Spatial relations (m) Groundtruth. (n) Before the KFs application. (o) After the KFs application. X Building X Beach X Slikke X Wooden area X X Field Dune X Water X Salt meadow Cédric Wemmert Knowledge integration in image analysis 35/36

60 Knowledge integration in image analysis Many knowledge are available: methods, samples, databases, ontologies... It is difficult to choose which have to be used and can provide relevant information to have a better result How to store/organize the information? ontologies, databases, knowledge functions, fuzzy rules, multi-agent system... There exists no generic method or workflow to face any problem The problem of the correspondence between the geographic objects and the image objects is one of the keys Cédric Wemmert Knowledge integration in image analysis 36/36

61 Knowledge integration in image analysis Many knowledge are available: methods, samples, databases, ontologies... It is difficult to choose which have to be used and can provide relevant information to have a better result How to store/organize the information? ontologies, databases, knowledge functions, fuzzy rules, multi-agent system... There exists no generic method or workflow to face any problem The problem of the correspondence between the geographic objects and the image objects is one of the keys This is still a very active research field, your contributions are welcome! Cédric Wemmert Knowledge integration in image analysis 36/36