A Semantic Model for Multimodal Data Mining in Healthcare Information Systems. D.K. Iakovidis & C. Smailis

Transcription

1 A Semantic Model for Multimodal Data Mining in Healthcare Information Systems D.K. Iakovidis & C. Smailis Department of Informatics and Computer Technology Technological Educational Institute of Lamia, Greece Supported in part by the European Commission s Seventh Framework Information Society Technologies (IST) Programme, project DEBUGIT (no )

2 Multimodal / Multisource Medical Data Signals Images Reports

3 Data Mining Extraction of implicit information / knowledge from data Supervised data mining Requires prior knowledge about the data Classification Automatic data annotation Discovery of rules Unsupervised data mining Requires no prior knowledge about the data Clustering Discovery of similarities / relations between data

4 Data Mining Ontologically Represented Semantics Low-level feature extraction Clusters of feature vectors? Lungs Ribs Spinal cord Heart Findings Data Information abstraction

5 Multimodal / Multisource Data Mining Signals Feature spaces Reports? => Semantic Results Decisions Findings Rules Images

6 Multimodal / Multisource Data Mining Feature spaces Intensity Semantic Results Image? Lungs Ribs Texture => Spinal cord Heart Findings

7 Previous Works Feature level fusion Feature vector concatenation and other combinations Dissimilarity-based approaches (Bruno et al, 2009, Iakovidis et al, 2011 ) Issues Normalization of data/features obtained from different sources Selection of appropriate dissimilarity (distance) measures Semantic level fusion Ontological approaches Issues Domain specific or require much manual ontology development Different focus e.g. interoperability (Theodoro et al, 2011 )

8 Qualitative Spatial Semantics Objects in a 2-D or 3-D space have spatial relations Semantic model of fuzzy spatial relations for image interpretation Spatial relations are modeled as concepts, not as object properties (Hudelot et al, 2008) Object A Object B B_RightOf_A Image

9 Generalized Qualitative Spatial Semantics (GQSS) Generalization Clusters of feature vectors are spatial objects in an N-D space Spatial relations are considered per Axis of this space Axis 1 A Axis 3 + B Axis 2 (feature 2 values)

10 Generalized Qualitative Spatial Semantics (GQSS)

11 Generalized Qualitative Spatial Semantics (GQSS) A vector space may be defined by many axes that can also belong to other vector spaces as well VectorSpace ( definedby.axis) ( definedby.axis) Concept Axis represents an axis that may define one or more vector spaces at the same time Axis ( defines.vectorspace) ( defines.vectorspace)

12 Generalized Qualitative Spatial Semantics (GQSS) SpatialRelation refers to the set of spatial relations defined according to a reference object and a target object across an Axis SpatialRelation ( reference.object) ( target.object) ( hasaxis.axis) ( hasspace.vectorspace) ( reference.object) ( target.object) ( hasaxis.axis) ( hasspace.vectorspace) (=1 reference) (=1 target) (=1 hasaxis) (=1 hasspace) Each spatial relation must necessarily have one reference object and one target object as well as a Vector Space, within which it is defined.

13 Generalized Qualitative Spatial Semantics (GQSS) NumericValue indicates the number of intermediate objects (or their absence if this value represents zero e.g. Value-0), between the projections of two objects on this axis DirectionalRelation refers to the set of relations implying direction across an axis DirectionalRelation SpatialRelation ( numberofintermediateobjects. NumericValue) (= 1 numberofintermediate Objects).DirectionalRelation

14 Multimodal Mining of ICU Data Task: Learn and identify (annotate) pneumonia samples Dataset 25 patients Structured data/features Body temperature Blood gasses Chest X-Rays Intensity histogram features Gabor texture features Clustering Non-negative matrix factorization initialized by Fuzzy C-Means (Iakovidis et al., 2009) Reasoning Fact++ (

15 Supervised Data Mining based on GQSS Given a multimodal training set of an application domain: For each modality / data source Extract feature vectors Perform clustering Extract GQSS and associate them with semantics of the real-world Automatically generate ontology to represent knowledge Given a new multimodal dataset of the same domain For each modality / data source Extract feature vectors Perform clustering Extract GQSS and represent them as individuals of SpatialObject within the automatically generated ontology Perform reasoning to infer and assign labels to each cluster

16 Automatic Ontology Generation 2D Sammon s Mapping of Spaces Generated Ontology Intensity Space Fixed part Normal Pneumonia Intensity Texture Texture Space Dynamic part

17 Automatic Ontology Generation ClusterA target.( ( reference.r-ab) ( hasaxis.{axisy}) ( hasspace. {Modality1Space}) Equal) target( ( numberofintermediateobjects.{value-0}) ( reference.r-ab) ( hasaxis.{axisx}) ( hasspace.{modality1space}) NegativeDirectionalRelationship)

18 Automatic Ontology Generation ClusterA object instances need to participate as target object in two spatial relations. - The first one is an equal topological relation that takes place on Y-Axis of Modality1Space and has a reference object of the type R-AB. - - The second one, is a Negative directional Relationship that takes place on X-Axis of Modality1Space and has an object of type R- AB as a reference object. - In X-axis, zero intermediate objects exist between the non-classified object and R-AB and the non classified object.

19 Results Clustering average accuracy 94.1% Cross-validation 10% Training, 90% Testing Identification of pneumonia cases (i.e. automatic annotation of the clusters) 100%

20 Results (additional) Simulated data Randomly generated clusters Cardinality 150 Number of clusters 2,3,4,5 0 15% overlap Modalities / Feature spaces Number of modalities 1,2,3 Cross-validation 10% Training, 90% Testing Results Automatic annotation of clusters 100%

21 Conclusions The proposed semantic model Is based on Generalized Qualitative Spatial Semantics (GQSS) Is generic, not limited by any application domain Enables a uniform representation of knowledge across different modalities Multimodal data mining based on GQSS Simplifies the dissimilarity selection problem Does not suffer from normalization issues Copes efficiently with the semantic gap by directly associating the objects in feature spaces with formally represented semantics Its application for classification of multimodal ICU data Verified its effectiveness for automatic data annotation Small training set, not necessarily ellipsoidal clusters

22 Demonstration GQSS has been integrated into Ratsnake annotation tool

23 Thank you