How To Create A Medical Information System From A Medical Record

Transcription

1 N d ordre 2010-ISAL-0120 Année 2010 THESE Présentée devant L Pour obtenir LE GRADE DE DOCTEUR Spécialité : INFORMATIQUE Ecole doctorale : INFORMATIQUE ET MATHEMATIQUES Par Hossam Aldin JUMAA Titre AUTOMATISATION DE LA MEDIATION ENTRE XML ET DES BASES DE DONNEES RELATIONNELLES XML AND RELATIONAL DATABASES MEDIATION AUTOMATION Soutenue le 16 décembre 2010 devant le Jury composé de : Jury MM. Bruno DEFUDE Professeur, Institut Telecom, Telecom SudParis Examinateur Jocelyne FAYN Ingénieur de recherche HDR, MTIC, INSERM, Bron Co-directrice de thèse Frank MORVAN Professeur, IRIT, Univ. Paul Sabatier, Toulouse Rapporteur, examinateur Paul RUBEL Professeur Emérite, MTIC, INSA de Lyon Directeur de thèse Christine VERDIER Professeur, LIG, Univ. Joseph Fourier, Grenoble Rapporteur, examinateur Laboratoire MTIC: Méthodologies de Traitement de l'information en Cardiologie Université Claude Bernard Lyon 1, INSA de Lyon, Bron, France

2

3 INSA Direction de la Recherche - Ecoles Doctorales Quadriennal SIGLE ECOLE DOCTORALE NOM ET COORDONNEES DU RESPONSABLE CHIMIE E.E.A. E2M2 EDISS CHIMIE DE LYON M. Jean Marc LANCELIN Insa : R. GOURDON ELECTRONIQUE, ELECTROTECHNIQUE, AUTOMATIQUE M. Alain NICOLAS Insa : C. PLOSSU ede2a@insa-lyon.fr Secrétariat : M. LABOUNE AM Fax : EVOLUTION, ECOSYSTEME, MICROBIOLOGIE, MODELISATION M. Jean-Pierre FLANDROIS Insa : H. CHARLES INTERDISCIPLINAIRE SCIENCES- SANTE Sec : Safia Boudjema M. Didier REVEL Insa : M. LAGARDE M. Jean Marc LANCELIN Université Claude Bernard Lyon 1 Bât CPE 43 bd du 11 novembre VILLEURBANNE Cedex Tél : Fax : lancelin@hikari.cpe.fr M. Alain NICOLAS Ecole Centrale de Lyon Bâtiment H9 36 avenue Guy de Collongue ECULLY Tél : Fax : eea@ec-lyon.fr Secrétariat : M.C. HAVGOUDOUKIAN M. Jean-Pierre FLANDROIS CNRS UMR 5558 Université Claude Bernard Lyon 1 Bât G. Mendel 43 bd du 11 novembre VILLEURBANNE Cédex Tél : Fax e2m2@biomserv.univ-lyon1.fr M. Didier REVEL Hôpital Cardiologique de Lyon Bâtiment Central 28 Avenue Doyen Lépine BRON Tél : Fax : Didier.revel@creatis.uni-lyon1.fr INFOMATHS INFORMATIQUE ET MATHEMATIQUES M. Alain MILLE M. Alain MILLE Université Claude Bernard Lyon 1 LIRIS - INFOMATHS Bâtiment Nautibus 43 bd du 11 novembre VILLEURBANNE Cedex Tél : Fax infomaths@bat710.univ-lyon1.fr - alain.mille@liris.cnrs.fr Matériaux MATERIAUX DE LYON M. Jean Marc PELLETIER Secrétariat : C. BERNAVON M. Jean Marc PELLETIER INSA de Lyon MATEIS Bâtiment Blaise Pascal 7 avenue Jean Capelle VILLEURBANNE Cédex Tél : Fax Jean-marc.Pelletier@insa-lyon.fr MEGA MECANIQUE, ENERGETIQUE, GENIE CIVIL, ACOUSTIQUE M. Jean Louis GUYADER Secrétariat : M. LABOUNE PM : Fax : M. Jean Louis GUYADER INSA de Lyon Laboratoire de Vibrations et Acoustique Bâtiment Antoine de Saint Exupéry 25 bis avenue Jean Capelle VILLEURBANNE Cedex Tél : Fax : mega@lva.insa-lyon.fr ScSo ScSo* M. OBADIA Lionel Insa : J.Y. TOUSSAINT M. OBADIA Lionel Université Lyon 2 86 rue Pasteur LYON Cedex 07 Tél : Fax : Lionel.Obadia@univ-lyon2.fr *ScSo : Histoire, Geographie, Aménagement, Urbanisme, Archéologie, Science politique, Sociologie, Anthropologie

4

5 Automatisation de la Médiation entre XML et des Bases de Données Relationnelles Résumé XML offre des moyens simples et flexibles pour l'échange de données entre applications et s'est rapidement imposé comme standard de fait pour l'échange de données entre les systèmes d'information. Par ailleurs, les bases de données relationnelles constituent aujourd hui encore la technologie la plus utilisée pour stocker les données, du fait notamment de leur capacité de mise à l échelle, de leur fiabilité et de leur performance. Combiner la souplesse du modèle XML pour l'échange de données et la performance du modèle relationnel pour l archivage et la recherche de données constitue de ce fait une problématique majeure. Cependant, l'automatisation des échanges de données entre les deux reste une tâche difficile. Dans cette thèse, nous présentons une nouvelle approche de médiation dans le but d automatiser l'échange de données entre des documents XML et des bases de données relationnelles de manière indépendante des schémas de représentation des données sources et cibles. Nous proposons tout d abord un modèle d architecture de médiation générique des échanges. Pour faciliter la configuration d interfaces spécifiques, notre architecture est basée sur le développement de composants génériques, adaptés à n'importe quelle source XML et n'importe quelle base de données relationnelle cible. Ces composants sont indépendants de tout domaine d'application, et ne seront personnalisés qu une seule fois pour chaque couple de formats de données sources et de stockage cible. Ainsi notre médiateur permettra la mise à jour automatique et cohérente de toute base de données relationnelle à partir de données XML. Il permettra aussi de récupérer automatiquement et efficacement les données d'une base de données relationnelle et de les publier dans des documents XML (ou messages) structurés selon le format d'échange demandé. La transformation en XML Schema d'un modèle relationnel constitue l un des éléments clé de notre médiateur. Nous proposons une méthodologie basée sur deux algorithmes successifs : l un de stratification des relations en différents niveaux en fonction des dépendances fonctionnelles existant entre les relations et les clés des relations, le deuxième de transformation automatique du modèle relationnel en XML Schema à partir de la définition d un ensemble de fragments types d encodage XML des relations, des attributs, des clés et des contraintes référentielles. La méthodologie proposée préserve les contraintes d'intégrité référentielles du schéma relationnel et élimine toute redondance des données. Elle a

6 été conçue pour conserver la représentation hiérarchique des relations, ce qui est particulièrement important pour la génération de requêtes SQL correctes et la mise à jour cohérente des données. Notre approche a été appliquée et testée avec succès dans le domaine médical pour automatiser l échange de données entre une représentation XML du protocole de communication standard SCP-ECG, une norme ISO décrivant un format ouvert de représentation de bio-signaux et métadonnées associées, et un modèle relationnel européen de référence qui inclut notamment l archivage de ces données. L'automatisation de la médiation est particulièrement pertinente dans ce domaine où les électrocardiogrammes (ECG) constituent le principal moyen d investigation pour la détection des maladies cardio-vasculaires et doivent être échangés rapidement et de manière transparente entre les différents systèmes de santé, en particulier en cas d'urgence, sachant que le protocole SCP-ECG a de nombreuses implémentations puisque la plupart des sections et des champs de données sont optionnels. Mots-Clés: XML, Bases de données relationnelles, Médiation, Echange de données automatisé.

7 XML and Relational Databases Mediation Automation Abstract XML has rapidly emerged as a de facto standard for data exchange among modern information systems. It offers simple and flexible means to exchange data among applications. In the meanwhile, relational databases are still the most used data storage technology in almost all information systems because of their unique features of scalability, reliability and performance. Thus, a crucial issue in the data management is to bring together the flexibility of the XML model for data exchange and the performance of the relational model for data storage and retrieval. However, the automation of bi-directional data exchange between the two remains a challenging task. In this dissertation, we present a novel mediation approach to automate data exchange between XML and relational data sources independently of the adopted data structures in the two data models. We first propose a generic mediation framework for the data exchange between any XML document and any existing relational database. The architecture of our proposed framework is based on the development of generic components, which will ease the setup of specific interfaces adapted to any XML source and any target relational database. The mediator components are independent of any application domain, and need to be customized only once for each couple of source and target data storage formats. Hence, our mediator provides automatic and coherent updates of any relational database from any data embedded in XML documents. It also allows to retrieve data from any relational database and to publish them into XML documents (or messages) structured according to a requested interchange format. The transformation from a Relational Model to XML represents a main key component of the proposed mediator. Thus, we proposed a methodology and devised two algorithms to efficiently and automatically transform the relational schema of a relational database management system into an XML schema. Our transformation methodology preserves the integrity constraints of the relational schema and avoids any data redundancy. It has been designed in order to preserve the hierarchical representation of the relational schema, which is particularly important for the generation of correct SQL updates and queries in the proposed mediation framework. Another key component is the automation of the SQL generation. Therefore, we devised a generic methodology and algorithms to automate the SQL queries generation that are required to update or retrieve data to/from the relational databases. Our proposed framework has been successfully applied and tested in the healthcare domain between an XML representation of SCP-ECG, an

8 open format ISO standard communications protocol embedding bio-signals and related metadata, and an European relational reference model including these data. The mediation automation is especially relevant in this field where electrocardiograms (ECG) are the main investigation for the detection of cardiovascular diseases, and need to be quickly and transparently exchanged between health systems, in particular emergency, whereas the SCP-ECG protocol has numerous legacy implementations since most of the sections and of the data fields are not mandatory. Keywords: XML, Relational databases, Mediation, Data exchange automation.

9 Table of Contents Chapter Introduction Introduction Context & Motivation Objective of Research Scientific Contributions Dissertation Organization... 5 Chapter State of the Art & Related Work Data Integration towards XML & Relational Database Integration Introduction Data integration: the basic concepts Data integration definition Data integration scenarios Data integration architectures Mediator based data integration architecture Data exchange architecture Peer-to-peer data integration architecture Mapping composition in data integration Data exchange vs. data integration Main open problems in data integration XML Introduction to XML XML schema languages (DTD vs. XSD) XML schemas advantages Document Type Definition language XML Schema Definition language XSD vs. DTD XML query languages XPath XSLT XQuery XQuery update facility XML parsers... 29

10 Contents DOM SAX XML data management and relational databases Introduction to XML data management XML vs. relational database XML & relational databases integration Relational publishing in XML XML storage into relational databases Relational databases update from XML Bidirectional data exchange between XML and relational databases Discussion Conclusion ehealth Challenges & Applications Introduction ehealth European research challenges in ehealth Standard Communications Protocol for Computer-Assisted Electrocardiography (SCP-ECG) project Open European Data Interchange and Processing for Electrocardiography (OEDIPE) Project Information Retrieval in Healthcare Interoperability in Healthcare Interoperability definition Interoperability in healthcare definition Electronic Health Record (EHR) EHR definition EHR benefits Implementation of the Electronic Health Record in France, called DMP EHR interoperability Standard Architecture for Healthcare Information Systems: HISA EHRcom standard HL7 standard Integrating the Healthcare Enterprise (IHE) Digital Imaging and Communications in Medicine (DICOM) Synthesis of the EHR standards Conclusion Chapter XML and Relational Databases Mediation Automation A Generic Mediation Architecture Introduction General data exchange scenarios Mediation framework design Mediator s components XSD Generator from XML XSD Converter from R Schema Schemata Manager SQL Generator Mediator functions HOSSAM JUMAA ii

11 Contents XML to DB function DB to XML function Discussion Conclusion Automation of the Transformation from Relational Models to XML Schemas Introduction Transformation methodology Relations stratification definitions Stratification algorithm Algorithm description XML Schema fragments templates Root fragment template Relation fragment template Data type and domain fragment template Attribute fragment template Primary key constraint fragment template Foreign key constraint fragment template Nesting fragment template Key reference fragment template Uniqueness constraint fragment template Non-nullable constraint fragment template Target XML Schema transformation algorithm Discussion Conclusion Automatic SQL Generation Introduction SQL auto-generation methodology SQL auto-generation to update relational DBs from any XML format Step1: SQL select generation Step2: SQL update/insert generation SQL auto-generation to query and retrieve data from any RDB and to embed them into any XML format Conclusion Chapter Validation & Application in ehealth Introduction SCP-ECG protocol OEDIPE reference relational model for the storage of ECG data Data acquisition sub-model XML based mediation for data exchange between SCP-ECG files and relational databases Meta-model manager Protocol meta-model Database meta-model Mapping meta-model Rule-base Mediator functions Protocol to database function Database to protocol function HOSSAM JUMAA iii

12 Contents 4.7 Discussion Conclusion Chapter Conclusion Summary Directions for future research HOSSAM JUMAA iv

13 Contents Table of figures Figure 2-1 Mediator based data integration architecture Figure 2-2 Data exchange architecture Figure 2-3 Peer to peer data integration architecture Figure 2-4 XML data management scenarios Figure 2-5 The interoperability framework of SIS architecture Figure 2-6 The HISA Standard Architecture [CEN/TC ] Figure 2-7 Extract from the standard reference model EHRcom around the root class EHR_EXTRACT [Kalra 2004] Figure 2-8 Defining the structure of an HL7 v3 message Figure 2-9 Preview of Clinical Document Architecture in the HL7 XML format Figure 3-1 The exchange scenarios between XML files and relational databases Figure 3-2 Overall architecture of the proposed data interchange XML-based mediator between XML documents and relational database management systems Figure 3-3 Global functional architecture of the mediation solution between XML documents and relational databases Figure 3-4 XML to DB function Figure 3-5 DB to XML function Figure 3-6 Relational database model to XML Schema transformation Figure 3-7 A principle example of stratification results into three levels (figure 3-7-a) from a physical relational model (figure 3-7-b) Figure 3-8 Automatic SQL generation schema (XML to RDB function) Figure 3-9 Automatic SQL generation schema (DB to XML function) Figure 4-1 SCP-ECG data structure [CEN/TC ] Figure 4-2 SCP-ECG section layout overview Figure 4-3 OEDIPE data acquisition sub-model (physical schema) [Fayn et al. 1994a], [Fayn et al. 1994b] Figure 4-4 XML based mediator architecture for automating the storage of SCP-ECG data into relational databases Figure 4-5 Overall view of the XML schema of the SCP-ECG protocol meta-model. 112 Figure 4-6 XMLSpy Editor snapshot of a part of section 1 (Header Informationpatient data/ecg acquisition data) of the SCP-ECG protocol XML schema Figure 4-7 Sample result of the OEDIPE ECG acquisition relational schema stratification according to algorithm 1 described in section We obtained four levels Figure 4-8 XMLSpy Editor snapshot of an excerpt of the target XML schema generated by the transformation algorithm 2 described in section Figure 4-9 Snapshot of the XSLT template that maps date of birth from the SCP-ECG protocol to the database meta-model instance Figure 4-10 The two main functions of the mediator HOSSAM JUMAA v

14 Contents HOSSAM JUMAA vi

15 Chapter 1 - INTRODUCTION 1.1 Introduction Thanks to the rapid development of Information and Communication Technologies (ICTs), we observe that the world changes from an industrial society to an information society where information is considered as one of the most important assets for any organization. The methods of communication have also changed considerably. Indeed, the mobile phone, electronic mail and videoconferencing offer new options to share information. This increases the possibility for individuals and organizations to share their experiences and knowledge. Therefore, in most domains, enterprises and organizations seek for sharing their data and open their systems to collaborate with their partners. That makes the design of more intelligent, pervasive and cost-effective information systems one of the most challenging goals. In fact, we have recently seen an extraordinary evolution in terms of design of new digital systems and intelligent communication, from mobile phone to computing and data grids, via the smart PDA and intelligent clothing incorporating a multitude of sensors. However, modern enterprises and organizations are still largely relying on their old legacy applications and systems to carry out their daily business. Therefore, the design of new information systems today becomes more challenging especially when they should integrate numerous heterogeneous and distributed systems equipped with a mix of old and new technologies. For instance, in the ehealth field, healthcare has become a cooperative process that involves several individuals with different but

16 CHAPTER 1 - Introduction complementary tasks and skills (e.g. general practitioners, specialists, nurses, pharmacists, social assistants, etc.). Moreover, people mobility, long duration treatment, stakeholders multiplicity, huge volume of data generated and exchanged are factors that require new, more intelligent and pervasive medical information systems that are able to retrieve the patient related information from any Electronic Health Record (EHR) or any medical information system in order to improve the quality of care and of the patients follow-up [Verdier 2006]. Indeed, there are many problems to be faced in the medical informatics and ehealth domain. The main ICT challenge in the healthcare field is to enable more informed decisionmaking and more cost-effective use of resources [Dzenowagis and Kernen 2005] by developing new medical information systems capable to give the right information, when and where needed. To fulfil these objectives, ICTs must track and provide patient data and ensure that they are correct and complete. In addition, they must ensure an effective and timely communication, between patients and professionals as well as between healthcare professionals, to enable efficient sharing of medical knowledge and increase the quality of care. Finally, the ICTs must be able to deliver high quality services despite distance and time. To achieve the goal of designing more intelligent, pervasive and cost-effective information systems by adopting and using the new ICTs, we aim to design new generic models and new methodologies which shall facilitate the access to the patient s related information, when and where needed, from any electronic health record or medical system anywhere and anytime, and to assist the healthcare providers in taking the right decision about the health of the patient. Therefore, we face different challenging issues: information retrieval, interoperability, data integration from different sources and data exchange. 1.2 Context & Motivation Data integration and data exchange are major research challenges in the domain of information systems. Nowadays, XML is a key technology in this field. Enterprises increasingly collaborate through XML based data exchange technologies and Web services. However, enterprise information systems mainly store data in relational databases (RDB) which are a wellexperimented, dominant technology for structured data storage and retrieval. RDBs allow to efficiently store, query and retrieve a huge volume of data. They are still the most important type of data sources. This situation has triggered several attempts to bring together the flexibility of the XML model as a data exchange standard and the performance of the relational model as a data storage and retrieval HOSSAM JUMAA 2

17 CHAPTER 1 - Introduction technology. Therefore, many systems and techniques have been developed for publishing relational databases in XML format and for storing XML data into new, compliant relational databases. However, the automation of bi-directional data exchange between the two remains a challenging task. In fact, there is not only a need to publish relational data in XML format, but also to update the existing relational databases from XML data. Hence, the problem of updating existing relational databases from XML data remains a challenge and has not received much attention. This operation should be automatic and transparent to the user, and should dynamically support open structures of incoming data. Indeed, it is all the more important to preserve business oriented database models because of their efficiency for data retrieval via legacy human-machine interfaces. Motivating example In this context, we were motivated by a real scenario from the cardiology field where the Electrocardiogram (ECG) remains one of the most important non-invasive diagnostic tools for early recognition of heart diseases. In fact, on the first hand, the patient s ECG data are often recorded by different acquisition devices and stored according to either proprietary or to the SCP-ECG format. SCP-ECG is an international standard communication protocol to exchange ECG data between health systems [ISO/DIS ]. It is an open protocol and may have many implementations since most of the sections and of the data fields are not mandatory. On the other hand, pre-existing relational databases store the ECG data according to heterogeneous schemas. A reference core database model is the OEDIPE model [Fayn et al. 1994a] which has been proposed within the framework of a European project for the storage of serial ECG data and related metadata. The model, composed of 50 tables and 200 attributes, has been designed to support a modular implementation in various sub-models according to different scenarios of use of the ECG data. Therefore, we need to find an efficient solution for storing any ECG data file created by any device using a proprietary format or compliant with the SCP-ECG protocol into any already existing relational database such as a sub-model of the OEDIPE relational model. A generic interface able to extract data from any SCP-ECG file coming from any manufactured device, and to automatically store these data, transparently for the user, into any relational database of health records, is thus needed. The outcomes of our research should be generic enough to be applicable to a wide range of other applications and domains. The e- commerce, for example, is another key domain where our research should HOSSAM JUMAA 3

18 CHAPTER 1 - Introduction have an important impact. Enterprises in that domain are engaged in business workflows and exchange XML-based data (e.g. Web service based business workflows). However, most of the exchanged data are stored in relational databases inside the enterprise information system. For instance, the enterprise information system may export some information about orders in XML format to its partners. The partners may then change that information (e.g. the quote, billing information, delivery address, etc) in the XML documents and return it back to the enterprise. Then the changed information in the returned XML documents (e.g. quote) must be imported correctly and transparently into the underlying relational databases. 1.3 Objective of Research We aim to design and develop an intelligent mediator, which allows to connect and map different representations (e.g. XML and RDB) to automate the data exchange. Thus we will tackle the problem of the interoperability in the data exchange between RDBs and XML and we will particularly focus on the problem of data exchange between collaborating and loosely coupled information systems. Hence, in this dissertation, we will present our research result about designing and developing new generic information models, IT architectures and methodologies to enable automatic and transparent data exchange between XML data format and relational databases. We are particularly interested in taking advantage of the new ICTs in order to access the right information when and where needed, by ensuring an efficient and transparent data exchange between different representations of data from different heterogeneous systems. Therefore, we propose a generic XML-based data mediation framework to exchange data between any open structure of XML documents and any relational database of a relational database management system. The proposed framework will then be applied and tested on an application from the cardiology domain to demonstrate the validity of our proposal. However, our solution may be used in other application domains. 1.4 Scientific Contributions In this dissertation, we aim to present a novel mediation approach to automate data exchange between XML and relational data sources independently of the adopted data structures in the two data models. The contributions of this thesis are as follows: We first propose a generic mediation framework for the data exchange between any XML document and any existing relational database. The architecture of our proposed framework is based on the development of HOSSAM JUMAA 4

19 CHAPTER 1 - Introduction generic components, which will ease the setup of specific interfaces adapted to any XML source and any target relational database. The mediator components are independent of any application domain, and need to be customized only once for each couple of generic source and target data storage formats. Hence, our mediator will provide automatic and coherent updates of any relational database from any data embedded in XML documents. It will also allow retrieving data from any relational database and publishing them into XML documents (or messages) structured according to a requested interchange format. The transformation from a relational model to XML represents a main key component of our proposed mediator. Thus, we will propose a methodology and devise two algorithms to efficiently and automatically transform the relational schema of a relational database management system into an XML schema. Our transformation methodology preserves the integrity constraints of the relational schema and avoids any data redundancy. It will be designed in order to preserve the hierarchical representation of the relational schema, which is particularly important for the generation of correct SQL updates and queries in the proposed mediation framework. Another key component is the automation of the SQL generation. Therefore, we will devise a generic methodology and algorithms to automate the SQL queries generation that are required to update or retrieve data to/from the relational databases. Finally, our proposed framework will be applied and tested in the healthcare domain between XML representations of SCP-ECG, an open format ISO standard communications protocol embedding bio-signals and related metadata, and a European relational reference model including these data. The mediation automation is especially relevant in this field where electrocardiograms (ECG) are the main investigation for the detection of cardiovascular diseases, and need to be quickly and transparently exchanged between health systems, in particular in emergency, whereas the SCP-ECG protocol has numerous legacy implementations since most of the sections and of the data fields are not mandatory. 1.5 Dissertation Organization The remainder of this dissertation is organized as follows: In chapter 2, we review the different works related to the interoperability challenges in both data integration and the ehealth domains. Therefore, this chapter is divided into two main parts: In the first part, we introduce the basic concepts from the data integration and the data HOSSAM JUMAA 5

20 CHAPTER 1 - Introduction exchange fields and introduce XML as a key technology in these fields. Then, we give a detailed literature review of previous works on the integration between XML and relational databases. In the second part, we review the particularity of the interoperability problem in the ehealth field and analyse the previous related work done in this field. In chapter 3, the core of our contributions, we present our new mediation approach for the automatic bi-directional data exchange between any XML source and any relational database. This chapter is divided into three main parts: In the first part (section 3.1), we describe the architecture of the proposed generic mediation framework to automate the data exchange between any XML file and any relational database. This framework supports the automatic updating of any relational database from any XML file as well as the retrieval of data from any relational database and their storage into any predefined or required XML format. In the second part (section 3.2), we describe a methodology for the automatic transformation of the relational model into XML schema. This transformation methodology has been adopted and used in our proposed mediator to automatically generate an XSD schema from any relational database model. In the third part (section 3.3), we present the automatic SQL generation methodology that has been used through our mediation framework. In chapter 4, we give a case study from the health domain where we have successfully applied and tested our proposed framework between an XML representation of the SCP-ECG standard communications protocol and the OEDIPE relational reference model including these data, in addition to other data and related meta-data. In Chapter 5, we provide some concluding remarks and discuss directions for future research. HOSSAM JUMAA 6

21 Chapter 2 - STATE OF THE ART & RELATED WORK This chapter presents the state of the art of the recent works related to the interoperability challenges in the data integration domain. In the first part, we introduce the basic concepts from the data integration and the data exchange fields and introduce XML as a key technology in these fields. Then, we give a detailed literature review of previous works about the integration between XML and relational databases. In the second part, we review the particularity of the interoperability problem in the ehealth domain and present some previous related work done in this field. 2.1 Data Integration towards XML & Relational Database Integration Introduction In today s modern information systems, data integration becomes a pervasive challenge, where querying data across multiple autonomous and heterogeneous collaborative information systems and data sources is crucial. Enterprises seek for more business opportunities in highly competitive markets. Thus, to meet the business requirements, the paradigm of new dynamic and cost-effective data integration for existing information systems becomes more and more important. Indeed, data integration facilitates information access and reuse, and allows to satisfy the business and customers needs by providing more complete information.

22 CHAPTER 2 - State of the Art & Related Work Data integration, as one of the oldest database research fields, has been well-studied and yields a large number of theoretical works in the literature [Lenzerini 2002], [Ziegler and Dittrich 2004], [Lenzerini 2005], [Halevy et al. 2006], [Ziegler and Dittrich 2007]. It has emerged shortly after the database systems were introduced into the business world. Many sub-fields and directions have been developed from data integration research. Authors in [Halevy et al. 2006] divide these directions into: generating schemas mapping, query answer processing, the role of XML, model management, peer to peer data management and the artificial intelligence role. In spite of the enormous research works that have been realized in each of these fields, there remain a lot of open problems and challenges to be addressed and solved. Extensible Markup Language (XML) [W3C 1998] has had an important role in the data integration domain over the last decade. There are several integration systems and solutions that have been developed using XML as the underlying data model and XML query as the query language. Indeed, XML plays increasingly an important role in the exchange of a wide variety of data on the Web and elsewhere. It offers a mechanism for structuring data for large-scale electronic publishing thanks to a common syntactic format. The XML technology and its related languages and derivatives (XML Schema, XSLT, XQuery ) now provide powerful tools for sharing, converting and exchanging information via networked computers, taking into account both the syntax and the semantic of the data. The remaining of this section is organized as follows: First, we review the main theoretical aspects and works in the data integration field. We introduce the problem of the data integration by its definition and its main scenarios. Next, we present the main data integration architectures: the classical mediation architecture, the data exchange architecture, and the peer to peer data integration architecture. Then, we give a formal comparison between the data exchange and data integration systems. After that, we review the different approaches and languages used in the mapping composition of the data integration systems: Global as View (GaV), Local as View (LaV), Global Local as View (GLaV) and BYU-Global-Local-as- View (BGLaV). Next, we summarize some of the remaining open problems in the data integration field. Secondly, we introduce the XML technology and its related languages and standards: XML schema languages (DTD and XSD), XML query languages (XPath, XSLT, XQuery and XQuery Updates Facilities languages) and the XML parsers (DOM and SAX). HOSSAM JUMAA 8

23 CHAPTER 2 - State of the Art & Related Work Finally, we review in detail the XML data management role in the data integration domain and focus on the problem of data integration when relational databases are used as the main data storage technology and XML is used as the main data exchange technology Data integration: the basic concepts Data integration is one of the oldest research fields in databases and it has been the focus of extensive theoretical work. However, numerous open problems of data integration remain to be solved [Ziegler and Dittrich 2004], [Halevy et al. 2006], [Ziegler and Dittrich 2007]. Hereafter, we will summarize some of the main recent papers in this research area Data integration definition Data integration is the process of combining data residing at different sources and providing the user with a unified and transparent access to these data [Ullman 2000], [Halevy 2001], [Lenzerini 2002]. Data integration is an important issue in business and scientific scenarios. In the business domain, it is often required to integrate the customer data that may be stored in different databases located at different companies. In scientific scenarios, it is often needed to combine the research results (i.e. experiments, findings, etc) that may be scattered across different repositories. Data integration aims to facilitate information access and reuse through a single virtual information access point and to have a more comprehensive view to satisfy the need. This problem becomes more important when the volume of data and the need of sharing are increasing Data integration scenarios We distinguish two main scenarios of data integration: (1) The tightly coupled approach, commonly known as data warehousing. This scenario of data integration consists in the replication of the data of interest from the external sources on a local server for locally working on these data. (2) The loosely coupled approach, known as data mediation. This scenario relies on querying a virtual global representation of data and providing query rewriting mechanisms to fetch just the necessary data from the sources at the query time. Indeed, in the first approach, data sources are largely dependent on each others. It is often used when latency is an important factor in the integrated system. However, most of the time, this approach is very expensive in terms of money and the efforts required to set up the integration system. Its architecture is not very flexible and it does not HOSSAM JUMAA 9

24 CHAPTER 2 - State of the Art & Related Work promote the reusability of code. Also, it makes changes much more difficult to do. In contrast, in the second scenario, data sources operate independently of each other. Communications between them is often done via a mediating queuing system. Therefore, the later approach is more used especially for its cost-effectiveness and flexibility Data integration architectures From the literature review we distinguish three types of data integration architectures: The mediator based, called classical data integration architecture. The data exchange architecture. The peer-to-peer data integration architecture. In the following sub-sections we will present each of these three different architectures Mediator based data integration architecture The mediator based architecture in [Wiederhold 1992] is one of the several architectures that have been proposed to deal with the problem of integrating heterogeneous information systems. This architecture, known also as classical data integration architecture, has been schematically depicted in Figure In this architecture the goal is to query heterogeneous data in different sources via a virtual global schema, such that all queries are expressed over this virtual global schema (called mediated schema) [Lenzerini 2005]. The data are stored in a set of heterogeneous and autonomous sources which are wrapped by wrappers or translators to provide a uniform data model view of the data stored in the sources. The mediators combine the answers coming from wrappers and/or other mediators. Figure 2-1 Mediator based data integration architecture HOSSAM JUMAA 10

25 CHAPTER 2 - State of the Art & Related Work Indeed, a mediator is a smart software needed to improve access to the type and to the quality of the information needed, for decision making for instance. Mediators are common in software design but their architectures and their implementations are various. The main advantage of mediation is to manage complexity and ensure the maintainability of the data integration system. According to [Lenzerini 2002], a data integration system I is formally a triple <G, S, M> where: G is the global schema, i.e. a set of global relational symbols, each one with an associated arity, plus a set of integrity constraints expressed over such relational symbols. S is the source schema, i.e. a set of relational symbols (disjoint from G), that constitutes a relational representation of the data stored at the sources. M is the mapping between G and S, constituted by a set of assertions of the form [qs4 qg], where: qs is a conjunctive query over the source schema and qg is a conjunctive query over the global schema (for a review of the different approaches in the mapping compositions see section 2.1.4). Starting from a source database instance D, the semantics of a data integration system I is defined by Lenzerini as the set of all the global databases B that respect the following conditions: B is a legal database instance for G (it respects the relational structure of G, and satisfies the integrity constraints expressed on it). B satisfies the mapping assertions with respect to the source instance D. With respect to the mapping assertions, different assumptions can be made about the quality of the mapping. In particular, if we assume that the mapping is sound, then the data provided by the sources are a subset of the global data (the extension of qs is contained into the extension of qg). Conversely, if the mapping is considered to be complete, the data provided by the sources are a superset of the global data (the extension of qs contains the extension of qg). Finally, we assume that a mapping is exact, when it is both sound and complete. We point out that, due to the general characteristics of the sources, which are distributed, autonomous and independent, the sound mapping assumption is more reasonable in a data integration environment Data exchange architecture The previous work on data mediation formed the foundations for new researches on data exchange [Fagin et al. 2005], [Kolaitis 2005], [Kolaitis et al. 2006], [Giacomo et al. 2007]. Indeed, data exchange is an old, but HOSSAM JUMAA 11

26 CHAPTER 2 - State of the Art & Related Work recurrent, database problem. Figure 2-2 shows the general architecture of a typical data exchange system. The goal is to transform data structured according to a source schema into data structured according to a different target schema. In other words, the problem here is how to transform or to move data from a source schema to a target schema, according to a given specification. Figure 2-2 Data exchange architecture According to [Fagin et al. 2005], given a source schema S, a target schema T with a set of target dependencies t and a set of source-to-target dependencies st, then the target dependencies are Tuple-Generating Dependencies (TGDs) and Equality Generating Dependencies (EGDs), whereas source-to-target dependencies are TGDs mapping the source schema to the target schema. Thus the data exchange problem for a given source instance can be formalized as the following: is there a target instance that satisfies the constraints of the schema mapping (expressed in terms of source-to-target dependencies) and the target dependencies constraints? Formally, given a source instance I, is there a target instance J such that: <I, J> = st and J = t? Then we call such J a solution for I. Furthermore, for a schema mapping specified by embedded implicational dependencies, this problem is solvable in polynomial time by assuming that the schema mapping is kept fixed and the constraints of the schema mapping satisfy a certain structural condition, called weak acyclicity Peer-to-peer data integration architecture The emergence of peer-to-peer sharing systems inspired the data management research community to consider P2P architectures for data HOSSAM JUMAA 12

27 CHAPTER 2 - State of the Art & Related Work sharing. Many systems and works have been proposed for the data integration and exchange in the peer to peer environment. These systems are called Peer Data Management Systems PDMSs [Bernstein et al. 2002], [Nejdl et al. 2002], [Halevy et al. 2003], [Halevy et al. 2004], [Huebsch et al. 2005], [Fuxman et al. 2006], [Giacomo et al. 2007], [Al King et al. 2009]. The architecture for data integration and exchange in Peer to Peer networks and systems can be schematically represented as shown in Figure 2-3. This architecture of data integration is decentralized, dynamic and presents a data-centric coordination between autonomous organizations. In P2P data integration systems, each peer has a schema that is mapped into its own local source or/and into other external peer schemas. Queries are posed over one peer. In such a P2P system, each peer models an autonomous information site. Each peer: exports its information content in terms of a peer schema, by combining its own data from its local source or/and data coming from other peers, relates its local source to its schema by means of local mappings, relates to other peers by means of a set of P2P mappings. For a survey of the related works on the data management issues and challenges in the peer to peer systems we refer the reader to [Blanco et al. 2006]. Figure 2-3 Peer to peer data integration architecture HOSSAM JUMAA 13

28 CHAPTER 2 - State of the Art & Related Work In fact, the peer to peer networks offer two additional benefits in the context of data integration. First, organizations often want to share data, but none of them wants to take the responsibility of creating a mediated schema, maintaining it and mapping sources to it. Thus the P2P architecture offers a truly distributed mechanism for sharing data. Every data source needs to only provide semantic mappings to a set of neighbours it selects, and more complex integrations emerge as the system follows semantic paths in the network. Second, it is not always clear that a single mediated schema can be developed for a data integration scenario. Let us for example consider data sharing in a scientific context, where data may involve scientific findings from multiple disciplines, bibliographic data, drug related data and clinical trials. The variety of the data and the needs of the parties interested in sharing are too diverse to be represented by a single mediated schema. With a P2P architecture there is never a single global mediated schema, since data sharing occurs in local neighbourhoods of the network Mapping composition in data integration A crucial issue in the design of a data integration system is to specify the method of mapping between the global schema and the sources. Two basic approaches have been proposed in the literature: the LaV (local-as-view) approach [Ullman 2000], [Levy et al. 1996] and the GaV (global-as-view) approach [Garcia-Molina et al. 1997], [Halevy 2001]. Recently two other approaches have been proposed: the GLaV 1 approach [Friedman et al. 1999], which is a generalization of the LaV approach, and the BGLaV 2 combined approach [Xu and Embley 2004]. In the following we will try to expose the advantages and disadvantages of these different approaches. 1 Global Local as View 2 BGLaV is an acronym for BYU-Global-Local-as-View. HOSSAM JUMAA 14

29 CHAPTER 2 - State of the Art & Related Work The GaV approach for data mediation was first used in the TISMMIS 1 project [Garcia-Molina et al. 1997] at Stanford & IBM Labs, around In this approach, the global schema is defined from the sources (i.e. the relations in the mediated schema are defined as queries over the source schemas). The main advantage is that the query answering process is easy and can be done by means of simple unfolding techniques. However, its structure is not well suited for adding or removing sources (i.e. every change in the source schema may lead to redesign the mapping). Moreover, the building of a global schema for a set of heterogeneous sources is tedious and a hard task. The LaV approach was introduced on the initiative of the Information Manifold Project [Levy et al. 1996], at AT&T Bell Laboratories, around In this approach, the sources are defined from the global schema (i.e. the relations in the source schemas are defined as queries over the mediated schema). The main advantage is that the integration of new sources becomes easier. No knowledge is needed about other sources and relationships between them. As well, specifying constraints over the sources is easier than over the mediated schema, so the description of sources could be more precise. On the other hand, query answering is more complex than only simple unfolding queries and includes the problem of the reformulation of each user query into subqueries over the sources, in addition to the hard task of building the mediated schema. The GLaV approach [Friedman et al. 1999] is a generalization of the LaV approach. The sources are described in terms of queries over a mediated schema, but it takes into account more general cases where designers have to associate a general query over the source relations "qs" to a general query over the global relations "qg". Thus, GLaV mappings are more expressive and well suited to represent complex "data webs" 1 Stands for The Stanford-IBM Manager of Multiple Information Sources HOSSAM JUMAA 15