Tecnologias de Sistemas de Informação baseados na WEB

Transcription

1 Tecnologias de Sistemas de Informação baseados na WEB 9 Metadata Repository TSIW 2006/2007 Warning! Authors " Ricardo Martins Ferreira (rmf@uninova.pt) " With contributions of! João Moura-Pires (jmp@di.fct.unl.pt)! This material can be freely used for personal or academic use and without previous author permission, as long as this declaration by the authors is included.! The use of this material for commercial matters (for example, paid courses) requires the explicit authorization of the authors. 9-2

2 Summary! Part 1: Introduction " Motivation " Information Model " Requirements " Solution! Part 2: Core functionalities Demo! Part 3: Distributed functionalities Demo! Part 4: Case Studies! Part 5: Conclusions 9-3 Motivation! Metadata = data about data! Gaining importance in last years " Domain modeling: Data Warehousing " Technical modeling: Complex systems! Usage of Metadata in Information Systems provides: " Representation of system behaviour # Metadata-driven applications " Consistency " Maintainability " Documentation "

3 Motivation (2)! Metadata Standards " Try to model solutions for common technical problems " Limited scope, targeted at specific areas " Focus mostly on interchange and interoperability between tools " Examples:! CASE Data Interchange Format (Computer Aided Software Engineering)! Common Warehouse Metamodel (Data Warehousing)! Dublin Core Metadata Initiative (Resource Management)! Evaluation " Allow little or no extension " Don t cover all cases 9-5 Motivation (3)! However, metadata use goes beyond interchanging! Metadata can describe not only technical data, but also business data and rules " Metadata = Knowledge 9-6

4 Motivation (4) MME = Managed Metadata Environment Source: Marco, D. and M. Jennings, Universal Meta Data Models, ed. W.C. Publishing Motivation (5)! Metadata is not only for interchanging! Can play an active role in Information Systems through their lifecycle! Information Systems require a Metadata Repository solution: " Flexible and Extensible! Handle any kind of Metadata: Technical or Business! Standards support! In-house standards support " Lightweight " Easy Integratable " Inexpensive! Approach " XML technologies for metadata " Open-source technologies " Web Services 9-8

5 Information Model! Meta Object Facility (MOF) as standard metamodelling architecture! M0 Layer " Objects in a given reality! M1 Layer " Models - finite description of objects for a specific purpose " Instances (XML)! M2 Layer M3 Meta- Metamodel M2 Metamodel M1 Model M0 Instance Object Rules (XML Schema + SchemaTron + Java) " Metamodels - types of models by the definition of a language " Concepts (XML Schema + optional SchemaTron)! M3 Layer Concepts (XML Schema + SchemaTron) Instances (XML) External Data " Meta-Metamodels types of metamodels, their common components and syntax rules " Rules (XML Schema + SchemaTron + Java code) 9-9 Information Model (2)! Roles of each technology in the Information Model:! M1 Layer (Instances) " XML documents metadata information! M2 Layer (Concepts) " XML Schema vocabulary / language for the instances " SchemaTron additional validation rules over instances (not expressable by XML Schema)! M3 Layer (Rules) " XML Schema common parts of languages to be used in all concepts " SchemaTron validation rules for the concept schemas " Java further run-time validations on concepts such as checking references to existing concepts 9-10

6 Requirements! Storage " Concepts " Instances! Validation " All the instances must follow the language defined in their concept " All concepts languages must follow a set of rules! Concept language standardization predefined elements, attributes, etc. (defined in Rules)! Querying and transforming " Flexible and powerful querying system " Possibility of transforming the query results into other formats! Versioning / Change management " Instance versioning " External notification mechanism 9-11 Requirements (2)! Relationships " Support for instance relationship ensuring referencial integrity! Interfacing " Simple and easy to use interfaces! Management Console " Administration of Metadata Repository " Management, visualization and navigation on all stored metadata! Security " Access Control mechanisms " Resource Permissions! Fault-Tolerance " Transaction support " Backups 9-12

7 Solution! Storage " exist Native XML database for storing XML resources! Instances (XML)! Concepts (XML Schema + SchemaTron)! Management Information (XML)! Querying and transforming " Query using XQuery over exist database " Transform query results with XSLT(s)! Interfacing " Service based architecture " Java Web Service running in Apache Tomcat 5.0! Management Console " Windows GUI console (.NET) 9-13 Solution High Level Architecture Metadata Repository Apache Tomcat 5.0 Web Server Management Console Web Service Metadata Repository Logic exist database Change Notifications Metadata Importers Information System / Enterprise Application Web Service 9-14

8 Solution Low Level Architecture Metadata Repository Logic Web Service Access Control Instance Change Notification Mechanism Instances Instance Versioning Concepts Querying and Transforming Backup and Restore Instance Relation Management exist Database Transaction Manager Resource Access Permissions exist Database Connector exist Database 9-15 Summary! Part 2: Core functionalities Demo " Concepts " Instances " Instance Relations " Querying " Transform! Part 3: Advanced functionalities Demo! Part 4: Case Studies! Part 5: Conclusions 9-16

9 Concepts! A concept is defined by an XML Schema! Rules to write concepts (XML Schemas): " Definition of the namespace with the cd prefix " Inclusion of a provided schema with predefined types, elements and attribute groups (base schema) " Identification the concept root element (@id = root ) " Concept root elements are unique " Usage of special annotation in root element! Concept Information (Name, Type, URL)! Authoring Information (Author, Creation/Modification Dates, Version) 9-17 Concepts (2)! Rules to write concepts: " The concept root element must define a complex type with a sequence grouper " The first two childs of the sequence must be:! Reference to the identificationelementsgroup! Reference to the documentationelementsgroup! (both defined in the base schema) " The concept root element must have a set of attributes:! Reference to the identificationattributesgroup! (defined in the base schema) " Concept custom elements are defined inside this sequence freely! Every element must include documentation information (annotation/documentation)! These rules are checked when the concept is uploaded to the Metadata Repository 9-18

10 Concept Template <?xml version="1.0" encoding="utf-8"?> <xs:schema elementformdefault="qualified" attributeformdefault="unqualified" xmlns:cd=" Concept Definition namespace declaration xmlns:xs=" <xs:include schemalocation="base.xsd"/> Base Schema Inclusion <xs:element name="concepttemplate" id="root"> Concept Root Element identification <xs:annotation> <xs:documentation>this is the concept template</xs:documentation> <xs:appinfo source="conceptinfo"> <cd:conceptinfo> <cd:name>concepttemplate</cd:name> <cd:type>domain</cd:type> <cd:documentationurl> </cd:conceptinfo> </xs:appinfo> <xs:appinfo source="conceptauthoring"> <cd:conceptauthoring> <cd:author>author Name</cd:Author> <cd:creationdate> t12:00:00</cd:creationdate> <cd:modificationdate> t12:00:00</cd:modificationdate> <cd:version>1.0</cd:version> <cd:status>proposal</cd:status> <cd:comments/> </cd:conceptauthoring> </xs:appinfo> </xs:annotation> Elem. documentation Concept Information Concept Authoring 9-19 Concept Template (2) <?xml version="1.0" encoding="utf-8"?> <xs:schema elementformdefault="qualified" attributeformdefault="unqualified" xmlns:cd=" xmlns:xs=" <xs:include schemalocation="base.xsd"/> <xs:element name="concepttemplate" id="root"> <xs:annotation>... </xs:annotation> <xs:complextype> <xs:sequence> <xs:group ref="identificationelementsgroup"/> Concept Root element complex type Sequence grouper First elements are two <xs:group ref="documentationelementsgroup"/> predefined element groups Concept specific elements can be added after this line  </xs:sequence> <xs:attributegroup ref="identifierattributesgroup"/> </xs:complextype> </xs:element> </xs:schema> Predefined attributes group for concept root element 9-20

11 Concept Template (3)! Predefined element groups for concept root element 9-21 Concept Template (3)! Predefined attribute group for concept root element <xs:attributegroup name="identifierattributesgroup"> <xs:attribute name="globalid" type="globalidtype" use="optional"/> <xs:attribute name="version" type="xs:string" use="optional"/> <xs:attribute name="subscribedfrom" type="xs:string" use="optional"/> </xs:attributegroup> For the complete identification of an instance version <xs:simpletype name="globalidtype"> <xs:restriction base="xs:normalizedstring"> <xs:pattern value="[0-9]{2}.[0-9]{6}"/> </xs:restriction> </xs:simpletype> Definition of the Global Identifier (GlobalId) type: (ServerId.InstanceId) ##.###### 9-22

12 Concept Template (4)! XMLSpy visualization of the concept template 9-23 Product Development Metamodel! Sample metamodel based on product development! Concepts: " Entity An organization, institute or company that participates in product development " Consortium A group of entities that participate together in product development " Product A specific product developed by an entity or a consortium 9-24

13 Product Development Metamodel Entity! Entity An organization, institute or company that participates in product development! DEMO: " Concept creation " Metadata Manager Console introduction " Concept addition " Concept visualization " Sample instances creation " Instance Upload " Instance visualization 9-25 Product Development Metamodel Consortium! Consortium A group of entities that participate together in product development! To model the consortium, a relationship is required " A consortium must refer a set of entities.! The Metadata Repository supports relationships between instances.! These are declared and configured in the concept schema 9-26

14 Concepts Relation elements! To declare a relation element in the schema use the predefined type relationtype! Include a specific annotation to configure the relation " Cardinality! Minimum (0, 1, #)! Maximum (0, 1, #, N) " Target! All concepts or...! Concept List! Targets must be defined when the concept is uploaded 9-27 Concepts Relation elements (2)! Example for the consortium schema: <xs:element name="entities" type="relationtype"> <xs:annotation> <xs:documentation>entities that compose the consortium</xs:documentation> <xs:appinfo source="relation"> <cd:relation> <cd:cardinality> <cd:min>2</cd:min> <cd:max>n</cd:max> </cd:cardinality> <cd:target> <cd:conceptlist> <cd:concept>entity</cd:concept> </cd:conceptlist> </cd:target> </cd:relation> </xs:appinfo> </xs:annotation> </xs:element> relationtype for the element Description of the relation (like in all other elements) Appinfo for configuring Relation Relation Cardinality configuration In this case: Minimum = 2 Maximum = N Relation target configuration In this case: Only instances of the Entity concept 9-28

15 Intances Relations syntax! Example for the Caparica Consortium including FCT/UNL and UNINOVA XML instance: <Consortium xmlns:cd=" xmlns:xsi=" xsi:nonamespaceschemalocation="consortium.xsd"> <Name>Caparica consortium</name> <ShortName>CC</ShortName> <Description/> <AuthoringInfo> <Author>Ricardo Ferreira</Author> <CreationDate> T11:00:00</CreationDate> <ModificationDate> T11:00:00</ModificationDate> <Status>Proposal</Status> <Comments/> </AuthoringInfo> <Entities type="relation"> <Relation name="fct/unl" semantic="consortium member"/> <Relation name="uninova" semantic="consortium member"/> </Entities> <URL> </Consortium> type = relation is a fixed attribute defined in relationtype target instance target instance 9-29 Intances Relations syntax (2)! In relations instances are identified by their name! Instance name is a key! Global Identifiers are internal! Allows related instances not yet existing in the Metadata Repository to be uploaded in batch! Relation cardinality and target validity is checked upon instance upload! Referential integrity is assured " Referenced instances can t be removed 9-30

16 Product Development Metamodel Consortium and Product! Consortium A group of entities that participate together in product development! Product A specific product developed by an entity or a consortium! DEMO: " Concept creation " Concept addition " Concept visualization " Sample instances creation " Instance Upload " Instance visualization " Relation navigation 9-31 Querying and Transforming! Queries " XQuery executed directly in the exist database! May return XML or any other format! Transforms " XSLT to be applied on XML query results to produce other outputs! Transform single transformation! Transform Pipeline multiple transformations executed in pipeline " Generaly used for outputing HTML exist XML Database Query (XQuery) Query (XQuery) XML TXT / HTML / other Transform (XSLT) Transform Pipeline (XSLT[1]) XML / HTML / other XML Transform Pipeline (XSLT[n])... XML / HTML / other 9-32

17 Querying and Transforming (2)! Support for parameters in both Queries and Transforms! Queries and Transforms are stored in the Metadata Repository to be executed by external applications! WebService functions: " String executequery(string name) " String executetransform(string name) " String executetransformpipeline(string name)! DEMO: " Query Groups " Queries " Transforms " Transform Pipelines " System Instances creation 9-33 Summary! Part 3: Advanced functionalities Demo " Notification mechanism " Metadata subscriptions! Part 4: Case Studies! Part 5: Conclusions 9-34

18 Notifications! Triggers to external web services to inform changes about metadata " Instance change " Instance addition / removal from a certain concept! Provides better integration with external systems! Consists on an assynchronous evaluator launched automatically after every update! Applications " Data Warehouse dimension updates " alerts "...! DEMO: " notificator for product changes 9-35 Subscriptions! Allowing the subscription of instances from other servers! Subscribed instances are read-only! Local instances can relate with subscribed instances! If an instance is modified in its source metadata repository all subscriptions are automatically updated.! Implementation details: " Assynchronous mechanism " Uses Really Simple Syndication (RSS) feeds " Resources are downloaded by exist webdav service 9-36

19 Subscriptions (2)! How the subscription process starts? Repository A Repository B Legend: Declare Repository B Set instance shares for Repository B (*) Create Publication to Repository B Set subscriptions Generate RSS with publication to Repository B RSS Updated Declare Repository A Set subscriptions from the available shares (*) User actions Server actions Download RSS Download and create subscribed instances 9-37 Subscriptions (3)! How are subscriptions updated automatically when instances are modified in the source? " RSS update is invoked on the source server everytime any published instance is modified " When the publication is created, a notification is created:! Notification over the published instances! Target web service is the local! Method call to update the RSS for that server " Once this method is called from the notification mechanism, the end of the previous process is repeated 9-38

20 Subscriptions (4)! How are subscriptions updated automatically when instances are modified in the source? Repository A Update instances involved in Publication to Repository B Repository B Legend: User actions Server actions Notification Evaluator Update RSS Generate RSS with publication to Repository B RSS Updated Download RSS Download and refresh subscribed instances 9-39 Subscriptions (5)! DEMO: " Remote server definition " Sample subscription 9-40

21 Summary! Part 4: Case Studies " Space Environment Support System " Aspects Specification for the Space Domain! Part 5: Conclusions 9-41 Space Environment Support System! ESA project! DEIMOS Space UNINOVA INTA! Client: Galileo! Information System for the Space Domain! Provide to mission operators a set of tools for analysing Space Weather conditions and Spacecraft telemetry " Space Weather = Sun + Earth environment data such as solar activity, earth magnetic field. " Space Weather can affect spacecraft life-time and performance.! Complex domain " Spacecrafts " Parameters (space weather and telemetry measures) " Events (types of occurrences) 9-42

22 Space Environment Support System (2)! Architecture with multiple components developed by different teams " Data Processing Module (UNINOVA) " Data Integration Module (DEIMOS Space) " Client Tools (UNINOVA)! Reporting and Analysis Tool! Monitoring Tool " Integration...! Multi-Mission Architecture " Each mission has a private architecture " A common architecture is available for public data (controlled by ESA) " Data subscriptions between architectures! Large amounts of business and technical metadata Space Environment Support System (3)! Single Infrastructure Architecture Public Data Service Providers (http / ftp) SESS Infrastructure Architecture Metadata Repository txt... txt txt Private Data Service Provider (http / ftp) Data Processing Module Data Integration Module Monitoring Tool Reporting and Analysis Tool 9-44

23 Space Environment Support System (4)! Multiple Infrastructure Architecture Public DSP XMM DSP ENVISAT DSP INTEGRAL DSP Common Infrastructure: ESA Public data subscription Mission Infrastructure: XMM Mission Infrastructure: ENVISAT Mission Infrastructure: INTEGRAL Metadata Repository DPM DIM Client Tools Private data publication Metadata Repository DPM Metadata DIMRepository Client Tools DPM Metadata DIMRepository Client Tools DPM DIM Client Tools 9-45 Space Environment Support System (5)! Metadata Repository is the integrator of all tools helping system maintenance and consistency! Contains all technical and domain metadata of the system! Metadata-driven system components! Data Subscriptions between infrastructures is controlled by Metadata Subscriptions! Notifications mechanisms used to refresh Data Warehouse dimensions in Data Integration Module! Requirements and Tests stored as metadata # Generation of documentation! Success!! Project in operational state! February 2007 Final Presentation to Galileo staff 9-46

24 Aspect Specification for the Space Domain! ESA Project! UNINOVA FFCT EADS Test & Services DEIMOS Engenharia! Study the applicability of Aspect-Oriented analysis methodologies on space domain projects! Definition of a methodology (based on existing ones)! Development of a tool to support this metodology, and provide automatic analysis results 9-47 Aspect Specification for the Space Domain (2)! Metadata Repository acts as supporting infrastructure for the project! Client tool is a specialized metadata editor! Concepts identified from the methodology! High use of relation mechanisms! XQuery + XSLT generates the system analysis automatically! Success!! February 2007 Project end 9-48

25 Future Work! Core functionalites " Standards importing " Namespace support for concepts! Distributed functionalites " Replication for fault-tolerance " Load-balancing 9-49 Questions?! Thank you 9-50

26 References! Marco, D. (2000). Building and Managing the Meta Data Repository: A Full Lifecycle Guide.! Marco, D. and M. Jennings, Universal Meta Data Models, ed. W.C. Publishing ! Tannenbaum, A. (2002). Metadata Solutions Using Metamodels, Repositories, XML and Enterprise Portals to Generate Information on Demand.! Ferreira, R., J. Moura-Pires, et al. (2005). XML based Metadata Repository for Information Systems. 12th Portuguese Conference on Artificial Intelligence.! W3C (2006) from ESA. (2005). "Space Environment Support System for Telecom/Navigation Missions (SESS)." from UNINOVA. (2005). Aspect Specification for Space Domain from References (2)! Ferreira, R. and J. Moura-Pires (2007). Extensible Metadata Repository for Information Systems And Enterprise Applications. Submitted.! Raminhos, R. and J. Moura-Pires (2007). Extraction and Transformation of Data from Semi- Structured Text Files using a Declarative Approach. Submitted.! Ferreira, R., R. Raminhos, et al. (2005). Metadata Driven Aspect Specification. ACM/IEEE 8th International Conference on Model Driven Engineering Languages and Systems! Agostinho, S., R. Ferreira, et al. (2007). A Metadata-Driven Approach for Aspect-Oriented Requirements Analysis. Submitted.! Marques, A., R. Raminhos, et al. (2007). Aspect-Oriented Analysis Applied to the Space Domain. Submitted. 9-52