Transformation architecture From production data base to INSPIRE data using WFS: potential methods Inspire KEN Paris 2013-10-08 Frank Arntsen and Morten Borrebæk
Background Based on ELF / ESDIN experiences and on going work connected to national data services and ELF. The work is an attempt to test potential production lines for the development of Inspire download services with Norwegian data content Involves transformation of content according national data models to INSPIRE compliant data
ESDIN and ELF project ESDIN econtentplus Underpinning the European Spatial Data Infrastructure with a Best Practice Network NMCA s (as data providers for large scale data): ELF The European Location Framework Competitiveness and innovation framework programme NMCA s (as data providers for large scale data): Denmark Finland France Netherlands Norway Sweden Belgium Chech Republic Denmark Finland France Germany Netherlands Norway UK Poland Slovenia Spain Sweden (IGN-B) (CUZK) (GST) (NLSFI) (IGNF) (BKG) (CADNL) (KARTV) (OSBG) (MAC) (SMA-SL) (CNIG and SDGC) (NLS-S) Excel matching tables are produced for these NMCA s where they can provide data, and is a starting point for transformations.
Transformation in two levels 1. Application Schema Level (Model level) National specifications (Data models) Documentation of the transformation INSPIRE Data specification Resources: Domain experts and data modelling experts 2. Data level National database Technical implementation of the transformation INSPIRE valid GML via WFS Resources: Database experts and service developers The result of level 1 is used as input for level 2
Model mapping - ELF / ESDIN tool
Mapping the model elements Guidelines are produced in ESDIN and ELF project: ELF_MatchingTableGuidelines.docx ELF_MatchingTableguidelines.pptx
Transformation in two levels 1. Application Schema Level (Model level) National specifications (Data models) Documentation of the transformation INSPIRE Data specification Resources: Domain experts and data modelling experts 2. Data level National database Technical implementation of the transformation INSPIRE valid GML via WFS Resources: Database experts and service developers The result of level 1 is used as input for level 2
Content transformation Architecture - potential methods - Client applications that use INSPIRE WFS/GML Web Feature Service (INSPIRE Download Service) Distribution database Other methods Database views Database tools Transformationsoftware Production database or replicated production database
Our focus while working with different methods Delivery point close to the source data. As few steps as possible from the source database to the Inspire valid GML output. Actual and authoritative information directly from the official source. Deliver full INSPIRE valid GML. Use open source software if possible.
Content transformation Architecture - potential methods - Client applications that use INSPIRE WFS/GML 1 Web Feature Service (INSPIRE Download Service) Transformation as part of the WFS configuration Distribution database Other methods Database views Database tools Transformationsoftware Production database or replicated production database
1 Transformation as part of the WFS configuration The transformation is handled by the WFS server and included in the configuration of the service. Transformation is processed live for each request. No customization at the database level needed. May not handle complicated transformations, but appropriate when the transformation is relatively simple. The source database has to be quite similar to the Inspire model with respect to structure and content. Resources: Mainly involves service developers and knowledge about the source database.
1 Example of WFS config-file - mapping INSPIRE model concepts and national database concepts - deegree WFS server (open source). <FeatureTypeMapping name="cp:cadastralparcel" table="testgruppe1s.eiendomsteig">. <Complex path="cp:estimatedaccuracy"> <Primitive path="text()" mapping= nøyaktighet"/> </Complex> <Complex path="cp:geometry"> <Geometry path="." mapping= område"/> </Complex> <Complex path="cp:inspireid"> <Complex path="base:identifier"> <Primitive path="base:localid" mapping="objectid"/> <Primitive path="base:namespace" mapping="'no.statkart.matr'"/> </Complex> </Complex>
Content transformation Architecture - potential methods - Client applications that use INSPIRE WFS/GML Web Feature Service (INSPIRE Download Service) Distribution database Transformation using database views Other methods 2 Database views Database tools Transformationsoftware Production database or replicated production database
2 Transformation using database views In this method the transformation is taking place on the database level and implemented as database views. The database views are design to fit the Inspire data model. Standard SQL is used to select information from different tables, change content values and structure. Transformation is processed "live" for each request. The transformation is done efficiently close to the original data. Can handle complicated transformations. The data owner and database expert is responsible for the transformation. Cooperation between database expert and service developer.
2 Transformation using database views Norwegian Cadastral Parcels example Production database Internal model INSPIRE model Database tables Database views or materialized views matrikkel nummer matrikkel nummer gyldig fra område areal slutt dato national Cadastral Reference valid From vali dto geom etry area Value
2 Norwegian Cadastral Information example - delivery system - Production system Delivery system Access control Norwegian Central cadastral system server (View Service) Download service Database views INSPIRE model -INSPIRE data specification Central Cadastral Information database Oracle Replicated Cadastral database Oracle
Content transformation Architecture - potential methods - Client applications that use INSPIRE WFS/GML Web Feature Service (INSPIRE Download Service) Database views Distribution database 3 Database tools Transformation as separate database Transformation- process software Other methods Production database or replicated production database
3 Transformation as a separate database process Data is transformed with SQL-scripts and database functions. The transformed content is loaded into a separate distribution database structured according the Inspire model. The transformation is performed as scheduled process. Can handle complicated transformations. The transformation is done efficiently using database tools. The transformation work can be handled separate from the production system. Can give better performance than database views since the database can be more specialized for delivery.
3 Geographical Names example - delivery system - Production system Delivery system Access control Norwegian Place Name System INSPIRE Download service WFS 2.0 -INSPIRE data specification Norwegian Place Name Database Oracle Transformation SQL script and Database link Scheduled Distribution Database for Inspire Geographical Names
Content transformation Architecture - potential methods - Client applications that use INSPIRE WFS/GML Web Feature Service (INSPIRE Download Service) Database views Specialised transformation tools Distribution database Database tools 4 Transformationsoftware Other methods Production database or replicated production database
4 Specialized software for data transformation Transformation is taking place outside the database as a separate process with specialized software. Often as part of a production line where results are stored in a database for delivery via WFS. The transformation is often performed as scheduled process. Could also have a WFS interface as output. Great advantage for complicated transformations. Can be integrated in a complex production line.
4 Transformation of Inspire theme: Transport Networks Kartverket: National node for distribution of Inspire datasets and services Planned Distribution database WFS 2.0 WMS 1.3.0 Atom Feed Geo- Syncronisation -ChangeLog - update -... Planned WFS 2.0 WFS 2.0 WFS 2.0 INSPIRE specification Road transport networks Rail transport networks Air transport networks Water transport networks Transformation Specialized software National specifications Data provider National Road Data Bank Norwegian Public Roads Administration Railways Air Roads Jernbaneverket Avinor Norwegian Coastal Administration
Experiences It is possible and efficient to transform and deliver INSPIRE data directly from the original source in quite few steps. Transformation at the database level is an efficient method. Live transformation of both request and response. To solve the transformation task in multidisciplinary teams have been very helpful. Important considerations when selecting method: The complexity of transformation. Requirements for live and updated information. How the method fits into the production line. Possibilities for optimisation and tuning of the system. How easy the system is to maintain. How the method fit into the organisation.
Content transformation Architecture - potential methods - Client applications that use INSPIRE WFS/GML Web Feature Service Distribution database Other methods Database views Database tools Transformationsoftware Production database or replicated production database