Environment Canada Data Management Program Paul Paciorek Corporate Services Branch May 7, 2014
EC Data Management Program (ECDMP) consists of 5 foundational, incremental projects which will implement a suite of data management tools, services, standards and best practices that will strengthen the management of EC s environmental data in support of program activities, decision-making, and open government. Project Deploy Products Service Realize Benefits Page 3 September-10-14
Examples of early successes EC Data Catalogue Contributions to Joint Oil Sands Monitoring Portal Page 4 September-10-14
EC Data Management Program (ECDMP) Five interdependent, foundation projects: 1. Data Governance and Architecture 2. Data Catalogue 3. Data Access and Sharing 4. Data Consolidation 5. Data Integration & Preservation Page 5 September-10-14
An incremental journey... Action Plan Approved ECDMP Working Group Data Stewardship Data Standards Data-centric Architecture Data Governance and Architecture (P1) Ongoing Data Catalogue (P2) Data Discovery Metadata Management & Publishing Completed / Deployed Service Page 6 September-10-14
An incremental journey Data Warehouse & Archive Ongoing Geospatial Visualization Data Consolidation (P4) Authoritative Repositories Data Integration & Preservation (P5) Standards-based Data Access (EC Data Mart) Streamlined Data Publishing Data Access and Sharing (P3) Page 7 September-10-14
P1: Data Governance & Architecture - Data Stewardship (1) A data steward is a person within the organization who is accountable for the management of data within a specified domain. This includes: Ensuring data is properly managed throughout its lifecycle, including its quality requirement; Ensuring conformance to policies, processes, and standards; Coordinating with other Data Stewards and Stakeholders to establish common data requirements, definitions, business rules, and data quality metrics. 8 Page 8 September-10-14
P1: Data Governance & Architecture - Data Stewardship (2) Data stewardship is the formalization of accountability for the management of GC data assets. EC Data Stewardship Model A model that defines data stewardship roles and responsibilities Registry of Data Stewards (under development) Planned component of the EC Data Catalogue for identifying EC data stewards associated with datasets Data Stewardship Handbook (under development) Guide for EC employees and managers involved in data management Page 9 September-10-14
P1: Data Governance & Architecture - Data Standards (1) System A Data System B Problem: Systems have been developed and evolved independently, resulting in: Disparate data terminology and formats Difficulty to exchange, share and compare data Difficulty to acquire, integrate & display data from different sources Need to develop unique, custom applications for each type of data Goal: Interoperability: Enable the sharing or exchange of information between multiple parties in a way that guarantees that the interacting parties share the same understanding of what is represented Reusable data management tools for collection, processing, analysis, visualization, and storage Ability to integrate and compare various data from many sources to support decision making, analysis and reporting Page 10 September-10-14 10
P1: Data Governance & Architecture - Data Standards (2) Data standards improve data usability, interoperability and comparability, and enable development of reusable data tools. EC Enterprise Data Model Defines EC s core data subject areas, common data definitions and structures Data Cataloguing Standards Implemented GC and international standards that enable interoperable data discovery & sharing Data Exchange Standards (Phase 1: Environmental Monitoring) Developing standards for organizing and formatting data Observation & Measurement Data, Monitoring Site Data Data Classification Hierarchy, Data Naming conventions Leveraging international OGC and ISO standards Page 11 September-10-14
P2: Data Catalogue - Completed/Deployed Solution for describing, publishing and discovering EC s environmental data. http://donnees-data.intranet.ec.gc.ca Page 12 September-10-14
How it works? 1. Describe Data Stewards use standardsbased metadata creation tools to quickly and easily create metadata that makes their data discoverable 2. Publish Publish Request Internal EC Data Catalogue External Data.gc.ca GC-AB Oil Sands Portal Data Stewards use standardsbased publishing process to publish data sets internally or externally Submit Publication Request 3. Discover Search, discover, and access EC data Slide 13
Federated, Standardized Data Publishing Open Data Portal (Data.gc.ca) Canada-Alberta Oil Sands Portal Other Applications, Departments, Partners LINK Standard Metadata External (Internet) Data Catalogue Interface (API) Data Access Internal (EC Intranet) EC Data Catalogue Data Access Slide 14
P3: Data Access & Sharing - Ongoing Developments EC Data Mart: Suite of departmental tools for providing online access to and visualization of EC s environmental data File-based Data Access Access to data in open, machine readable file formats Web Service-based Data Access Access to data using standards-based web services for web mapping and data access Geospatial Visualization (internal) Dynamic, interactive geospatial data visualization of EC s geospatiallyformatted data Self-Serve Data & Metadata Publishing Page 15 September-10-14
Ongoing Geospatial Developments Web Services OGC compliant Web Services (WMS, WFS, WCS) KML + ArcRest Users Upload Shapefile via EC Data Catalogue Files Visualize Open, machine readable, proprietary free data access CSV, JSON + GeoJSON conversions Internal GIS Visualization Metadata ISO 19115 NAP compliant, bilingual Metadata
Target Enterprise Data Management Capabilities Capabilities: Describe, Publish, Search, Discover Capabilities: Access, Download, Visualize, Analyze Data Catalogue Data Mart Data Repositories Data Standards Capabilities: Store, Archive, Integrate Capabilities: Structure, Transform, Exchange
Key to Success: Partnership Within Environment Canada Partnerships across EC s program areas to strengthen data management Collaborative implementation facilitated by horizontal working groups and communities of practices Leveraging an enterprise data management approach to deliver specific EC program priorities Within Government of Canada Partnered implementation with the Federal Geospatial Platform initiative Collaboration with TBS on Open Data Participation in GC data management and geospatial committees And Beyond Page 19 September-10-14
Thank you! Paul Paciorek Paul.Paciorek@ec.gc.ca Corporate Services Branch May 7, 2014