1 Air Quality Monitoring: Time Series Management in AQM Data Centers MTWE 2014, Brussels/BE Edgar Wetzel, 23rd October 2014
2 Brief company presentation KISTERS Group 10 subsidiaries in Europe, North America, Australia, Asia employees (full time equiv): 520+ established in years: environmental software Customers B2B / B2G customers worldwide thousands of software licenses installed Core Competencies software to manage measured + calculated data time-series data + raster data + meta data KISTERS AG 11/11/
3 Why talk about Air Quality? Air Quality affects Public Health Meteorology Affects Air Quality Influencing Climate Change D.J. Jacob, D.A. Winner / Atmospheric Environment 43 (2009) 51 63
4 AQM Data Center: Main Tasks Why monitor? determining the level of contaminants in an airshed comparing pollutant concentrations with standards and guidelines obtaining exploratory data + conducting air quality research obtaining data for air quality modelling providing air quality information for policy or strategy development reporting on the state of the environment (inform the citizen, other institutions)
5 Air Quality Monitoring (AQM) emission sources industry traffic domestic heating livestock & agriculture natural emissions monitoring PM10 PM2,5 O3 CO NO2 SO2 other gases odour WD + WS T RH ATM pressure radiation assessment Monitoring data Emission inventory Dispersion modelling
6 Air Quality Monitoring Station monitoring station geographic location (incl. mobile) set of smart analyzers + meteorological sensors PM2,5 PM10 toluene PAHs network of monitoring stations numerous stations installed at strategic locations in a defined (political/service) territory station types: urban, rural, rural background road traffic industrial sources
7 Air Quality Monitoring monitoring station time = continuous space = permanent location monitoring station time = temporarily continuous space= mobile monitoring data <site><parameter><timestamp><value><quality-flags> time series data (TS) a continuous stream of monitoring data KISTERS AG 11/11/
8 Air Quality Monitoring: Big Data typical monitoring stations 15 parameters 5 scan-rate 3 aggregates calculated locally every 10 : average, min, max small network: 10 stations 23 million data points/yr large network: 60 stations 140 million data points/yr
9 AQM Data Center Software Architecture Presentation Tier Reporting AquisNet REP Validation AquisNet DMO Dispersion models Web Services, file transfer Time Series Data Management Server Application Tier JDBC Data Tier metadata time series data
10 AQM Data Center: core functions Configuration & Definitions system wide catalogues/key list regulatory compliance rules (EC, US EPA, UBA...) predefined and userdefinable data groups data access permissions & data views Operational Meta Data Management site station (measurement location) parameter (channel) time series definition Time Series Data Management data feeds (import/export) data validation and editing calculation of derived time series data publishing Time Series Data Analysis statistics and air quality indicators manual or automated reporting graphics and numeric tables Performance Reporting (KPI) KISTERS AG 11/11/
11 Integration of Air Quality Monitoring Data heterogeneous monitoring network telemetry TS data station A PSTN PULL PUSH Cloud TS data station Ω SODAair homogeneous TS data in the data centre harmonized TS data Product, brand and company names are the property of the respective owners.
12 AQM Data Center: Monitoring Data Arrives application server Cloud 1h TS 1h TS TS updating auto qual check TS updating TS calculator TS updating metadata time series(ts) data Monitoring station 1 Monitoring station k 1h production TS n 1h original TS n 1h production TS n-1 1h original TS n-1 1h production TS n-x 1h original TS n-x TS aggr. 1 TS aggr. 2
13 AQM Data Center: Data Validation Data validation increases data quality searches for erroneous and unusual measurements calibration: in the monitoring station (DAS); calibration data send to data center plausibility checks + visual examination All validation steps... are recorded in a versioning system allowing traceability & rollback are explained in full-text comments change the quality state of the data (<quality-flags> in the TS) Validated data represent the final data set to be used in the processing and assessment processes.
14 Data Validation: typical validation procedure(eu) Level 0 daily calibrations Daily calibration cycles are controlled by the monitoring station DAS software. Level 1 plausibitychecks algorithm-based plausibility checks on all data entering the database Level 2 daily visual inspection each morning the data of the past data validation experts visually examine the data of the previous day. Level 3 monthly visual inspection A full month of data is visually examined and corrections are applied on the basis of new knowledge (reports on incidents, fires, fireworks, etc.). Level 4 yearly visual inspection In early January, the data of the past year is visually examined. Further corrections applied (long-term transport effects). Comparison with so-called reference monitoring method and application of calibration factors.
15 AQM Data Center: Data Validation (Editing) user action: visual validation edit data application server Cloud TS selection generate graph generate table TS editing TS calculator TS updating metadata time series(ts) data Monitoring station 1 Monitoring station k 1h production TS n 1h original TS n 1h production TS n-1 1h original TS n-1 1h production TS n-x 1h original TS n-x TS aggr. 1 TS aggr. 2
16 AQM Data Center: Big-Data Processing data security by means of controlled redundancy all incoming data are write-protected a copy is automatically produced for validation and further processing automated data aggregation derived time series (mean, max, min, percentiles,...) are calculated automatically as data is updated on the fly calculation of air quality indicators statistical indicators temporary time-series for ad-hoc analysis
17 AQM Data Center: Ad-hoc User Request (calc.) command: on-the-fly statistics command: close statistics application server Cloud TS selection TS calculator TS updating generate graph generate table TS updating metadata time series (TS) data Monitoring station 1 Monitoring station k 1h production TS n 1h original TS n 1h production TS n-1 1h original TS n-1 1h production TS n-x 1h original TS n-x TS aggr. 1 TS aggr. 2 temp. TS
18 AQM Data Center: Processing aggregation rule short trigger event unknown U Import average AA Calculation/Import minimum MIN Calculation maximum MAX Calculation percentile PC Calculation sum SUM Calculation vector wind direction VWD Calculation vector Wind speed VDS Calculation scalar wind direction SWD Calculation scalar wind speed SDS Calculation exceedance EX Calculation free formula FF Calculation AOT40 (hourly) AOT Calculation minimum indicators ICMIN Calculation maximum indicators ICMAX Calculation exceedance indicators AICEXC Calculation statistical indicators ICSTAT Calculation priority formula PF Calculation epsilon beta calibrationebcal Import 2 point calbration (Auto 2PCAL GPT) Import multi point calibration MPCAL Import Goal Index GI Calculation AQI AQI Calculation regional air qualityindexraqi Calculation LIMS_PARAMETER LP Calculation LIMS CONCENTRATIONL Calculation LIMS TOXIC PARAMETERLTP Calculation nth highest maximum NMAX Calculation statistical summary SS Calculation statistical boxplot SBP Calculation 2PC Calibration (Auto PC) 2PAPC Calculation multimax MMAX Calculation Import Sample Text IST Import Import Sample IS Import nth highest max indicators INMAX Calculation weighted average RAA Calculation weighted sum RSUM Calculation rated copy RC Calculation exceedance indicators BEXEV Calculation percentage valid valuespvv Calculation psi_index PSI Calculation dynamic exceedance DEXC Calculation
19 AQM Data Center: Analysis Why monitoring data analysis? determine the extent of pollution, ensure compliance with national or supra-national (EU, WMO) regulations, facilitate decision making process/understand collected air pollution data air quality modeling Which tools? effective data editing/manipulation: tables and graphics Identify air quality indicators, correlation/regression and data quality visualization ( box-whisker ) dynamic graphics(line, bar, box-plot, polar plots)
20 AQM Data Center: Data Reporting reports user-defined pre-determined scheduled or manual report generation private or public numeric tables or graphs or a combination of both file formats standard formats (EC, US EPA, UBA, etc.) standard file formats (csv, pdf, jpg,...) user-defined formats (optional)
21 AQM Data Center: Web Publishing (examples) stations AQI colour coded legend map: dynamic map: stations with wind + concentrations zoom & pan map background map selection
22 AQM Data Center: Web Publishing (examples) tables: stations with their concentrations AQI colour coded legend line graphs: pollutants and/or meteorological data zoom, print, download navigate by hour/week/month
23 AQM Software Integration Into GIS
24 Each Data Point has an Economic Value! The cost to air quality monitoring: hardware (analyzers, stations, communication, servers, ) software people maintenance Big data analysis reveals hidden scientific and economic value data discovery strategic planning, prevention, warning + alarms transparency and early critical situation detection citizens and stakeholders engagement better decision making and sustainable air pollution management KISTERS AG 11/11/
25 Summary Important elements: The hardware/software system must provide all steps of processing scheme. Automation is important for big data analysis. Visual data validation is important: algorithms cannot eliminate all erors, nor explain all effects visible in the data. Data editing must be possible in both graphs and numeric tables. Communication requires tools adapted to the target audience.
26 Thank you for your attention. Nous vous remercions de votre attention. Vielen Dank für Ihre Aufmerksamkeit. KISTERS AG Pascalstr Tel Aachen Fax Germany
PROJECT FINAL REPORT Grant Agreement number: 212117 Project acronym: FUTUREFARM Project title: FUTUREFARM-Integration of Farm Management Information Systems to support real-time management decisions and
United States Office of Solid Waste EPA 550-B99-003 Environmental Protection and Emergency Response November 1999 Agency (5104) www.epa.gov/ceppo/ RMPs Are on the Way! How LEPCs and Other Local Agencies
Topic 11 Implementation and follow up Objectives To explain the role and contribution of implementation and follow up measures within the EIA process. Training session outline To understand the procedures
EN HORIZON 2020 WORK PROGRAMME 2014 2015 5. Leadership in enabling and industrial technologies iii. Space Revised This Work Programme was adopted on 10 December 2013. The parts that relate to 2015 (topics,
January 2013 Page 1 This document provides instruction for using the feature in the DeltaV process control system. Increase Operator alarm response effectiveness with instant access to in-context alarm
Managed Video as a Service RFP Template Managed Video as a Service Request for Proposal Choosing the right managed video surveillance solution isn t easy. When evaluating a managed video solution, there
FACTS AND FIGURES (JANUARY 2008) B R U S S E L S. D U B A I. F R A N K F U R T. G R E N O B L E. H O N G K O N G. N E W Y O R K. PA R I S. S Ã O PA U L O. S E O U L. S I N G A P O R E. S Y D N E Y. W A
TÜV RHEINLAND IMMISSIONSSCHUTZ UND ENERGIESYSTEME GMBH Akkreditiertes Prüfinstitut DAP-PL-3856.99 Report on the suitability test of the ambient air quality measuring system BAM-1020 with PM 2.5 preseparator
SKF @ptitude Observer Condition monitoring software optimized to collect and analyze continuous measurement data from critical rotating machinery SKF @ptitude Observer is the on-line application in a family
SI-Consulting S.A. Manage Customer Relationships Our competence is SAP CRM Best-run businesses use SAP solutions We have the honor to support them in achieving success! The consultants of SI-Consulting
General Principles of Software Validation; Final Guidance for Industry and FDA Staff Document issued on: January 11, 2002 This document supersedes the draft document, "General Principles of Software Validation,
Joint UNECE/Eurostat/OECD Work Session on Statistical Metadata (METIS) Generic Statistical Business Process Model Version 4.0 April 2009 Prepared by the UNECE Secretariat 1 I. Background 1. The Joint UNECE
Institute for Visualization and Interactive Systems University of Stuttgart Universitätsstraße 38 D 70569 Stuttgart Fachstudie Nr. 162 Social Media Analysis for Disaster Management Dang Huynh Nils Rodrigues
Release 5.2 Voice Firewall User Guide DOC-FW-ETM521-2007-0504 About SecureLogix Corporation SecureLogix Corporation enables secure, optimized, and efficiently managed enterprise voice networks. The company
EVALUATE AND IMPROVE The WMO Strategy for Service Delivery and Its Implementation Plan DESIGN SERVICES ENGAGE USERS DELIVER WMO-No. 1129 WMO-No. 1129 World Meteorological Organization, 2014 The right of
The Twining project, an institutional cooperation between Italy and Turkey, is co-financed by the European Union and the Republic of Turkey. EU TWINNING PROJECT Improving Data Quality in Public Accounts
The Critical Security Controls for Effective Cyber Defense Version 5.0 1 Introduction... 3 CSC 1: Inventory of Authorized and Unauthorized Devices... 8 CSC 2: Inventory of Authorized and Unauthorized Software...
IBM SPSS Direct Marketing 21 Note: Before using this information and the product it supports, read the general information under Notices on p. 105. This edition applies to IBM SPSS Statistics 21 and to
IBM SPSS Direct Marketing 20 Note: Before using this information and the product it supports, read the general information under Notices on p. 105. This edition applies to IBM SPSS Statistics 20 and to
WHITE PAPER FOR PUBLIC TRANSPORT STAKEHOLDERS Based on the lessons learned in SECUR-ED This White Paper benefits from the conclusions of FP7 PROTECTRAIL (www.protectrail.eu) Content 1. Introduction and
ORACLE HEALTH SCIENCES INFORM: COMPREHENSIVE CLINICAL DATA CAPTURE AND MANAGEMENT CLOUD KEY BENEFITS Accelerate clinical trial timelines while reducing trial cost and risk Collect and deliver higher-quality
Cloud Service Level Agreement Standardisation Guidelines Brussels 24/06/2014 1 Table of Contents Preamble... 4 1. Principles for the development of Service Level Agreement Standards for Cloud Computing...