Successful Management of Database Changes and Workflow. Chris Mickle Charles Richardson

Size: px

Start display at page:

Download "Successful Management of Database Changes and Workflow. Chris Mickle Charles Richardson"

Blaise Lane
7 years ago
Views:

1 Successful Management of Database Changes and Workflow Chris Mickle Charles Richardson International Conference for Environmental Data Management. May 14, 2014

2 Agenda > About Cardno > Overview DBTracker

3 About Cardno Founded in 1945, Cardno partners with public and private clients to solve engineering, environmental and social challenges. With over 230 offices in the Americas and 300 offices worldwide, Cardno provides clients with access to over 8,000 professionals focused on delivering customized consulting solutions to plan, design and construct social and physical infrastructure. Cardno currently ranks 26th on Engineering News-Record's (ENR s) 2013 Top 500 Design Firms and 30th on ENR s Top 200 Environmental Firms. Visit to learn more.

delivering customized consulting solutions to plan, design and construct social and physical infrastructure.

4 Professional Staff Chris Mickle Environmental Database Practice Leader at Cardno ENTRIX in the US. Chris serves as a project and client manager for environmental data management and information solutions projects while setting the strategic direction for Cardno s environmental data management practice. Chris has been working in the environmental consulting business for over 20 years and has extensive experience in designing and implementing information solutions for infrastructure and the environment. Charles Richardson Senior Environmental Database Manager and Lead Web Applications Developer at Cardno ENTRIX. Charles has developed many database and web applications for clients. Charles is the lead developer of the DB Tracker and has been instrumental in developing processes and systems for implementing structured and unstructured data management following rigorous quality assurance and quality control requirements.

5 Key Services > Human Health Assessment Physiologically Based Pharmacokinetic (PB-PK) modeling Biostatistics Dose-response modeling Consumer product exposure assessment REACH > Statistical Analyses Chemical fingerprinting Environmental statistics and geostatisics Dose reconstruction and similar exposure group assessment Data mining and database design > Environmental Fate and Transport Modeling Air dispersion and indoor air modeling Occupational exposure modeling Vapor intrusion modeling Groundwater and soil to groundwater transport modeling Particulate and soil vapor emission modeling Food web modeling

exposure group assessment Data mining and database design > Environmental Fate and Transport Modeling Air dispersion and indoor air modeling

6 Key Services > Geospatial Information Technology Web Mapping Services Geoprocessing Mobile Mapping GIS Program Management > Environmental Compliance Management > Asset Management > Training Management > Readiness and Mission Support > Automated Workflow Tools > Comprehensive Planning > Modeling and Simulations > Control Systems > Pavement Management

Management > Training Management > Readiness and Mission Support > Automated Workflow

7 Web-Mapping Services Overview Extending GIS over the web with ArcGIS Server

8 Web-Mapping Enhancements: Geoprocessing > Run complex spatial queries from within the web-mapping application Evaluate parcels against land cover, proximity to other entities, tax data Delineate a temporary area of interest and use as input into query Use query results to dynamically generate custom forms, charts and tables

other entities, tax data Delineate a temporary area of interest and use as input

9 ArcGIS Server & Mobile Mapping in the Field > ArcGIS App for ios and Android > Collect waypoints & fill out attribute fields > ~ 2.5m accuracy with Bluetooth GPS receiver > Wireless signal required > ipad Compatible

10 ArcGIS Server & Mobile Mapping in the Field > ArcGIS Mobile runs on Trimble Handheld Collect waypoints & fill out attribute fields Accuracy based on Trimble receiver and can be set to real-time correction in the field Data is stored temporarily on the handheld When unit is docked the data transfers automatically to ArcGIS Server Transferred data can be mapped immediately, no delay for post-processing or data conversions

the field Data is stored temporarily on the handheld When unit is docked the data transfers automatically

11 Environmental Data Management Client Types > Global Industrial Clients Chemicals Defense Energy Mining Oil and Gas > State and Local Government Agencies Remediation Services Watershed Management and Planning Waste Handling > Incident Response

Gas > State and Local Government Agencies Remediation

12 About DBTracker The DBTracker was designed by Cardno for the coordination of large environmental data management projects where several data managers, subject matter experts, working on the team are spread out across the U.S. in different time zones. DBTracker provides a centralized location for environmental database decision information. The system documents how data was received and loaded into the database. DBTracker is used for logging data issues and any resolutions if needed

in different time zones. DBTracker provides a centralized location for environmental database decision information.

13 DBTracker Business Requirements > Provide a controlled environment for tracking inputs and outputs, managing change to the overall systems data and architecture. > Keep data legally and technically defensible following quality management procedures outlined in a data quality management plan > Save time and money keeping information centralized Helps get users to the details of changes faster

> Keep data legally and technically defensible following quality management procedures outlined in

14 Components of DBTracker > Lab Tracker > Validation Tracker > EDD Loading Tracker > Dataset Loading Tracker > Issue Tracker > Change Tracker > QA Tracker > Report Tracker

15 Lab Tracker > The Analytical Services Coordinator (ASC) is the primary role > Maintain laboratory profiles and contact information. > Labs upload their unvalidated EDD files > ASC approves and prioritizes unvalidated data for load into the database > ASC assigns unvalidated data packages to third party validators > EDD Data Managers load unvalidated data into the database

> Labs upload their unvalidated EDD files > ASC approves and prioritizes unvalidated data for

16 Validation Tracker > The Analytical Services Coordinator (ASC) is the primary role > Maintain validation firm profiles and contact information. > Validators upload their validated EDD files > ASC approves and prioritizes validated data for load into the database > ASC assigns unvalidated data packages to third party validators > EDD Data Managers load unvalidated data into the database

> Validators upload their validated EDD files > ASC approves and prioritizes validated data for

17 EDD Load Tracker > The EDD Data Manager is the primary role > Access unvalidated and validated EDD files for upload into the database > Document loading or rejection of EDDs

18 Dataset Loading Tracker > Not all data comes in a standardized EDD format Observational data Biological data > Dataset Loading Coordinator (DLC) is the primary role. > DLC approves and assigns datasets to Environmental Data Managers for loading to the database > Dataset loading tool prepares scripts for import of new datasets into staging database tables > Database Architect approves DDL scripts for updating or changing the database structure > Once structure changes are approved the EDM can migrate the data into the core dbo schema

> DLC approves and assigns datasets to Environmental Data Managers for loading to the database > Dataset loading tool prepares scripts

19 Issue Tracker > Data users and members of the data verification QA Team are the primary role > Data users document any issues or findings they see in the data during it s use > QA Team reviews and promotes issues to a Change Request if necessary

document any issues or findings they see in the data during

20 Change Tracker > QA Team is the primary role > QA Team documents changes needed to data in the database based on data verification quality checks > Subject Matter Experts review and provide technical approval for changes > Senior database managers and coordinators provide database approval for changes > Database Architect approves changes to database structure > Data Managers implement changes > QA team performs final review of data changes

> Senior database managers and coordinators provide database approval for changes > Database Architect

21 Change Request Stages 1 Pending 2 Approved 3 To Database 4 Working 5 On Hold 6 Final QA 7 Complete System is customizable based on the workflow of your environmental data management program International Conference for Environmental Data Management May 13 14, 2014

22 QA Tracker > QA Team are the primary role > QA Tracker is used to track structured and unstructured data verification activities on chemistry and biological observational dataset Field EDD Verification Lab Sample Verification Spatial Data Verification Unstructured data/documents verification > System is customizable to track the data verification QA/QC standard operating procedures for a given client or program.

23 Report Tracker > Data Users and Data Managers are primary roles > Data users request data in certain formats > Data Managers identify existing reports which fulfill the request > Data Managers customize reports and queries to Data Users specifications and log query and report in Report Tracker International Conference for Environmental Data Management May 13 14, 2014

24 DBTracker Analytics > Summarize EDD Load Status > Summarize DV Status > Summarize DB Changes by Type International Conference for Environmental Data Management May 13 14, 2014

25 DBTracker Functional Requirements > Web Portal Dashboard based on user login roles > Workflows for moving EDDs, datasets, issues, and changes through their life cycles > Ability to upload the following document types EDDs Lab Reports SQL Scripts correspondence Dataset files -.csv,.txt,.xlsx (Source Data) International Conference for Environmental Data Management May 13 14, 2014

26 Document Management Integration > Enterprise Content Management Systems > Microsoft SharePoint > Windows File Shares International Conference for Environmental Data Management May 13 14, 2014

27 Roles and Responsibilities Data Providers > Field Teams > Laboratories > Data Validators > Data Managers SQL Scripts > SMEs - Approvals Technical Approvers (SMEs) > Environmental Scientists > Chemists > QA/QC Data Analysts Database Approvers > Systems Administrator > Database Administrator > Database Architect International Conference for Environmental Data Management May 13 14, 2014

28 DBTracker Technical Requirements > Accessible via Internet > File share for document management > User security Logins Roles and Responsibilities User Configurations for Reporting Configuring notification Integration of EQuIS data through reports or Enterprise user management International Conference for Environmental Data Management May 13 14, 2014

29 DBTracker Demo International Conference for Environmental Data Management May 13 14, 2014

Portal for ArcGIS. Satish Sankaran Robert Kircher

Portal for ArcGIS. Satish Sankaran Robert Kircher Portal for ArcGIS Satish Sankaran Robert Kircher ArcGIS A Complete GIS Data Management Planning & Analysis Field Mobility Operational Awareness Constituent Engagement End to End Integration Collect, Organize,