Modeling Quality Assurance Project Plans (QAPPs) Sandra Arismendez TCEQ Laboratory and Quality Assurance Section July 8, 2015

Size: px

Start display at page:

Download "Modeling Quality Assurance Project Plans (QAPPs) Sandra Arismendez TCEQ Laboratory and Quality Assurance Section July 8, 2015"

Betty Carson
8 years ago
Views:

1 Modeling Quality Assurance Project Plans (QAPPs) Sandra Arismendez TCEQ Laboratory and Quality Assurance Section July 8, 2015

2 QAPPs: Why Write Them? Because they are required EPA Order CIO (formerly A2) sets forth requirement for all organizations conducting environmental data operations on EPA s behalf State agencies (including TCEQ and TSSWCB) and TCEQ contractors follow suit So that projects can be planned and implemented effectively

3 Importance of Modeling QAPPs Helps ensure that the model(s) (and underlying data) are sufficient for achieving project objectives Provides documentation of specifications and activities

4 QAPP Requirements Graded Approach Level of detail commensurate with nature of the work and intended use of data

5 Tools for QAPP Development EPA Requirements for Quality Assurance Project Plans (EPA QA/R-5) is governing requirement ( EPA Guidance for Modeling QAPPs (EPA QA/G-5M) ( QAPP Program Shells Other QAPPs

6 QAPP Content Section A Project Management Section B Measurement and Data Acquisition Section C Assessment and Oversight Section D Data Validation and Usability

7 Section A Project Management A1, A2, A3 Title Page, Signature Page, TOC, Distribution List A4 Project/Task Organization A5 Problem Definition/Background A6 Project/Task Description and Schedule A7 Quality Objectives and Criteria A8 Special Training/Certification A9 Documents and Records

8 A5 Problem Definition/Background Define problem, place modeling activities into overall context with the project, and explain why modeling is being conducted Provide justification for model(s) to be used Summarize what questions will be answered and/or what decisions will be made

9 A6 Project/Task Description and Schedule Summarize work to be done Detail project tasks, schedule, and milestones Development, verification, and validation of code or software is a task (if applicable) Provide enough detail so it is clear as to what versions and implementations of packaged models will be used If multiple models are being used, any linkages should be discussed (preferably diagrammed)

10 A7 Quality Objectives and Criteria for Model Inputs/Outputs Inputs: Quantity, quality, and data type are relevant Quantity and quality: Precision, Bias, Accuracy, Sensitivity, Representativeness, Comparability, Completeness Type: data collection technique Outputs: Qualitative and quantitative criteria and measures Quantitative: Goodness of fit, error and other forms of uncertainty, sensitivity, graphical analysis Qualitative: Reasonableness (what is reasonable should be defined)

11 Sensitivity and Uncertainty Analyses Sensitivity analysis evaluates the effect of changes in input values or assumptions on a model s results Uncertainty analysis investigates the effects of a lack of knowledge or other potential sources of error in the model When used in combination, sensitivity and uncertainty analysis allow model users to be more informed about the confidence that can be placed in model results * Text above copied directly from Guidance on the Development, Evaluation, and Application of Environmental Models, Council for Regulatory Environmental Modeling (EPA/100/K-09), USEPA, 2009.

12 A9 Documents and Records Specify records, their contents, storage location, retention period, formats, and backup protocols Examples: QAPPs, User s Manual(s), various reports, raw files, code, calibration records, verification/validation records

13 Section B Measurement and Data B7 Calibration Acquisition B9 Non-direct Measurements (Data Acquisition Requirements) B10 Data Management and Hardware/Software Configuration

14 B7 Calibration Identify calibration parameters (including literature values) If calibration process is iterative and/or stepwise, spell out calibration steps Specify criteria/guidelines used to determine whether calibration is successful Also, state what will be done if minimum requirements are not met (e.g., recalibration with additional data, consideration of different modeling approach, etc.)

15 Section B9 Non-Direct Measurements (Secondary Data) This section is critical as primary data collection is rarely described in modeling QAPPs Tables can be extremely helpful Don t forget the graded approach! If some data will be acquired that are not yet known, this may be built into the QAPP In this case, it is especially important to specify criteria for acceptance and use of those data

17 B10 Data Management Details about data storage, production, management, storage, and archival should be provided If SOPs already exist, these may be referenced instead If data coding or software development is taking place, some reference to these procedures should be provided Hardware and software requirements/specifications should be included Flow diagrams can be very instructive

18 Section C Assessment and Oversight C1 Assessments and Response Actions C2 Reports to Management

19 C1 Assessments and Response Actions Describe surveillance activities, audits, and the process for identifying, documenting, and correcting problems Also, describe assessments of the model performance, and the schedule for these activities

20 Section D Data Validation and Usability D1 Departures from Validation Criteria D2 Validation Methods D3 Reconciliation with User Requirements

21 Section D D1: Describe how results are evaluated and departures from procedures presented D2: Describe steps in the validation process Performance criteria from A7 should be referenced D3: Describe uses of data and how limitations will be reported

22 Questions?

Elements A7 Project Plan - Modeling Needs and Requirements

Elements A7 Project Plan - Modeling Needs and Requirements United States Office of Environmental EPA/240/R-02/007 Environmental Protection Information Agency Washington, DC 20460 Guidance for Quality Assurance Project Plans for Modeling EPA QA/G-5M Quality Quality