Third Party Data Validation: What you need to know and do!

Transcription

1 Third Party Data Validation: What you need to know and do! Anand Mudambi & Pamela Wehrmann 2012 DoD EMDQ Workshop San Diego, CA

2 Course Objectives/Outline The course will provide an overview of third party data verification and validation for EPA and DoD Environmental Projects It will cover many aspects including: Data requested from laboratories Stages of review that can be performed and their uses Communicating the level of data validation and its benefits March 2012 Third Party Data Validation - Training 2

3 Third Party (External Organizations) Third Parties are defined as organizations (including Governmental entities, contractors, or vendors) that conduct analytical data review, verification, and validation activities and that are not part of the immediate laboratory that generates the subject analytical data (but that are part of the overall project-specific data review process). March 2012 Third Party Data Validation - Training 3

4 Course Overview Why Verify/Validate Data? Project Planning & Documentation Validation scenarios (how much is enough?) Laboratory Analytical Data Packages Labeling Validated Data Benefits of Labeling & Examples However!!! Won t teach the how of data validation! March 2012 Third Party Data Validation - Training 4

5 Why Verify / Validate Data? Additional resources are used by DoD/EPA to review (i.e., verify and validate) laboratory analytical data packages Reviews are conducted to ensure adequate quality and usability for site decision making Often different procedures used to evaluate laboratory data quality across programs Goal for this course is to encourage use of consistent terminology to describe the scope and content of validation for data users. March 2012 Third Party Data Validation - Training 5

6 Verification & Validation - Definitions Analytical data verification generally consists of a completeness check to confirm that all data requested from the laboratory have been received and comply with specified requirements. Analytical data validation generally consists of an analyte and sample specific process for evaluating compliance of the laboratory data received with methods, procedures or contract requirements. Data review commonly used for any combination of the above. Not limited to analytical data but should include items like sampling documentation and field analytical. (UFP-QAPP Manual section 5). March 2012 Third Party Data Validation - Training 6

7 Verification & Validation Where to Start Project Planning Process: DQOs; Assessment criteria (regulatory levels (MCLs); screening criteria(rsls)or eco risk assessment Sampling strategy Methods; DQIs; LOD/LOQ; Type of data needed from lab Level of data verification and/or validation Where to document data validation planning? Use UFP-QAPP WS# March 2012 Third Party Data Validation - Training 7

8 UFP-QAPP Manual Data Review Process March 2012 Third Party Data Validation - Training 8

9 Project Planning Establish validation needs and procedures based on the project DQOs and phase of the project; Verification and validation procedures should address the following: Process used to verify sample collection, handling, field analysis etc. Validation scenario for project laboratory data. Must be documented in the QAPP to ensure that data are evaluated correctly, completely, and consistently; Should be used to determine analytical data needs from the lab (contract requirements). March 2012 Third Party Data Validation - Training 9

10 Criteria to Guide Project Planning Level of risk associated with the target analytes/cocs at the site (not always known in the planning stage). Cost and schedule demands of the overall project The specific decisions the data is supporting (initial screening (SI) or risk assessment (RA) Complexity of analysis (a more comprehensive validation may be appropriate for highly complex analyses). March 2012 Third Party Data Validation - Training 10

11 Criteria to Guide Project Planning Ability to identify critical/significant) samples and focus data review on those samples. Project-specific audits suggest that data quality problems exist or alternatively the lab is performing high-quality work (long term monitoring). Sampling events that include recurring samples (i.e., long-term monitoring of the same chemicals could reduce validation for these events). Proximity of results to action levels. levels that are close to action levels = may require a higher level of confidence (and a greater amount of validation) levels considerably above action levels = validation is not likely to show a difference in the presence or absence of risk. March 2012 Third Party Data Validation - Training 11

12 Project Budget or How Much Is Enough? Validation can take into account budget & schedule efficiencies (examples): All data will be validated electronically. Validate a specific percentage (e.g., 10-20%) of all data sets (SDGs) unless a problem is identified. Validate a specific percentage of all data sets and all critical decision samples (as identified in the UFP-QAPP) will have full sample validation. March 2012 Third Party Data Validation - Training 12

13 Determining Data Sets for Validation Project-specific Data Quality Objectives (DQOs) defined in the QAPP Sample collection period of time within a Project, Site or Operable Unit (e.g., site investigation, remedial action, semi-annual monitoring event) Laboratory (multiple project labs or specialty analysis) Laboratory determined sample groupings (e.g., Sample Delivery Group, laboratory reporting batch, preparation batch, method batch) Combination of the preceding factors March 2012 Third Party Data Validation - Training 13

14 Data Validation Software Data review software - both data verification and validation are sequentially performed with software such as ADR.NET Efficiencies in batch processing & input error reduction result from use of EDDs and data review software. Compliance checks of electronic data (sampling and analytical) prior to import into project data base, e.g. are there results for every sample collected? March 2012 Third Party Data Validation - Training 14

15 Inputs To Review Process - Planning Item Step I WS#34 Verification Step IIa WS# 35/36 Compliance Step IIb WS# 35/36 Comparison Evidence of required approval of QAPP X Identification of personnel (those involved in conducting verification steps) X Laboratory identification X Methods (sampling and analysis) X X Performance requirements (including QC criteria) X X X Project quality objectives X X Reporting forms X X Field Sampling Plans: location, maps, grids, and sample ID numbers X X SOPs (sampling and analytical) X X List of project-specific analytes X X UFP-QAPP, Manual V1, March 2005, Data Review Elements March 2012 Third Party Data Validation - Training 15

16 Lab Data Package Requirements Data Package Item** Step I WS#34 Verification Step IIa WS# 35/36 Compliance Step IIb WS# 35/36 Comparison Laboratory name / Case narrative / Signature Page X X Chain of custody X X Sample receipt / storage records X X Communication logs X X Sample chronology (lab receipt, extraction, analysis) X X Laboratory sample identification numbers X X Identification of QC samples (sampling and lab) X X Copies of laboratory notebook, records, prep sheets X X Sample results / reporting units / dilution factors X X X Sample QC results (surrogate spike, internal standards) X X X Method QC results (e.g., spike, duplicate, LCS) X X X Associated PT sample results (batch or periodic) X X X Instrument calibration reports X X X Corrective action reports and results X X X Definitions of laboratory qualifiers X X X Documentation of any laboratory method deviations X X X QC sample raw data X X X Sample raw data X X X Standards traceability records (standard source) X X X Electronic data deliverables X X X ** DoD QSM data reporting requirements (version 4.2, Oct. 2010, section 5.10 and Appendix E) March 2012 Third Party Data Validation - Training 16

17 Verification (Step I) Process - WS#34 Step I (verification): is a completeness check of all of the sampling and analytical data collected for the project. Who conducted by the environmental laboratory (for analytical data) and The prime contractor / DoD component PDT Should address both sample collection and analytical data. March 2012 Third Party Data Validation - Training 17

18 Verification (Step I) Process - WS#34 UFP-QAPP Manual Section Verify both data packages & Electronic Data Deliverables Verify other data information operations: field operations e.g. sample collection & shipping documentation field analytical records and correlation studies database input March 2012 Third Party Data Validation - Training 18

19 Example 1 Data Verification (WS#34) Verification Input Chain-of- Custody and Shipping Forms Description Chain-of-Custody forms and shipping documentation will be reviewed internally upon their completion and verified against the packed sample coolers they represent. The shipper s signature on the Chain-of-Custody forms will be initialed by the reviewer, a copy of the Chain-of- Custody retained in the project file, and the original and remaining copies taped inside the cooler for shipment. See Field SOP-29 (Sample Handling and Custody) in Attachment 1 for further details. Internal/ External Internal Person(s) Responsible for Verification Contractor Field Team Leader Field Notebooks Laboratory Data Field notes will be reviewed internally at the end of each working day and placed in the project file. Laboratory data packages will be verified internally by the laboratory performing the work for completeness and technical accuracy prior to submittal. Received data packages will be validated internally by the Contractor Project Chemist or designee. Internal Internal/ External Contractor Field Team Leader Laboratory QA Officer Contractor Chemist Project Reports Project reports will undergo a QA review by Contractor senior staff with applicable expertise. Internal Contractor Project Team March 2012 Third Party Data Validation - Training 19

20 Example 2 Data Verification (WS#34) Verification Input Sample Receipt and Chain of Custody Description The condition of shipping coolers, temperature preservation and condition / sample ID of enclosed sample containers will be documented upon receipt at the analytical laboratory through use of a cooler receipt form. The completed cooler receipt form will be transmitted by upon receipt of samples in the lab. The cooler receipt form will also be submitted with the final analytical results from the laboratory. Internal/ External External Responsible for Verification Project Lab Analytical Data Package and Electronic Data Deliverable The lab will perform in-house analytical data reduction under the direction of the Laboratory QA Manager. Completeness of all requirements and submittals (Data Packages / EDDs) will be verified by the Laboratory PM. External Project Lab QA Project Lab PM Laboratory Data Packages and Electronic Data Deliverables Verify Data Package for completeness for the following: Laboratory Case Narrative All samples analyzed for requested methods Sample condition upon receipt Holding times Sample Results Blanks MS/MSDs LCSs Surrogate recovery Initial and Continuing Calibration summary and raw data Sample analysis raw data including sample and CCV standard chromatography Verify the completeness and identify any errors in the EDD. Internal Project Chemist March 2012 Third Party Data Validation - Training 20

21 Validation (Steps IIa) Process (WS#35) UFP-QAPP Manual Section Step IIa Validation Activities address compliance with: published methods, procedures (lab SOPs etc.), contract requirements. March 2012 Third Party Data Validation - Training 21

22 Validation (Steps IIb) Process (WS#35) Step IIb Validation activities address comparison with measurement performance criteria DQIs documented in the QAPP LOD/LOQs for project DoD QSM method specific criteria (tables in Appendix G and F) Laboratory derived performance criteria (LCS criteria) March 2012 Third Party Data Validation - Training 22

23 Example 1 Data Validation (WS#35) Step IIa/IIb Validation Input Description IIa Field SOPs Verify that the sampling SOPs were followed Person(s) Responsible for Validation Contractor Field Team Leader IIa Analytical SOPs Verify that the analytical SOPs were followed Laboratory QA Officer IIa Results Verify that the required QC samples were run and met required limits Laboratory QA Officer Contractor Project Chemist IIa/IIb Data Validation Validate 100 percent of the data to confirm quality as defined in Worksheet #14 (Summary of Project Tasks) Contractor Project Chemist IIa/IIb Data Usability Evaluation Evaluate data based on precision, accuracy, representativeness, comparability and completeness for project objectives Contractor Project Chemist IIb Field Documentation Verify accuracy and completeness of field notes Field Team Leader IIb Sample Results Verify that the required field QC samples were run and met required limits Contractor Project Chemist IIb Quantification Limits Verify that the sample results met the project quantification limit specified in the QAPP. Contractor Project Chemist March 2012 Third Party Data Validation - Training 23

24 Example 2 Data Validation (WS#35) Step IIa IIb Validation Input sample receipt, preservation, holding times Initial Calibration Description Verify that all samples were analyzed for the methods requested. Check holding times for compliance to DoD QSM Verify criteria in DoD QSM (Table 7-2) & WS #24 were met. Recalculate the calibration factors and % RSD for at least one target compound and surrogate for each standard. Recalculate at least one target compound and/or surrogate in instances where average calibration factor was not used for quantitation (i.e., linear regression, or other). Responsible for Validation Chemist Chemist IIb ICV and CCV Verify criteria in Worksheet #24 were met. Recalculate the calibration factors and % Difference for at least one target compound and surrogate (including confirmation column, if applicable) for each standard. Verify and recalculate at least one target and/or surrogate in instances where average calibration factor was not used for quantitation Chemist IIb Method and Field Blanks Verify that criteria in Worksheet #28 are met for all blanks. Review raw data for the presence of target compounds that are above the lab LOD. Chemist IIb MS/MSD Inspect the MS/MSD Recovery summary and verify that the recovery and RPD results meet criteria specified in WS#24. Recalculate at least one target compound and surrogate from the MS/MSD. Recalculate at least one RPD. Chemist IIb Surrogates Verify that the Surrogate recovery results are within criteria specified in Worksheet #28. Recalculate at least one surrogate recovery from the raw data in each field sample and QC sample. Chemist IIb LCS Verify that the LCS is within criteria specified in QAPP Worksheet #28. Recalculate at least one LCS recovery from the raw data. Chemist March 2012 Third Party Data Validation - Training 24

25 Validation Summary (Steps IIa & IIb) WS#36 Step IIa and Step IIb (validation): The amount of data / type of information to be validated will vary on a project-by-project basis DQOs for the project data The project phase (SI data or confirmation data after RA?) Budget & schedule: usually always a factor for analytical data validation Ensure sufficient representativeness for data quality across methods and analytes of concern. March 2012 Third Party Data Validation - Training 25

26 Example 1 Validation Summary (WS#36) QAPP Worksheet #36 Analytical Data Validation (Steps IIa and IIb) Summary Table Step IIa/IIb Matrix Analytical Group Validation Criteria Data Validator IIa and IIb IIa and IIb Soil Explosives 8330B Soil Metals 6010C/ 6020A QC criteria specified in this document, DOD QSM requirements, EPA Methods, USACE , Guidance for Evaluation Performance Based Data (June 30, 2005), USACE Shell Document Tables and EPA Contract Laboratory Program, and National Functional Guidelines for Organic Data Review (EPA, 2008) QC criteria specified in this document, DOD QSM requirements, EPA Methods, USACE , Guidance for Evaluation Performance Based Data (June 30, 2005), USACE Shell Document Tables and EPA Contract Laboratory Program, and National Functional Guidelines for Inorganic Data Review (EPA, 2004) Contractor Project Chemist automated data review system or third party data validation company Notes: 1. Step IIa denotes compliance with methods, procedures, and contracts. Step IIb denotes comparison with measurement performance criteria in the UFP-QAPP. March 2012 Third Party Data Validation - Training 26

27 Example 2 Validation Summary (WS#36) Step IIa/IIb Matrix Analytical Group Validation Criteria Responsible for Validation IIa & IIb soil PCB Aroclors SW8082 WS# 24 and 28 IIa & IIb soil VOCs SW8260C WS# 24 and 28 Project chemist ADR.NET ver IIa & IIb soil Metals 6020A WS# 24 and 28 March 2012 Third Party Data Validation - Training 27

28 Documentation and Communication Typical Project flow 1. Planning 2. Data collection 3. Validation processes 4. Database 5. Data users 6.? March 2012 Third Party Data Validation - Training 28

29 Validation Checks Why it is the way it is Checks are usually done in a sequence that gets the biggest bang for your buck (to use efficiently use validation resources) Sequence of checks may change depending on what parts of the validation process can be automated This is turn depends on what laboratories can provide electronically March 2012 Third Party Data Validation - Training 29

30 How does this play out when validating analytical results Critical factors affecting the results(sometimes legal not analytical - like holding times) are done first Followed by reviewing QC results that affect a large number of results (e.g., Calibration, Laboratory control samples) within a sample batch Validation generally completed by representative checks that affect fewer results but are needed for more assurance (e.g., analyte identification & recalculations to confirm a contaminant of concern and its concentrations) March 2012 Third Party Data Validation - Training 30

31 Validation Process Types Manual Review of hard copy data package submitted by the laboratory Automated Review of an electronic data deliverable by review software Combination of the above March 2012 Third Party Data Validation - Training 31

32 6. How are the validation checks and process used communicated with data results? March 2012 Third Party Data Validation - Training 32

33 Labeling Validated Data: The Black Box Opened!

34 OR Tell me what you did when you validated my data AND don t take all day about it!!! March 2012 Third Party Data Validation - Training 34

35 Typical Project Data Generation and Review Process QAPP ANALYTICAL REQUIREMENTS SENT TO LABORATORY Project Scope Defined Quality Assurance Project Plan (QAPP) Requirements Identified Field Activities SAMPLES + CHAIN OF CUSTODY INFORMATION Laboratory Analysis and Reporting FIELD LOGS INFORMATION ANALYTICAL DATA PACKAGE (with Laboratory Qualifiers) Data May Support Future Projects Data Repository e.g., Database (Project or General) DATA USED IN PROJECT DECISION (S) Data Quality Assessment of Field and Laboratory Data VALIDATED ANALYTICAL DATA External Party Validation of the Laboratory Analytical Data March 2012 Third Party Data Validation - Training 35

36 The Analytical Data Validation Black Box Each year Federal Programs spend considerable resources to validate laboratory analytical data. Validation guidelines vary from program to program. No consistent mechanism to indicate what was actually checked during the validation process. March 2012 Third Party Data Validation - Training 36

37 Opening up the Black Box! Guidance for Labeling Externally Validated Laboratory Analytical Data for Superfund Use Location on the web: pdfs/epa-540-r pdf Length: 21 pages Main document: 7 pages 5 Appendices: 14 pages March 2012 Third Party Data Validation - Training 37

38 Guidance Goals Improve communication within Superfund about scope and content of lab analytical data verification and validation. Encourage appropriate use of data for Task at hand Project phase Future decisions Guidance is not limited to EPA Superfund! March 2012 Third Party Data Validation - Training 38

39 Approach Taken Facilitate communication to data users using labels that summarize verification and validation checks. Checks are grouped into stages. Each stage builds on the checks from previous stage. Labels also describe nature of review process (manual and/or electronic) Based on the general guidelines used for data validation March 2012 Third Party Data Validation - Training 39

40 Stages of Validation Checks Completeness Compliance Sample-related QC Instrument-related QC Recalculation Instrument output review Examples of checks in each stage are given in Appendix A of the Guidance Document. March 2012 Third Party Data Validation - Training 40

41 Completeness Checks To make sure that the requested data deliverables are provided. To determine that data requested are actually present in the deliverables. To ensure consistency within the deliverable (e.g., between hardcopy and electronic deliverables). March 2012 Third Party Data Validation - Training 41

42 Compliance Checks To compare analytical Quality Control (QC) results with the acceptance criteria, requirements or guidelines present in the regional data validation documents, analytical methods or contract. Sample-Related QC (e.g., blank contamination, surrogate recoveries) Instrument-Related QC (e.g., instrument calibrations, tunes) March 2012 Third Party Data Validation - Training 42

43 Recalculation Checks The laboratory reported values (e.g., sample results, instrument calibration results) are verified by recalculation using instrument output data reported by the laboratory. Confirms that correct formulae and values were used in calculation of results. March 2012 Third Party Data Validation - Training 43

44 Instrument Output Checks Actual instrument outputs should be checked to ensure that the laboratory reported analytes have been correctly identified and quantitated (e.g., are mass spectra properly identified? Are peak integrations correct?). March 2012 Third Party Data Validation - Training 44

45 Process Labels Electronic review only Manual validation Electronic and manual E M EM March 2012 Third Party Data Validation - Training 45

46 Validation Checks as Stages Stage 2a Validation: completeness and compliance checks of sample receipt conditions and ONLY sample-related QC results. Stage 2b Validation: completeness and compliance checks of sample receipt conditions, BOTH sample-related AND instrument-related QC results. Stage 3 Validation: completeness and compliance checks of sample receipt conditions, BOTH sample-related AND instrument-related QC results, and recalculations of sample results from laboratory instrument responses. Stage 4 Validation: completeness and compliance checks of sample receipt conditions, BOTH sample-related AND instrument-related QC results, recalculations of sample results from laboratory instrument responses, and instrument outputs March 2012 Third Party Data Validation - Training 46

47 Validation Checks, Stage, and Labels Checks Validation Stage Label Completeness Stage 1 S1V Completeness & Sample QC Stage 2a S2AV Completeness, Sample QC & Instrument QC Completeness, Sample QC, Instrument QC & Recalculations Stage 2b Stage 3 S2BV S3V Completeness, Sample QC, Instrument QC, Recalculations & Instrument Outputs Stage 4 S4V March 2012 Third Party Data Validation - Training 47

48 Data Packages for Validation Stages Type of Data Package Results (with sample receipt conditions) Results with sample receipt conditions AND sample related QC information Data Validation Stage for Data Package Type Stage N/A N/A N/A N/A 1 Stage 1 Stage 2A N/A N/A N/A Results with sample receipt conditions and BOTH sample related and instrument related QC information Results with sample receipt conditions and BOTH sample related and instrument related QC information, and summary instrument data used for calculating sample results. Results with sample receipt conditions. BOTH sample related and instrument related QC information, summary instrument data used for calculating sample results, and instrument outputs (e.g., chromatograms, spectra) Stage 1 Stage 1 Stage 1 Stage 2A Stage 2A Stage 2A Stage 2B Stage 2B Stage 2B N/A Stage 3 Stage 3 N/A N/A Stage 4 March 2012 Third Party Data Validation - Training 48

49 Example Labels For Validated Data Stage 2b validation by electronic tools only: S2BVE Stage 3 validation by both electronic and manual processes: S3VEM Note: Labels combining all validation stages and processes are given in Appendix B of the Guidance March 2012 Third Party Data Validation - Training 49

50 Software Requirements for Validation Labels Data review software must: automatically append the appropriate label to each analyte in the output electronic deliverable have the capability for data reviewers to manually change the label for an analyte or group of analytes based on additional manual review of a data package or SDG. ensure that the label(s) transfer(s) with the data qualifier (flag) with each analyte into any database that will be maintained for the project March 2012 Third Party Data Validation - Training 50

51 Data Storage Requirements End Database must have a field to store validation label For uniformity it is recommended that this field be called Analytical Data Validation Stage This will be addition to the Data Qualifier field March 2012 Third Party Data Validation - Training 51

52 List of Data Validation Label Codes LABEL Stage_1_Validation_Electronic_Only Stage_1_Validation_Manual Stage_1_Validation_Electronic_and_Manual Stage_2a_Validation_Electronic_Only Stage_2a_Validation_Manual Stage_2a_Validation_Electronic_and_Manual Stage_2b_Validation_Electronic_Only Stage_2b_Validation_Manual Stage_2b_Validation_Electronic_and_Manual Stage_3_Validation_Electronic_Only Stage_3_Validation_Manual Stage_3_Validation_Electronic_and_Manual Stage_4_Validation_Electronic_Only Stage_4_Validation_Manual Stage_4_Validation_Electronic_and_ Manual Not_Validated CODE S1VE S1VM S1VEM S2AVE S2AVM S2AVEM S2BVE S2BVM S2BVEM S3VE S3VM S3VEM S4VE S4VM S4VEM NV March 2012 Third Party Data Validation - Training 52

53 Labeled EDD (ADR.NET) FIELD SAMPLE ID SAMPLING DATE MATRIX LAB SAMPLE ID QC TYPE METHOD PREP DATE ANALYSIS DATE VALIDATION CODE METHOD TYPE DUPL /9/2011 AQ FD 6010C 9/15/2011 9/19/2011 S2AVE METALS MW-78 9/8/2011 AQ C 9/15/2011 9/16/2011 S2AVE METALS MW-81 9/8/2011 AQ C 9/15/2011 9/19/2011 S2AVE METALS MW-81D 9/8/2011 AQ C 9/15/2011 9/19/2011 S2AVE METALS MW-82 9/8/2011 AQ C 9/15/2011 9/19/2011 S2AVE METALS MW-82D 9/8/2011 AQ C 9/15/2011 9/19/2011 S2AVE METALS MW-86 9/8/2011 AQ C 9/15/2011 9/19/2011 S2AVE METALS MW-86D 9/8/2011 AQ C 9/15/2011 9/19/2011 S2AVE METALS MW-89 9/9/2011 AQ C 9/15/2011 9/19/2011 S2AVE METALS MW /9/2011 AQ C 9/15/2011 9/19/2011 S2AVE METALS MW /8/2011 AQ C 9/15/2011 9/16/2011 S2AVE METALS MW /7/2011 AQ C 9/15/2011 9/16/2011 S2AVE METALS MW /7/2011 AQ C 9/15/2011 9/16/2011 S2AVE METALS MW-LF26-1 9/9/2011 AQ C 9/15/2011 9/19/2011 S2AVE METALS MW-LF26-2 9/8/2011 AQ C 9/15/2011 9/16/2011 S2AVE METALS MW-LF26-3 9/8/2011 AQ C 9/15/2011 9/16/2011 S2AVE METALS MW-LF26-4 9/8/2011 AQ C 9/15/2011 9/16/2011 S2AVE METALS MW-LF26-4MS 9/8/2011 AQ QC609298MS MS 6010C 9/15/2011 9/16/2011 S2AVE METALS MW-LF26-4MSD 9/8/2011 AQ QC609299MSD MSD 6010C 9/15/2011 9/16/2011 S2AVE METALS PZ-09 9/9/2011 AQ C 9/15/2011 9/19/2011 S2AVE METALS QC /15/2011 AQ QC MB 6010C 9/15/2011 9/16/2011 COA METALS QC /15/2011 AQ QC LCS 6010C 9/15/2011 9/16/2011 COA METALS QC /15/2011 AQ QC LCSD 6010C 9/15/2011 9/16/2011 COA METALS March 2012 Third Party Data Validation - Training 53

54 Data Review Sample Summary Report by Analysis Method Lab Batch ID: 01_AW Approved by: DN (3/15/2012) Laboratory: LDC Client Sample ID Lab Sample ID Matrix Sample Type Preparation Method Collection Date Validation Code Method: 6010C (TOT) MW AQ N 3010A 10/6/2008 S2AVE MW AQ N 3010A 10/6/2008 S2AVE MW AQ N 3010A 10/6/2008 S2AVE MW AQ N 3010A 10/6/2008 S2AVE Method: 8081A MW AQ N 3520C 10/6/2008 S2AVE MW AQ N 3520C 10/6/2008 S2AVE MW AQ N 3520C 10/6/2008 S2AVE MW AQ N 3520C 10/6/2008 S2AVE Method: 8260C MW AQ N 5030B 10/6/2008 S2AVE MW AQ N 5030B 10/6/2008 S2AVE MW AQ N 5030B 10/6/2008 S2AVE MW AQ N 5030B 10/6/2008 S2AVE Method: 8270D MW AQ N 3520C 10/6/2008 S2AVE MW AQ N 3520C 10/6/2008 S2AVE MW AQ N 3520C 10/6/2008 S2AVE MW AQ N 3520C 10/6/2008 S2AVE March 2012 Third Party Data Validation - Training 54

55 Example of Labeled Project Report Location ID / Analyte Lab Result Qualifier LOD LOQ Units MW-L66-1 Diesel C10-C28 MW-L66-1 Motor Oil C28-C36 MW-081 Diesel C10-C28 MW-081 Motor Oil C28-C36 MW-L66-3 Diesel C10-C28 MW-L66-3 Motor Oil C28-C36 MW-L66-6 Diesel C10-C28 MW-L66-6 Motor Oil C28-C36 MW-01 Diesel C10-C28 MW-01 Motor Oil C28-C36 Validation Code 15 J µg/l S2BVE <300 UJ µg/l S2BVEM 41 J µg/l S2BVE <300 UJ ug/l S2BVEM 45 J µg/l S2BVE <300 UJ µg/l S2BVEM 22 J µg/l S2BVE <300 UJ µg/l S2BVEM 30 J µg/l S2BVE <300 UJ µg/l S2BVEM March 2012 Third Party Data Validation - Training 55

56 March 2012 Third Party Data Validation - Training 56

57 March 2012 Third Party Data Validation - Training 57

58 Desired Outcome Third party reviewers/validators associate the validated data with its validation stage as data is shared with decision makers. Data users quickly recognize the nature of review performed on data prior to use. Future use of data is facilitated by labels that travel with data (e.g., kept associated with data in repositories) March 2012 Third Party Data Validation - Training 58

59 Benefits of Providing Validation Labels Opens up the Validation Black Box by telling data recipients and users in a short and succinct manner: Checks used to validate the data Process used to validate the data Supports future use of data March 2012 Third Party Data Validation - Training 59

60 Benefits of Providing Validation Labels Allows for automation of some stages of the data validation process (e.g., use of electronic tools for compliance checks and recalculations) Helps integrate manual and electronic processes used for data validation March 2012 Third Party Data Validation - Training 60

61 Summary & Things to Think About. Determine data sets and level of validation as part of the Project Planning Process (e.g., document in UFP-QAPP Worksheet) Ensure Data Review software can provide the labels and allows for manual editing of labels based on further validation (for EDDs) Retain Data Validation Labels by ensuring data repositories have fields for the labels which are linked to the validated analytical data March 2012 Third Party Data Validation - Training 61

62 Implementation Status US Army Corps of Engineers through use of ADR.NET software (puts validation label after automated review) US Army Corps of Engineers FUDS CHEM Database (has a field for the validation label) EPA Regions (may need some prodding!) March 2012 Third Party Data Validation - Training 62

63 FINAL STEP!!! UP TO YOU Repeat after me: I will never use validated data that does not have a label attached to it!!! March 2012 Third Party Data Validation - Training 63

64 Contact Information Pam Wehrmann Phone: Anand Mudambi Phone: March 2012 Third Party Data Validation - Training 64

65 Questions? March 2012 Third Party Data Validation - Training 65