CONTENTS STANDARD OPERATING PROCEDURES FOR THE CAEAL PROFICIENCY TESTING PROGRAM CAEAL Inc., 2001. All rights reserved. INTRODUCTION AND SCOPE DEFINITION OF PROFICIENCY TESTING (PT) PROFICIENCY TESTING AND ACCREDITATION SUPPORTING STANDARDS, GUIDELINES AND REFERENCES REVISION HISTORY SECTION A - PT PROGRAM REQUIREMENTS SECTION B - CAEAL DATA EVALUATION PROCEDURES SECTION C - CAEAL CALCULATION ROUTINES SECTION INTRODUCTION AND SCOPE This document describes the program requirements and the standard operating procedures for the routine analysis and evaluation of inter-laboratory data generated by the CAEAL Proficiency Testing (PT) Program. These procedures apply to all the test groups/parameters offered in the CAEAL PT Program, with the exception of Asbestos, which is offered in partnership with the Asbestos Quality Assurance Program (AQAP). The details of this program are located in the Asbestos Program Description. DEFINITION OF PROFICIENCY TESTING (PT) CAN-P-1593: 2000 Guidelines for the Recognition of Proficiency Testing Schemes, Introduction: Proficiency testing is the use of inter-laboratory comparisons to determine the performance of individual laboratories for specific tests or measurements. PROFICIENCY TESTING AND ACCREDITATION ISO/IEC Guide 58: (most recent version) Calibration and testing laboratory accreditation systems - General requirements for operation and recognition, clause 6.8.3: Accredited laboratories shall participate in proficiency testing or other inter-laboratory comparisons as required by the accreditation body. Their performance in such tests shall meet the requirements of the accrediting body. PT15 SOP for PT Rev. 1.3 Page 1 of 15
ILAC-G10: 1996 Harmonised Procedures for Surveillance and Reassessment of Accredited Laboratories, Introduction: Laboratory accreditation is the best mechanism to provide assurance to customers on the quality and competence of the laboratory This assurance can be achieved through a mechanism of regular surveillance and reassessment visits enhanced, where appropriate, by other surveillance activities and regular participation in proficiency testing. PT15 SOP for PT Rev. 1.3 Page 2 of 15
SUPPORTING STANDARDS, GUIDELINES AND REFERENCES ISO/IEC Guide 43: 1997 - Proficiency testing by inter-laboratory comparisons Part 1: Development and Operation of Laboratory Proficiency Testing Schemes Part 2: Selection and Use of Proficiency Testing Schemes by Laboratory Accreditation Bodies ISO/IEC Guide 58: (most recent version) Calibration and testing laboratory accreditation systems. ISO/IEC 17025: 1999 General requirements for the competence of testing and calibration laboratories. CAN-P-4D: 2000 General requirements for the competence of testing and calibration laboratories. ILAC-G13: 2000 Guidelines for the Requirements for the Competence of Providers of Proficiency Testing Schemes. ILAC-G10: 1996 Harmonised Procedures for Surveillance and Reassessment of Accredited Laboratories. CAN-P-1593: 2000 Guidelines for the Recognition of Proficiency Testing Schemes. CAN-P-1594: 2000 Checklist for the Assessment of Proficiency Testing Providers. ISO Guide 35: (most recent version) Certification of reference materials General and statistical principles. IUPAC/ISO/AOAC 1993. The International Harmonized Protocol for the Proficiency Testing of (Chemical) Analytical Laboratories. Grubbs, F. E., 1969. Procedures for Detecting Outlying Observations in Samples. Technometrics, Vol. II, No. 1, pp. 1-21. PT15 SOP for PT Rev. 1.3 Page 3 of 15
REVISION HISTORY Rev. 1.2 Added reason for selection of data analysis procedures to Section B. Rev. 1.3 Added copyright to document. Updated Supporting Standards, Guidelines and References to include current versions. Added revised scoring scheme for toxicology analyses (Section A and C). Added the method of assessing the test material homogeneity and stability for microbiological samples (Section A). Added requirement that verification data reports are received, filed and that periodic checks of assigned property values are carried out (Section B). Added procedures for the evaluation of reference lab homogeneity and stability data to ensure that when there is evidence of analytical or other errors, the proper Corrective Actions are taken (Section A). Added requirement to refrain from manually adjusting the data for best fit by visual examination of the data generated by the software (Section B). Redefined the formulation of the z score to be consistent with ISO/IEC Guide 43 (Sections A and C) Redefined and outlined the specific processes for the statistical analysis of the data and the basis for the performance evaluation techniques (Sections B and C). Documented the acceptance criteria for determining the suitability of test items for evaluation (Section A). Added the requirement to modify the PT reports issued to participants to clearly indicate which sample/test invoked the Exception Clause (i.e. use of precision function data) for the calculated z scores during the statistical analysis of the data (Section A). Removed reference to the issue of Preliminary PT reports (Section B). PT15 SOP for PT Rev. 1.3 Page 4 of 15
SECTION A PT PROGRAM REQUIREMENTS Program Operation Requirements Co-ordinator - day to day operations (CAEAL Proficiency Testing Manager) Advisory Panel and Program Committee - advice on program design, implementation and delivery Program Development Requirements Test Samples Test Methods Sample Management Data Management Statistical Design TEST SAMPLE REQUIREMENTS The PT samples must be: homogenous - to ensure that all participating laboratories are treated fairly and that the CAEAL program is seen to be equitable to all stable -requirements for chemical preservation, storage temperatures, holding times, etc. must be addressed matrix matched - test samples used must have a matrix which is generally similar to the matrix of the corresponding routine samples Note 1: There may be circumstances when it is necessary to cancel the results for one or more of the four samples in the set. The acceptance criteria for determining the suitability of the samples are as follows: - the verification analyses from the current study are not significantly different from the design/target values. - the consensus data (i.e. mean and standard deviation) are not significantly different from the precision function data. - the samples used in the CAEAL program have been demonstrated to be stable for the period of each study. PT15 SOP for PT Rev. 1.3 Page 5 of 15
- the homogeneity analysis has demonstrated that there is no systematic variability from the first to the last sample in the production run. Note 2: the microbiology samples are guaranteed to be stable for a period of 72 hours. The stability and homogeneity of these samples will be assessed by reviewing the data tables and pictures of the plates submitted to CAEAL by the reference lab. The bacterial strains may also be verified by an independent third-party lab (in addition to the verification lab) to ensure they perform well on all media types that are typically used by labs participating in the microbiology PT studies. Note 3: the reference lab homogeneity and stability data are evaluated to ensure that when there is evidence that analytical or other errors have occurred, the following Corrective Actions are taken: - The PT manager will initiate a CAR/PAP (see section 4.7 of the Quality Manual and Q24) and forward it to the reference lab responsible for preparing the samples. - The reference laboratory must respond within 5 working days and then take corrective action to resolve the problem, to the satisfaction of the PT manager and/or investigator. - The PT Manager and two members of the Program Committee will make the decision on whether to allow the shipment to proceed or whether the batch of PT samples should be withheld and prepared again. TEST METHOD REQUIREMENTS Laboratories must use a specified analytical method in those cases where: - the test method defines the result (e.g. oil and grease determination) Otherwise laboratories may use their method of choice provided that: - the method is not known to produce (i) results which are biased relative to other methods or (ii) results which are not normally distributed SAMPLE MANAGEMENT REQUIREMENTS production and sub-sampling packaging identification and labelling shipping and storage PT15 SOP for PT Rev. 1.3 Page 6 of 15
DATA MANAGEMENT REQUIREMENTS Results reported by participating laboratories must conform to routine practice. For example, duplicate determinations should not be carried out if this is not routine practice, as specified by the test method. This requirement to treat all samples as routine is included in the Technical Requirements (section 3.5.1.5) of ILAC-G13:2000, Guidelines for the Requirements for the Competence of Providers of Proficiency Testing Schemes. Reports issued to participants must, as a minimum, contain the following: - the actual test results provided by the laboratory - a performance score - summary of results for all participating laboratories The PT reports issued to participants will be modified to clearly indicate which samples/test groups invoked the Exception Clause described below (i.e. use of precision function data) for the calculated z scores during the statistical analysis of the data. STATISTICAL DESIGN REQUIREMENTS 1) Overall Study Specifications 2) Establishing the Assigned Reference Value 3) Removal of Outlier Data 4) Performance Statistics 1) OVERALL STUDY SPECIFICATIONS Studies should (i) accommodate required concentration ranges, (ii) adequately accommodate the on-going maintenance and evaluation of performance and (iii) provide results which are statistically meaningful. CAEAL PROGRAM Concentration Range - number of samples per study 4 On-going Performance - study frequency (months) 6 Statistical Validity - minimum number of participating labs 20 PT15 SOP for PT Rev. 1.3 Page 7 of 15
2) ESTABLISHING THE ASSIGNED REFERENCE VALUE Definition: The best available estimate of the true value. Options for its determination: 1. Consensus Value - provided by all participating laboratories 2. Reference Lab Value - provided by the reference/collaborator labaoratory 3. Design Value - traceability is obtained by ensuring all reagents are of certified purity and all gravimetric and volumetric measurements are traceable to national measurement standards (e.g. NIST) 4. Direct Comparison Value - traceability to national measurement standards is demonstrated by simultaneously analyzing the test sample and a similar certified reference material (CRM) and obtaining results for the CRM that do not differ significantly from the certified value(s). CAEAL PROGRAM The Consensus Value is generally chosen as the assigned reference value. The Reference Lab Value may be employed in situations where the Consensus Value is deemed unreliable (i.e. when there are < 20 labs or when the majority of labs report < MDL for a given parameter). 3) REMOVAL OF OUTLIER DATA Possibilities include: - Dixon Test - Grubbs Test - Huge Error Test - Manual Inspection CAEAL PROGRAM Grubbs Test: - sort data in order of increasing value PT15 SOP for PT Rev. 1.3 Page 8 of 15
- calculate mean (X) and standard deviation (σ) - compute T for x 1 and x n ; i.e.: T = x i - X / σ x I = x 1, x n - select 95% confidence interval and if T exceeds the corresponding statistical value (obtained from a statistical table) the data point in question is rejected. - repeat process until no further data is rejected. 4) PERFORMANCE STATISTICS - Assigned Reference Value (X). The best available estimate of the true value. - Target Standard Deviation (σ). A numerical value for the standard deviation of a measurement result, which has been designated as a goal for measurement quality. Performance of single test results: - Deviation, or error; D = (x i - X) - Deviation adjusted for random error, (Z score); Z = D/σ Combined Performance Scores: - composite score for the same test Z < 2 = satisfactory performance 2 Z 3 = questionable performance Z > 3 = unsatisfactory performance - composite score for a group of tests (e.g. organic tests) - composite score for the laboratory CAEAL PROGRAM 1. Assign s using either the observed study standard deviation (σ) from the consensus data or the standard deviation calculated from precision function data (see Exception Clause below). Exception clause: precision function data (the relationship of s and X from studies previously conducted by CAEAL or other appropriate organizations, such as the U.S. PT15 SOP for PT Rev. 1.3 Page 9 of 15
EPA or NIST) will be used for the calculated z scores during the statistical analysis of the data only under the following circumstances: - when a special study is being conducted (i.e. when a lab is being re-evaluated following a failure for a specific parameter/test group) - when a reliable consensus value cannot be obtained from the data set (i.e. when there are less than 20 labs participating for a specific parameter/test group or when the majority of labs report < MDL) 2. Calculate the Z score (Z = x X / s) and assign points as follows. Z Score Pts Assigned Code Assigned Category 1.00 5 OK satisfactory 1.01-2.00 4 OK* satisfactory 2.01-3.00 2 L or H questionable > 3.00 0 VL or VH unsatisfactory Toxicology: the following scoring scheme is used for toxicology analyses (including trout, daphnia and microtox) effective as of the March 2002 study. Z Score Pts Assigned Code Assigned Category 1.00 5 OK satisfactory 1.01-2.00 5 OK satisfactory 2.01-3.00 3 L or H questionable > 3.00 0 VL or VH unsatisfactory 3. Calculate the PT Score PT Score = Total Points / # of samples * 100 / 5 PT Score Category 70 satisfactory < 70 unsatisfactory PT15 SOP for PT Rev. 1.3 Page 10 of 15
SECTION B CAEAL DATA EVALUATION PROCEDURES This section outlines the procedures performed by the Proficiency Testing (PT) Manager in the process of evaluating the data submitted by the participating laboratories. The data analysis techniques selected for the CAEAL program are based on approved procedures in Annex A of ISO/IEC Guide 43-1 (1997). Note: Refer to Section C for a more extensive description of all the CAEAL Calculation Routines. The general data evaluation procedures are as follows: Verify that all reference lab data reports are received and filed and that periodic checks of assigned property values are carried out. If there is evidence of analytical or other errors, ensure the proper Corrective Actions are taken (see Section A). Remove outliers using the Grubbs Test Exclude any values preceded with a < sign Compare the mean, adjusted mean and median: - Mean = consensus mean - Adjusted mean = consensus mean after outliers removed by Grubbs test - Median = consensus median after outliers removed by Grubbs test Compare the reference values: - Consensus mean (Con) - Reference lab mean (PFx) - The PT Manager and two members of the Program Committee will make the decision whether to use the Reference lab mean in place of the Consensus mean - The Reference Lab Value may be employed in situations where the Consensus Value is deemed unreliable (i.e. when there are < 20 labs or when the majority of labs report < MDL for a given parameter) Compare the s values: - Consensus data (Con) - Precision function data (PFx) - The PT Manager and two members of the Program Committee will make the decision whether to use the Precision function data in place of the Consensus data (see Exception Clause in Section A) or whether the sample(s) should be eliminated from the data analysis PT15 SOP for PT Rev. 1.3 Page 11 of 15
Precision function data are used (whenever possible) to calculate a value of s for two reasons: - May be used for test groups with a small sample size (i.e. < 20 participating labs) or for a special PT study - Also used to compare the consensus data with precision function data Refrain from manually adjusting the data for best fit by visual examination of the data generated by the software Confirm design (target) values with verification data from reference lab Check for duplicates (confirm with reference lab data) - Average the consensus means of duplicate samples so the duplicates are equivalent Examine the performance summaries for each parameter Add the outliers back into the database in order to prepare the Final PT Reports for all the participating laboratories Modify the PT reports issued to participants (e.g. on the cover letter) to clearly indicate which samples/test groups invoked the Exception Clause. Paper copies of the analyses are maintained on file by the PT Manager. Any decisions or actions required are also documented and filed by the PT Manager. PT15 SOP for PT Rev. 1.3 Page 12 of 15
SECTION C CAEAL CALCULATION ROUTINES This section describes the statistical analyses performed by the Data and Information Manager on the data submitted by the participating laboratories. PT Calculation Routines 1. Identify results x i provided by n laboratories at the sample I.D./parameter level. 2. Remove outlier data points from the data set using Grubbs Test. Grubbs Test: T = x S - X / σ X S = x 1 and x n Calculations: 1. Calculate X and σ. 2. Calculate T for x 1 and x n. Identify largest T. 3. If largest T > value from Grubbs Table, point is removed. 4. Continue until no more points are removed. Note (1) < values: set to zero for purposes of calculating averages and applying the Grubbs test. 3. Calculate the Adjusted Mean and Median Adjusted Mean = mean provided subsequent to removal of outlier data points. Median = median provided subsequent to removal of outlier data points. If n is even, the average of the middle two data points is used. 4. Identify the assigned reference value (X) (a) calculate assigned reference value using: Assigned reference value = (adjusted mean + median) / 2 (b) the Reference Lab mean may be used in place of the Consensus mean, if the Consensus mean is deemed to be unreliable (i.e. when there PT15 SOP for PT Rev. 1.3 Page 13 of 15
are < 20 labs or when the majority of labs report < MDL for a given parameter). 5. Calculate the s value (a) (b) initially use the observed study standard deviation (σ) from the consensus data the standard deviation calculated from precision function data may also be used (see Exception Clause) 6. Calculate the Z score Z score = x - X / s - Calculate Flags (exception: toxicology analyses; see section A). Case Z Score Flags 1. < 1.00 OK 1.00-2.00 OK* 2.01-3.00 L or V > 3.00 VL or VH 2. >2.00 Flag 7. Calculate the PT Score - identify the number of PT samples for a particular test group/parameter (number of PT samples usually = 4) - calculate points assigned to each PT sample according to the following table (exception: toxicology analyses; see section A): Z score Points assigned Points assigned (tox) 1.00 5 5 1.01-2.00 4 5 2.01-3.00 2 3 > 3.00 0 0 Note (1) < Values: if either the ref value or reported value is a < value, apply the following rules: (a) one < value where < value is less than the other value: (i) (ii) use < value to calculate deviation calculate points assigned PT15 SOP for PT Rev. 1.3 Page 14 of 15
(b) all other cases, including two < values: (i) do not calculate deviation (ii) points assigned = 5 Note (2) No Result: if no result is reported do not enter any values (leave blank): (i) do not calculate deviation (ii) points assigned = 0 - calculate PT Score for test group/parameter - if there is no data (i.e. lab did not provide results) PT score = 0 8. Calculate status for a particular test group/parameter (a) (b) identify PT score assign a Pass or Fail according to: PT score 70 = Pass PT score < 70 = Fail (c) subsequent to a Pass, assign status as follows: one fail = possible suspension (PS) two successive fails = suspension (S) three successive fails = withdrawal (W) PT15 SOP for PT Rev. 1.3 Page 15 of 15