USING CLSI GUIDELINES TO PERFORM METHOD EVALUATION STUDIES IN YOUR LABORATORY Breakout Session 3B Tuesday, May 1 8:30 10 am James Blackwood, MS, CLSI David D. Koch, PhD, FACB, DABCC, Pathology & Laboratory Medicine, Emory University School of Medicine
Learning Objectives Identify the seven performance characteristics that should be evaluated prior to reporting results from a new test method. Verify the claims of manufacturers regarding analytical performance by following CLSI guidelines. Demonstrate ongoing compliance with the method evaluation criteria contained in accreditation guidelines. Who is CLSI and what are these guidelines? Outline Method evaluation basic definitions and experiments Use of the CLSI Evaluation Protocols Use of StatisPro software in method evaluation
Who is CLSI? Clinical and Laboratory Standards Institute ANSI-accredited, global, nonprofit standards development organization CLSI has over 2,000 members organizations such as IVD manufacturers, hospital laboratories, reference laboratories, universities, professional associations, and government agencies We promote the development and use of voluntary consensus standards and guidelines within the health care community. Our documents help health care organizations meet their responsibilities with efficiency, effectiveness, and global acceptance.
We Make the Blue Books
Standards in the Clinical Laboratory Goal of standardization in the laboratory: The right laboratory test at the right time with the right result leads to quality diagnostics and improved patient care, and improved public health around the world. Standardized test Standardized procedure Standardized reporting Improved outcomes
How are CLSI Documents developed? CLSI documents are developed by volunteer experts from three distinct constituencies: professions, government, and industry. Under the supervision of a consensus committee, these volunteers work on: Document Development Committees or Standing subcommittees and Working groups
CLSI Consensus Process Government Balance Industry Professions
Why are CLSI Guidelines Important? The US Food and Drug Administration(FDA) recognizes over 100 CLSI documents. The College of American Pathologists (CAP) recognizes 80 CLSI documents. The Joint Commission recognizes over 144 CLSI documents. All Evaluation Protocol guidelines in this presentation are recognized by all three groups.
Why are Evaluation Protocols Important? Provide clear and thorough guidance. Evaluation protocols are guidelines for clinical laboratories and manufacturers to characterize the performance of their analytical systems. Ensure consistency with good laboratory practice. Good laboratory practice requires clinical laboratories to verify performance claims before reporting results used for decisions about patient care. Help you to comply with the law! Evaluation of performance characteristics is required by regulatory and accreditation bodies in the United States. See http://www.cms.hhs.gov/clia ( 493.1255).
CLSI and Evaluation Protocols CLSI has over 25 Evaluation Protocol Guidelines. These include: EP05 Evaluation of Precision EP06 Evaluation of Linearity EP09 Evaluation of Bias and Comparability Using Patient Samples EP10 Preliminary Evaluation (Bias, Carryover, Drift, Linearity) EP15 Verification of Precision and Trueness EP17 Limits of Detection and Limits of Quantitation C28 Defining, Establishing, and Verifying Reference Intervals
Performance Characteristics The seven performance characteristics that should be evaluated before reporting results of a new test method include: 1. Precision 2. Accuracy (measured bias) or comparability (measured differences) 3. Linearity over the measuring interval or analytical measurement range (AMR) 4. Limit of detection (LoD) and limit of quantitation (LoQ or analytical sensitivity) 5. Specificity or interference 6. Reagent or sample (analyte) carryover 7. Reference interval or decision value (interpretive information)
Precision & Accuracy
Apply a clinical perspective; set a target, an analytical goal, before you begin Perform experiments that gather representative data about a method s analytical performance Convert data into statistical estimates of errors Compare error estimates to specifications for medically allowable error for an objective assessment of the errors
Introducing a New Method Establish a Need Define the Quality Goal Method Selection Submit Specimen Method Evaluation Method Development Implementation Preventive Maintenance Routine Analysis Quality Control Practices Report Result
APPROACH IN METHOD EVALUATION: Evaluate imprecision and inaccuracy IMPRECISION Refers to Random Analytic Error (Lack of repeatability, reproducibility) INACCURACY Refers to Systematic Analytic Error (Lack of trueness) 1. Constant 2. Proportional TOTAL ERROR Combined error for a single result
RELIABLE DECISIONS ABOUT PERFORMANCE REQUIRE: 1. Standards for acceptable performance 2. Experimental protocols to estimate performance reliably 3. Criteria for comparing estimated performance with performance standards
PERFORMANCE STANDARD (PS) Specify: EA... Allowable error XC... Decision level Format: PS = EA at XC
ALLOWABLE ERROR (EA) The amount of error that can be tolerated without invalidating the medical usefulness of the result or causing the test to fail a proficiency testing event
DECISION LEVEL (XC) Any concentration of the analyte that is critical for medical interpretation whether for diagnosis, monitoring, or therapeutic decisions. Laboratory data are most carefully interpreted at these decision level concentrations.
DECISION LEVELS FOR GLUCOSE 1 0 20 50 2 80 126 3 160 200 plasma glucose, mg/dl 260 300 340
DECISION LEVELS FOR GLUCOSE Performance standards for Glucose PS1 = 6.0 mg/dl @ 50 mg/dl Medical Decision Hypoglycemia PS2 = 10% = 12.6 mg/dl @ 126 mg/dl Impaired glucose control PS3 = 30 mg/dl @ 300 mg/dl Poorly controlled diabetes
Sources of Allowable Errors 1. Proficiency testing requirements for acceptable performance 2. Literature guidelines a. based on physician surveys e.g.: Karon, Boyd & Klee, Glucose Meter Performance Criteria for Tight Glycemic Control Estimated by Simulation Modeling, Clin Chem, 2010; 56: 1091-97 b. based on intra-individual biological variation of analyte Ricos C et al., Scand J Clin Lab Invest,1999; 59: 491-500 Fraser C, Biological Variation: From Principles to Practice, AACC, 2001 Internet at http://www.westgard.com/biodatabase1.htm 3. Input from clinicians and/or professional judgment
Formulation of Criteria to Judge Analytic Errors General form: compare observed analytic error to the specification for allowable analytic error Performance is acceptable when: observed error < allowable error Performance is not acceptable when: observed error > allowable error
Performance Characteristics: Precision CLSI Guidelines for Precision EP15: a five-day procedure to verify that imprecision meets the claims of a measurement procedure (EP15 is most frequently used by clinical laboratories for method evaluation.) EP05: a 20-day procedure to establish the imprecision for a measurement procedure
Replication Experiment 1. Time period: within-run within-day day-to-day 2. Number of samples: minimum of 20 3. Sample matrix: simulate patient sample 4. Analyte concentration: medical decision limit 5. Calculations: mean, standard deviation (SD), coefficient of variation (CV)
Performance Characteristics: Accuracy Accuracy [Trueness] (Measured as Bias) Bias: Estimate of a systematic measurement error; a quantitative measure of the average difference between results from a measurement procedure and results from an accepted reference measurement procedure. When a reference measurement procedure is not available for an analyte, a best-available comparative method may be used to measure bias. Frequently, clinical laboratories perform a comparison of patient sample results between a new and an existing measurement procedure. ( correlation studies )
Performance Characteristics: Accuracy CLSI Guidelines for Trueness (Measured as Bias) EP15: a method comparison to verify that a new method conforms to a manufacturer s claim for comparability to another procedure. (minimum of 20 patient samples) EP09: a method comparison to establish a claim for method comparability. (minimum of 40 patient samples)
Comparison of Methods Experiment CLSI EP9-A: User Comparison of Quantitative Clinical Laboratory Methods Using Patient Samples 1. Choice of comparative method: critical for the conclusions which can be made 2. Number of test samples: minimum N = 40 uniform distribution (EP9-A includes a table) a bin-box approach
Comparison of Methods Experiment Bin-box approach: 5 Number of samples 10
Comparison of Methods Experiment 3. Replicates: required for EP9-A: desirable, but not always practical 4. Time period: minimum of 5 days 5. Data analysis: review daily Check for maximum allowable differences between methods EP9-A includes a test for outliers within and between methods 6. EP9-A has a section on establishing manufacturer s claims
Three Approaches to Analyzing Comparison of Methods Data 1. correlation coefficient 2. t-test statistics 3. regression statistics
Sensitivity of Statistical Parameters to Errors Parameter Random Constant Proportional LEAST SQUARES SLOPE Y-INTERCEPT STD. ERROR no no yes no yes no yes no no BIAS sd no yes yes no yes yes yes no no T-TEST CORRELATION COEFFICIENT r
Effect of range on the correlation coefficient H H Range 0 to 300 70 to 110 Random Error 10 units 10 units 0.986 0.764 Corr. Coef., r
Correlation coefficient, r Responds to random error. Value depends on the range of data. Does not estimate analytical bias or random error between methods. Merely presents the relationship of the range of the data to the scatter of the data between methods. Therefore, the correlation coefficient should NOT be used to judge acceptability of analytical methods in method comparison studies.
Linear regression statistics Subject to certain limitations: Data must be linear Outliers must be carefully examined Range of data must be wide: a. r > 0.99 (Waakers et al.) b. r > 0.975 (CLSI EP9-A)
Recommendations for Method Comparison Summary Present graph of data Present slope, y-intercept, and Sy/x Present mean and standard deviation of X data Present correlation coefficient ONLY to show that least squares regression is applicable; if not, use Deming or Passing-Bablock regression statistics
Performance Characteristics: Linearity Linearity Measuring Interval or Analytical Measurement Range (AMR) A linearity study is used to establish or verify the measuring interval for a measurement method. Measuring Interval: the interval between lower and upper numerical values for which a method can produce quantitative results suitable for the intended clinical use. The measuring interval is verified by demonstrating a linear relationship between the measured and expected concentration relationships. CLSI Guideline for Linearity Measuring Interval EP06: procedures to verify or establish the linear measuring interval of a measurement procedure.
Performance Characteristics: LoD/LoQ Limit of Detection (LoD) & Limit of Quantitation (LoQ) (sometimes referred to as Analytical Sensitivity ) LoD: the lowest amount of analyte (measurand) in a sample that can be detected with a stated probability. LoQ: the lowest amount of analyte (measurand) in a sample that can be quantified with acceptable precision and bias under stated experimental conditions. Usually, laboratories review and accept the manufacturer s claims for LoD and LoQ. But these characteristics can be tested by laboratories using: CLSI Guideline for LoD and LoQ EP17: procedures for verifying or establishing the LoD and the LoQ
Performance Characteristics: Interference Interference: an artifactual increase or decrease in the apparent quantity of an analyte due to the presence of a substance that reacts nonspecifically with the measuring system. Most manufacturers evaluate a large number of substances known or suspected to be potential interferents. They report this information in the Instructions For Use (IFU). It is not practical for most clinical laboratories to repeat such an investigation and inspection of the manufacturer s information is frequently sufficient. But these characteristics can be tested by laboratories using: CLSI Guideline for Interference EP7: procedures for testing constant error due to interference
Interference Experiment: Factors 1. See CLSI EP7-A2 2. What to test: Literature review Always test hemolysis, lipemia, bilirubin Tube additives 3. Concentrations to test: Interferent: highest compatible with life Analyte: at medical decision levels 4. Volume of interferent <10% of sample 5. Replicates: Based on Effect / Stm (see EP7) 6. Validate technique with current method
Interference Experiment: N=? Number of Measurements / Replicates: at least several samples per interferent at least duplicates per sample EP7 lists a table of N as a function of bias/stm (EA,I/Stm ), with which one can determine how many replicates are necessary to reach 95% probability of observing a certain magnitude of error: EA,I/Stm No. Replicates EA,I/Stm No. Replicates 0.8 1.0 1.1 1.2 1.3 1.4 41 26 22 18 16 14 1.5 1.6 1.8 2.0 2.5 3.0 12 10 8 7 6 3
Performance Characteristics: Carryover Carryover: the discrete amount of reagent or analyte carried by the measuring system from one test into subsequent test(s), thereby erroneously affecting test results. Reagent carryover among different measurement procedures on multichannel automated analyzers is an evaluation that is usually conducted by measuring system manufacturers. But this characteristic can be tested by laboratories using: CLSI Guideline for Carryover EP10: includes an assessment of sample carryover along with other parameters. NOTE: EP10 is intended to determine if a device has unacceptable performance. It is recognized in the CAP Chemistry Checklist as an acceptable way to measure carryover.
Performance Characteristics: Reference Intervals Reference Interval: interpretive information for laboratory test results that is frequently provided as the central 95% interval of results for a group of well-defined reference individuals. Laboratories can produce reference intervals in a variety of ways, including testing procedures found in CLSI Guideline for Reference Intervals or Decision Value C28: procedures for establishing a reference interval or verifying the suitability of a manufacturer-proposed reference interval
Reference Interval Determination Transference of established reference intervals to an individual laboratory or a new method may be accomplished in a variety of ways: 1. Subjective assessment by a responsible individual; the Medical Director (sometimes called by divine judgment ) 2. Donor testing a. Verify with ~ 20 donor samples b. Validate/Estimate using ~ 60 donor samples c. Establish using ~ 120 donor samples 3. Calculation use regression statistics from a comparison of methods study to calculate reference limits for the new method (Y) that correspond to the reference interval limits of the former method (X). Y=a+b X
CLSI Makes Life Easier with StatisPro In October 2010, CLSI released StatisPro software: Direct, faithful implementation of CLSI Evaluation Protocol Guidelines Study Advisor step-by-step help for each study Four steps to complete a study: Definition, Data Input, Analysis, and Signoff
StatisPro Pick a Study Type
StatisPro Study Design Study Goal Identifying Information Details of the Study Performance Claim to be Verified Description of Materials Used
StatisPro Data Entry Copying and Pasting from any spreadsheet application or Windows application with clipboard support is easy.
StatisPro Analysis 1 - Inspect group: Evaluate the data visually using various plots and tables. You can choose to show or hide excluded observations. 2 - Outliers group: Select an observation to exclude from the calculations. 3 - Study-specific group: Select commands that continue to evaluate the data and reach a study conclusion. 4 - Sign Off group: Add any comments, your name, and a signature line to the study report so it is ready for a handwritten signature when printed.
StatisPro Study Advisor
StatisPro Demonstration Demonstrate EP15 (method comparison) and EP06 (linearity).
User Experience with StatisPro StatisPro is useful when introducing new methods into your laboratory. StatisPro is useful when performing six-month linearity or calibration verification studies. By using StatisPro: You are demonstrating compliance with regulatory and accreditation bodies. You are ensuring that your laboratory delivers accurate results.
Thank You Questions?