Eileen M. Burd Ph.D., D(ABMM) Director, Clinical Microbiology. Atlanta, GA

Transcription

1 Eileen M. Burd Ph.D., D(ABMM) Director, Clinical Microbiology Emory University Hospital Atlanta, GA

2

3 In vitro diagnostic devices (IVDs) Substantial equivalence to existing device FDA cleared Novel agent, new method, public health threat (e.g., Mycobacterium tuberculosis) FDA approved Labeled: For in vitro diagnostic use

4 FDA cleared or approved tests that have been modified Changes in test components, procedural parameters, cutoff values, specimen types, collection devices, etc. Tests currently not subject to FDA clearance or approval Standardized textbook procedures Tests developed in the laboratory FDA has authority to regulate but has exercised enforcement discretion and has chosen not to do so Test systems in which performance specifications are not provided by the manufacturer ASRs (analyte specific reagents)

5 For research only, not for diagnostic purposes Not medical devices essentially unregulated Must be labeled bld For research use only. Not for use in diagnostic procedures. Basic science research Research related to product development Can be used as components of LDTs if FDA approved/cleared or ASRs are not available

6 Required clinical investigation before a manufacturer can submit an application to FDA May be distributed only for use in well controlled clinical trials to establish performance characteristics monitored and documented Label: CAUTION Investigational Device Limited by Label: CAUTION Investigational Device. Limited by Federal (or U.S.) law to investigational use.

7 VALIDATION FDA and ISO: confirmation by examination and provision of objective evidence that the particular requirements for a specific intended use can be consistently fulfilled" CLIA does not use the term validation but refers to establishment of performance specifications. at the time of assay development prior to reporting patient results. includes determination of accuracy, precision, reportable range, reference interval, analytical sensitivity and analytical lti l specificity. it VERIFICATION FDA and ISO: confirmation through the provision of objective evidence, that specified requirements have been fulfilled CLIA: Verification of performance specifications. Each laboratory that introduces an unmodified FDA cleared or approved test must demonstrate and document that it can obtain performance specifications comparable to those established by the manufacturer for the following performance characteristics: accuracy, precision, reportable range of test t results for the test t system, verify that t reference intervals (normal values) are appropriate for the laboratory s patient population. ongoing quality control and quality assessment

8 Performance Characteristic Reportable Range linearity study (quant assays) Analytical Sensitivity limit of detection study Precision replication experiment Analytical Specificity interference study FDA approved/cleared LDT + + N/A not required by CLIA N/A not required by CLIA + Accuracy (trueness) + + comparison of methods study Reference Interval + + For details: Burd, E.M. Validation of Laboratory Developed Molecular Assays for Infectious Diseases. Clin. Microbiol. Rev. 23:50 576, 2010.

9 To avoid diagnostic use of laboratory tests with unproven performance characteristics Use of such tests may mislead healthcare providers and could cause serious adverse health consequences ces to patients, ts, who are not aware a that they are being diagnosed with tests without established performance characteristics

10 Definition Span of test result values over which the laboratory can establish or verify the accuracy of the instrument or test system measurement response. Does not apply to qualitative tests May only report results within the reportable range Synonyms: y Measuring interval, analytical measurement range(amr), linear range Study suggestions Linearity experiment: series of samples (n=7 11) of known concentration or dilution of known concentration, 2 4 replicates to check for imprecision Sample Sources Reference panels, standard solutions, characterized patient samples Matrix for each specimen type Data analysis measured values compared to assigned values linear regression polynomial regression (preferred)

11 7 concentrations of analyte prepared by dilution of a high concentration standard tested in triplicate Assigned values (converted to log 10 ) plotted on the x axis Measured values (converted to log 10 ) plotted on the yaxis Linear regression y=0.9613x r 2 = Second order polynomial y= 0.028x 0028x x 1937x r 2 = Third order polynomial y= x x x r 2 = Second and third order polynomials not significantly different from linear equation Reportable range; 30 copies/ml through 3,000,000 copies/ml

12 Definition The ability of the assay to detect very low concentrations of target in a specimen Qualitative and quantitative assays limit of detection (LOD) consistently detected (not necessarily quantified) with acceptable precision Study suggestions Statistical: signal distinguished from background limit of blank (plus testing of low level level positive) Empirical: test serial dilutions of known concentration in range of expected detection limit Sample Sources Known concentration: standards, d QC materials, il proficiency i samples, cell lines containing target etc. Data analysis Probit analysis commonly used

13 A series of six samples was prepared by diluting a high concentration standard. Copies/ml Log 10 copies/ml No. of Replicates No. Positive % Positive Probit value 1, NA Each sample was tested eight times NA Probit regression analysis : probit value = 6.65 converts to a C 95 value of Limit of detection is copies/ml samples containing 80 copies/ml would be detected 95% of the time NA NA

14 Definition How well a given measurement can be reproduced when a test is applied repeatedly to a single homogeneous sample. Random analytical error Remember: A measurement may be very precise but not very accurate Study suggestions Replication experiment within run run to run *day to day consider different operators, multiple reagent lots, multiple laboratories Qualitative: 20% below LOD, LOD, 2o% above LOD in replicates up to 40 Quantitative: high, low, LOD in duplicate over 20 days Sample Sources standards, QC materials, proficiency testing samples, patient specimens Data analysis depends d on experimental design standard deviation, coefficient of variation

15 Example: CMV Quantitative i Assay 3 samples spanning the reportable range In duplicate, one run per day over 20 days, 2 operators Overall %CV Less precise at lower concentrations 5 fold change may represent imprecision rather than a true biological difference Conc copies/ml Log 10 conc %CV total SD 95% CI Log 10 viral load 250, , Log Fold change change 95% CI 95%CI

16 Definition Ability of an assay to detect only the intended target Cross reactivity related or interfering nucleic acids organisms with similar genetic structure normal flora organisms that cause similar disease states or clinically relevant co infections Interfering substances specimen related (e.g., hemolyzed, icteric, lipemic) Study suggestions Interference screen: Add potentially interfering substance to specimens containing the analyte of interest analyze specimens with and without Comparative measurement: Evaluate bias: patient specimens containing i the potential il interfering i substance and analyte of interest test using new method and comparator method Data analysis Pi Paired t test, t repeated measures test, t paired difference diff test t Difference between means and controls with allowable error that is clinically significant for the test

17 Definition Closeness of agreement between the average value from a large series of measurements and the true (or accepted) value Study suggestions Comparison of Methods Study specimens tested t by new method and valid comparative method in duplicate is recommended singly OK if close agreement is expected Recovery Study if no comparison method is available specimens with known amount of analyte can also assess proportional error that can occur as concentration of analyte increases Sample Sources residual patient specimens, positive and negative, over range of assay No fewer than 20, typically or more Data analysis scatter diagram regression analysis (slope, y intercept) Bland Altman difference plot

18 Quantitative real time PCR assays new extraction methods A B 72 patient samples tested by old and new extraction methods. 43 specimens had numerical results Linear regression: y=0.9834x R 2 = (good) Slope close to 1.0 no proportional bias Y intercept close to origin (0.0) no constant systematic bias Bland Altman mean bias 0.01 log (no systematic bias) C D 46 positive specimens Linear regression: y=0.9698x R 2 = (good) Slope close to 1.0 no proportional bias Y intercept significantly away from origin (0.0) > constant systematic bias Bland Altman mean bias 0.41 log (about 2.5 fold) not statistically significant since 95% confidence interval contains zero, but? clinically significant

19 Definition Normal range rangeof values typically found din individuals who do not have the disease Study suggestions Qualitative assays: If the nucleic acid target is always absent in a healthy individual normal range is negative or not detected no study cite literature or other pertinent information Quantitative assays: < LOD, LLOQ or clinical decision limit (asymptomatic vs disease). Reference Interval Study: test healthy subjects in intended population (n=120) use results to determine normal range

20 LOD LLOQ Upper Limit of Linearity Analyte concentration Low Medium High Reportable Range (for Quantitative Assays) Analytical Sensitivity (LOD) Precision (qualitative assay) Precision (quantitative assay) Analytical Specificity Accuracy Reference Interval X X X X X X X X X X X X X X X X X X X X X X Comments: Reportable range: 7 11 concentrations across anticipated measuring range; 2 4 replicates on same day Analytical Sensitivity: 8 12 replicates of 4 5 samples at the low concentration end over 5 days. Precision Qualitative: Use concentrations at LOD, 20% above and 20% below. Test in duplicate over 15 days (include data from Analytical sensitivity runs to provide data over 20 days). Precision Quantitative: Use high, low and LOD concentrations. Test in duplicate over 19 days (include data from Reportable Range study as Day 1 to provide data over 20 days).

21

22

23 Table of Contents Subpart K_Quality System for Nonwaived Testing Sec Standard: Control procedures. (a) For each test t system, the laboratory is responsible for having control procedures that t monitor the accuracy and precision of the complete analytic process. (b) (b) The laboratory must establish the number, type, and frequency of testing [[Page 578]] control materials using, if applicable, the performance specifications verified or established by the laboratory as specified in Sec (b)(3). (c) (c) The control procedures must (1) Detect immediate errors that occur due to test system failure, adverse environmental conditions, and operator performance. (2) Monitor over time the accuracy and precision of test performance that may be influenced by changes in test system performance and environmental conditions, and variance in operator performance. (d) Unless CMS approves a procedure, specified in Appendix C of the State Operations Manual (CMS Pub. 7), that provides equivalent quality testing, the laboratory must (1) Perform control procedures as defined in this section unless otherwise specified in the additional specialty and subspecialty requirements at Sec. Sec through (2) For each test system, perform control procedures using the number and frequency specified by the manufacturer or established by the laboratory when they meet or exceed the requirements in paragraph (d)(3) of this section. (3) At least once each day patient specimens are assayed or examined perform the following for (i) Each quantitative procedure, include two control materials of different concentrations; (ii) Each qualitative procedure, include a negative and positive control material; (iii) Test procedures producing graded or titered results, include a negative control material and a control material with graded or titered reactivity, respectively; (iv) Each test system that has an extraction phase, include two control materials, including one that is capable of detecting errors in the extraction process; and (v) Each molecular amplification procedure, include two control materials and, if reaction inhibition is a significant source of false negative results, a control material capable of detecting the inhibition.... (ii) Include at least one control material on each plate lt or card, as applicable, which h must be processed through each step of patient testing, including extraction processes (ii) The laboratory may use the stated value of a commercially assayed control material provided the stated value is for the methodology and instrumentation employed by the laboratory and is verified by the laboratory. (iii) Statistical parameters for unassayed control materials must be established over time by the laboratory through concurrent testing of control materials having previously determined statistical parameters (g) The laboratory must document all control procedures performed. (h) If control materials are not available, the laboratory must have an alternative mechanism to detect immediate errors and monitor test system performance over time. The performance of alternative control procedures must be documented. [68 FR 3703, Jan. 24, 2003; 68 FR 50724, Aug. 22, 2003

24 POSITIVE CONTROL(S) To see if the PCR has worked Obtain commercially, prepare in house, other sources A specific nucleic acid fragment containing tii the entire sequence to be amplified including the primer binding site (e.g., previously amplified and or a cloned DNA fragment that has been sequenced for confirmation Purified total nucleic acid from the organism containing the sequence of interest Patient specimens containing the target nucleic acid Patient specimens spiked with target The whole organism Do not use calibrators (Note: CLIA allows use of calibrators if other control material is not available use different lot number Matrix that matches specimens as closely as possible For each specimen type NEGATIVE CONTROL To detect contamination Blank control (e.g., water or buffer) Specimen containing known nontarget nucleic acid (preferred)

25 QUALITATIVE ASSAYS Positive for each analyte for each specimen type Negative QUANTITATIVE ASSAYS Positive at least 2 controls of different values (generally high and low) Negative Frequency: Daily Control testing is not necessary on days when testing is not performed Multiplex assays: controls for each analyte are included in each run or rotated so that all analytes are tested periodically (MIC.63264)

26 How do you define a low positive control? 1.) Within 1 log of LLOD/LLOQ 2.) CT value between ) Within 1,2, 3 standard deviations of LLOD/LLOQ 4.) 4) Other

27 FDA cleared/approved, /pp not modified Quantitative: 2 levels, run daily Qualitative: a run daily Validation studies: comparison of external and built in controls, 25 sample minimum. External surrogate controls are run for each new lot number or shipment, after major system maintenance and after software upgrades also as recommended by the manufacturer or at least every 30 days, whichever is more frequent.

28 Detect errors in the extraction process Whole bacteria or virus as it would appear in a patient sample (best) Purified nucleic acid can be used Seed into appropriate matrix at low level Run in parallel with patient samples * if the positive extraction control is taken through all steps of * if the positive extraction control is taken through all steps of the assay it can dually serve as an extraction control and an amplification control

29 CLIA requires in assays in which reaction inhibition is a significant source of false negative results Testing for inhibition may be relaxed or discontinued if Sufficient data ( samples) Inhibition rates are found to be within acceptable limits considering the medical implications of a false negative test If added d before sample preparation can also serve as an extraction control Must be detected for a negative result to be considered valid

30 EXTRINSIC INTERNAL CONTROLS Homologous extrinsic controls Unmodified intended target Small amount added to second aliquot of specimen Easy to use Cost and space occupied by a second reaction Modified intended target Can be amplified with same primers as intended target Contain inserts that can be distinguished from the intended target by size ( bp longer) or unique sequence Can be co amplified with intended target in same reaction vessel Heterologous extrinsic controls Non target controls Different primers/probes required INTRINSIC INTERNAL CONTROLS Heterologous intrinsic controls housekeeping genes host genome present in patient specimens in low copy number Different set of primers Same or separate reaction β globin, β actin, gamma interferon, glyceraldehyde 3 phosphate dehydrogenase (GAPDH), etc. Can also be used to establish the presence of cellular material in a samples Concern: cumber of copies may be much higher than target amplification advantage not accurately test for inhibition

31 Minimum frequency at least each day To qualify each shipment or lot to lot change of reagents Each time there is a major preventive maintenance or replacement of a critical part that may influence test performance For assays that contain electronic/procedural/built in controls, are FDA approved and not modified by the laboratory new lot and shipment only

32 Negative controls at the end of the run the last sample to which reagents are added For large runs place evenly throughout the run (e.g., every 50 samples) or distribute randomly For many FDA approved/cleared assays, the position of the controls is established by the manufacturer and cannot be changed

33 Tolerance and acceptability limits must be defined and monitored for all control procedures Stay tuned

34 CLIA definition: the assaying of calibration materials in the same manner as patient specimens to confirm that the calibration of the instrument, kit, or test system has remained stable throughout the laboratory s reportable range for patient test results. According to CAP, to fulfill this requirement requires two processes: 1. calibration verification 2. verification of the analytical measurement range (AMR, reportable range in CLIA terminology)

35 This section of the echecklist stonly yapplies to quantitative tat tests for which appropriate external materials exist. Materials must have assigned values When: At changes of reagent lots, unless the laboratory can demonstrate that the use of different lots does not affect the accuracy of patient/client test results and the range used to report patient/client test data QC fails to meet established criteria After major maintenance or service When recommended by the manufacturer At least every 6 months

36 If standard curve is run with every run First point, slope and intercept statistics f d d i d If standard curve is stored Two points are run with every run statistics

37 **NEW** 06/15/2009 MIC Quantitative Cut Off Phase I For qualitative tests that use a cut off value to distinguish positive from negative, the cut off value is established initially, and verified with every change in lot or at least every 6 months. NOTE: The limit of detection that distinguishes a positive from a negative result should be established or verified when the test is initially placed in service, and verified with every change in lot (e.g. new master mix), instrument maintenance, or at least every six months thereafter. Note that a low positive control that is close to the limit of detection can satisfy this checklist requirement, but must be external to the kit (e.g. weak positive patient sample or reference material prepared in appropriate matrix). Evidence of Compliance: Written procedure for initial establishment and verification of the cut off value AND Records of initial establishment and verification documented at defined frequency

38 **NEW** 06/15/2009 MIC AMR Validation Criteria Phase II Criteria are established for validating the analytical measurement range and compliance is documented. NOTE: If the materials used for calibration or for calibration verification include low, midpoint, and high values that are near the stated AMR, and if calibration verification data are within the laboratory's acceptance criteria, the AMR has been validated; no additional procedures are required. If the calibration and/or calibration verification materials do not include the full AMR, or the laboratory extends the AMR beyond the manufacturer's stated range, g, the AMR must be validated by assaying materials reasonably near the lowest and highest values of the AMR. Evidence of Compliance: Written procedure defining the method, frequency and acceptability criteria for AMR validation