Verification/Validation of Non- FDA Approved Tests

Transcription

1 Verification/Validation of Non- FDA Approved Tests Michael Loeffelholz, Ph.D. ABMM Professor, Dept. Pathology Director, Clinical Microbiology Laboratory University of Texas Medical Branch, Galveston 1

2 Faculty Disclosure The Association of Public Health Laboratories adheres to established standards regarding industry support of continuing education for healthcare professionals. The following disclosures of personal financial relationships with commercial interests within the last 12 months as relative to this presentation have been made by the speaker(s): Michael Loeffelholz: Biofire Diagnostics (advisory board) Pro-Lab Diagnostics (grant funding) 2

3 Outline Definitions Review of regulatory requirements and guidance documents Verification/validation examples 3

4 Verification and Validation Definitions CLIA, CLSI CAP Validation On going process; tests continue to perform as per specs (aka quality assurance) One time process; modified tests and LDTs Verification One time process to demonstrate that test system performs according to specifications; all tests One time process; unmodified tests 4

5 I Will Discuss the One-Time I will not discuss Process Reagent QC New lot/new shipment QC Establishing internal control in lieu of external controls New instrument qualification 5

6 What are Non-FDA Approved Tests? Modifications (off label use) of FDAcleared/approved tests (ModTs) Laboratory-developed tests (LDTs) 6

7 Test Modification Examples* Pre-analytical Change in specimen handling instructions Specimen type or source Analytical Change or elimination of procedural step Change in cutoff Different calibration material Post-analytical Qualitative results reported as quantitative 7 *Not a complete list of test modifications.

8 CLIA and CAP Essentially Do Not Distinguish Between ModT and LDT 8

9 CLIA ModTs and LDTs (b)(2) Each laboratory that modifies an FDAcleared or approved test system, or introduces a test system not subject to FDA clearance or approval (including methods developed in-house and standardized methods such as text book procedures) must, before reporting patient results, establish performance specifications 9

10 Establish versus Verify CLIA regulations refer to establishment of performance specifications, for ModT or LDTs Breaking new ground. No extensive clinical trial data Could argue that standardized methods (e.g. bacterial or viral culture) have extensive data Going outside of package insert Essentially creating own package insert 10

11 CLIA ModTs and LDTs (b)(2) The establishment of method performance specifications should provide evidence that the accuracy, precision, analytical sensitivity, and analytical specificity of the procedure is adequate to meet the clients needs as determined by the laboratory director... Accuracy and precision studies are required of FDAcleared tests Analytical sensitivity and analytical specificity are additional elements, required for ModT and LDT 11

12 CAP All Common Checklist COM Body fluid validation Testing specimens using methods intended for other specimen types validated by the laboratory for accuracy, precision, analytical sensitivity, analytical interferences, and reportable range. COM LDT reporting Reports contain a description of the method, and statement that the assay was developed by the laboratory COM LDT clinical claims validation All clinical claims made about (LDT) are validated. COM Analytic methodology changes If the laboratory changes methodology so that test results or their interpretations may be SIGNIFICANTLY different, the change is explained to clients. 12

13 CAP Microbiology Checklist Standards related to verification/validation are limited to Molecular subsection MIC.64770; sample types or uses collection devices other than those listed in package insert, the laboratory performs validation studies MIC new cut-off value has been validated. MIC modifies an FDA-cleared/approved assay, the modified procedure has been validated equivalent or superior performance. MIC validation adequate number samples for each type of specimen MIC LDT report. contain description of method, statement that assay was developed by the laboratory MIC ASR report. report includes disclaimer required by federal regulations. 13

14 CAP Microbiology Checklist Molecular subsection is not well outlined, to clearly distinguish requirements for unmodified tests versus ModTs and LDTs Assay validation section (MIC ) refers to analytical and diagnostic sensitivity and specificity, precision, linearity, reportable and reference ranges. However, All Common checklist: Analytical sensitivity (COM.40400). may consist of data from manufacturer Analytical specificity (COM.40450). For modified or LDTs. Result is confusion over what the laboratory must do to validate a new test 14

15 15

16 Cumitech 31A ModTs and LDTs accuracy (diagnostic tests) should be verified minimum of 50 (+) specimens 100 (-) specimens identification tests minimum of 200 isolates very small modifications may not require the extensive testing Cumitech 31A, Verification and Validation of Procedures in the Clinical Microbiology Laboratory ASM Press 16

17 So What Must I Do? Unmodified, FDA-cleared/approved tests Accuracy (i.e. diagnostic sensitivity and specificity) Reproducibility/precision Reportable range Reference range Others required for test performance (e.g. linearity for quantitative tests) ModTs and LDTs Above, plus Analytical sensitivity Analytical specificity/interfering substances 17

18 How Should I Do It? Accuracy Specimen panel with expected results From another method in your lab Specimens provided by another lab Tests should have expected concordance Reproducibility/precision Repeat testing (within run, between run, between operators) gives consistent results Reportable range Test both weak and strong positive specimens (in panel above) 18

19 How Should I Do It? Reference range Determine cutoff such that specimens from healthy/normal subjects are negative Analytical sensitivity Limit of detection Analytical specificity/interfering substances Factors that can give false positive results (other organisms, specimen matrix components) False negative results (specimen components) 19

20 Examples of ModT/LDT Validation/Verification 20

21 #1. Change from Culturette swabs to flocked swab for bacterial cultures Is validation/verification required? 21

22 #1. Swab Change for Bacterial Culture..regulatory agencies require the validation of newly incorporated test systems, the specimen collection swab, which is not a component of the test system, is neither a test nor a system. It is in fact more like a scalpel or urine cup, neither of which requires validation before use. Swabs for general specimen collection. CAP and CLIA laboratory surveyors have no grounds to suggest that validation of swabs is necessary. 22 Source: Miller, Campbell, Loeffelholz Changing swabs: to validate or not to validate? J Clin Micro. 51(11):3910

23 #2. Rapid virus antigen detection test (FDA-cleared). Kit provides specified dacron swabs. Change to flocked swab Is validation/verification required? 23

24 #2. Swab Change for FDA-Cleared Test a modification to the FDA clearance and would require a validation to ensure that (the test using) the new collection swab works at least as well as the swab recommended by the manufacturer. You are validating the performance of the virus antigen test (with this modification), NOT the swab 24 Source: Miller, Campbell, Loeffelholz Changing swabs: to validate or not to validate? J Clin Micro. 51(11):3910

25 #2. Swab Change for FDA-Cleared Test Suggested validation approach Parallel comparison of specimens collected with FDA-cleared swabs and flocked swab. Can include contrived specimens (accuracy) As this is a minor change (same specimen source), smaller number of specimens may be adequate Test several specimens in duplicate, and with different operators to demonstrate reproducibility Include both weak and strong positives in panel, to verify reportable range 25

26 #2. Swab Change for FDA-Cleared Test Again, this is a minor change (no change in sample source, patient population, or analytical changes) Reference range, analytical sensitivity and specificity unlikely to be affected 26

27 #3. Swab Change for Fungal Culture Flocked swabs (Eswab; Copan) are FDA-clear for aerobes, anaerobes and fastidious bacteria (package insert) Use for recovery of fungi would be outside package insert claims (modification) However, swabs aren t a test system. Regulations re. validation refer to test system Relevant issue is survival of fungi in liquid transport system. Science dictates that validation/verification of this characteristic be performed. Spike several yeast, molds into liquid transport medium Hold at conditions that simulate transport (time, temperature) Perform quantitative culture (T0, T24 h, T48 h) 27

28 #4. Specimen Change for FDA- Cleared Test A more rigorous validation than a swab change Test characteristics (esp. diagnostic accuracy, analytical sens) are unknown and must be established Assemble panel of positive and negative specimens 28

29 #4. Specimen Change for FDA- Cleared Test Validation approach Number of specimens Twenty is probably not sufficient Up to 50+ves and 100-ves Local regulatory agencies may specify # Appropriate matrix and source of positives Actual patient specimens Spiked (e.g. create rectal swabs; spike with Neisseria gonorrhoeae from culture plate or external QC material) Aliquot of specimens from lab that has validated specimen source 29

30 #4. Specimen Change for FDA- Cleared Test Validation approach Accuracy Patient specimens- second, reference method Spiked specimens- expected result Shared specimens- versus other lab Reportable range Include specimens with both low and high levels of analyte Reference (normal) range Negative patients, or non-spiked samples should test negative 30

31 #4. Specimen Change for FDA- Cleared Test Validation approach Reproducibility/precision Repeat several positive specimens (inter-, intra-run, inter-operator) Analytical sensitivity Test several replicates of different amounts of analyte Analytical specificity/interference If applicable, compare assay internal control (e.g. molecular test), new vs. FDA-approved specimen Review manufacturer s package insert data for relevance to evaluated sample type Spiking study if necessary (e.g. blood) 31

32 #5. Change in Specimen Processing Goal- validate unextracted or crude specimens in place of nucleic acid extraction procedure Saves time, $ Risk: decreased sensitivity (diagnostic and analytical); specificity (assay interference from crude specimens); reproducibility 32

33 #5. Change in Specimen Processing Validation approach Same specimens as FDA-cleared assay Accuracy Assemble panel of specimens previously tested by unmodified assay (extracted specimens) Reportable range Include both strong and weak +ves in above panel Reference (normal) range -ves in panel above 33

34 #5. Change in Specimen Processing Reproducibility/precision Retain several positive specimens for rpt testing Analytical sensitivity Serial dilutions of several positives Spike with different amounts of analyte Analytical specificity/interference If applicable, compare assay internal control (e.g. molecular test), new vs. FDA-approved specimen Spiking study if necessary (e.g. blood) 34

35 #6. Validation of LDT Real-time PCR for virus X Pre-validation Literature review Identify specimen types Determine intended use. This is important! (Screening? Diagnostic? Symptomatic? Asymptomatic?) Select primers, probes. Identify vendor [may be analyte specific reagents (ASRs)] Sample processing and PCR conditions optimized and determined 35

36 #6. Validation of LDT Rigorous validation is required establishing performance characteristics (for your laboratory) Assemble panel of specimens Up to 50+ves and 100-ves From your laboratory (accuracy vs. another method) Can be spiked (accuracy vs. expected result) From another laboratory (lab to lab accuracy) 36

37 #6. Validation of LDT Reference and reportable range Ensure that weak positives are reliably detected May need to adjust threshold (C T value) for cutoff Reproducibility As for modified tests Analytical sensitivity Test several replicates of different amounts of analyte Analytical specificity Monitor PCR internal control Test relevant microorganisms Likely found in specimen source Genetically related viruses Test substances that could interfere (e.g. blood) 37

38 Acknowledgements Richard Clark Michael Lewinski Bob Tibbets Susie Sharp Alice Weissfeld 38

39 39