Clinical Microbiology Principles for validation & quality control Clinical Microbiology Mission Identify etiologic agents/microbes in clinical samples Provide evaluation/interpretation of lab results to the clinician, infection control dept etc Anders Bergqvist, Uppsala University Hospital 2 Important tasks for clinical lab Scientfiic quality of analysis (e.g. accuracy) Communication with clinician Time Cost Qualitative vs Quantitative tests Qualitative Yes or No response Specify microbe Quantitative Numerical response Specify and quantify microbe 3 4 Validation Verification The documented act of demonstrating that a procedure or process will consistently lead to the epected results. Refers to meeting the needs of the intended end-user or customer. The act of reviewing, inspecting or testing, in order to establish and document that a product, service or system meets regulatory standards Clinical lab use should verify that approved test works as epected in that specific contet 5 6
Government-approved assays In vitro diagnostic devices (IVDs) intended for use in diagnosis of disease or in cure, treatment or prevention of disease in man Commercially distributed for in vitro diagnostic use Lab should verify that peformance specifications can be reproduced Accuracy Precision Reportable range Lab-developed (in-house or home-brew) assays Not approved by authorities Lab must establish performance characteristics Full validation required including Accuracy Precision Reportable range Analytical sensitivity Analytical specificity 7 8 Lab sample Lab vs Clinical sample E.g. cultured microbe, plasmid, synthesized oligonucleotide, in-vitro tr RNA. More defined (pure, known genotype etc) Clinical sample Sample from patient Less defined (dirty, less known genotype etc) The real thing Analytical sensitivity The ability to detect very low concentration in a biological specimen The smallest amount in a sample that can accurately be measured Often referred as Limit of detection (LOD) 9 10 Diagnostic sensitivity Reportable range The proportion of individuals with a given disorder that can be identified (more later...) The span where accuracy can be verified Only used in quantitative tests 11 12
Precision (reproducibility) Accuracy/trueness The agreement of obtained value and true value or accepted reference value. Epressed numerically as bias/lack of agreement Evaluated by comparison with reference method or proficiency testing samples How well a measurement of single sample can be reproduced 13 14 Trueness method comparison Method comparison eample 2 Scatter (y) plot Method in evaluation Scatter (y) plot Bland-Altman bias plot Reference method 15 16 Precision vs Accuracy Limit of detection High Accuracy Low High Precision Low Precision and accuracy are independent of each other The lowest concentration that can be detected (>95 % of samples) with acceptable precision 17 18
Limit of detection Limit of detection - probit regression analysis The lowest concentration that can be detected (>95 % of samples) with acceptable precision Testing muliplicates of serial dilutions of target in range of epected detection limit Probit analysis enables linear regression 19 20 Analytical sensitivity The ability to detect very low concentration in a biological specimen The smallest amount in a sample that can accurately be measured Often referred as Limit of detection (LOD) Diagnostic sensitivity The proportion of individuals with a given disorder that can be identified. Gives information about frequency of false negatives in a clinical setting. 21 22 Analytical specificity Diagnostic specificity The ability to measure one particular organism, rather than others, in a sample. Lack of cross-reactivity (e.g. genetically similar organisms) The proportion of individuals who do not have a given disorder and are identified as negative. Gives information about frequency of false positives in clinical setting 23 24
Determination of diagnostic sensitivity & specificity Determination of diagnostic sensitivity & specificity Sample Gold standard Method M 1 + + 2 - + 3 - - 4 - - 5 + + 6 - - Method A Sample Gold standard Method M 1 + + 2 - + 3 - - 4 - - 5 + + 6 - - + - Total Gold standard + - Total Sensitivity: True positive results/(true positive results+false negative results) Specificity: True negative results/(true negative results+false positive results) 25 26 Determination of diagnostic sensitivity & specificity Positive predictive value Method A Sample Gold standard Method M 1 + + 2 - + 3 - - 4 - - 5 + + 6 - - Gold standard + - Total + 2 1 3-0 3 3 Total 2 4 The probability that a person whose test result is positive truly has the disease of interest Sensitivity: True positive results/(true positive results+false negative results) = 2/(2+0) = 2/2 = 100 % Specificity: True negative results/(true negative results+false positive results) = 3/(3+1) = 4/4 = 75 % 27 28 Negative predictive value The probability that a person whose test result is negative does not have the disease of interest Effects of prevalence and specificity on positive predictive value Large effects of suboptimal specificity on PPV at low prevalence 29 30
Effects of prevalence and specificity on positive predictive value What decides the predicitive value? Strong diagnostic sensitivity improves negative predictive values, and strong diagnostic specificity improves positive predictive values Assays that have very high analytical sensitivity and specificity but have low diagnostic sensitivity and specificity have a poor diagnostic predictive value. Predictive value is a product of diagnostic performance and prevalence of microbe in the population Large effects of suboptimal specificity on PPV at low prevalence 31 32 Realtime PCR Quantitation over long range Standard In house standards Internationally recognised standards (WHO etc) Commercial standards Determination of cycle treshold (Ct) instead of final amount Reportable range: >10 7 33 34 Frequently used quantitation standards Plasmid, linear PCR product Pooled cdna Genomic DNA Commercial RNA In Vitro Trans RNA None Plasmid, not linear Other Bacteria Plasma Plasmid + Std How is the standard itself quantitated? Absolute: Physical/chemical Spectrophotometry Mass spec Atomic abs Relatative: Calibration against accepted standard Form of NA is critical for correct quantitation 35 36
Future perspectives Large centralised labs Cost efficient Standardised methods Increased capacity Infection-control surveillance Local facilities, point-of-care Better communication Short transport Quick Future perspectives two opposite dierections Large centralised labs Cost efficient Standardised methods Increased capacity Infection-control surveillance Local facilities, point-of-care Better communication Short transport Quick 37 38 Suggested reading Burd, EM., Validation of laboratory-developed molecular assays for infectious diseases, Clin. Micro. Rev. 23:550-576 qpcr guide, Booklet from Eurogentec Regarding what we know and not......as we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns the ones we don't know we don't know. Donald H. Rumsfeld, 2002 39