Sigma Metrics in the Clinical Laboratory: From Theory to Practice D R. D A N A B A I L E Y U N I V E R S I T Y O F T O R O N T O G A M M A - D Y N A C A R E
Quality indicators are everywhere 1
Objectives To convince you that sigma-metrics provides an intuitive, encompassing, snap-shot of method performance suitable for use in quality management planning To convince you that sigma-metrics are not that difficult 2
Setting the scene: What we will cover Sigma metrics as a quality indicator Theory: The use of Sigma metrics for quantitative assays Examples from the Clinical Chemistry laboratory: How to calculate Sigma using laboratory data What you can do with your Sigma values Generating a laboratory scorecard Revamping QC procedures 3
Sigma metrics as a quality indicator of qualitative assays
In the business world: Six Sigma is a management strategy that seeks to improve the quality of process outputs by identifying and removing the causes of defects (errors) and minimizing variability in manufacturing and business processes. What is Six Sigma? 5
In the laboratory: Six Sigma is a quality management strategy that seeks to improve assay quality by identifying biased and/or imprecise assays so that appropriate quality monitoring strategies can be used and assay performance can be addressed What is Six Sigma? 6
Sigma metrics as a quality indicator 7 1σ 2σ 3σ 4σ 5σ 6σ 691,462 308,538 66,807 6210 2833 3.4 DPM
Sigma metrics as a quality indicator 8 0 1 2 3 4 5 6 7 Airline Safety Baggage handling Departure Delays Hospital fatal errors HAI Pre-analytical sample Hemolyzed specimens Landrigan, NEJM 2010; Llopis, CCLM 2011
The application of sigma metrics to quantitative assays
Defining sigma 10
Defining sigma 11
Defining sigma 12
Defining sigma 13
Defining sigma 14
Defining sigma 15 3σ: Clinically unacceptable results are produced 67,000 times in a million (6.7%)!
Defining sigma 16 6σ: Clinically unacceptable results are only produced 3.4 times in a million (0.00034%)!
The advantage of Sigma Metrics 17 Sigma metrics places analytical characteristics within the framework of clinical requirements Defect rate Sigma values can be calculated for both qualitative and quantitative assays The Sigma scale is easily interpreted and appreciated by laboratorians, management, and laymen The Sigma scale provides guidelines for assay improvement and monitoring
Applying sigma concepts to clinical laboratory data E X A M P L E S F R O M T H E C L I N I C A L C H E M I S T R Y L A B O R A T O R Y
Applying sigma metrics to the clinical laboratory 19 1. Define acceptability criteria: Total allowable error 2. For qualitative assays, count violations and assign sigma value 3. For quantitative assays a) Determine imprecision b) Determine bias c) Calculate sigma value 4. Adjust QC protocol according to sigma value 5. Address poor assay performance 6. Continue to monitor sigma values
What impact does TAE have on my sigma metric? 20
What impact does TAE have on my sigma metric? 21
What impact does TAE have on my sigma metric? 22
What impact does TAE have on my sigma metric? 23
Strategies for selecting TAE 24 Rank Strategy Subclasses 1 2 Impact on specific clinical decision making Impact on general clinical decision making Quality specifications in specific clinical situations A. Based on biological variation B. Based on medical opinions 3 Professional recommendations 4 EQAS recommendations A. International expert groups B. Expert individuals A. Regulatory B. EQAS 5 Published data on state of the art A. Published by PT/EQAS B. Published individual method Stockholm Consensus
Strategies for selecting TAE 25 Rank Strategy Subclasses 1 2 Impact on specific clinical decision making Impact on general clinical decision making Quality specifications in specific clinical situations A. Based on biological variation B. Based on medical opinions 3 Professional recommendations 4 EQAS recommendations A. International expert groups B. Expert individuals A. Regulatory B. EQAS 5 Published data on state of the art A. Published by PT/EQAS B. Published individual method http://www.westgard.com/biodatabase1.htm
Strategies for selecting TAE 26 Rank Strategy Subclasses 1 2 Impact on specific clinical decision making Impact on general clinical decision making Quality specifications in specific clinical situations A. Based on biological variation B. Based on medical opinions 3 Professional recommendations 4 EQAS recommendations A. International expert groups B. Expert individuals A. Regulatory B. EQAS 5 Published data on state of the art A. Published by PT/EQAS B. Published individual method
How to determine assay imprecision 27
How to determine assay bias Compile data from PT/EQA surveys (e.g. 1 year) Compute the % bias %bias = ((lab value reference value)/ref x 100%) 28 Calculate average % bias for the 3 (or more) samples with concentrations closest to the QC material
How to determine assay bias 29
How to calculate Sigma and score your assays Sigma = (TAE - bias)/cv Strive for 6 sigma >4 sigma is ideal <3 sigma cannot be controlled with statistical QC protocols 30
Graphical representation of Sigma metrics 31
What to do with your Sigma values
Adjusting QC practices according to Sigma values 33 Reduce QC rules on six-sigma assays Schoenmakers, Clin Chem Lab Med, 2011
Reducing QC frequency on Six Sigma assays 34
Strategy to reduce QC frequency Step 1: Determine QC rule(s) based on sigma metric Step 2: Overlay new QC practice on old QC practice Monitor (3 months) Number false-alarms, true alarms Step 3: Determine effectiveness of change-inpractice & implement 35
Work toward assay improvement 36
Grade platforms prior to purchase 37
Summary 38 Sigma metrics places the analytical characteristics of assays on the same scale Requires definition of a defect Sigma value is inherently dependent on TAE definition Sigma metrics are intuitive and easy to interpret Sigma metrics provides a benchmark to select QC protocols and target assay improvement σσσσσσ 4 sigma Room for improvement
If it matters measure it. Q U E S T I O N S & C O M M E N T S
Using sigma metrics to address quality concerns 41
42
43
Sigma metrics as a quality indicator 44