Sigma Metrics in the Clinical Laboratory: From Theory to Practice

Size: px

Start display at page:

Download "Sigma Metrics in the Clinical Laboratory: From Theory to Practice"

Rebecca Lawrence
10 years ago
Views:

1 Sigma Metrics in the Clinical Laboratory: From Theory to Practice D R. D A N A B A I L E Y U N I V E R S I T Y O F T O R O N T O G A M M A - D Y N A C A R E

2 Quality indicators are everywhere 1

3 Objectives To convince you that sigma-metrics provides an intuitive, encompassing, snap-shot of method performance suitable for use in quality management planning To convince you that sigma-metrics are not that difficult 2

performance suitable for use in quality management

4 Setting the scene: What we will cover Sigma metrics as a quality indicator Theory: The use of Sigma metrics for quantitative assays Examples from the Clinical Chemistry laboratory: How to calculate Sigma using laboratory data What you can do with your Sigma values Generating a laboratory scorecard Revamping QC procedures 3

Clinical Chemistry laboratory: How to calculate Sigma using laboratory data What

5 Sigma metrics as a quality indicator of qualitative assays

6 In the business world: Six Sigma is a management strategy that seeks to improve the quality of process outputs by identifying and removing the causes of defects (errors) and minimizing variability in manufacturing and business processes. What is Six Sigma? 5

and removing the causes of defects (errors) and minimizing

7 In the laboratory: Six Sigma is a quality management strategy that seeks to improve assay quality by identifying biased and/or imprecise assays so that appropriate quality monitoring strategies can be used and assay performance can be addressed What is Six Sigma? 6

imprecise assays so that appropriate quality monitoring strategies

8 Sigma metrics as a quality indicator 7 1σ 2σ 3σ 4σ 5σ 6σ 691, ,538 66, DPM

9 Sigma metrics as a quality indicator Airline Safety Baggage handling Departure Delays Hospital fatal errors HAI Pre-analytical sample Hemolyzed specimens Landrigan, NEJM 2010; Llopis, CCLM 2011

10 The application of sigma metrics to quantitative assays

11 Defining sigma 10

12 Defining sigma 11

13 Defining sigma 12

14 Defining sigma 13

15 Defining sigma 14

16 Defining sigma 15 3σ: Clinically unacceptable results are produced 67,000 times in a million (6.7%)!

17 Defining sigma 16 6σ: Clinically unacceptable results are only produced 3.4 times in a million ( %)!

18 The advantage of Sigma Metrics 17 Sigma metrics places analytical characteristics within the framework of clinical requirements Defect rate Sigma values can be calculated for both qualitative and quantitative assays The Sigma scale is easily interpreted and appreciated by laboratorians, management, and laymen The Sigma scale provides guidelines for assay improvement and monitoring

qualitative and quantitative assays The Sigma scale is easily interpreted and appreciated by

19 Applying sigma concepts to clinical laboratory data E X A M P L E S F R O M T H E C L I N I C A L C H E M I S T R Y L A B O R A T O R Y

20 Applying sigma metrics to the clinical laboratory Define acceptability criteria: Total allowable error 2. For qualitative assays, count violations and assign sigma value 3. For quantitative assays a) Determine imprecision b) Determine bias c) Calculate sigma value 4. Adjust QC protocol according to sigma value 5. Address poor assay performance 6. Continue to monitor sigma values

For qualitative assays, count violations and assign sigma value 3.

21 What impact does TAE have on my sigma metric? 20

22 What impact does TAE have on my sigma metric? 21

23 What impact does TAE have on my sigma metric? 22

24 What impact does TAE have on my sigma metric? 23

25 Strategies for selecting TAE 24 Rank Strategy Subclasses 1 2 Impact on specific clinical decision making Impact on general clinical decision making Quality specifications in specific clinical situations A. Based on biological variation B. Based on medical opinions 3 Professional recommendations 4 EQAS recommendations A. International expert groups B. Expert individuals A. Regulatory B. EQAS 5 Published data on state of the art A. Published by PT/EQAS B. Published individual method Stockholm Consensus

26 Strategies for selecting TAE 25 Rank Strategy Subclasses 1 2 Impact on specific clinical decision making Impact on general clinical decision making Quality specifications in specific clinical situations A. Based on biological variation B. Based on medical opinions 3 Professional recommendations 4 EQAS recommendations A. International expert groups B. Expert individuals A. Regulatory B. EQAS 5 Published data on state of the art A. Published by PT/EQAS B. Published individual method

27 Strategies for selecting TAE 26 Rank Strategy Subclasses 1 2 Impact on specific clinical decision making Impact on general clinical decision making Quality specifications in specific clinical situations A. Based on biological variation B. Based on medical opinions 3 Professional recommendations 4 EQAS recommendations A. International expert groups B. Expert individuals A. Regulatory B. EQAS 5 Published data on state of the art A. Published by PT/EQAS B. Published individual method

28 How to determine assay imprecision 27

29 How to determine assay bias Compile data from PT/EQA surveys (e.g. 1 year) Compute the % bias %bias = ((lab value reference value)/ref x 100%) 28 Calculate average % bias for the 3 (or more) samples with concentrations closest to the QC material

30 How to determine assay bias 29

31 How to calculate Sigma and score your assays Sigma = (TAE - bias)/cv Strive for 6 sigma >4 sigma is ideal <3 sigma cannot be controlled with statistical QC protocols 30

32 Graphical representation of Sigma metrics 31

33 What to do with your Sigma values

34 Adjusting QC practices according to Sigma values 33 Reduce QC rules on six-sigma assays Schoenmakers, Clin Chem Lab Med, 2011

35 Reducing QC frequency on Six Sigma assays 34

36 Strategy to reduce QC frequency Step 1: Determine QC rule(s) based on sigma metric Step 2: Overlay new QC practice on old QC practice Monitor (3 months) Number false-alarms, true alarms Step 3: Determine effectiveness of change-inpractice & implement 35

37 Work toward assay improvement 36

38 Grade platforms prior to purchase 37

39 Summary 38 Sigma metrics places the analytical characteristics of assays on the same scale Requires definition of a defect Sigma value is inherently dependent on TAE definition Sigma metrics are intuitive and easy to interpret Sigma metrics provides a benchmark to select QC protocols and target assay improvement σσσσσσ 4 sigma Room for improvement

40 If it matters measure it. Q U E S T I O N S & C O M M E N T S

42 Using sigma metrics to address quality concerns 41

43 42

44 43

45 Sigma metrics as a quality indicator 44

Six Sigma Metric Analysis for Analytical Testing Processes

Six Sigma Metric Analysis for Analytical Testing Processes Sten Westgard, MS, Westgard QC Introduction Laboratories seek objective assessment and comparison of analytical methods and instrumentation performance.