COMPARABILITY OF METHODS AND ANALYSERS Nora Nikolac

Size: px

Start display at page:

Download "COMPARABILITY OF METHODS AND ANALYSERS Nora Nikolac"

Dennis Griffith
7 years ago
Views:

1 COMPARABILITY OF METHODS AND ANALYSERS Nora Nikolac Zagreb, October 24-25, th EFLM Continuing Postgraduate Course in Clinical Chemistry and Laboratory

2 When? 2 Introducing new method or analyzer Multiple analytical systems in laboratory Using services of another laboratory

3 Why? 3 To increase patient safety To assure that method change is not going to influence laboratory result for the patient.

4 How? 4 Experimental procedures following protocols CLSI EP09-A3: Measurement procedure comparison and bias estimation using patient samples 1. Number of samples 2. Measurement range 3. Time of analysis 4. Data analysis 5. Data interpretation

5 1. Number of samples 5 Min: 40 samples Optimal: 100 samples To identify unexpected errors from sample matrix or interferences Measurements in duplicate

6 2. Measurement range 6 Cover 90% of the method measurement range Method B Good agreement between methods Difference in higher concentration range Method A Measurement range

7 2. Measurement range 7 Overlaping measurement range for both methods Method A Method A determined using dilution protocol Method B reported as LOQ Method B Glucose concentration

8 3. Time of analysis 8 Measurements done within 2 hours Not for: glucose, lactate, ammonia, blood gass testing... Measurements done over 5 days Better over longer period of time Collecting samples over period of time (first method) and analyzing in batch using second method

9 4. Analyzing results 9 Several statistical aproaches: Correlation Paired test for difference Linear regression Deming regression Passing-Bablok regresion Bland-Altman analysis

10 4. Analyzing results 10 Comparison of two methods for direct bilirubin concentration measurement Analyzer, method Summary data Method 1 N=40 Architect (Abbott) Diazo method Method 2 N=40 AU 680 (Beckman Coulter) DPD method Min-Max Mean ± SD 65.5 ± ± 83.6 Median (IQR) 38.5 ( ) 42.4 ( ) P (normality)

11 4.1 Correlation 11 Spearman coefficient of correlation r (95% CI) = 0.97 ( ) Excellant correlation

12 What is the meaning of this result? 12 Methods are significantly associated Linear relation between methods of Method A associated with of Method B Nothing about amount of increase! Same correlation coefficient!

13 4.2 Significance of difference 13 Wilcoxon test (normality failed) P < Significant difference between methods

14 What is the meaning of this result? 14 Calculating differences for each pair of measurement Comparing number of negative and positive differences If there is no difference between methods, number of differences is equal More measurements were higher using Method 2

4.3 Linear regression 15 High correlation Linear relationship Equation to describe relationship between methods Determine proportional and constant

15 4.3 Linear regression 15 High correlation Linear relationship Equation to describe relationship between methods Determine proportional and constant error Deming regression Passing and Bablok regression Bilić-Zulle L. Comparison of methods: Passing and Bablok regression. Biochem Med (Zagreb) 2011;21:49-52.

16 Linear regression 16 Method B y y = x 95% confidence intervals α tg (α) = b Intercept = a Regression equation y = a + bx Regression equation y = a (95% CI) + b (95% CI) x Method A x

17 Constant and proportional error 17 y Method B Regression equation y = a (95% CI) + b (95% CI) x Excluding 0 Constant error y = x Excluding 1 Proportional error α tg (α) = b Intercept = a Method A x

18 Deming regression 18 Includes analytical variability of both methods (CV) Assumes that errors are independent and normally distributed Both methods prone to errors y = 1.74 (-1.77 to 5.24) (1.16 to 1.30) x No constant error Proportional error

19 Passing-Bablok regression 19 Non-parametric method No assumptions about distributions of samples No assumptions about distributions of errors Not sensitive to outliers

20 Why don t we recalculate results? 20 Direct bilirubin (Method 2) = 1.23 x Direct bilirubin (Method 1) Direct bilirubin (Method 1) = Direct bilirubin (Method 2) / 1.23

21 Residual analysis 21 How well data fit to the regression model > 0 > 0 0 x 0 x < 0 Y F(x) F(X) < 0 Y F(X) y y

22 Residual analysis 22 Differences between measured and calculated values

23 4.3 Bland-Altman analysis 23 Graphical method to compare two measurements technique Analyzing differences between measurement pairs Giavarina D. Understanding Bland Altman analysis. Biochem Med (Zagreb) 2015;25(2):

24 Mountain plot 24 More positive differences Normal distribution of differences

25 method 2 method Bland-Altman analysis 25 Plotting differences against: Mean of two methods (no reference method) One method (reference method) s 95% CI 95% CI s 0 Limits of agreement Mean difference Limits of agreement Mean of method 1 and method 2

26 method 2 method 1 LoA and mean difference 26 Absolute units Constant bias Wide LoA = poor agreement s 95% CI 95% CI s 0 Limits of agreement Mean difference Limits of agreement Including 0 = no constant bias Excluding 0 = Constant bias Percentage (%) Proportional bias Mean of method 1 and method 2 Narrow LoA = good agreement

27 Bland-Altman analysis 27 Plotting against mean difference No constant bias Plotting against % difference Proportional bias

28 5. Data interpretation 28 Statistical significance Clinical significance Comparing values with predefined acceptance criteia

Method comparison 29 Important laboratory procedure for verification Included into validation protocols for new reagents Comparison with the

29 Method comparison 29 Important laboratory procedure for verification Included into validation protocols for new reagents Comparison with the reference method Comparison with different manufacturers Comparison with same manufacturer Results are presented in manufacturers declarations

30 Can we rely on manufacturers declarations? 30 Correlation for Comparing 7 insert sheets for glucose concentration measurement determination of agreement Manufacturer N Unit r Intercept (95% CI) Slope (95% CI) A 102 mg/dl (?-?) 1.06 (?-?) B 117 mmol/l (?-?) (?-?) C 75 mmol/l (?-?) (?-?) D 43 mg/dl (?-?) (?-?) E? mg/dl (?-?) 0.99 (?-?) F 40 mg/dl (?-?) 0.98 (?-?) No 95% CI for evaluation of bias No BA analysis G 60 mmol/l (?-?) (?-?)

31 To conclude 31 Clinically relevant criteria Interpretation of results Laboratory methods Linear regression analysis Bland-Altman plot Data analysis Verification procedure Number of samples Measurement range Time of analysis Data analysis Data interpretation

32 Take a home massage Comparability of methods and analyzers Coefficient of correlation doesn t allow conclusions about comparability of methods, but only about linear association between them, even when it is very high (close to 1) Regression equation: Y = 0.67 ( ) ( ) x is an example of proportional bias between methods (95% CI for slope not including 1) without constant bias between methods (95% CI for intercept including 0) Regression equation for glucose concentration: Y = 0.07 ( ) ( ) x (mmol/l) is an example of statistically significant, but clinically non-significant constant bias. Value of 0.07 ( ) mmol/l glucose is lower than conventional analytical performance of the test

USING CLSI GUIDELINES TO PERFORM METHOD EVALUATION STUDIES IN YOUR LABORATORY

USING CLSI GUIDELINES TO PERFORM METHOD EVALUATION STUDIES IN YOUR LABORATORY Breakout Session 3B Tuesday, May 1 8:30 10 am James Blackwood, MS, CLSI David D. Koch, PhD, FACB, DABCC, Pathology & Laboratory