Proficiency Testing with FAPAS. Understanding PT Statistics Ken Mathieson Senior Proficiency Analyst

Transcription

1 Proficiency Testing with FAPAS Understanding PT Statistics Ken Mathieson Senior Proficiency Analyst

2 Statistics?!

3 Where you want to go? If you don t know where you are going, you won t get there! The International Harmonised Protocol for the Proficiency Testing of Analytical Chemistry Laboratories [1] The point of a PT is to give a performance assessment to its participants Against what standard will such a performance assessment be made?

4 Fitness-for-Purpose Fitness-for-purpose is at the heart of the statistical model used by FAPAS Definition: a simple expression that takes lots of words, much arm waving and at least one diagram to explain but once grasped, a very succinct way of conveying the concept of fitness-for-purpose

5 Making Data Usable All analyses are variable, you never get the same answer twice The end use of the data should dictate the limits of acceptable variability In simple terms the more effort you put in to the analysis, the lower the uncertainty of the final answer In contrast, the greater the effort, the greater the cost (time = money!)

6 Fitness-for-Purpose Illustrated Uncertainty Time Effort Expended

7 Fitness-for Purpose Quantified Fitness-for-purpose represents reasonable uncertainty a tolerance on a result that is small enough to make the data meaningful and useful To put it another way, it represents a range / spread of possible results. Statistically, a standard deviation, σ p the standard deviation for proficiency assessment

8 Std. Deviation for Proficiency Assessment FAPAS sets σ p using data external to the observed performance i.e. σ p is prescriptive NOT descriptive Not all PT schemes do this

9 Sources of σ p The inter-laboratory variance (Reproducibility) from a method validation study is an indicator of best practice sd R, RSD R, R Predictive models e.g. modified Horwitz equation [2] Expert judgement of what is fit-for-purpose

10 Horwitz (original) ppth % ppm % 1 ppm Original Horwitz Equation = 0.02 c σ RSD, % ppb 200 ppb 150 ppb 120 ppb 100 ppb 50 ppb 20 ppb 10 ppb 1 ppb concentration

11 Modification Below 120ppb Horwitz (<120ppb) Horwitz (original) σ = 0.22c RSD, % ppb 150 ppb 120 ppb 100 ppb 50 ppb 20 ppb 10 ppb 1 ppb concentration

12 Modification Above 13.8% σ = 0.01c RSD, % 6 Horwitz (original) Horwitz (>13.8%) % 10 % 13.8 % 20 % concentration 50 %

13 Are You Sitting Comfortably? Then I ll begin

14 Homogeneity Testing All test materials are heterogeneous What we want is sufficient homogeneity In other words: the differences between individual test portions must not be large enough to materially affect the outcome of the PT Otherwise the results would simply reflect the many levels in the test portions and not the accuracy of the participating lab

15 Testing for sufficient homogeneity - 1 The statistical test is based around ANOVA typically using the results from 10 samples analysed in duplicate For the statistics to be meaningful the results have to be obtained under specified conditions: random selection, random analytical order, same time, etc. i.e. under repeatability conditions

16 Testing for sufficient homogeneity - 2 It is NOT just a comparison of the variance within and between pairs Experience has shown that type of simple test is limited over-sensitive when repeatability is very good under-sensitive when repeatability is poor

17 Testing for sufficient homogeneity rep 1 rep rep 1 rep

18 Testing for sufficient homogeneity - 4 FAPAS uses a more sophisticated protocol Fearn and Thompson [3] Seek to reject material that displays heterogeneity above a set limit limit derived from fitness-for-purpose Fully worked example in Prof. Thompson s published paper

19 Testing for sufficient homogeneity - 5 Assume homogeneity reasonable, lots of time and effort goes into making test materials Scrutinise the results for any obvious problems e.g. trends, possible outliers Use Cochran s test to formally check the variance of the worst pair

20 Testing for sufficient homogeneity rep 1 rep rep 1 rep

21 Testing for sufficient homogeneity - 7 Carry out ANOVA to obtain: the analytical variance, san 2 the sampling variance, ssam 2 Calculate the allowable sampling variance σall 2 = 0.3σp Calculate critical value c = F1σall 2 + F2 san 2 (F1 and F2 from a given table) If s sam 2 > c then the test indicates is a lack of sufficient homogeneity

22 Testing for sufficient homogeneity rep 1 rep

23 The Assigned Value Note, the assigned value, not the true value the true value is an ideal we ll never know The use of the word assigned indicates we are setting the value The assigned value should be the best estimate of the true value

24 Deriving the Assigned Value FAPAS usually derives the assigned value from the consensus of submitted results other options are a cert. ref. or a formulation value Using the most appropriate measure of central tendency: robust mean [4, 5] median mode [6] but not necessarily in that order

25 Simple vs Robust Mean Descriptive Statistics Variable: ass. value Anderson-Darling Normality Test A-Squared: P-Value: Mean StDev Variance Skewness Kurtosis N Robust Mean % Confidence Interval for Mu Minimum 1st Quartile Median 3rd Quartile Maximum % Confidence Interval for Mu % Confidence Interval for Sigma % Confidence Interval for Median 95% Confidence Interval for Median

26 Limitations of a Robust Mean / Median Descriptive Statistics Variable: afm1 Anderson-Darling Normality Test A-Squared: P-Value: Mean StDev Variance Skewness Kurtosis N E Robust Mean % Confidence Interval for Mu Minimum 1st Quartile Median 3rd Quartile Maximum % Confidence Interval for Mu % Confidence Interval for Sigma % Confidence Interval for Median 95% Confidence Interval for Median

27 Bump-hunting Adaptive kernel density plot - afm1 6 5 mode = Density Analytical result 1.0

28 Identifying Poor Methodology Adaptive kernel density plot - chloride Density Analytical result 1500

29 Poor or Just Different Performance? Adaptive kernel density plot - peanut prote 0.2 Density Analytical result 20

30 z-scores (at last!) This is a score that compares a participant s result to the true value x - X Then standardises it against a measure of acceptable analytical variation (x - X)/sd

31 More formally z = ( x Xˆ σ p ) where : x = Xˆ σ p participant' s result = the assigned value = std dev for proficiency assessment

32 Non-Normal Distributions - 1 z-scores rely on the results being normally distributed Microbiological results are known to be nonnormally distributed (Poisson distribution) log-transformation prior to calculating z-scores

33 Non-Normal Distributions - 2 GeMMA PT results invariably are skewed, with a long tail to the high end Review [7] of two GM schemes, commissioned by UK Food Stds Agency, confirmed: the non-normal distributions log-transformation prior to calculating z-scores, as the most appropriate way to treat the results

34 Non-normal distributions - 3 More formally z = (log x log σ p Xˆ ) where : x = Xˆ σ p participant' s result = the assigned value = std dev for proficiency assessment, expressed in log10

35 Understanding z-scores z-scores embody the concept of fitness for purpose If the level of the determinand and/or the allowable variation around this level are inappropriate for your work your z-scores have no worth e.g. oil content of soya beans, assaying this to determine its commercial value is not the same as checking the oil content for nutritional purposes

36 Interpreting z-scores - 1 z-scores look simple but z-scores are statistics and, as with any statistic, interpretation requires experience Such experience gives you the edge with your managers, competitors, customers and accreditation assessors

37 Interpreting z-scores - 2 Superficially z-scores can be interpreted as: z <= 2 satisfactory z >2 but <= 3 questionable z > 3 unsatisfactory However, there is more to it! You must consider the probabilities a questionable score has about a 1 in 20 chance of being a perfectly good result, from the edge of the distribution!

38 Interpreting z-scores std dev -3 std dev -2 std dev -1 std dev mean + 1std dev +2 std dev +3 std dev +4 stdev

39 Your z-score What is fit for YOUR purpose?

40 References [1] M. Thompson, S. Ellison and R. Wood, The International Harmonised Protocol for the Proficiency Testing of (Chemical) Analytical Laboratories, Pure Appl. Chem., Vol.78, No.1, pp , [2] M. Thompson, Recent trends in inter-laboratory precision at ppb and sub-ppb concentrations in relation to fitness for purpose criteria in proficiency testing, Analyst, 2000, 125, [3] T. Fearn and M. Thompson, A New Test for Sufficient Homogeneity, Analyst, 2001, 126, [4] Analytical Methods Committee, Robust Statistics How not to reject outliers Part 1. Basic Concepts, Analyst, 1989, 114, [5] ISO 13528:2005, Statistical methods for use in proficiency testing by interlaboratory comparisons, Annex C [6] P.J. Lowthian, and M. Thompson, Bump-hunting for the proficiency tester searching for multimodality, Analyst, 2002, 127, [7] Thompson, M., et al, 2006, Scoring in GMO Proficiency Tests based on log-transformed results, J. AOAC Int., 89(1),