Uncertainty Associated with Microbiological Analysis

Transcription

1 Appendix J STWG Pat 3 Uncetainty Page 1 of 31 Uncetainty Associated with Micobiological Analysis 1. Intoduction 1.1. Thee ae only two absolute cetainties in life: death and taxes! Whateve task we undetake, no matte how menial o how sophisticated, we ae faced with a lack of cetainty in the outcome! It is theefoe essential to have a common undestanding of what is meant by uncetainty in elation to ou specific tasks in defining BPMM. 1.. In micobiological laboatoy pactice, we can identify many causes of vaiability, fo instance: The ability of an isolate to give typical eactions on a diagnostic medium; 1... The use of the incoect ingedients in a cultue medium; The consequence of changing bands of commecial media; Use of non-standad conditions in the pepaation, steilisation and use of a cultue medium; Equipment and human eos in weighing, dispensing, pipetting and othe laboatoy activities; The toleance applied to the shelf life of test eagents; The elative skill levels of diffeent technicians; The elative well-being of any technician who is undetaking analyses; and so on, and so on. ad infinitum! 1.3. These ae but a few tite examples of biological, instumental and pesonal bias that affect the accuacy, pecision and hence the uncetainty of micobiological tests; a situation that constantly faces scientists involved in laboatoy management To intepet popely the esults obtained using any analytical pocedue, whethe physical, chemical o biological, equies caeful consideation of the divese souces of actual o potential eo associated with the esults obtained. Any analytical esult is influenced by a complex of thee majo eo goups: Random eos, associated with the oiginal sample matix, the analytical (test) sample, the cultue media, etc; Inheent systematic eos associated with the analytical pocedue; and Modification of the systematic eos due to a paticula laboatoy s envionment and equipment togethe with individual analysts pesonal taits in caying out the test pocedue.

2 Appendix J STWG Pat 3 Uncetainty Page of Accuacy and Pecision Accuacy is a qualitative concept (VIM, 1993). In simple tems, accuacy can be defined as the coectness of a esult, elative to an expected outcome; whilst pecision is a measue of the vaiability of test esults Accuacy is defined (ISO3534-:003) as "the closeness of ageement between a test esult o a measuement esult and the tue value." Accuacy is a combination of tueness and pecision (a combination of andom components and systematic eo o bias components). This diffes fom the definition given by VIM (1993): "the closeness of ageement between the esult of a measuement and a tue value of a measuand" Accuacy is essentially absence of eo ; the moe accuate a esult the lowe the associated eo of the test. It is impotant to note that the tem accuacy applies only to esults and can not be applied to methods, equipment, laboatoies o othe geneal mattes Tueness is defined (ISO, 003) as, the closeness of ageement between the aveage value obtained fom a lage seies of test esults and an accepted efeence value Tueness is equivalent to an absence of bias, which is the diffeence between the expectation of the test esults and an accepted efeence value and is a measue of total systematic, but not andom, eo Tueness, unlike accuacy, may coectly be contasted with pecision Pecision is defined as the closeness of ageement between independent test esults obtained unde stipulated conditions Pecision depends only on the distibution of andom eos and does not elate to a tue value o a specified value The measue of pecision is expessed usually in tems of impecision and computed as a standad deviation of the test esults Lowe pecision is eflected by a lage standad deviation Independent test esults means esults obtained in a manne not influenced by any pevious esults on the same o simila test object Quantitative measues of pecision depend citically on the stipulated conditions. Repeatability and epoducibility conditions ae paticula sets of exteme stipulated conditions (ISO 3534: 3.14) Fig 1 illustates schematically the elationships between tueness, accuacy, pecision and uncetainty (AMC, 003).

3 Page 3 of 31 Fig 1. Relationships between tueness, accuacy, pecision and uncetainty in analytical esults (AMC, 003). (Repoduced by pemission of the Royal Society of Chemisty, London) 1.8. The concepts of accuacy and tueness must take account of eo and pecision. Uncetainty estimates (qv) povide a simple way to quantify such needs. Howeve, since in a eal-life situation we neve know what the tue o coect answe is, tueness can be assessed only in a validation-type tial against an accepted efeence value. This is much moe complex in micobiology than it is in physics, and chemisty.. Uncetainty of Measuement.1. The ISO/Euachem (000) definition of Uncetainty of a Measuement is.. A paamete associated with the esult of a measuement that chaacteises the dispesion of the values that could easonably be attibuted to the measuand. The tem measuand is a bueaucatic way of saying analyte..3. Tanslated into simple English this definition can be ewitten, as Uncetainty is a measue of the likely ange of values that is indicated by an analytical esult..4. Fo quantitative data (e.g. colony counts, MPNs o LOD 50 values) a measue of uncetainty may be any appopiate statistical paamete associated with the test esult. Such paametes include the standad deviation, the standad eo of the mean o a confidence inteval aound that mean..5. Measues of epeatability and epoducibility ae the cone stones of estimation of analytical uncetainty. They ae defined (ISO 004) as:

4 Page 4 of Repeatability is a measue of vaiability deived unde specified epeatability conditions, i.e. independent test esults ae obtained with the same method on identical test items in the same laboatoy by the same analyst using the same equipment, batch of cultue media and diluents, and tested within shot intevals of time..5.. Repoducibility is a measue of pecision deived unde epoducibility conditions i.e. test esults ae obtained with the same method on identical test items in diffeent laboatoies with diffeent opeatos using diffeent equipment. A valid statement of epoducibility equies specification of the conditions used Intemediate Repoducibility (ISO 575-:1994 ) is defined as a measue of epoducibility deived unde epoducibility conditions within a single laboatoy Standad Uncetainty of a measuement (u(y)) is defined (GUM, 000) as the esult obtained fom the values of a numbe of othe quantities, equal to the positive squae oot of a sum of tems, the tems being the vaiances o covaiances of these othe quantities weighted accoding to how the measuement esult vaies with changes in these quantities.5.5. Expanded Uncetainty (U) is defined as the quantity defining an inteval about a esult of a measuement expected to encompass a lage faction of the distibution of values that could easonably be attibuted to the measuand The Expanded Uncetainty values ae deived by multiplying the SD s with a coveage facto to povide confidence intevals fo epeatability and epoducibility aound the mean value. Routinely, a coveage facto of is used to give appoximate 95% distibution limits (confidence inteval) aound the nomalised mean value..6. Fo qualitative data (e.g. pesence o absence tests) uncetainty measues cannot be deived in the same way. Howeve, othe pocedues e.g. use of the standad eo associated with deived values fo e.g. LOD 50 (qv) and by binomial analysis of the elative popotions of positive and negative esults in a compaative evaluation of methods (see 3.4 below). 3. How is uncetainty estimated? 3.1. Thee ae two totally diffeent appoaches to the estimation of uncetainty: The bottom up appoach in which the eos associated with all the elevant steps undetaken duing an analysis ae used to deive a value fo the combined standad uncetainty associated with a method (Euachem 000; Niemelä, 00). Essentially this appoach povides a boad indication of the possible level of uncetainty associated with method athe than a measuement; ISO TC34 SC9 consides the appoach always to undeestimate the extent of vaiation since it cannot take into account eithe matixassociated eos o the actual day-to-day vaiation seen in a laboatoy. Fo

5 Page 5 of 31 these easons, ISO has ecommended that this appoach is not appopiate fo micobiological analyses The top-down appoach is based on statistical analysis of data geneated in inta- o inte-laboatoy collaboative studies on the use of a method to analyze a divesity of matixes. It theefoe povides an estimate of the uncetainty of a measuement associated with the use of a specific method Statistical aspects of the pocedues, togethe with woked examples, fo both appoaches ae summaised in Annexes I & II A eview of measuement uncetainty in quantitative micobiological analysis is cuently in pess (Coy et al, 006). 3.. Quantitative Tests. Fo quantitative data (e.g. colony counts and MPN estimates), measues of epeatability and epoducibility ae deived as the standad deviations of epeatability (s ) and epoducibility (s R ). Howeve Micobiological data do not nomally confom to a nomal distibution, and usually equie mathematical tansfomation pio to statistical analysis. Fo most puposes, a log 10 tansfomation is used to nomalise the data but in cases of significant ove-dispesion the use of a negative-binomial tansfomation may be necessay (Javis, 1989; Niemelä, 00). If thee is eason to believe that data confom to a Poisson distibution, then a squae oot tansfomation is equied, since the vaiance (σ ) is numeically equal to the mean (m) value Statistical analyses of collaboative tial data ae geneally done by Analysis of Vaiance (ANOVA) afte emoving any outlying values, as descibed by Youden & Steine (1975) and by Howitz (1995). Howeve, it has been agued (e.g. AMC 1989, 001) that it is wong to eliminate outlie data and that application of Robust Methods of analysis is pefeable One appoach to obust analysis is a obusticised ANOVA pocedue based on Hube s H15 estimatos fo the obust mean and standad deviation of the data (AMC, 1989, AMC 001, ISO 575-5:1998) An altenative appoach is that of the Recusive Median (REMEDIAN) pocedue (ISO 000; Wilich, 005) Woked examples of taditional and obust analyses ae shown in Annexe III A majo dawback to use of these obust techniques fo inte-laboatoy tials is that they do not pemit the deivation of Components of Vaiance. A novel appoach to ovecome this disadvantage is by the use of stepwise obust analysis fo nested tial data, as descibed by Hedges & Javis (006) Intemediate Repoducibility of Quantitative Tests. Simila pocedues may be used to estimate intemediate (inta-laboatoy) epoducibility associated with the use of an analytical pocedue in a single laboatoy. Even data obtained, fo instance, in laboatoy quality monitoing can be used to povide an estimate of inta-laboatoy

6 Page 6 of 31 epoducibility. ISO/PTDS 19036:005 (Pat 6) descibes a statistical pocedue fo analysis of paied data. A woked example is shown in Annex IV Qualitative Tests. Estimation of uncetainty associated with qualitative (e.g. pesence o absence) methods has not been well documented and is cuently the subject of discussion within ISO Many of the potential eos that affect quantitative methods also affect qualitative methods; but thee ae also some additional potential eos that ae inheent in the analytical pocedue. Fo example: In taking a sample fo analysis, it is of citical impotance to have knowledge of the pobable distibution of oganisms in the test matix, especially when testing fo oganisms at the limit of detection of a method. Whilst it may be possible to ensue easonable confomity with a Poisson (andom) distibution of index oganisms in atificial test matixes, such distibution should not be assumed to occu in natual matixes and equies confimation (e.g. using an Index of Dispesion Test such as that descibed by Fishe et al, 19) befoe using such matixes in collaboative studies. In eal life testing, eoneous decisions can esult fom an assumption that all micooganisms ae distibuted andomly at low level thee ae some well-documented examples whee ove dispesion of oganisms (e.g. due fo instance to clumping) has esulted in a significant level of genuine false negative suveillance data Thee is an intinsic need to ensue effective gowth of the index oganism to citical levels duing all pe-enichment, enichment and diffeential/diagnostic cultue stages so cultue medium composition, incubation times & tempeatues, etc ae citical to the success of the test It is citical to ensue that the confimatoy stages of a test potocol do actually identify the index oganism Knowledge of the potential effect of competitive oganisms is of majo impotance fo all cultual and confimatoy stages of a test potocol The decision on use of eithe tue pais o non-paied samples is of geat impotance in the intepetation of potential false negative o false positive esults fo method validation studies The output of qualitative tests is a seies of positive and negative esponses. One appoach to seeking to quantify such data was the deivation of the Accodance and Concodance concept (Langton et al, 00) that sought to povide measues equivalent to the conceptual aspects of epeatability and epoducibility. Howeve, it is now consideed that this appoach is not sufficiently obust to be used in the manne poposed and adds no value to the oiginal data Povided that a sufficient numbe of paallel tests has been undetaken at each of seveal levels of potential contamination, then it is possible to quantify the

7 Page 7 of 31 test esponses in tems of an estimated Level of Detection fo (e.g.) 50% positives [LOD 50 ](fo details see Hitchins, 005) This statistical appoach essentially estimates the Most Pobable Numbe of oganisms at each test level and then analyses the elative MPN values using the Speaman-Kabe appoach Altenative appoaches including Pobit and Logit analyses may also be appopiate in specific cicumstances What these methods have in common is an ability to tansfom puely qualitative data into a quantitative fomat fo which eo values can be deived so pemitting an estimate of the uncetainty of the test esult An extapolation of the appoach would be to detemine also the LOD 0 and LOD 90 values such that a dose-esponse cuve can be deived. This may be of impotance in diffeentiating between methods capable of detecting specific oganisms at a simila LOD 50 level but fo which the absolute limit of non-detection (LOD 0 ) and a selected highe limit of detection (e.g. LOD 90 ) diffe An altenative appoach is to estimate the uncetainty associated with the popotions of test samples giving a positive esponse, based on the binomial distibution Examples of the way in which such appoaches to analysis of qualitative data can be used ae illustated in Annex V. 4. Repoting of Uncetainty 4.1. The expession of uncetainty is of some impotance in intepetation of data. Assuming a mean aeobic colony count (ACC) = 5.00 (log 10 ) cfu/g and a epoducibility standad deviation of ± 0.5 (log 10 ) cfu/g, then the expanded uncetainty is given, fo instance, by: Aeobic colony count on poduct X is 5.00 ± 0.50 (log 10 ) cfu/g; o Aeobic colony count on poduct X is 5.00 (log 10 ) cfu/g ± 10% 4.. It is impotant not to efe to analytical methods as having a pecision of e.g. ± 10% based on uncetainty estimates. Uncetainty is a measue of vaiability i.e. a measue of the lack of pecision. 5. The use of uncetainty measues in assessing compliance of a test esult with a defined citeion is of some impotance and has been consideed by the Euopean Commission (Anon, 003). Javis et al (004) and Javis & van de Voet (005) have discussed the intepetation of data in elation to micobiological citeia fo foods. Fo moe infomation, please contact Basil Javis at basil.javis@btconnect.com.

8 Page 8 of Refeences cited Analytical Methods Committee (1989) Robust Statistics How Not to Reject Outlies. Pat 1: Basic concepts, The Analyst 114, Pat : Inte-laboatoy tials, The Analyst 114, Analytical Methods Committee (001) Robust statistics: a method of coping with outlies. AMC Bief No.6, Royal Society of Chemisty, London. Analytical Methods Committee (003) Teminology - the key to undestanding analytical science. Pat 1: Accuacy, pecision and uncetainty. AMC Bief No.13, Royal Society of Chemisty, London. Anon (003) The elationship between analytical esults, the measuement uncetainty, ecovey factos and the povisions in EU food and feed legislation. Repot to the EU Standing Committee on the Food Chain and Animal Health Woking Goup Daft, 5 June 003 Coy, J, Javis, B, Passmoe, S and Hedges, A (006) A citical eview of measuement uncetainty in the enumeation of food micooganisms. Food Micobiology (in pess) Euachem (000) Quantifying Uncetainty in Analytical Measuement. nd edition, Laboatoy of the Govenment Chemist, London. Fishe, R A, Thonton, H G & Mackenzie, W A (19) The accuacy of the plating method of estimating the density of bacteial populations. Annals Applied Biology, 9, Hedges, A & Javis, B (006) Application of obust methods to the analysis of collaboative tial data using bacteial colony counts.. J Micobiological Methods (in pess). Hitchins, A J (005) Poposed Use of a 50 % Limit of Detection Value in Defining Uncetainty Limits in the Validation of Pesence-Absence Micobial Detection Methods. BPMM Repot fo the Statistics WG (010705). Howitz, W (1995) Potocol fo the design, conduct and intepetation of method pefomance studies. Pue & Applied Chemisty, 67, ISO 575-:1994: Accuacy (tueness and pecision) of measuement methods and esults Pat : Basic method fo the detemination of epeatability and epoducibility of a standad measuement method. Intenational Standadisation Oganisation, Geneva ISO 575-5:1998: Accuacy (tueness and pecision) of measuement methods and esults Pat 5: Altenative methods fo the detemination of the pecision of a standad measuement method. Intenational Standadisation Oganisation, Geneva ISO FDIS 16140:000 Micobiology of food and animal feeding stuffs Potocol fo the validation of altenative methods. Intenational Standadisation Oganisation, Geneva. ISO :003 Statistics Vocabulay and Symbols. Intenational Standadisation Oganisation, Geneva.

9 Page 9 of 31 ISO 16140:003 Micobiology of food and animal feeding stuffs Potocol fo the validation of altenative methods. Intenational Standadisation Oganisation, Geneva. ISO TS 1748:004 Guidance fo the use of epeatability, epoducibility and tueness estimates in measuement uncetainty estimation Intenational Standadisation Oganisation, Geneva ISO PTDS 19036:005 Micobiology of food and animal feeding stuffs Guide on estimation of measuement uncetainty fo quantitative deteminations. Intenational Standadisation Oganisation, Geneva Javis, B (1989) Statistical analysis of the micobiological analysis of foods. Pogess in Industial Micobiology Vol 1. Elsevie, Amstedam. Javis, B (000) Sampling fo Micobiological Analysis. In The Micobiological Safety and Quality of Food. Ed B M Lund, A C Baid-Pake & G W Gould. Vol II, pp Aspen Pub Inc, Gaithesbug, MA. Javis, B, Hedges, A, Coy, J E L & Wood, R (004) Cetainty o Uncetainty? The impact of uncetainty on the intepetation of colony count data in elation to micobiological citeia. Poste pesented at the Society fo Applied Micobiology confeence on Food and Daiy Micobiology, Cok, July 004. Javis, B & van de Voet, H (005) Guidelines on the use of uncetainty measuements in the assessment of data fo compliance with quantitative micobiological citeia fo foods. Daft woking pape fo ISO TC34 SC9 Statistics WG. LaBae, D D, Zelenke, D & Flowes, R (005) Inta-laboatoy and Inte-laboatoy Vaiability. BPMM Sampling WP document (daft 5 6/9/05) Langton, S.D, Chevennement, R, Nagelkeke, N, Lombad, B. (00). Analysing collaboative tials fo qualitative micobiological methods: accodance and concodance. Intenational Jounal of Food Micobiology 79, Niemelä, S.I, 00 Uncetainty of quantitative deteminations deived by cultivation of micooganisms. nd edition, Cente fo Metology and Acceditation, Advisoy Commission fo Metology, Chemisty Section, Expet Goup fo Micobiology, Helsinki, Finland, Publication J3/00 Niemelä, SI (003) Measuement uncetainty of micobiological viable counts. Acceditation and Quality Assuance, 8: NMKL (00). Measuement of uncetainty in micobiological examination of foods. NMKL Pocedue no 8, nd ed., Nodic Committee on Food Analysis Rousseeuw, P. J., and Coux, C. (1993): Altenatives to the median absolute deviation. J. Am. Stat. Ass. 88, SMITH, P.A. & KOKIC, P. (1996) Winsoisation in ONS business suveys. Woking pape no. at the UN Data Editing Confeence 1996, Voobug.

10 Page 10 of 31 Van de Voet, H (005) Altenative models fo measuement uncetainty of micobiological count data. Pape pesented to ISO TC34 SC9 Statistics WG meeting, Pama, Apil 004. VIM (1993) Intenational vocabulay of basic and geneal tems in metology. (ISO 1993) Wilich, P-T (005a) Robust estimates of the theoetical standad deviation to be used in inte-laboatoy pecision expeiments. Discussion pape fo the ISO TC34 SC9 Statistics Goup Meeting, Pais, Apil 005. Wilich, P-T (005b) The detemination of pecision of measuement methods with qualitative esults by intelaboatoy expeiments. Discussion pape fo the ISO TC34 SC9 Statistics Goup Meeting, Pais, Apil 005. Youden, W J and Steine, E H (1975) Statistical Manual of the AOAC. AOAC, Washington.

11 Page 11 of 31 Annex I Top-Down Pocedue Fo Estimation Of Uncetainty 1. The basis of the top down appoach descibed by GUM (Euachem 000) is to identify and take account of all pocedual stages of an analytical method. The vaiance associated with each individual stage is combined with the vaiances all the othe stages and inteactions that make up an analytical pocedue in ode to estimate a geneic level of uncetainty fo a method. This is illustated diagammatically in the schematic below. Sample Matix Sampling Pocedue Analytical Method Analytical Result. Conside fist the sample matix: what ae the likely eos that will affect the analytical esult?.1. The lagest potential eo souces will be: the spatial distibution of the micooganisms (andom, unde- o ove dispesion as exemplified by evidence of clumping); the condition of the micooganisms (viable and vital, sublethally damaged, non-cultivatable); the effects of competitive oganisms on the ecoveability of specific types; whethe the oganisms ae located pimaily on the suface of, o moe geneally distibuted thoughout, the matix; etc... Howeve, the intinsic natue of the matix will also affect the esults of an analysis. 3. How epesentative is an analytical sample taken fom a matix? 3.3. Should the analytical sample be totally epesentative of the whole matix, o should it elate only to a specific pat, e.g. the suface of a meat cacass? If the fome should the matix be homogenized pio to taking a sample; if the latte should the suface laye be excised, swabbed, insed o tested using a eplica plating technique? What eve the method of sampling to what extent is the micofloa in the analytical sample epesentative of both the numbe and types of micooganisms pesent in the oiginal matix If the matix is a composite food, should the sample epesent the whole o individual pats of the food matix (e.g. in the case of a meat pie should the pasty and the meat be analysed sepaately)? 3.5. What size of sample should be tested? Inceasing the size of an analytical sample esults in a decease in the standad eo associated with the mean weight of sample taken. Similaly, inceasing the weight of sample taken tends to incease the appaent colony count whilst educing the oveall vaiance of the mean count (Javis, 1989).

12 Page 1 of At its simplest, the analytical pocess consists of taking an analytical sample, suspending that sample in a defined volume of a suitable pimay diluent, maceating the sample, pepaing seial dilutions, plating measued volumes onto o into a cultue medium, incubating the plates, counting and ecoding the numbes of colonies and deiving a final estimate of colony foming units (cfu) in the oiginal matix. At all stages thoughout this pocess, eos will occu Some eos, e.g. those associated with the accuacy of weighing, the accuacy of pipette volumes, the accuacy of colony counting, etc, etc can be quantified and measues of the vaiance can be deived. 4.. Some eos can be assessed, but not necessaily quantified; fo instance, laboatoy quality contol pocedues can be used to assess the extent to which a cultue medium will suppot the gowth of specific oganisms. Such data may potentially povide a coection facto fo the yield of oganisms on a paticula cultue medium; whethe o not the use of a coection facto should be employed in micobiological pactice is a matte of debate! 4.3. Howeve, othe eos, such as those associated with individual technical pefomance on a day, cannot be quantified. 5. Some analytical eos associated with micobiological pactices ae possibly not significant when compaed to othe eos, but how do you know this if the eos cannot be quantified? To assess the uncetainty of an analytical micobiological pocedue fom the top down equies a full evaluation of all potential souces of eo fo each and evey stage of an analytical pocedue. 6. Estimation of the standad uncetainty of an analytical pocedue, once a eliable schedule of quantifiable eos has been poduced, is done simply by combining the eos: sr = sa + sb sx + sy + s z whee s R = epoducibility vaiance of the method and s a... z = vaiance of any stage (a.z) within the oveall method. By definition, the epoducibility standad deviation ( of the vaiance: s R ) is deived fom the squae oot s = s + s s + s + s z R a b x y 7. The expanded uncetainty is deived by multiplying the standad uncetainty by a coveage facto k, which has a value fom to 3. A value of is nomally used to give appoximate 95% confidence limits; hence U = k. sr =. sr 8. Niemelä (00, 003) gives a moe detailed explanation of the top down appoach to assessment of measuement uncetainty in micobiological analysis.

13 Page 13 of 31 Annex II Bottom-up Appoach to Estimation of Uncetainty 1. Taditionally, the paametes used to deive uncetainty measues ae estimated fom the pooled esults of a valid inte-laboatoy collaboative study, o in the case of intemediate epoducibility, fom an inta-laboatoy study. Appopiate pocedues to ensue that the study design is valid have been descibed inte alia by Youden & Steine (1975) and by ISO (1994, 1998).. The data fom all paticipating laboatoies ae subjected to analysis of vaiance (ANOVA) afte fist checking fo:.1. Confomance with a nomal distibution eithe by plotting the data o by application of appopiate tests fo nomality... Identification and emoval of outlies using the methods descibed by Youden and Steine (1975) o Howitz (1995), followed if necessay by epeating the tests fo confomance with nomality. 3. Quantitative micobiological data (e.g. colony counts and MPNs) do not confom to a nomal distibution and equie tansfomation to nomalise the data befoe analysis. 4. Tansfomations ae done by conveting each of the aw data values (x i ) into the log 10 value (y i ) whee y i = log 10 x i. Stictly, it is moe coect to use the natual logaithmic tansfomation (i.e. y i = ln x i) (van de Voet, 004). 5. Fo low level counts (typically < 100 cfu/g) that confom to the Poisson distibution (mean value (m) = vaiance (s )), the data ae tansfomed by taking the squae oot of each data value (i.e. y i = x i ). 6. Howeve, because of poblems of ove-dispesion fequently associated with micobial contamination, it may be pefeable to test fo (o to assume) confomance with a negative binomial distibution. Some statistical packages (e.g. Genstat) include a facility to make this tansfomation (using the Maximum Likelihood Method pogamme RNEGBINOMIAL), but such pocedues ae not univesally available and it can be vey time-consuming to calculate manually (Javis, 1989; NMKL, 00, Niemelä, 003; van de Voet, 004). 7. Assuming a fully nested expeimental design (e.g. duplicate testing of duplicate samples by A analysts in each of L laboatoies), the esidual mean vaiance (i.e. the vaiance of the eplicated analyses on each sample) of the ANOVA povides an estimate of epeatability vaiance ( s ). The estimate of epoducibility vaiance ( s R ) fist equies computation of the contibutions to vaiance of the samples, analysts and laboatoies. This is illustated below. 8. The epeatability standad deviation (s ) and the epoducibility standad deviation (s R ), being the squae oot values of the espective vaiances, ae the measues of standad uncetainty fom which the expanded uncetainty estimates ae deived.

14 Page 14 of Statistical Pocedue to Deive Component Vaiances fom an ANOVA Analysis Assume: tial consists of (p) laboatoies (p=0) in each of which analysts test eplicate samples and make duplicate analyses of each sample. Hence, each laboatoy caies out 8 eplicate analyses and the total numbe of analyses = 8p = 160. Each data value (y pijk ) is allocated to a cell in the data table in the sequence laboatoy (p), analyst (i), sample (j) and eplicate (k), as shown below, and ae then analysed by multivaiate analysis of vaiance. Laboatoy (p = 1 0) Analyst (i =1) Analyst (i = ) Sample (j = 1) Sample (j = ) Sample (j = 1) Sample (j = ) (k = 1) (k = ) (k = 1) (k = ) (k = 1) (k = ) (k = 1) (k = ) 1 y 1111 y 111 y 111 y 11 y 111 y 11 y 11 y 1 y 111 y 11 y 11 y 1 y 11 y 1 y 1 y 3 y 3111 y 311 y 311 y 31 y 311 y 31 y 31 y 3 4 y 4111 y 411 y y 0111 y 011 y 011 y 01 y 011 y 01 y 01 y 0 Souce of Vaiation ANOVA table fo a fou-facto fully-nested expeiment Sum of Squaes Degees of feedom Mean Squae Expected Mean Squae Components* Laboatoies SS lab p-1 = 19 SS lab /19 = MS lab σ + σ + 4σ + 8σ sam ana lab Analysts SS ana p = 0 SS ana /0 = MS ana σ + σsam + 4σ Samples SS sam p = 40 SS sam /40 = MS sam σ σ Residual SS es 4p = 80 SS es /80 = MS es σ + sam ana Total Total SS 8p 1=159 * The components ae shown as population vaiances since this is an expectation table. The esidual mean squae (MS es = s ) povides the epeatability vaiance between duplicate analyses done on the same eplicate sample.

15 Page 15 of 31 The vaiance due to l samples ( s sam )is given by [MS sam - s ]/ The vaiance due to analysts ( ) is given by [ MS s s ]/4 s ana The vaiance due to is laboatoies ( s lab The Repoducibility Vaiance ( s R ) is given by ana sam ) given by [ MS s 4s s lab sam ana 1 [ ssam + sana + slab + s ] ]/8 The Repoducibility Standad Deviation is given by s + s + s + s 1 sam ana lab The Repeatability Standad Deviation is given by s. WORKED EXAMPLE (10 Labs x Analysts x Samples x analyses) Log tansfomed colony counts (Log 10 cfu/g) Laboatoy Analyst (i = 1) Analyst (i = ) Sample (j=1) Sample (j=) Sample (j=1) Sample (j=) (k = 1) (k = ) (k = 1) (k = ) (k = 1) (k = ) (k = 1) (k = ) Tests fo nomality (e.g. Shapio-Wilk, W = , p= ) did not dispove the hypothesis that the log 10 tansfomed data confom easonably (although not pefectly) to a nomal distibution. Howeve, application of the Cochan Test (Howitz, 1995) identified Laboatoy 7 as an outlie; subsequently evaluation using the Gubbs test did not eliminate othe laboatoies although laboatoies 3 & 4 appeaed to be possible outlies.

16 Page 16 of 31 ANOVA table fo the fou-facto fully nested expeiment (All data included) Souce of Vaiation Sum of Squaes Degees of feedom Mean Squae (ounded to 4 places) Mean Squae Components Laboatoies s + s + 4s + 8s Analysts sam ana lab s + s + 4s sam ana Samples Residual ssam s s Total The esidual mean squae (MS es = s = ) povides the epeatability vaiance between duplicate analyses done on the same eplicate sample. Component Vaiances Sample vaiance ( s sam ) = [MS sam - s ]/ = [ ]/ = Analyst vaiance ( ) = [ MS s s ]/4 = [ ]/4 = s ana ana sam Laboatoy vaiance ( )= [ MS s 4 s s ]/8 = [ ]/8 = s lab Hence, Repoducibility Vaiance ( s R ) = lab sam ana 1 [ sam + ana + lab + ] s s s s = [ ] = Repoducibility Standad Deviation = s s s s s = = ± R = sam + ana + lab + Repeatability Standad Deviation = s = The mean colony count = (log 10 ) cfu/g s = = ± Hence, Relative Standad Deviation of Repoducibility (RSD R ) = 100 x /5.668 = 8.4% and, Relative Standad Deviation of Repeatability (RSD ) = 100 x /5.668 =.08% Fom these values the 95% expanded uncetainty of epoducibility is given by: U = sr = x = ± ±0.93 (log10) cfu/g

17 Page 17 of 31 The uppe and lowe limits of the 95% Confidence Inteval on the mean colony count ae: U L = = 6.60 (log 10 ) cfu/g L L = = 4.74 (log 10 ) cfu/g Souce of Vaiation Repeat analyses fo 9 laboatoies( afte elimination of data fo laboatoy 7) Sum of Squaes Degees of feedom Mean Squae (ounded to 4 places) Mean Squae Components Laboatoies s + s + 4s + 8s Analysts sam ana lab s + s + 4s sam ana Samples Residual ssam s s Total The component vaiances wee deived as: Repeatability vaiance ( s ) = Sample vaiance ( s ) = 0.07 Analyst vaiance ( ) = Laboatoy vaiance ( ) = s ana Hence, Repoducibility Vaiance ( s R ) = 0.79 Repoducibility Standad Deviation = s R = 0.79 = ± Repeatability Standad Deviation = s = = ±0.111 The mean colony count = (log 10 ) cfu/g Hence, Relative Standad Deviation of Repoducibility (RSD R ) = 8.35% and, Relative Standad Deviation of Repeatability (RSD ) = 1.95% Fom these values the 95% expanded uncetainty of epoducibility is given by: U = sr sam s lab = x = ± ±0.95 (log10) cfu/g The uppe and lowe limits of the 95% Confidence Inteval on the mean colony count ae: U L = = 6.64(log 10 ) cfu/g L L = = 4.7 (log 10 ) cfu/g

18 Page 18 of 31 Compaison of ANOVAs with and without emoval of outlie laboatoy The table below shows that emoval of one set of data (fom the outlie laboatoy) maginally inceased the mean colony count and educed the component vaiances fo epeatability, samples, analysts and laboatoies. Howeve the oveall effect, in this specific example, was maginal in elation to the deived values fo epeatability and epoducibility; and hence thee was little effect on the level of expanded uncetainty. Paamete 10 Laboatoies 9 Laboatoies Mean Colony Count (log 10 cfu/g) Repeatability Vaiance Sample Vaiance Analyst Vaiance Laboatoy Vaiance SD epeatability (SD ) ± ±0.111 Relative SD.08% 1.95% SD epoducibility (SD R ) ± ± Relative SD R 8.4% 8.35% Expanded Uncetainty (U) ±0.93 ±0.95 Uppe Limit of 95% CI (log 10 cfu/g) Lowe Limit of 95% CI (log 10 cfu/g)

19 Page 19 of 31 Annex III Estimation of Intemediate Repoducibility based on Routine Monitoing Data 1. Inta-laboatoy uncetainty estimates can be made eithe by caying out a full intenal collaboative tial, with diffeent analysts testing the same samples ove a numbe of days o, fo instance, using diffeent batches o even diffeent bands of commecial cultue media. In such a case the statistical pocedue of choice is that descibed in Annex II.. Howeve, if a laboatoy undetakes outine quality monitoing tests, it is possible to estimate epoducibility fom these test data. One appoach is to use a 1-way ANOVA and to take the mean esidual squae as the estimate of epoducibility. A pefeed, and simple pocedue, is descibed fully in ISO19036: 005; this detemines the vaiance fo each set of tansfomed eplicate data values. 3. The epoducibility standad deviation is deived fom the squae oot of the sum of the duplicate vaiances divided by the numbe of data sets. The equation is: ( 1 ) = n yi yi / SR i= 1 n whee y and i1 y i ae the log tansfomed values of the oiginal duplicate counts (x1 and x ) and n is the numbe of pais of counts. 4. A woked example (based on log 10 tansfomation) is pesented below. 5. Confusion sometimes aises between epeatability and intemediate epoducibility. It must always be emembeed that epeatability equies all stages of the eplicated tests to be done only by a single analyst, caying out epeat deteminations on a single sample in a single laboatoy, using identical cultue media, diluents, etc within a shot time peiod e.g. a few hous. If moe than one analyst undetakes the analyses and/o tests ae done on diffeent samples and/o on diffeent days then the calculation deives a measue of intemediate epoducibility. The pocedue can be used to detemined aveage epeatability estimates fo individual analysts povided all the epeatability citeia ae met.. 6. Intenal laboatoy quality management is aided by the use of statistical pocess contol (SPC). The estimates of intemediate epoducibility povide a souce of data that is amenable to SPC.

20 Page 0 of 31 Woked Example (modified fom ISO 19036:005) The data below wee deived fom enumeation of aeobic mesophilic floa in mixed poulty meat samples. The duplicate data values (x ia and x ib ) ae log tansfomed to give y ia and y ib, espectively. The mean log 10 counts ( y ) ae deived fom (y ia + y ib )/; the vaiances (S Ri ) ae deived fom (y ia - y ib )/; and the RSD values fom 100* S Ri / y. Test(i) Colony Count A (cfu/g) Colony Count B (cfu/g) Log count A Log count B Mean log Count Absolute Diffeence in log count Vaiance Relative Standad Deviation (%) x ia x ib y ia =log 10 (x ia ) y ib =log 10 (x ib ) y y ia - y ib S Ri RSD Ri i=1 6.70E E % i= 7.10E E % i=3 3.50E E % i=4 1.00E E % i=5 1.90E E % i=6.30e E % i=7 5.30E E % i=8 1.00E E % i=9 3.00E E % i= E+08.0E % Σ Aveage Using the log 10 -tansfomed data (y ij ), the epoducibility standad deviation is deived fom: S R n (y y ) / i1 i i= = = = 0, 0193 ± 0,15 log cfu/g 10 n 10 Aveage % Relative Standad Deviation (RSD av )= 100*( SR/ y)=100*(0.15/6.18)=.39% Individual tests (i = 1.10) gave RSD values anging fom 0.47% to 5.98%, with an oveall value of.39%. Note: it is incoect to take the aveage of the individual RSD values. Once sufficient data ae available, a moving RSD av can be detemined and used in a statistical pocess contol system.

21 Annex IV Appendix J STWG Pat 3 Uncetainty Page 1 of 31 Application of Robust Methods of Statistical Analysis 1. Because of the poblems with the occuence of outlie data, seveal altenative appoaches to the Analysis of Vaiance have been developed, based on Robust Methods of Statistical Analysis.. Rathe than elying on identification and emoval of outlying data (which values could actually be valid esults, albeit consideably diffeent fom most of the data) and then estimating the vaiance aound the mean, altenative obust pocedues ely on estimation of the vaiation aound the median value. 3. A mean value will be affected significantly by one o moe high (outlie) values within a data set, wheeas the median value is not affected. Conside the following examples: A. 1, 4, 3, 6, 3, 5, 6, 3, 4, 5 n = 10, = 40, Mean = 4.0 Median = 4.0 B. 1, 4, 3, 6, 3, 5, 6, 3, 4, 5 n = 10, = 60, Mean = 6.0, Median = 4.0 C. 1, 4, 3, 6, 3, 5, 6, 3, 4, 15 n = 10, = 70, Mean = 7.0, Median = 4.0 D. 1, 4, 3, 6, 3, 5, 16, 3, 4, 15 n = 10, = 170, Mean = 17.0,Median = 4.0 E. 1, 4, 3, 6, 3, 5, 3, 4, n = 8, =9, Mean = 3.6, Median = The pesence of one o moe high values (Examples B, C, D) has a significant effect on the mean value but no effect on the median value. Removal of the high outlies (E) educes both the mean and the median values. 5. A simila effect would be seen with low value outlies. Of couse, occuence of both high and low outlies could balance out the effect on the mean. 6. Thee ae two pimay altenative techniques of obust analysis cuently in use: The Analytical Methods Committee of the Royal Society Chemisty (AMC 1989, 001) descibes one appoach. The pocedue calculates the median absolute diffeence (MAD) between the esults and thei median value and then applies Hübe s H15 method of winsoisation. Winsoisation is a technique fo educing the effect of outlying obsevations on data sets (fo detail see Smith & Kokic, 1996). The pocedue can be used with data that confom appoximately to a nomal distibution but with heavy tails and/o outlies. An example is shown below. The pocedue is not suitable fo multimodal o heavily skewed data sets. The AMC website 1 povides downloadable softwae fo use eithe in Minitab o Excel (97 o late vesion) An altenative appoach, known as the Recusive Median is based on extapolation of the wok of Rousseeuw & Coux (1993). One vesion of this appoach 1

22 Page of 31 (descibed fully in ISO 16140:003) uses Rouseeuw s ecusive median S n. Howeve, Wilich (005a) ecommends a modified appoach to this pocedue also based on Rousseeuw s S n computation. Woked Example - Analysis of data set containing outlies Duplicate Seies of Colony Counts (as Log 10 cfu/g) done by 1 Analyst in each of 10 Laboatoies Laboatoy A B Laboatoy data look to be slightly low and laboatoy 3 data to be high when compaed with the othe data.

23 Page 3 of 31 Gaphical and Desciptive Analysis of the Data 4.5 n 10 4 Mean % CI 4.60 to Vaiance SD SE CV 15% Fequency Fequency Plot Median % CI to Range.79 IQR 0.49 Pecentile.5th - 5th th th th - Coefficient p Shapio-Wilk Skewness Kutosis Nomal Quantile Log Colony Count/g - A Box Plot showing outlies Nomality Plot Although thee is evidence of kutosis and positive skewness, the log-tansfomed data confom faily well to a nomal distibution. The Box plot shows the pesence of a potential low-level outlie (+) and a significant high-level outlie ( ). One-way Analysis of Vaiance (ANOVA) without emoval of outlies Souce of Vaiation SS df MS F P-value F cit Between Laboatoies E Within Laboatoies Total Repeatability SD = = Repoducibility SD = ( ) = =

24 Page 4 of 31 One-way Analysis of Vaiance (ANOVA) afte emoval of high outlie (lab 3) Souce of Vaiation SS df MS F P-value F cit Between Laboatoies E Within Laboatoies Total Repeatability SD = = Repoducibility SD = ( ) = = One-way Analysis of Vaiance (ANOVA) afte emoval of both low and high outlies (labs & 3) Souce of Vaiation SS df MS F P-value F cit Between Laboatoies Within Laboatoies Total Repeatability SD = = Repoducibility SD = ( ) = 0. = Analysis of Vaiance using the AMC Method Softwae fo this analysis, compatible with Micosoft Excel, can be downloaded fom Royal Society of Chemisty statistical softwae. A vesion fo use in Minitab is also available. ROBUST ESTIMATES Paamete Gand Mean Within-laboatoy/epeatability SD Between-laboatoy SD Repoducibility SD c=1.5: Convcit= Repeatability SD = Repoducibility SD = Value

25 Page 5 of 31 Compaison of data analyses by ANOVA, without and with emoval of the high (*) outlie and both the high and low outlies (**), by Robust ANOVA (AMC 1989, 001) and by Recusive Median (ISO 16140:003) Paamete* ANOVA ANOVA* ANOVA** ROBUST RECMED Mean Median 5.05 SD RSD.45%.65%.73%.31%.8% SD R RSD R 0.45% 13.33% 9.4% 9.70% 11.07% * SD = Standad Deviation of epeatability; SD R = Standad Deviation of epoducibility RSD = % elative standad deviation of epeatability RSD R = % elative standad deviation of epoducibility The effect of the outlie values on the Standad Deviation of epoducibility is clea fom the above data. Removal of the high outlie (*) educes both the mean and the SD R ; emoval of both the high and low outlies (**) educes both the mean value and the SD R to a level simila to than that seen in the Robust ANOVA. The Recusive Median technique (woking data not shown) poduces a simila value fo SD but a somewhat highe SD R value than does the Robust Method.

26 Page 7 of 31 ANNEX V Uncetainty Associated with Qualitative Methods 1. By definition, a non-quantitative method meely povides an empiical answe to a question egading the pesence o absence of a specific index oganism o a goup of elated oganisms in a given quantity of a epesentative sample.. Povided that multiple samples ae analysed, and on the assumption that the test method is pefect, then the numbe of tests giving a positive esponse povides an indication of the incidence of defective samples within a lot..1. Fo instance, if a test on 10 paallel samples found 4 positive and 6 negative samples then the peceived incidence of defectives would be 40% (sic of the samples analysed)... Howeve, if no positive samples wee found the appaent incidence of defectives in the lot would be zeo. Howeve, it is not possible to say that the lot is not contaminated because the tue incidence of defective samples will be geate than zeo. 3. Sampling theoy fo occuence of defectives is based on the binomial distibution, in which the pobability of an event occuing (p) o of not occuing (q) can be deived and an eo estimate can be made based on a ealistic numbe of samples analysed. Unfotunately, in laboatoy pactice it is not usually possible to analyse a ealistic numbe of samples fo the pesence of specific micooganisms Table 1 below shows the statistical pobability of occuence of 0, 1, o defective units in 10 sample units fom lots containing fom 0.1 to 30% tue defectives. Fo a lot having only 0.1% defective units, the pobability of detecting one o moe defective (sic positive) samples is only 1 in 100 whilst fo a lot having 5% tue defectives thee is still only a 40% pobability of obtaining a positive esult; even with 0% tue defectives thee is still a 0% chance of not finding defective units when testing 10 sample units. 3.. Table shows the pobability of detecting 0, 1 o defective units with inceasing numbes of sample units tested when the tue incidence of defectives is 10%. The pobability of finding no defective samples is 59% if only 5 samples ae tested, 35% with 10 samples and 1% with 0 samples These examples illustate a basic chaacteistic of undetaking qualitative tests fo specific oganisms: unless the likelihood of contamination of the matix is high, and the numbe of sample units tested is consideable and the analytical test itself is pefect, then the pobability of detecting positive samples in food matix is vey low.

27 Page 8 of 31 Table 1. Binomial Pobability of detecting 0, 1 o defective units in 10 sample units tested with inceasing incidence of tue defectives (mod fom Javis, 000) Tue Incidence (%)of Defective Units in a lot Pobability (p) of detecting defective units % < % <0.01 5% % % % Table. Binomial pobability of detecting defective units with inceasing sample units fom a lot having 10% tue defectives (mod fom Javis, 000) Numbe of Sample units (n) tested Pobability of detecting the following numbe of defective units < Maximum Incidence and Level of Contamination. Even when all test esults ae negative, use of the binomial distibution concept pemits the deivation of a pobable maximum contamination limit fo a test lot Assuming that esults on all (n) sample units ae negative, then fo a given pobability (p) the maximum incidence (d) of defective units is given by: d = 100(1 n (1 p) Hence, if n = 10 and p = 0.95, then 100(1 10 n d = (1 0.95) = 100(1 0.05) = 100( ) = 5.88%.

28 Page 9 of Knowing the maximum incidence of defective sample units and the size of the sample units we can deive a Maximum Contamination level (C) fom: C = ( d/100)(1/ W) oganisms pe g, whee W is the weight of the sample unit tested. Fo the example given above and assuming that each of the 10 samples weighed 5g, then the maximum contamination level would be given by C =(5.88/100)(1/ 5) = oganisms/g 10.4 oganisms/kg In othe wods, the failue to detect a positive in 10 paallel tests meely indicates, at a 95% pobability, that the index oganism would be pesent in not moe than 6% of simila samples thoughout the lot; and that the maximum contamination level would be 11 oganisms/kg of poduct It might be thought that such a level of poduct secuity is insufficient, in which case it would be necessay to analyse a geate numbe of sample units and ideally to test lage quantities of sample. It is essential also to ecognise that this pesupposes that the test method is pefect Multiple Test Most Pobable Numbe Estimates. If some test esults ae positive, then we can deive an estimate of population density (the basis fo deivation of a Most Pobable Numbe) fo multiple tests even at a single dilution level The following equation povides the deivation of the MPN: 1 s M =.ln, whee M = Most pobable numbe, V = quantity of sample, s = V n numbe of steile tests out of n tests inoculated Assume 10 tests ae set up on eplicate 5g samples of poduct, 3 tests ae positive and 7 ae negative. Then the MPN of contaminating oganisms is: M 1000.ln 7 = = = oganisms/kg 14 oganisms/kg Unfotunately, it is not possible to deive an estimate of the eo of the MPN when tests ae done at a single dilution level Level of Detection Estimates. The equation used in 3.5 is also the basis fo deiving MPN values fo use in the Speaman-Kabe pocedue to estimate the LOD 50 fo a test. This is the level of oganisms that will give 50% positive esults when tested by an appopiate potocol. Details of the pocedue togethe with woked examples ae given in the epot by Hitchins (005). This method of quantification has the benefit that it is possible to deive a value fo the standad eo of the mean (sic LOD) estimate. The pocedue can be used to compae pefomance of two o moe