technical guide Uncertainty of Measurement, Precision and Limits of Detection in Chemical and Microbiological Testing Laboratories

Transcription

1 techical guide Ucertaity of Measuremet, Precisio ad Limits of Detectio i Chemical ad Microbiological Testig Laboratories Published by: Iteratioal Accreditatio New Zealad 66 Great South Road, Greelae, Aucklad 1005 Private Bag 8 908, Remuera, Aucklad 1136, New Zealad Telephoe Facsimile ifo@iaz.govt.z Iteret: AS TG5, May 004 ISBN: Copyright Iteratioal Accreditatio New Zealad 003

2 Cotets 1 Itroductio 3 Some Defiitios 4 3 IANZ Policy 6 4 Geeral Iterpretatio Ad Guidace 6 5 Estimatig Ucertaity Of Measuremet Step By Step 1 6 Expadig Ucertaity Ad Cofidece Limits 1 7 Comparig A Test Result With A Fixed Value 3 8 Comparig A Test Result With A Natural Populatio 5 9 Comparig Two Test Results 6 10 Criterio Of Detectio 6 11 Limit Of Detectio 7 1 Limit Of Quatificatio 7 13 Ackowledgemet 8 14 Refereces 9

3 1 Itroductio Decisios relatig to idustrial productio ad also legal decisios are ofte based o test results. They ca ivolve large sums of moey or eve a perso s liberty. It is therefore importat, ot oly for the aalyst, but also for their cliets or the decisio makers to kow how reliable test results are. What cofidece ca be placed o results, which idicate that a product is usatisfactory or that a perso has committed a offece? Some New Zealad chemical ad microbiological testig laboratories do ot kow what cofidece they ca place i their results ad therefore misiterpretatio of isigificat differeces is resultig i icorrect busiess decisios. Whe aalysts look at the results, they are faced with a large list of possible sources of variatio, some causig a systematic bias to the results ad some beig radom i ature. The possible combied extet of these variatios eeds to be kow for each test method at typical aalyte levels before the aalyst ca kow whether their method ad results are fit for purpose or ot. Kow biases should be corrected (uless that is ot the covetio for the method). The remaiig compoets of variatio should be evaluated so that a overall ucertaity estimate may be made. The geeral accreditatio criteria for laboratories ISO/IEC 1705: 1999, states that, Testig laboratories shall have ad shall apply procedures for estimatig ucertaity of measuremet. I certai cases the ature of the test method may preclude rigorous, metrologically ad statistically valid calculatio of ucertaity of measuremet. I these cases the laboratory shall at least attempt to idetify all the compoets of ucertaity ad make a reasoable estimatio, ad shall esure that the form of reportig of the results does ot give a wrog impressio of the ucertaity. Reasoable estimatio shall be based o kowledge of the performace of the method ad o the measuremet scope ad shall make use of, for example previous experiece ad validatio data. This Iteratioal Accreditatio New Zealad (IANZ) guide explais the steps, which may be take to idetify ucertaity compoets ad to estimate the ucertaity of measuremets i the fields of Chemistry ad Microbiology. It also explais the use of precisio i such estimates. Limits of detectio ad compliace with specificatio are also discussed. Later sectios cotai some useful formulae ad examples for these fields. Readers are referred to the Eurachem / CITAC Guide for more details ad further examples o this subject. This documet is obtaiable from the Eurachem web site. The Eurachem documet suggests two approaches that may be used for the estimatio of ucertaity i chemical measuremets. The first is to idetify each source of ucertaity, estimate the ucertaity of each source separately ad combie them i the correct way to arrive at a overall ucertaity. The secod approach is to look for available precisio data or obtai suitable such data, the to idetify the sources of ucertaity which are ot icluded i this data, estimate the ucertaity of each additioal source ad the combie these with the precisio stadard deviatio. Telarc / IANZ sice the mid 1980s has bee ecouragig chemical testig laboratories to obtai precisio data ad to estimate stadard deviatios, cofidece limits ad limits of detectio from these data. It is therefore assumed that most such laboratories i New Zealad will have estimates of precisio ad preferably itermediate precisio for their tests. May will also have precisio data from iter-laboratory compariso programmes i which they participated or from published data. Microbiology laboratories will have or ca obtai test results for replicate samples aalysed preferably by differet people, media, icubators, etc but usually o the same day (morig the afteroo). Differet days ad iter-laboratory comparisos are a problem i microbiology because of the lack of stability of samples. Because of the availability of these data, the mai approach described here for the estimatio of ucertaity of measuremet i chemical ad microbiological testig is the secod Eurachem oe, startig with precisio data ad combiig this with ay missig compoets. Where chemistry laboratories are ivolved i characterisig referece materials or require a more rigorous approach, the estimatio of ucertaity for each cotributig item ad the the combiig of these results will be the preferred approach. 3

4 May New Zealad chemical ad microbiological testig laboratories are also ivolved i moitorig evirometal samples. I most cases, either the Health Departmet of a regioal coucil will have set limits for levels of pollutats that samples may cotai. Frequetly, laboratory results idicate that oe of a particular pollutat was detected. Such iformatio is, of course, useless if the limit of detectio for the method is above the limit set by the authority. This situatio may occur whe a laboratory has ot correctly calculated the limits of detectio for its methods. The ucertaity estimates for low levels may be exteded readily for the calculatio of limits of detectio ad this is preseted i the later sectios of this guide. Some Defiitios.1 Ucertaity of Measuremet (ISO / VIM 1993) A parameter associated with the result of a measuremet that characterizes the dispersio of the values that could reasoably be attributed to the measurad. Notes 1. The parameter may be, for example a stadard deviatio (or a give multiple of it), or the half-width of a iterval havig a stated level of cofidece.. Ucertaity of measuremet comprises, i geeral, may compoets. Some of these compoets may be evaluated from the statistical distributio of the results of series of measuremets ad ca be characterized by experimetal stadard deviatios. The other compoets, which ca also be characterized by stadard deviatios, are evaluated from assumed probability distributios based o experiece or other iformatio. 3. It is uderstood that the result of the measuremet is the best estimate of the value of the measurad ad that all compoets of ucertaity, icludig those arisig from systematic effects such as compoets associated with correctios ad referece stadards, cotribute to the dispersio.. Measuremet Traceability (ISO / VIM 1993) The property of the result of a measuremet whereby it ca be related to stated refereces, usually atioal or iteratioal stadards, through a ubroke chai of comparisos all havig stated ucertaities. ISO/IEC 1705 requires that For testig laboratories the requiremets give for calibratio laboratories apply for measurig ad test equipmet with measurig fuctios used, uless it has bee established that the associated cotributio from the calibratio cotributes little to the total ucertaity of the test result. Whe this situatio arises, the laboratory shall esure that the equipmet used ca provide the ucertaity of measuremet eeded. Where traceability of measuremets to SI uits is ot possible ad/or relevat, the same requiremets for traceability to, for example, certified referece materials, agreed methods ad/or cosesus stadards are required as for calibratio laboratories..3 Measuremet The set of operatios havig the objective of determiig a value of a quatity with a stated level of ucertaity Where a test yields, or is based o, a result, which is expressed i umerical terms, the testig process will be regarded as a measuremet for the purposes of this documet ad for the applicatio of the ucertaity clause i ISO/IEC Measurad The particular quatity that is subject to the measuremet..5 Empirical Test Method A test method where the result is ot traceable to a SI uit, but depeds o the defied steps of the method. A differet test method (ofte for the same amed aalyte) may give a differet result. A empirical test method is a test method iteded to measure a property, which is depedet o the test method used to measure it. Differet methods may retur differet results, which may ot be related. I may cases, the method caot be verified usig aother test method. Examples of empirical test methods are the leachable cocetratio of chemicals ad the hardess of a material. For the former, differet extractat ad leachig coditios will produce differet results. 4

5 For the latter, differet ideter shapes ad applied forces will produce differet results. A ratioal test method is a test method iteded to measure a property, which is defied idepedetly of ay test methods. There is a objective true value to that property ad the method ca be verified usig other test methods. Examples of ratioal test methods are those where the total cocetratio of a compoud i a sample is measured. It is recogised that although there is a objective true value, it may be very difficult to measure that value. All microbiology test methods are empirical as are may (most) chemical test methods (particularly where there is a digestio, extractio, chemical reactio or clea-up step or where steps i the method ivolve icomplete recovery of a aalyte ad there is o correctio for bias). Correctios for spike recovery results or use of similar compoud iteral stadards are attempts at correctig for bias ad makig the method closer to a ratioal method..6 True Value The value that would be obtaied by a perfect measuremet. This is idetermiate..7 Covetioal True Value The value attributed to a particular quatity ad accepted, sometimes by covetio, as havig a ucertaity appropriate for a give purpose. Frequetly, a umber of results of measuremets of a quatity are used to establish a covetioal true value..8 Accuracy or Trueess of a Measuremet This is the closeess of the agreemet betwee the result of a measuremet ad a true value of the measurad. Accuracy, ad particularly the extet of iterfereces must be assessed primarily durig the selectio ad, if ecessary, developmet of a test method. This requires detailed cosideratio of the ways i which various sample types may affect the measuremet. For well-researched methods, these aspects should have bee covered for the rage of sample types specified ad therefore specific checks withi each laboratory usig that method may ot be required. However, it is prudet to iclude accuracy checks such as ruig spiked samples ad referece samples, or split samples i iter-laboratory trials, for which the covetioal true values are well established. Whe itroducig a wellresearched method, the laboratory must still check that it gives reliable results, usually by use of certified referece materials ad replicates. Where a ew test method is developed or a method is applied to sample types for which its accuracy has ot bee assessed, detailed iterferece ad accuracy checks are ecessary. These may iclude; Detailed cosideratio of the chemistry ad physics of the method i relatio to various sample types Cross-checkig of results with completely idepedet methods Checkig the effect of addig iterferig substaces to kow samples Spike recovery tests Reaalysig samples from which the measurad has bee removed. Iter-laboratory comparisos Certified Referece Materials.9 Bias The systematic differece betwee results of measuremets ad the true value of the measurad..10 Precisio Closeess of agreemet amogst results of successive measuremets of the same measurad. Precisio icreases as radom variatios decrease. It is possible to have results, which are precise but ot accurate. They may be all close to a mea or average value but because of some systematic affect there is a bias ad all are much higher (or lower) tha the true value..11 Repeatability r Closeess of the agreemet betwee the results of successive measuremets of the same measurad carried out uder the same measuremet coditios (same laboratory, same sample, same method, same equipmet, same materials, same staff, same time) e.g. from duplicates withi the same batch. 5

6 .1 Reproducibility R Closeess of agreemet betwee the results of measuremets of the same measurad carried out uder chaged coditios of measuremet (differet laboratory, same sample, same method, differet equipmet, differet materials, differet staff, differet time) e.g. from iter-laboratory compariso results..13 Itermediate Precisio Closeess of agreemet betwee the results of measuremets of the same measurad carried out uder chaged coditios of measuremet but withi the oe laboratory (same laboratory, same sample, same method, differet equipmet, differet materials, differet staff, differet time) e.g. from replicate aalysis of Quality Cotrol samples or matrix referece materials over time..14 Criterio of Detectio This is the lowest result at which the aalyst may have (say 95%) cofidece that some aalyte has bee detected rather tha oe. This is about 1.7 times the stadard deviatio of low-level results (the sigle sided statistic is used)..15 Limit of Detectio If a test result is just below the criterio of detectio the the aalyst caot say that he has foud othig. The true value will be expected (say with 95% cofidece) to lie betwee zero ad two times the criteria of detectio (the cofidece iterval aroud the criteria of detectio). Therefore all that ca be said is that the true result is less tha this upper cofidece limit viz. twice the criteria of detectio. Therefore the Limit of Detectio is twice the Criteria of Detectio or about 3.4 times the stadard deviatio of low-level results..16 Limit for Quatificatio For a umerical value to be placed o a lowlevel test result, the lower cofidece limit associated with this value should be sigificatly above zero. For a result at the limit of detectio (say 10 uits), for a ormal distributio ad reasoable degrees of freedom, the 95% cofidece limits will be plus ad mius /3.4 of the result (e.g. 10 uits +/- 5.9 uits). Such a result may ot be helpful to the cliet. It may therefore be reasoable to specify that the cofidece limits for a reported umerical test result should be o more tha say oe quarter of that result. Therefore to achieve this the limit for quatificatio should be about 8 times the stadard deviatio of low-level results..17 Certified Referece Material Referece material accompaied by a certificate, oe or more of whose property values are certified by a procedure which establishes traceability to a accurate realisatio of the uit i which the property values are expresses, ad for which each certified value is accompaied by a ucertaity at a stated level of cofidece. 3 IANZ Policy It is the policy of IANZ that accredited testig laboratories shall comply with the requiremets of ISO/IEC 1705 i relatio to the estimatio ad reportig of ucertaity of measuremet i testig. The requiremets are stated maily i Clauses ad c) of ISO/IEC Procedures ad recommedatios stated i the documets referred to i the otes followig Clause are ot requiremets. The measuremet ucertaity defiitio i ISO VIM 1993 applies. 4 Geeral Iterpretatio Ad Guidace The followig guidace ad iterpretatio are based o the APLAC TC 005 Guide. 4.1 Tests for which Ucertaity Applies Where a test produces umerical results, or the reported result is based o a umerical result, the ucertaity of measuremet for those umerical results shall be estimated. I cases where the ature of the test method precludes rigorous, metrologically ad statistically valid estimatio of the measuremet ucertaity, a testig laboratory shall idetify all sigificat compoets of ucertaity ad make a reasoable attempt to estimate the overall ucertaities of those measuremets. This applies whether the test methods are ratioal or empirical. 6

7 Where results of tests are ot umerical (e.g., pass/fail, positive/egative or other qualitative expressios) estimates of ucertaity or other variability are ot required. However, laboratories are ecouraged to have a uderstadig of the variability of the results where possible ad especially the possibility of false egatives or false positives. The sigificace of the ucertaity of qualitative test results is recogised ad so is the fact that the statistical machiery required to hadle the calculatio of such ucertaity exists. However, i view of the complexity of the issue ad the lack of agreed approaches, IANZ does ot require laboratories to estimate the ucertaity of qualitative test results at this time but will keep this uder review. 4. Defiig the Measurad It is recogised that i chemical ad microbiological testig, the measurad is ofte defied i terms of the method (empirical method) ad is ot directly traceable to SI. Care is eed i defiig the measurad to esure that all ucertaity compoets will be idetified ad accouted for. Traceability for chemical test methods ofte has several compoets. Balaces, thermometers ad volumetric glassware are traceable to SI uits ad full ucertaity budgets should be available if such measuremets make a sigificat cotributio to the overall ucertaity. Traceability of the fial result calculatio is usually to a referece material which, although ofte ot complyig with the defiitio (VIM) of a certified referece material, should be the best available. Where specified sample preparatios, extractios, digestios, chemical reactios, clea-ups, temperatures, etc are icluded i the method ad o correctio for method bias (e.g. recoveries) is specified, the method is regarded as empirical ad is traceable to its specified istructios. Differet methods would give differet results. 4.3 Idetifyig the Compoets of Ucertaity The laboratory shall idetify all the sigificat compoets of ucertaity for each test. A idividual compoet cotributig less tha 1/5 to 1/3 of the total ucertaity of the measuremet will ot have much impact o the total ucertaity of the measuremet. However, it would become sigificat if the method icluded a umber of compoets of ucertaity of this size. Eve where reliace is to be made o overall precisio data, the laboratory shall at least attempt to idetify all sigificat compoets. This will provide iformatio to cofirm that the approach take is reasoable ad all sigificat compoets have bee accouted for. Flowchartig the steps of the test method ad usig fish-boe diagrams to preset the ucertaity compoets provide useful approaches. I some cases, groups of test method steps (similar sample type, same procedure, same weights ad same volumes) may be commo to several differet test methods ad oce a estimate of ucertaity has bee obtaied for that group of steps, it may be used i the estimates of ucertaities for all methods where the group applies e.g. sub-samplig ad sample preparatio compoets applied to a specific sample type. 4.4 Approaches to the Estimatio of Ucertaity There are various published approaches to the estimatio of ucertaity ad/or variability i testig. ISO/IEC 1705 does ot specify ay particular approach. Laboratories are ecouraged to use statistically valid approaches. All approaches, which give a reasoable estimate ad are cosidered valid withi the relevat techical disciplie, are equally acceptable ad o oe approach is favoured over the others. The followig paragraphs summarise of two approaches, which are described i the Eurachem documet Quatifyig Ucertaity i Aalytical Measuremet. Both the itermediate precisio ad reproducibility (from iter-laboratory comparisos) described i ISO 575 (see clause Note 3 of ISO/IEC 1705) may be used i estimatig testig ucertaity. However, these aloe may omit some ucertaity sources, which, if sigificat, should also be estimated ad combied. Method validatio ad verificatio data from repeat aalysis of matrix referece materials, i-house stadards, replicate aalyses, iterlaboratory compariso programmes, etc., will be useful i establishig method precisio. For chemical testig, a well-desiged itermediate precisio result or the precisio from a iterlaboratory compariso will ofte icorporate the major compoets of ucertaity. 7

8 For microbiological testig, i most practical cases, precisio will be the oly sigificat compoet ad the oly oe, which i practice ca be readily estimated. The ucertaity of physical measuremets, the purity of calibratio referece materials ad their ucertaities, the ucertaities associated with recovery (bias) trials (whe recovery factors are applied to results), as well as precisio data shall all be cosidered i the evaluatio of measuremet ucertaity for chemical testig. The ISO Guide to the Expressio of Ucertaity i Measuremet (GUM) (see clause Note 3 of ISO/IEC 1705) is regarded as the more rigorous approach to the estimatio of ucertaity. However, i certai cases, the validity of GUM estimates from a particular mathematical model may eed to be verified, e.g., through iter-laboratory comparisos. The Number of Sigificat Figures approach ad Note of ISO/IEC 1705 are ot cosidered suitable for the evaluatio of measuremet ucertaity i chemical or microbiological testig. 4.5 Degree of Rigour The degree of rigour ad the method to be used for estimatig ucertaity shall be determied by the laboratory i accordace with Note 1 of Clause of ISO/IEC To do this, the laboratory shall: Cosider the requiremets ad limitatios of the test method ad the eed to comply with good practice i the particular sector Esure that it uderstads the requiremets of the cliet (see clause 4.4.1a of ISO/IEC 1705). It is ofte the case that the cliet uderstads its problem but does ot kow what tests it requires or their ucertaity ad eeds guidace o the tests required for solvig its problem Use methods, icludig methods for estimatig ucertaity, which meet the eeds of the cliet (see clause 5.4. of ISO/IEC 1705). It should be oted that what cliets wat may ot be what they eed Cosider the arrowess of limits o which decisios o coformace with specificatio are to be made Cosider the cost effectiveess of the approach adopted. I geeral, the degree of rigour eeded is related to the level of risk that ca be tolerated. Rigorous cosideratio of idividual sources of ucertaity, combied with mathematical combiatio to produce a measuremet ucertaity is cosidered appropriate for the most critical work, icludig the characterisatio of referece materials. However, if a iappropriate model is used, this approach will provide a iadequate measuremet ucertaity ad is ot ecessarily better tha the followig approach. Estimatio of measuremet ucertaity based o the overall estimate of precisio through iter-laboratory studies, method validatio or other quality cotrol data, takig ito cosideratio additioal ucertaity sources will be commoly used for chemical ad microbiological testig. Additioal sources that eed to be cosidered may iclude sample homogeeity ad stability, calibratio/referece material, bias/recovery, equipmet ucertaity (where oly oe item of equipmet was used i obtaiig the precisio data). I geeral, if less rigour is exercised i estimatig measuremet ucertaity, the estimated measuremet ucertaity value should be larger tha a estimate obtaied from a more rigorous approach. Semiquatitative measuremets require less rigorous treatmet of measuremet ucertaity. Whe a compliace decisio is clear, the a less rigorous approach to measuremet ucertaity estimatio may be justified. If the estimated ucertaity i reported results meas that results will be uacceptable to the laboratory s cliet or decisio maker, or will be too large for determiatio of compliace with the specificatio, the laboratory should edeavour to reduce the ucertaity, e.g., through idetificatio of the largest cotributors to ucertaity ad workig o reducig these. 4.6 Collaborative Trials ad Proficiecy Testig Proficiecy testig precisio may ot always provide complete measuremet ucertaity data if sigificat aspects (e.g. samplig or sample homogeeity) have ot bee take ito accout. 8

9 Three examples are: Matrix differeces may occur betwee the proficiecy test samples ad the samples routiely tested by a laboratory. Result levels may ot be the levels routiely tested i a laboratory ad/or may ot cover the full rage of levels ecoutered i routie laboratory work. Participatig laboratories may use a variety of empirical methods (differet measurad) to produce the PT results. Statistical aalysis of PT results may give a idicatio of the precisio obtaiable from a particular method. 4.7 Ucertaity Arisig from Samplig Measuremet ucertaity strictly applies oly to the result of a specific measuremet o a idividual specime. Durig cotract review, there shall be cosideratio ad agreemet with the cliet as to whether the test result ad ucertaity are to be applied to the specific sample tested or to the bulk from which it came. Where samplig (or sub-samplig) is to be treated as part of the test (measurad), the ucertaity arisig from such samplig shall be cosidered by the laboratory. Estimatig the represetativeess of a sample or set of samples from a larger populatio requires additioal statistical aalysis. Where a test method icludes specific samplig procedures desiged to characterise a batch, lot or larger populatio, the measuremet ucertaities for idividual measuremets are ofte isigificat relative to the statistical variatio of the batch, lot or larger populatio. I cases where the measuremet ucertaity of idividual measuremets is sigificat i relatio to the samplig variatio, the measuremet ucertaity shall be take ito cosideratio whe characterisig the batch, lot or larger populatio. Where the test method icludes a specific subsamplig procedure, it is ecessary to aalyse the represetativeess of the sub-sample as part of the measuremet ucertaity estimatio. Where there is doubt about the represetativeess of a sub-sample, it is recommeded that multiple sub-samples be take ad tested to evaluate the represetativeess of each. Where oly oe sample is available ad is destroyed durig the test, the precisio of samplig caot be determied directly. However, the precisio of the measuremet system shall be cosidered. A possible method for estimatio of the precisio of samplig is to test a batch of homogeeous samples for a highly repeatable measurad ad calculate the samplig stadard deviatio from the results obtaied. 4.8 Reportig Measuremet Ucertaity If the laboratory has ot estimated the ucertaities of its measuremets the it will ot kow whether its test results are valid for the iteded use of the cliet. This is why ISO/IEC 1705 makes it a requiremet for laboratories to estimate ucertaities for all measuremets. However, there will be occasios whe the laboratory decides ot to report its ucertaities to its cliets. ISO/IEC 1705, however, requires reportig of ucertaities i some specific circumstaces. For quatitative test results, measuremet ucertaity shall be reported where required by clause b) of ISO/IEC 1705, which icludes the followig circumstaces: Whe it is relevat to the validity or applicatio of the result Whe a cliet s istructios so require Whe the ucertaity affects compliace with a specificatio limit Whe measuremet ucertaity is ot reported uder the provisio of the third paragraph of clause of ISO/IEC 1705, its absece shall ot affect the accuracy of the coclusio, clarity of the reported iformatio or itroduce ay ambiguity i the iformatio provided to the cliet. Measuremet ucertaity associated with results below the limit of quatificatio shall ot be reported, as curretly o reliable covetio for this purpose exists. The requiremet to report measuremet ucertaity whe it is relevat to the validity or applicatio of the test result will ofte eed to be iterpreted. I such cases, the cliet s eeds ad the ability of the cliet to use the iformatio may be take ito cosideratio. Although i the short term, some cliets will ot be i a positio to make use of measuremet ucertaity data, this situatio ca be expected to improve. 9

10 I this case, the laboratory may eed to provide a iterpretatio of the results if the cliet could be misled by the umerical results aloe. Whe reportig measuremet ucertaity the reportig format described i the GUM is recommeded. The results of the ucertaity estimatios would ormally be reported based o a level of cofidece of 95%. The idiscrimiate use of a coverage factor of is ot recommeded. Not all combied ucertaities are ormally distributed ad, where practicable, the ucertaity appropriate to the 95% cofidece level for the appropriate distributio should be used. The coverage factor used for calculatig the expaded ucertaity should also be reported. Whe reportig the test result ad its ucertaity, the use of excessive umbers of sigificat figures shall be avoided. Uless otherwise specified, the primary result shall be rouded to the umber of sigificat figures cosistet with the measuremet ucertaity. Whe the test method prescribes roudig to a level that implies greater ucertaity tha the actual measuremet ucertaity, the ucertaity implied by this roudig should be reported as the measuremet ucertaity of the reported result. Laboratories shall have the competece to iterpret measuremet results ad their associated measuremet ucertaity to their cliets. At preset, microbiological laboratories may decide ot to report ucertaity uless required by cliets. This is because ucertaities are very large ad cliets are ot yet prepared to uderstad ad accept these. Whe reportig microbiological ucertaities, a descriptio of the procedure used to estimate the ucertaity should also be icluded because, of the various methods used aroud the world, some give sigificat uderestimates of the actual ucertaity. 4.9 Determiig Compliace with Specificatio Decisios o whe ad how to report compliace or o-compliace vary accordig to requiremets of the cliet ad other iterested parties. However, the laboratory shall take its measuremet ucertaity ito cosideratio appropriately whe makig compliace decisios / statemets ad cliets shall ot be misled i relatio to the reliability of such decisios. I geeral, the priciples described i APLAC TC 004 shall be followed. Difficulties may arise where: Specificatios do ot quote the empirical method to be used ad a umber of methods for the aalyte but givig differet results are available. Where results are close to a specificatio limit oe laboratory may state that a sample complies with the specificatio (their cofidece iterval for the result excludes the specificatio limit) whereas aother laboratory may state that there is doubt as to whether or ot that same sample complies with the specificatio (their cofidece iterval icludes the specificatio limit) Assessmet for Accreditatio Durig assessmet ad surveillace of IANZ accredited laboratories, the assessmet team shall evaluate the capability of the laboratory to estimate the ucertaity of measuremets for all tests icluded i its accreditatio scope. They shall check that the estimatio methods applied are valid, all sigificat ucertaity compoets have bee icluded, ad all the IANZ criteria are met. The assessmet team shall also esure that the laboratory ca achieve the claimed limits of detectio Additioal Iterpretatios for Chemical Testig For empirical methods, the method bias is by defiitio zero ad oly idividual laboratory ad measuremet stadards bias effects eed to be cosidered. Where results are corrected for recovery (from spikig experimets), the ucertaity associated with these spikig results should be icluded. It is importat to esure that all appropriate effects are covered but ot double accouted. Where appropriate, effects of matrix type ad chages i cocetratio should be icluded i the measuremet ucertaity. Where proficiecy testig precisio data are used, it is importat to esure that the data used to estimate the measuremet ucertaity are relevat. I particular, they should relate to the same measurad (same test method ad matrix). 10

11 There may be issues if the proficiecy programme icludes laboratories with a wide rage of backgrouds ad skill levels. I chemical testig, it is customary to evaluate the ucertaity at various selected levels of the aalyte. However, whe a measuremet is beig made to test for compliace with limit values, it is ecessary to use a ucertaity value for test results close to the compliace limit. Therefore, it is useful to select the limit values as the levels at which the ucertaity is evaluated. This approach is most likely to provide the best estimate of the measuremet ucertaity at levels adjacet to the limit values. Professioal judgemet may be used for estimatig the magitude of ucertaity attributed to certai compoets where better estimates are ot available or readily obtaiable. I such cases, at least a shortterm precisio estimate of the compoet should be icluded i the evaluatio. Professioal judgemet should ot ormally be used for sigificat ucertaity compoets. Where professioal judgemet has to be used for sigificat compoets, it must be based o objective evidece or previous experiece. Measuremet ucertaity estimates cotaiig sigificat compoets evaluated by professioal judgemet shall ot be used for applicatios demadig the most rigorous evaluatios of ucertaity. 4.1 Additioal Iterpretatios for Microbiological Testig There are four mai types of microbiological tests; Geeral quatitative procedures MPN procedures Qualitative procedures Specialist tests, e.g. pharmaceuticals. Various approaches to estimatig ucertaity are available for geeral quatitative testig. Quatitative microbiological methods are empirical because results are depedat o method defied coditios such as temperature, icubatio period ad media. Therefore, they do ot have a method bias cotributor to their measuremet ucertaity. A few certified referece materials are available for quatitative microbiological tests. Where they are available, their certified results are mostly obtaied from collaborative studies. Therefore oly cosesus values relatig to the specified method are available ad, as with all other microbiological test methods, these methods are also empirical. These referece materials are useful i demostratig competece of the laboratory but should ot be used to quatify bias Use of Precisio Data The use of precisio data as described i the Eurachem / CITAC Guide is applicable for microbiology. Itermediate precisio as set out i ISO 575 is cosidered to icorporate most if ot all of the sigificat ucertaity compoets of microbiological tests. Ay compoets of ucertaity ot icluded i itermediate precisio, such as performace of differet batches of media, variatio i icubator coditios (where oly oe icubator is available), etc., may be examied for sigificace by other statistical meas or by redesig of the precisio exercise. The distributio of results from plate cout tests is ot ormal but skewed (log right tail). Such data may first be trasformed by takig the logarithm 10 of each result, to obtai a close to ormal distributio. The log stadard deviatio ad cofidece limits may the be calculated before ati-loggig each cofidece limit separately. Where other compoets of ucertaity such as samplig or equipmet variatios eed to be combied with a precisio result that was estimated usig logs ad ati-logs, sophisticated mathematical calculatios may be ecessary. For those plate cout tests where test results are less tha about 0 CFU, the result may well be below the limit of quatificatio (see Sectio 1) ad laboratories are cautioed about reportig umerical values for such results. At this stage, very little method performace data are available from collaborative trials or proficiecy testig, although this situatio may chage i the future. Proficiecy testig data may ot always provide suitable measuremet ucertaity data because sigificat aspects may ot have bee take ito accout: Matrix differeces may occur betwee the proficiecy test samples ad the samples routiely tested by a laboratory. Samples may have bee specially homogeised ad stabilised. 11

12 The populatio levels may ot be the levels routiely tested i a laboratory ad/or may ot cover the full rage of populatio levels ecoutered i routie laboratory work. Participatig laboratories may use a variety of empirical methods (differet measurad) to produce the PT results. However, statistical aalysis of PT results may give a idicatio of the precisio obtaiable from a particular method Most Probable Number (MPN) Procedures For MPN procedures it is traditioal to refer to McCrady s Tables to obtai the test result as well as the 95% cofidece limits. These data have bee established statistically but possibly without cosiderig all sources of ucertaity. Some versios of these tables also cotai sigificat errors arisig from the roudig of data durig their preparatio. Laboratories are ecouraged to idetify uusual combiatios of positive tubes ad to reject such results. If this is doe effectively, the the ucertaities quoted i the tables will, i the mea time, be regarded as a reasoable estimate of ucertaity for these methods. This may be regarded as a applicatio of Note of clause of ISO/IEC Note of Clause of ISO/IEC 1705 Applicatio of Note for some areas of specialist testig e.g., pharmaceutical microbiological assays, may also be applicable, as the methods cocered iclude validatio of the assay parameters, specify limits to the values of the major sources of ucertaity of measuremet ad defie the form of presetatio of calculated results Poisso Distributio The Poisso distributio ad cofidece limit approaches as described i BS 5763 ad ISO 718 may sigificatly uderestimate ucertaity as ot all sources of ucertaity are take ito accout. The distributio of particles / orgaisms i a liquid may be described usig the Poisso distributio but other compoets of ucertaity associated with the test procedure are ot icluded. Ideed some compoets of ucertaity such as dilutios ad cosistecy of readig plates will ot be described by a Poisso distributio. For oe-off aalyses, the Poisso distributio approach will give a quick estimate of the ucertaity (see Note to of BS 5763). However, this will likely give a sigificat uderestimate of the actual ucertaity Negative Biomial Model The egative biomial model described i ISO/TR may be more appropriate tha Poisso aloe as it covers the Poisso distributio plus over-dispersio factors. 5 Estimatig Ucertaity Of Measuremet Step By Step 5.1 Specify the measurad ad decide if it is empirical or traceable to SI A clear ad uambiguous statemet of what is beig measured must be prepared. Where the method is a empirical method (may chemical tests ad all microbiological tests), some careful thought is eeded because what is actually beig measured is usually ot what the ame of the method implies. For example, fat i milk by gravimetric determiatio (Roese Gottlieb) is actually all the substaces which extract from the milk sub-sample usig the specified solvet, time, temperature ad equipmet ad which remai after the dryig stage. If some oil or grease had spilled ito the sample these would also be measured as fat as would ay oily cotamiatios from reagets or equipmet. Some volatile fat compoets may also be lost durig the dryig stage. Maybe the cliet wats to kow the fat i the sample ad ot the sub-sample. This is a differet measurad ad the ucertaity of sub-samplig will eed to be take ito cosideratio. If the cliet wats to kow the fat i the shipmet the this a differet measurad agai ad homogeeity of the shipmet will eed to be take ito cosideratio. It is also oted that ot all coutries defie fat i terms of lipid cotet. They may also defie fat i terms of fatty acid cotet. Aother example is total petroleum hydrocarbos i soil. Perhaps the measurad is the material i the sub-sample, which is extracted by the specified solvet ad which absorbs ifra-red light at the specified wavelegth ad uder the specified coditios of dilutio, cell legth etc as compared with a specified mixture of hydrocarbos. Oe should ot forget the material, which is removed (presumably fats) if the clea-up step 1

13 is ivoked. But was t the cliet iterested i the whole sample ad ot the sub-sample? A microbiological example may be Listeria i Milk. The measurad may be the result of the specified calculatio from the coloies which are couted whe the sub-sample is resuscitated, diluted, plated, icubated (temperature ad time) ad otherwise treated as specified i the method Further examples of typical tests doe i chemical testig laboratories, which are empirical ad which require careful specificatio of the measurad are: Permagaate value Chemical Oxyge Demad Biochemical Oxyge Demad of Water Total Suspeded Solids of Water Weak Acid Extractable or Lead i Soil Available Phosphorus i Soil Crude or Dietary Fibre i Food Lea Meat i Sausages Total Orgaic Nitroge i Sausages Free Sulphur Dioxide i Wie Mercury i Fish Aflatoxis i Peauts Arseic, Atimoy, Cadmium i Ceramics Flash Poit of Avgas Acid ad Base Number of Oil Cholesterol i Blood Cyaide i Stomach Cotets Accelerats i Fire Debris. The list goes o coverig the large majority of all chemical test methods accredited by IANZ, which are empirical ad are ot traceable to the mole. Chage the method ad oe has a differet measurad givig a differet aswer. All require careful defiitio before oe ca decide which compoets of ucertaity should be icluded i ucertaity estimates. Questios also arise about whether the blak is pre-determied ad subtracted ad whether spike experimets give recovery results which are used to correct for method bias i a attempt to provide some traceability to the mole. 5. Write the formula for calculatig the results A example equatio for a pesticide i bread beig aalysed by solvet extractio followed by gas liquid chromatography usig a exteral stadard is, Isamp. cref. Vextr R = µ g/ g I. Rec. m Formula 1 Where R ref samp = Pesticide result i sample (mg/kg) I samp = Peak itesity of sample extract I ref c ref Vextr Rec = Peak itesity of referece stadard = Referece stadard cocetratio (ug/ml) = Fial volume of extract (ml) = Recovery Msamp = Mass of ivestigated sub-sample (g) 5.3 Idetify ad list all possible sources of ucertaity To assist with this process, it may be helpful to flowchart the etire method from samplig or sub-samplig to the fial result icludig listig all materials ad equipmet used. Items to ote are: Start at the begiig is samplig icluded i the measurad? Cosider all iputs such as set-up of equipmet ad preparatio of reagets Note cotrol items such as masses, volumes, temperatures, times, pressures cocetratios Note sub-samplig ad sample preparatio / dilutio / digestio / extractio steps Note referece materials ad their stated purities / values Are you usig stadard additios or spikes? Are you correctig for recoveries or usig these oly as quality cotrols? Are you doig duplicates ad reportig their meas? Are you correctig for a average blak or is a blak ru with each sample or batch? Are dilutios made prior to presetatio of samples to the istrumet? What are the plate coutig processes? Each step of the method should be examied to idetify possible sources of ucertaity. Variatios that relate to possible bias: Istrumet calibratio errors or drift Calibratio curve fittig errors Balaces, glassware or thermometer calibratios Referece material impurities Recoveries ad blaks. Radom variatios: Iadequate sample storage 13

14 Sample homogeeity, samplig, subsamplig, clumpig, dilutios Sample storage ad trasport variatios Sub-samplig Radom sample matrix effects, stability, variability, form of bidig of aalyte (differet to spike) Day to day differeces Reaget variatios Reactio stoichiometry departures from expected chemistry or icomplete reactios Measurig ad weighig Measuremet coditios temperature ad humidity effects o measuremets ad sample stability, etc Errors i equivalece poit detectio Variatios i volumetric glassware calibratios (toleraces) Repeatability of the method Istrumet settigs; readigs; other effects Electroic bleeps o istrumets ad computers Computatioal effects straight lie or curved calibratio ad roudig too soo Blak correctio blak ucertaity ad appropriateess to subtract blak Operator effects, colour chages, plate readigs Other radom effects (always iclude this). Errors which are ot cosidered for ucertaity estimates but which should be avoided or corrected: Icorrect sample Wrog cliet iformatio Cotaiers or exhibits mislabelled Samples preserved icorrectly Wrog method choice Errors o method card Wrog dilutios or dilutio factors Basic chemistry wrog Determiig icorrect species Matrix iterferece Wrog recovery factor applied Wrog chemicals or low quality chemicals Wrog uits used or reported Icorrect stadard solutios Calculatio programme errors Computer programme errors Reportig wrog species Number trasfer errors Calculatio of data etry errors Sample tube mix-ups Typig errors. 5.4 List available data Such data as calibratio certificate ucertaities, iter-laboratory, i-house or published precisio data, purity of referece materials, method validatio data, etc should be idetified. Examples are: Method validatio / verificatio data such as replicates of various samples or matrix referece materials uder various coditios (itermediate precisio) QC replicates or house stadard repeat aalyses uder various coditios (itermediate precisio) or same coditios (repeatability r ) Iter-laboratory compariso statistics (reproducibility R ) Published precisio data from method validatios (r ad R) Certificates of istrumet or referece material calibratios I-house equipmet calibratio data Published experimetal research data. 5.5 Select suitable precisio data Select available precisio data coverig the maximum umber of sources of ucertaity ad express these as a stadard deviatio. ILCP precisio (R) will probably iclude variatios from referece materials ad istrumet calibratios. However, it may ot iclude real sample ad sub-samplig variatios as samples are usually specially prepared, homogeised ad stabilised. Results from outlier laboratories may distort the R value. Itermediate precisio will probably ot iclude variatios i referece materials (same used for log periods) or larger equipmet (oly oe i the lab) or ucertaity i the calibratio of equipmet. If itermediate precisio is determied o real samples ad samples are split before the sub-samplig procedure the sub-samplig variatio will be icluded. Precisio data may ot iclude the full variatios i cotrol poits that are permissible withi the defied method, because most labs ted to set these poits to the cetre of the permissible rage (except durig robustess experimets for method validatio). Some measure of this may be desirable. Whe relyig o published precisio data (e.g. published R), the laboratory must demostrate that its ow precisio (r) is comparable with the published r. 14

15 If results are to be corrected for method bias / recovery (usig results from spikig studies) the the ucertaity of these spike tests should be estimated ad icorporated. Precisio frequetly varies sigificatly with the level of the result. The precisio data should be adjusted for cocetratio of the aalyte or a relatioship established betwee precisio ad cocetratio. For microbiological testig, precisio data derived from the skewed distributio will have the upper cofidece limit sigificatly further from the result tha the lower oe. 5.6 Calculate itermediate precisio Data quality The desig of the data collectio procedure is the most importat aspect of ay estimatio of itermediate precisio. Data must iclude variatios caused by differet days, operators, equipmet, laboratories, reagets etc if realistic estimatios of itermediate precisio are to be obtaied. Replicates aalysed by the same aalyst at the same time are likely to give a urealistically small precisio results (repeatability (r) oly). For estimatig reproducibility, results obtaied from several differet laboratories, each aalysig a sub-sample from the same sample or batch, are required. Itermediate precisio may be obtaied from aalyses by differet aalysts o differet days usig where possible differet equipmet, materials, stadards, etc ad usig stadard procedures usually applied i the laboratory. Use of a statistical formula without careful thought is udesirable. There eeds to be careful attetio to the sources of variability that should be cosidered i gettig a realistic desig for the collectio of data to estimate itermediate precisio. Because R ad itermediate precisio iclude r i additio to other sigificat compoets of precisio, a estimatio of oe or both of these will be the most useful. I some circumstaces a laboratory may be asked to certify the compositio of a batch of material. Here, samplig variatio ad product uiformity withi the batch become importat ad these variatios may be so large as to reder aalytical variatios isigificat Assumptios of ormality It is assumed throughout this paper that replicate results form a ormal distributio about a mea. There should, however, be coarse checks that this assumptio is reasoable. If there are few results, they should be plotted out alog a lie. If may, a histogram or similar display may be draw (see Figure 1). Alteratively, the use of ormal probability plots ca demostrate ormality of a data set. Figure 1 mea std dev Idicatios that the assumptio is ot valid are: several results lie well away from the rest a scatter of values that is quite asymmetric, e.g. values may be closely buched at the high or low ed of the scale. Microbiological replicate results are ofte foud to be buched at the low ed (log tail at the upper ed) of the scale. Such results may be trasformed by takig log 10 values of each test result to give a log distributio which is closer to ormal. Normal statistics may the be applied for the calculatio of precisio Replicate aalyses i oe laboratory I a good routie testig laboratory, it is ormal for at least a selectio of aalyses to carried out i duplicate or triplicate ( = or 3). This allows oly a very iaccurate estimate of stadard deviatio to be calculated for each sample. However, where a material such as a matrix referece material or a house stadard is aalysed repeatedly, ad especially uder varyig coditios, the estimatio of a stadard deviatio is more reliable. The formula is, ( y y ) S = Formula 1 15