Ελληνικό Στατιστικό Ινστιτούτο Πρακτικά 8 ου Πανελληνίου Συνεδρίου Στατιστικής (2005) σελ.44-449 UNCERTAINTY AND CONFIDENCE IN MEASUREMENT Ioannis Kechagioglou Dept. of Applied Informatics, University of Macedonia ABSTRACT Providing estimates of uncertainties related to engineering measurements involves the use of statistical methods and has become a necessity for all accredited laboratories. Accreditation bodies including the International Standards Organization (ISO) have introduced standards that require the above procedure. Various sources and types of uncertainty are present at any measurement undertaken but the total uncertainty at a certain confidence level has to be estimated. The choice of appropriate probability distributions for each partial and combined uncertainty is one of the main issues of concern. In this work the terms of standard uncertainty, standard combined uncertainty and expanded uncertainty are clarified and a procedure for obtaining estimates of these uncertainties is introduced. In this way a laboratory can design its uncertainty estimation procedure, where a compromise between speed and accuracy has to be made.. INTRODUCTION Obtaining estimates of measurement uncertainty has always been a critical task in engineering applications. In the past years however, accreditation bodies require that accredited laboratories should produce estimates of uncertainty of their measurements using accepted statistical and error analysis methods. Clause 5.4.6 of the ISO/IEC 7025:2000 standard, which focuses on electrical calibration procedures, is just an example that states clearly the above requirement. Apart from the typical requirements imposed by various national and international bodies, considering that measurement procedures may be involved in safety critical applications, one can understand the significance of this task. In this work a procedure is introduced with the aim of providing guidance with the estimation of measurement uncertainties in laboratories. Although electrical measurements were considered at all stages of this work, the results obtained can be easily modified to include cases of other types of measurement. - 44 -
2. EVALUATION OF TYPE A UNCERTAINTY COMPONENTS The uncertainty of measurement can be defined as "a parameter associated with the result of a measurement that characterizes the dispersion of the values that could reasonably be attributed to the measurand" [International Vocabulary of basic and general standard terms in Metrology (993)]. This parameter is usually a standard deviation or the width of a confidence interval. Experimental uncertainties that can be revealed by repeating measurements are called random or Type A. Statistical methods can give reliable estimates of random uncertainties through analysis of a series of observations. The relative importance of the random component of uncertainty is high, in relation to other contributions, if there is a significant spread in a sample of measurement results [Davis (2000)]. In this case the arithmetic mean or average of the results should be calculated. For n repeated independent values x,, x n for a quantity x, the mean value x is given by: n x = x j n j= The spread in the results is dependent on the instrument used, the method and probably the person taking the measurements and can be quantified by the standard deviation s(x) of the n values that comprise a sample of measurements taken. In general, the standard deviation σ of a finite population with N elements can be calculated as: σ = N N j= ( x j x) Given the statistical results of a single sample of n measurements, an estimate s(x) of σ is: s(x)= n n j= 2 ( x j x) If more samples are obtained, each will have a different value for the arithmetic mean and standard deviation. It follows from the central limit theorem that for large n, the sample mean approaches the normal distribution with mean the population mean. If samples of n measurements are taken after the estimation of the standard deviation of the population s, the standard deviation of the sample mean x is given by: s( x ) = s(x) / n (2) also called the standard error. In many cases it is not practical to repeat the measurement many times during a calibration or other measurement process. However, a prior measure of s can be used for obtaining an estimate of the random component of uncertainty [Hurll, Willson (2000)]. An experiment which involves a series of observations is run and an estimate of the standard deviation is obtained and used for all subsequent measurements. The 2 () - 442 -
instrument and method employed are therefore assessed through statistical analysis before any measurement is performed. In this way the standard deviation of the mean s( x ) can be obtained even if only one measurement is taken during the actual process. Equations () and (2) can be used for this purpose. Although reducing the number of measurements taken during the measurement process is beneficial for obtaining fast throughput times, it leads to greater standard errors therefore greater uncertainty values. Thus the sample sizes have to be determined according to the degree of accuracy and speed required for the process, where usually a compromise between these two parameters has to be made. When considering electrical measurements, for example, repeatability or random uncertainty components are mainly present due to electromagnetic interference and other kinds of electrical noise entering the system from the environment. 3. EVALUATION OF TYPE B UNCERTAINTY COMPONENTS Systematic or Type B components of uncertainty are associated with errors that remain constant while a sample of measurements is taken. These are usually determined based on scientific judgment using all the available information. This may include manufacturer s specifications, data provided in calibration reports and knowledge of the measurement process. While the normal distribution is used to describe random uncertainties, rectangular and triangular distributions are usually assumed for systematic uncertainties. If the upper and lower limits of an error are ±α without a confidence level and there is reason to expect that extreme values are equally likely, the rectangular distribution is considered to be the most appropriate (see Fig. ). In case that extreme values are expected to be unlikely, the triangular distribution is usually assumed [Bean (200)]. When an estimate of the maximum deviation which could reasonably occur in practice is determined, the value of the standard deviation has to be calculated in order to obtain the standard uncertainty value. The standard deviations are equal to α α for the rectangular and for the triangular distribution respectively. 3 6 -α μ-σ μ μ+σ +α Fig. : The rectangular distribution used to describe a Type B uncertainty based on the maximum expected error range ±α 4. COMBINED UNCERTAINTY Usually more than one parameter affects significantly the accuracy of a measurement being taken. All random or systematic sources of uncertainty associated with a measurement have to be quantified and then combined all together for the - 443 -
calculation of total uncertainty. Before combination all uncertainty contributions have to be expressed as standard uncertainties, which may involve conversion to the standard deviation from some other measure of dispersion [Guide to the Expression of Uncertainty in Measurement (995)]. Uncertainty components evaluated experimentally from the dispersion of repeated measurements are usually expressed in the form of a standard deviation, as described in section 2 and therefore no conversion is needed. When an uncertainty estimate is taken from specification data it may be in a form other than the standard deviation. A confidence interval is often given with a level of confidence (±α at p%) in specification sheets or calibration certificates. In this case the value α has to be divided by the appropriate divisor that corresponds to the given confidence interval in order to calculate the standard deviation. For example, a confidence level of 95% corresponds to two standard deviations, for a normally distributed variable. Therefore a division of the confidence interval by two will give us the value of the standard deviation. Uncertainties given in the form described above are called expanded uncertainties and will be further investigated in the next section. For some uncertainty components, error limits in the form ±α may be estimated. In this case the divisors 3 and 6 are used to determine the standard deviation for rectangular and triangular distributions respectively, as described in the previous section [UKAS (997)]. The divisors for the most commonly used probability distributions are summarized in Table. Probability Distribution Type Divisor Normal (confidence interval 68%) Normal (confidence interval 95%) 2 Normal (confidence interval 99%) 3 Rectangular 3 Triangular 6 Table : Common Probability Distributions and corresponding Di i Once the uncertainty components have been identified, estimated and expressed as standard deviations, the next stage involves calculation of the combined standard uncertainty u(y). The relationship between u(y) of a value y and the uncertainty of the independent parameters x, x 2,, x n upon which it depends is u(y) = n i= 2 c u( ) (3) 2 i x i where c i is a sensitivity coefficient calculated as c i = y/ x i, the partial derivative of y with respect to each of the standard uncertainty components x i. This coefficient describes how the value of y varies with changes in the parameters x i and therefore compensates for the contribution of a certain uncertainty component in the total uncertainty of a measurement [Barr, Zehna (983)] Equation (3) is also known as the law of propagation of uncertainty. - 444 -
5. EXPANDED UNCERTAINTY Usually a statement of confidence associated with a calculated total uncertainty is required. In practice this is the probability that a measured value and the corresponding uncertainty will define a range of values within which the true value of the measurand is included. The combined standard uncertainty is in the form of one standard deviation and therefore may not provide sufficient confidence [Laboratory Accreditation Bureau (200)]. For this reason the expanded uncertainty U is calculated by multiplying the standard uncertainty by a coverage factor k as follows: U = k u(y) (4) Expanded uncertainties provide intervals that encompass a larger fraction of the measurand value distribution, compared to that of the combined uncertainty. The result of a measurement is then reported in the form y ± U along with the confidence level, which depends on the coverage factor. According to the requirements of dominant accreditation bodies, the expanded uncertainty should provide an interval with a level of confidence close to 95% [ISO/IEC (2000)]. The choice of the coverage factor k is of critical importance as it determines the level of confidence associated with an uncertainty statement. Apart from the confidence level required, knowledge of the underlying distributions and the number of repeated measurements used for the estimation of random effects are issues to consider. In many cases the probability distribution characterized by the measurement result y and its combined standard uncertainty u(y) is approximately normal and u(y) itself has negligible uncertainty. If the above conditions are satisfied, a coverage factor of k = 2 defines an interval having a level of confidence of approximately 95 percent, and k = 3 corresponds to an interval having a level of confidence greater than 99 percent. This situation is encountered when the standard uncertainty estimates contribute comparable amounts to the combined standard uncertainty, random uncertainties are estimated using a sufficient number of observations and systematic errors are evaluated with high reliability. In this way the conditions of the Central Limit Theorem are satisfied and normality may be assumed. There are cases however that the combined standard uncertainty is dominated by a single contribution with fewer than six degrees of freedom. This usually happens when the random contribution to uncertainty is large in comparison to other contributions and the sample size used for its estimation is small [UKAS (997)]. It is then probable that the probability distribution will not be normal and the value of the coverage factor k will result in a confidence level that is smaller than the expected one. If this is the case, the value of the coverage factor that corresponds to the required confidence level has to be derived by considering the effective degrees of freedom v eff of the combined standard uncertainty. These can be evaluated using the Welsh-Satterthwaite equation: - 445 -
4 v u ( y) eff = (5) n 4 4 c j u j ( y) j= v j The effective degrees of freedom v eff are based on the degrees of freedom v j, which are equal to the number of measurements n taken during the prior evaluation of the variance, less. For systematic uncertainties, it is assumed that v j since the probability for the quantity considered laying outside the limits is very small. The combined standard uncertainty u(y), the sensitivity coefficient c i and the individual component uncertainties u j (y) are also included for the calculation of v eff as described in the above equation [Guide to the Expression of Uncertainty in Measurement (995)]. After the value of v eff is calculated using the above formula, the coverage factor k p, where p is the confidence probability in percentage terms, can be determined based on a t-distribution rather than a normal distribution. In general, the t-distribution is less peaked at the center and higher in the tails compared to the normal distribution, but it approaches it as the number of samples increases [Daniel, Terrell (995)]. Using the t-distribution tables, the entry value that is immediately lower than the calculated value of v eff is found in the degrees of freedom section and the corresponding value of t 95 is looked-up. The coverage factor k 95 is then taken equal to the t 95 value to ensure a 95% confidence level. 6. UNCERTAINTY ESTIMATION PROCEDURE Once all uncertainty components associated with an instrument are evaluated, manipulations have to take place on each of these values in order to obtain the combined and expanded uncertainty of a measurement. The procedure of calculating the expanded uncertainty from the individual component uncertainties is described in the flowchart of figure 2. After calculating the standard deviation of the repeatability uncertainty and obtaining tolerance limits for uncertainties related to systematic errors, the type of the probability distribution for each source of uncertainty has to be determined. All uncertainties can then be expressed as standard uncertainties using a divisor from table. The sensitivity coefficients c i are determined and formula (3) is then used to find the combined standard uncertainty. In case that the standard uncertainty of the repeatability component is more that 50% of the combined standard uncertainty, it is assumed that the latter is described by the t-distribution rather than the normal one [Davis (2000)]. Therefore the value of the coverage factor k is determined by looking at the t-distribution tables for the t 95 value that corresponds to the 95% level of confidence for the effective degrees of freedom of the combined standard uncertainty, as calculated using formula (5). Otherwise the value of the sample size of measurements has to be increased so that the standard deviation and hence the standard combined uncertainty of the repeatability component is reduced. If the standard uncertainty of the repeatability component is less than 50% of the combined standard uncertainty, normal distribution - 446 -
Determine probability distribution for each source of uncertainty Find corresponding divisor from table Determine coefficient c i Calculate all standard uncertainty components Calculate Combined Standard Uncertainty Is repeatability uncertainty > 50% of the Combined Standard Uncertainty? Yes Assume t-distribution for the Combined Standard Uncertainty No Calculate deg. of freedom for Combined Standard Uncertainty Assume that Combined Standard Uncertainty is normally distributed Find t 95 value from t- distribution tables and hence determine k Use k=2 to calculate normally distributed Expanded Uncertainty Use k= t 95 to calculate t- distributed Expanded Uncertainty Fig. 2: Flowchart for the Expanded Uncertainty calculation procedure. - 447 -
may be assumed and therefore a coverage factor of k= 2 may be used for a 95% level of confidence. Having determined the value of the coverage factor k, the expanded uncertainty for a confidence level of 95% is then obtained by substituting the values into formula (4). Its distribution type is assumed to be the same as the one for the combined standard uncertainty. Consider for example the uncertainty associated with a digital voltmeter. Type-A uncertainty components can be quantified by taking a series of ten measurements on a constant voltage source. Calculating the standard deviation we find the standard uncertainty related to repeatability u(x ) = 0.002Volts. Suppose that the error introduced due to limited resolution is ±0.0Volts. Also the manufacturer states an inaccuracy of ±0.05 Volts due to environmental effects. Since the distributions of both systematic inaccuracies may be assumed to be rectangular, the maximum errors have to be divided by 3 in order to find the standard errors u(x 2 ) = 0.0057Volts and u(x 3 ) = 0.029Volts. All three sensitivity coefficients can be set equal to one since all uncertainty components contribute equally. Formula (3) is then used to estimate the combined standard uncertainty u(y) = 0.0296Volts. Since the random component u(x ) is less than 50% of u(y) normal distribution may be assumed. If we assume that a confidence interval of 95% is required, we use a coverage factor k = 2 in equation (4) and the estimated expanded uncertainty is U = 0.0592Volts. 7. CONCLUSIONS Estimation of measurement uncertainty is a time consuming and demanding task for laboratories. A large number of experiments as well as research are required before all required uncertainties are obtained. This task however should not cause delays to other primary processes and operations. The requirement for uncertainty estimation should be considered even at the stage of work procedure and measurement method design so that all operations are performed in such a way as to reduce potential uncertainties and simplify their calculation. Since uncertainty estimation is based to a large extent on information provided by instrument manufacturers, only equipment with adequate information on associated uncertainties should be considered for use. It was also found that the calibration and drift since last calibration are much easier and more accurately calculated when the calibration of instruments is contracted to the same laboratory. The evaluation of both random and systematic uncertainties associated with measurement equipment becomes much easier once instruments of the same type are used by all bench engineers. Therefore consistency is required as far as subcontracting of calibrations and purchase of equipment is considered. The uncertainty estimation procedure presented in this paper has been implemented using an Excel spreadsheet supported by VBA code. The procedure was designed to facilitate its programming by setting a threshold for the decision variable. The estimation process can therefore be automated within any laboratory and become fully integrated within a measurement process. - 448 -
ΠΕΡΙΛΗΨΗ Η εκτίμηση της αβεβαιότητας που σχειτίζεται με μετρήσεις σε εφαρμογές μηχανικής περιλαμβάνει τη χρήση στατιστικών μεθόδων και καθιστάται απαραίτητη για κάθε διαπιστευμένο εργαστήριο. Τα σώματα πιστοποίησης, συμπεριλαμβανομένης της Διεθνούς Οργάνωσης Προτύπων (International Standards Organization, ISO), έχουν εισαγάγει προδιαγραφές που απαιτούν την παραπάνω διαδικασία. Διάφορες πηγές και τύποι αβεβαιότητας παρευρίσκονται σε κάθε μέτρηση που πραγματοποιείται. Η συνολική όμως αβεβαιότητα για συγκεκριμένο επίπεδο εμπιστοσύνης πρέπει να προσδιοριστεί. Η επιλογή των κατάλληλων πιθανοτικών κατανομών για κάθε μερική και συνδυασμένη αβεβαιότητα είναι απο τα κύρια ζητήματα αυτής της διαδικασίας. Σε αυτή την εργασία διευκρινίζονται οι έννοιες της τυπικής αβεβαιότητας, της συνδυασμένης αβεβαιότητας και της διευρυμένης αβεβαιότητας και εισάγεται μια διαδικασία για την εκτίμηση αυτών. Κατ αυτό τον τρόπο παρέχεται καθοδήγηση στα εργαστήρια ώστε να σχεδιάσουν μια κατάλληλη διαδικασία εκτίμησης της αβεβαιότητας, όπου αντιμετωπίζεται ένας συμβιβασμός μεταξύ ταχύτητας και ακρίβειας των εκτιμήσεων. REFERENCES Barr D. Zehna P. (983). Probability: Modeling Uncertainty. Addison-Wesley. Bean M. A. (200). Probability: The Science of Uncertainty. Brooks/Cole. Daniel, Terrell (995). Business Statistics, 7 th Edition. Houghton Mifflin. Davis P. W. (2000), ERA Technology Course Notes on Quality Systems in UKAS Accredited Laboratories. General requirements for the competence of testing and calibration laboratories (2000). ISO/IEC 7025:2000. Guidance for Documenting and Implementing ISO/IEC 7025 (200), Revision 3. Laboratory Accreditation Bureau. Guide to the Expression of Uncertainty in Measurement (995). Vocabulary of Metrology, Part 3. BSI. Hurll J. Willson S. (2000), Uncertainty of Measurement Training Course Material, UKAS. International Vocabulary of basic and general standard terms in Metrology. ISO, Geneva, Switzerland 993 (ISBN 92-67-075-) Mandel J. (984). The Statistical Analysis of Experimental Data. Dover Publishers, New York, NY. Taylor J. R. (997). An Introduction to Error Analysis, 2 nd Edition. University Science Books. The Expression of Uncertainty and Confidence in Measurement (997). UKAS Publications. - 449 -