It is now well understood that to make effective

Transcription

1 Factors Affecting Interpretation of Reliability Coefficients Leslie Russek, PT, PhD, OCS 1 It is now well understood that to make effective decisions based on patient data physical therapists need to know the psychometric properties of the clinical measurements they use. 5 What may not be as well understood is that the calculation of various reliability coefficients often reported in the literature are not all based on the same statistical concepts. This technical note explores how variability among subjects affects interpretation of various reliability coefficients and how different coefficients, used with the same data, can lead to paradoxical interpretation. Variability Among Subjects Researchers studying reliability have sometimes noted a difference between the intraclass correlation coefficient (ICC) calculated from the measurements of small angles, such as knee extension range of motion (ROM), and large angles, such as knee flexion ROM. Brosseau et al 1 state that To our knowledge, there are no studies that explain the difference between smaller and larger angles when considering intratester reliability, therefore, further research is needed. 1 In another study, researchers hypothesize that The lower reliability for knee extension could reflect the difficulty therapists have aligning the goniometer in extension. 6 Other researchers have hypothesized that the ICC is lower for knee extension than flexion because The knee extension arc is limited, and any error might, therefore, be magnified. Determining the anatomical landmarks may be difficult in patients with pathological changes in the knee. Knee extension itself may be highly labile and, therefore, hard to quantify Associate Professor, Physical Therapy Department, Clarkson University, Potsdam, NY. This work was initiated while author was a research physical therapist with Physiotherapy Associates, Glen Burnie, MD and completed at Clarkson University, Potsdam, NY. This research was supported by a demonstration grant from the APTA Section on Research and by Physiotherapy Associates. The protocol for this study was approved by the Institutional Review Board of Physiotherapy Associates. Address correspondence to Leslie Russek, Physical Therapy Department, Clarkson University, Potsdam, NY Lnrussek@ clarkson.edu The problem noted with reliability of small-value measurements is not limited to knee ROM. Researchers studying reliability of shoulder ROM have noted similar discrepancies between the ICC for shoulder flexion or abduction (large angles) and extension (smaller angles). 9 These researchers attributed the poor extension ICC to the fact that Extension may not be measured frequently in the clinic, which could have contributed to the poor intertester reliability. Although the cause of the poor intertester reliability for measurements of extension cannot be identified from our data... 9 Although researchers have often explained differences among ICCs between large and small joint angles as being due to physiological factors or measurement error, in actuality some or all of the difference among the magnitude of the coefficients may be due to differences in variability among subjects rather than variability in measurement error. Portney and Watkins emphasize that the ICC greatly depends on total subject variability and recommend that reliability studies use subjects having a wide range of scores. 8 However, because not all physical therapy measurements have the same amount of variability among subjects, not all reliability research has been able to follow this recommendation. Paradoxical Results Different reliability coefficients may give paradoxical results in which a measurement has good reliability according to one coefficient but poor reliability according to another. In particular, reliability estimates computed using the ICC may seem to contradict reliability estimates computed using the standard error of measurement (SEM). Stratford 13 notes a paradox in which the ICC suggests better reliability of knee flexion than extension, yet the SEM suggests better reliability of extension than flexion. He explains this paradox by saying that a good ICC indicates a measurement better for differentiating one subject from another, while a good SEM shows Journal of Orthopaedic & Sports Physical Therapy 341

2 that a measurement is better for differentiating a measurement in one subject from another measurement in that subject. 13 This paradox is based on the characteristics of the ICC and SEM. The ICC is based upon the results of an analysis of variance (ANOVA), which separates the error into variability between individuals and variability within an individual (error due to repeated measurements). Because the ICC is a ratio of the error from repeated measures and the total variability, the ICC varies between 0.0 and 1.0, where 0.0 reflects no reliability and 1.0 reflects perfect reliability. The SEM is derived from the SD of the measurement error and is proportional to the total variability. The SEM provides an estimate of reliability in the units measured (eg, degrees for ROM). 3,8,10,14 Because SEMs are reported in the units of the raw data, the SEMs for different types of measurements cannot be compared to one another as can ICCs. Table 1 provides a summary of the characteristics of the ICC and SEM. The SEM is clinically useful because it helps us determine the amount a second measurement would need to differ (from an initial measurement) to be confident that the change is not just due to test-retest error. For example, if investigators looking at the intrarater reliability of passive knee extension ROM measured using an electrogoniometer found an SEM of 0.988, the minimum difference to be confident of real patient change would be 2.7 (minimum detectable change is computed as SEM). 3,10 In this example, a second measurement would need to differ from the first by more than 2.7 for a therapist to be confident that the difference was due to real patient change and not just error in the 2 measurements. The current work addresses 2 issues related to reliability: (1) how variability among subjects affects interpretation of reliability coefficients and (2) how different coefficients can yield paradoxical results. In spite of warnings that the ICC is sensitive to total subject variability, reliability studies continue to attribute the lower ICC for knee extension to greater physiological TABLE 1. Comparison of the intraclass correlation coefficient (ICC) and standard error of measurement (SEM). ICC SEM Purpose Comparing differences between groups Comparing changes in an individual Units Normalized (unitless) In units of measurement Range Ideal Closer to 1 Closer to 0 Basic equation (S E)/(S+E) (S+E) (1 r) Abbreviations: S, subject variability; E, error variability; r, reliability coefficient. variability and measurement error. Few studies contrast ICCs and SEMs when these coefficients appear to show contradictory reliability. The current work uses actual and simulated knee ROM data to illustrate how the ICC and SEM reflect different aspects of reliability, even in the case when the test-retest error is the same. The objective is to provide readers with a clear understanding of the differences between the ICC and SEM, and to illustrate the effect of subject variability on these reliability coefficients. METHODS Subjects A total of 32 therapists in 9 clinics participated in 76 episodes of data collection on 38 subjects. Therapists collected data on between 1 and 6 subjects (mean, 2.4). Therapists had an average of 8.7 years of experience with 4.8 years in their current positions. Forty-nine percent had entry-level Bachelor s degrees and 51% had entry-level or advanced Master s degrees. Among the participating therapists, 82% worked full time. Subjects were drawn from patients whose evaluation or re-evaluation included ROM measurements of the knee. Potential subjects were excluded if they had acute or severe pain (arbitrarily defined as a pain rating greater than 6/10) to exclude patients whose condition might be exacerbated by the repeated measurements and to minimize the chance that increased pain would impair patient performance during retest measurements. Informed consent was obtained from each subject. A total of 38 subjects with a variety of orthopedic knee conditions were included. Only measurements from the injured limb were used; in case of bilateral involvement, only the measurement for the left limb was used. The present data were collected as part of a larger study of reliability of routinely collected clinical data. The protocol for this study was approved by the Institutional Review Board of Physiotherapy Associates. Procedure The therapist who initially evaluated the patient was the treating therapist ; the therapist repeating the measurements was the retest therapist. For clinics with more than 2 therapists, each therapist was given a unique, randomly generated list of names of other therapists in the clinic. Once a treating therapist was identified from the list, he/she was randomly paired with a retest therapist by selecting the next name on the random-pairing list, as described by Rothstein et al. 11 If the identified therapist was unavailable, the next name was selected; the unavailable therapist remained the next name on the list to be selected next. This process generated an approximately equal number of all possible pairings, minimizing pairing bias. 342 J Orthop Sports Phys Ther Volume 34 Number 6 June 2004

3 TABLE 2. Mean and SDs used to generate simulated data and error values. Simulated retest data were computed by adding error value to simulated initial measurement. data (ie, injured and noninjured knee flexion and extension). The simulated data are shown in Appendix 1. Mean for data generation SD for data generation SIE SIF SNE SNF Error Abbreviations: SIE, simulated injured (knee) extension; SIF, simulated injured (knee) flexion; SNE, simulated noninjured (knee) extension; SNF, simulated noninjured (knee) flexion. Therapists collected any combination of active or passive knee flexion or extension ROM measurements, as they deemed appropriate for that patient. Data were thus collected for active extension (AE), active flexion (AF), passive extension (PE), and passive flexion (PF). Therapists were asked to record range measurements to the nearest degree. Measurements were done in a repeated-measures (test-retest) manner; the treating therapist indicated which measurements had been performed and the retest therapist repeated these measurements, typically within 20 minutes. The order in which measurements were performed was not controlled. To avoid bias, therapists collected their data on separate sheets of paper and were not allowed to watch each other conduct the measurements or to consult with each other prior to completion of both sets of data. Generation of Simulated Data Simulated data were generated to more clearly illustrate how the different components of variability contribute to the differences among the reliability coefficients. All joint ranges were based on normally distributed random numbers, rounded to the nearest integer, with means and SDs selected to produce values similar to the actual data (Table 2). Hence, simulated injured flexion (SIF) and simulated injured extension (SIE) values were generated to reflect a distribution similar to the actual injured knee range of data presented here. Simulated noninjured flexion (SNF) and simulated noninjured extension (SNE) values were generated to reflect typical ranges seen in noninjured knees. Simulated error (difference between simulated initial measurement and the simulated retest measurement) was randomly generated based on a normal distribution. A mean of 0 and SD of 7 provided simulated error values that were similar to those seen in the actual data. The randomly generated difference was added to each test measurement to compute the simulated retest measurement. The same simulated error was used to compute the retest measurement for all simulated data. Consequently, the error between test and retest measurements is identical for each set of simulated Data Analysis Data were analyzed using ICC 1,1. ICC 1,1 was the most appropriate ICC model and form for the experimental design used in this study. 8,12 Model 1 (indicated by the first number) is appropriate when testers are selected at random from a larger pool of testers; here, 2 therapists were selected from the pool of participating therapists. The second number refers to the form, which in this case was a single measurement rather than a mean of several measurements. A separate 1-way analysis of variance (ANOVA) was computed for each actual data set (AF, AE, PF, and PE) and for each simulated data set (SIF, SIE, SNF, and SNE). Using the results of the ANOVA, ICC 1,1 for each data set was estimated using equation 1. 7,8 Equation 1. ICC 1,1 = BMS WMS BMS+(k 1)WMS, where BMS is the between mean square and WMS is the within-mean square from the 1-way ANOVA. In the current study k, the number of testers (therapists), is 2. The SEM was computed using equation 2. 8 Equation 2. SEM = 2 (1 ), where is the SD and is the reliability coefficient. The SD was computed for both test and retest data. The reliability coefficient used in computing the SEM is based on the intended use of the SEM 8 ; for this study, the corresponding ICC 1,1 value was used as the reliability coefficient because the goal was to compare the ICC and SEM as measures of interrater reliability. RESULTS Simulated and actual data are shown in Appendices 1 and 2, respectively. The actual and simulated data are similar for the injured knee, with knee extension angles clustered near 0 while knee flexion angles were distributed through a wide range of values. Simulated flexion and extension measurements were distributed over a wider range for injured knees than for noninjured knees. Tables 3 and 4 show the minimum, maximum, mean, and SD of the actual and simulated data, respectively. Equation (3) shows the computation of the ICC for the measurements of passive knee extension (PE), using relevant mean-square values from Table 5. The ICCs for the other data sets, computed using the mean-square results in Tables 5 and 6, are given in Table 7. J Orthop Sports Phys Ther Volume 34 Number 6 June

4 TABLE 3. Characteristics of actual data. Values are in degrees. PE PF AE AF Test Restest Error* Test Restest Error* Test Restest Error* Test Restest Error* Mean SD Minimum Maximum Abbreviations: PE, passive knee extension; PF, passive knee flexion; AE, active knee extension; AF, active knee flexion. * Error is the difference between test and retest; mean, SD, minimum and maximum are for the error values. TABLE 4. Characteristics of simulated data. Values are in degrees. SIE SIF SNE SNF Error* Test Retest Test Retest Test Retest Test Retest Mean SD Minimum Maximum Abbreviations: SIE, simulated injured (knee) extension; SIF, simulated injured (knee) flexion; SNE, simulated noninjured (knee) extension; SNF, simulated noninjured (knee) flexion. * Error is the difference between test and retest; mean, SD, minimum and maximum are for the error values. Equation 3. PE (ICC 1,1 )= BMS WMS = =0.5 BMS+(k 1)WMS (2 1)14.24 Using the relevant ICC as the reliability coefficient, equation (4) shows the computation of the SEM for knee PE (where the SD used is computer for combined test and retest data). The SEMs for the other data sets are given in Table 7. Equation 4. PE (SEM) = 2 (1 ) = (1 0.50) = 3.76 The ICCs for extension measurements ranged between 0.15 (SNE) and 0.67 (SIE); the ICCs for actual data were 0.50 and 0.59 for PE and AE, respectively. The ICCs for flexion measurements ranged between 0.46 (SNF) and 0.97 (SIF); the ICCs for actual data were 0.95 and 0.94 for PF and AF, respectively. The SEMs for the knee angle measurements made by physical therapists (PE and AE) were 3.8 and 4.9, respectively. The SEMs for PF and AF were 6.8 and 6.7, respectively (Table 7). The SEMs for simulated knee flexion and extension data were all between 4.9 and 5.0. DISCUSSION Variability Among Subjects The present work indicates that the test-retest ICC was better (higher) for knee flexion than extension for both actual and simulated data. Previous literature also reports higher ICCs for knee flexion than for extension. Brosseau et al 1 found ICCs of 0.43 to 0.52 for knee ROM data with values close to 0 (ie, extension) and 0.82 to 0.88 for knee ROM values not close to 0 (ie, flexion). Clapper and Wolf 2 similarly found higher ICCs for knee flexion (0.95) than knee extension (0.85) and Watkins et al 16 reported a smaller difference between ICCs for knee flexion and knee extension (0.90 and 0.86, respectively). Hayes et al 6 found ICCs of 0.95 to 0.99 for knee flexion and 0.71 to 0.86 for knee extension in knees with osteoarthritis. Rothstein et al 11 reported ICCs of 0.84 to 0.92 for knee flexion (depending on goniometer and trial) and ICCs of 0.59 to 0.80 for knee extension. Rothstein et al 11 did a post hoc analysis separating those therapists who used similar measurement techniques from those who did not; results showed that the ICC was higher for those who used the same measurement technique ( ) relative to the ICC for those using different technique ( ). They concluded that the low ICC for extension could be attributed to differences in patient positioning. 11 However, others have analyzed the importance of patient position and found that it had only slight impact on reliability. 10 The current study shows that physiological or methodological differences between flexion and extension measurements are not necessary to have wide variation in ICC. The constant error in the simulated data was a larger proportion of the extension measurements (which were near 0) than the flexion measurements. The simulated data therefore had lower ICCs for extension than for flexion, even 344 J Orthop Sports Phys Ther Volume 34 Number 6 June 2004

5 though the simulated error did not have sources of variability, such as patient positioning. The low ICCs reported in the literature for knee extension and other small values may therefore reflect the impact of limited subject variability on the computation of the ICC. Fritz et al 4 compared reliability for ROM in people with injured and uninjured knees and found that the ICC for the injured knee was higher than for the uninjured knee in both flexion and extension. The ICC for the current simulated injured versus noninjured data shows that when injured joints have more intersubject variability, they will have higher ICCs because of the greater intersubject variability rather than because of a difference in actual testretest error (which was the same for the simulated injured and noninjured data). Reliability measures are, therefore, specific to the patient population tested and should only be applied to that population. Paradoxical Results Contrary to what one might expect, the current work found that the smaller (ie, better) SEMs did not correspond to measurements with higher (ie, better) ICCs. In spite of the differences in ICCs in the TABLE 5. One-way analysis of variance data for actual knee range of motion. Sum of Squares Mean Square Degrees of Freedom Passive extension (PE) Between subjects Within subjects Passive flexion (PF) Between subjects Within subjects Active extension (AE) Between subjects Within subjects Active flexion (AF) Between subjects Within subjects TABLE 6. One-way analysis of variance data for simulated knee range of motion data. Sum of Squares Mean Square Degrees of Freedom Injured extension (SIE) Between subjects Within subjects Injured flexion (SIF) Between subjects Within subjects Noninjured extension (SNE) Between subjects Within subjects Noninjured flexion (SNF) Between subjects Within subjects TABLE 7. Intraclass correlation coefficient (ICC) and standard error of measurement (SEM) for actual and simulated knee range of motion data. Motion ICC 1,1 SEM n Passive extension (PE) Passive flexion (PF) Active extension (AE) Active flexion (AF) Simulated injured extension (SIE) Simulated injured flexion (SIF) Simulated noninjured extension (SNE) Simulated noninjured flexion (SNF) n n J Orthop Sports Phys Ther Volume 34 Number 6 June

6 simulated data presented, the simulated data sets had equal SEMs (the small differences can be attributed to computation and rounding errors) because absolute error was the same for each simulated data set. The SEMs for actual data in the present study were smaller for passive and active knee extension (3.3 and 4.5, respectively) than for flexion (6.9 and 6.0 ). The smaller SEMs suggest that extension measurements were more reliable than flexion measurements in the present study. Fritz et al 4 found the same pattern of SEMs for knee extension and flexion: the SEM for injured knees was 1.7 for extension and 3.9 for flexion. Only 2 reports could be found in the literature comparing SEM and ICC data for knee ROM. Fritz et al 4 obtained a similar paradoxical result: higher ICCs in measurements with greater variability compared to data with less variability; lower SEMs in measurements whose value was small (relative to the magnitude of error) compared to measurements with values that were large (relative to the magnitude of error). However, they did not address this discrepancy in their discussion. Stratford and Goldsmith 14 noted a similar phenomenon by computing SEMs from the data presented in Hayes et al 6 to demonstrate that ICCs are better for distinguishing among subjects while SEMs are better for assessing error in repeated measures. The simulated data presented here illustrate how the different nature of the SEM and ICC results in this paradox. The ICC compares the error due to repeated measures to the total variability in the data (ie, a ratio of variances). Mathematically, this comparison is done by dividing the error from repeated measures (S E) by the total variability (S + E). This can be seen in (equation 5), which is a simplified version of the ICC computation provided in equation 1. Equation 5. ICC 1,1 = S E S+E, where S is the variability among subjects and E is the error variability. If S equals BMS and E equals WMS, equation 5 reduces to equation 1 for a single measurement repeated 1 time. When the subject variability is large relative to the error variability (S E), the ICC approaches 1.0; this is the case with flexion data and simulated data from injured knees. When the error variability is the same order of magnitude as the subject variability (S E), the ICC approaches 0.0; this is the case with extension data and simulated data from noninjured knees. In contrast, equation 6 shows that the SEM is proportional to the SD, or total variability in the data (S + E). Equation 6. SEM=(S+E) 1 ICC Because total variability (subject plus error) was much smaller for knee extension than for flexion, the SEM was smaller for knee extension than for flexion. The SEM and ICC therefore have an almost inverse relationship to one another. 15 The different characteristics of these 2 reliability coefficients may create confusion about measurement error and have implications for clinical decision making. 5 CONCLUSION The simulated data here show that some or all of the differences previously noted among ICC for large and small value data could be due to differences in variability among subjects rather than differences due to physiological variability or measurement error. The ICC and SEM reflect different aspects of reliability: a ratio of variances versus consistency of measurement. As a result, the ICC and SEM can contradict one another: measurements may have good reliability as determined by their SEM, but may have poor reliability as determined by their ICC, and vice versa. As Roebroeck et al 10 have stated, Reliability is not an absolute quality of a measurement, but is dependent on the way a measurement will be interpreted. ACKNOWLEDGMENTS I would like to thank Michael Wooden, PT, MS, OCS of Physiotherapy Associates for his assistance in recruiting participating clinics, and all of the physical therapists who contributed data for this study. REFERENCES 1. Brosseau L, Tousignant M, Budd J, et al. Intratester and intertester reliability and criterion validity of the parallelogram and universal goniometers for active knee flexion in healthy subjects. Physiother Res Int. 1997;2: Clapper MP, Wolf SL. Comparison of the reliability of the Orthoranger and the standard goniometer for assessing active lower extremity range of motion. Phys Ther. 1988;68: Eliasziw M, Young SL, Woodbury MG, Fryday-Field K. Statistical methodology for the concurrent assessment of interrater and intrarater reliability: using goniometric measurements as an example. Phys Ther. 1994;74: Fritz JM, Delitto A, Erhard RE, Roman M. An examination of the selective tissue tension scheme, with evidence for the concept of a capsular pattern of the knee. Phys Ther. 1998;78: ; discussion Hayes KW. The effect of awareness of measurement error on physical therapists confidence in their decisions. Phys Ther. 1992;72: ; discussion Hayes KW, Petersen C, Falconer J. An examination of Cyriax s passive motion tests with patients having osteoarthritis of the knee. Phys Ther. 1994;74: ; discussion J Orthop Sports Phys Ther Volume 34 Number 6 June 2004

7 7. Hobbs FD, Parle JV, Kenkre JE. Accuracy of routinely collected clinical data on acute medical admissions to one hospital. Br J Gen Pract. 1997;47: Portney LG, Watkins MP. Foundations of Clinical Research. 2nd ed. Norwalk, CT: Appleton and Lange; Riddle DL, Rothstein JM, Lamb RL. Goniometric reliability in a clinical setting. Shoulder measurements. Phys Ther. 1987;67: Roebroeck ME, Harlaar J, Lankhorst GJ. The application of generalizability theory to reliability assessment: an illustration using isometric force measurements. Phys Ther. 1993;73: ; discussion Rothstein JM, Miller PJ, Roettger RF. Goniometric reliability in a clinical setting. Elbow and knee measurements. Phys Ther. 1983;63: Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psych Bull. 1979;86: Stratford P. Reliability: consistency or differentiating among subjects? Phys Ther. 1989;69: Stratford PW, Goldsmith CH. Use of the standard error as a reliability index of interest: an applied example using elbow flexor strength data. Phys Ther. 1997;77: Streiner D, Norman G. Health Measurement Scales. Oxford, England: Oxford University Press; Watkins MA, Riddle DL, Lamb RL, Personius WJ. Reliability of goniometric measurements and visual estimates of knee range of motion obtained in a clinical setting. Phys Ther. 1991;71:90-96; discussion J Orthop Sports Phys Ther Volume 34 Number 6 June

8 Appendix Appendix 1. Simulated knee range of motion data used in analysis. Values are in degrees. SIE SIF SNE SNF Subject Error* Test Retest Test Retest Test Retest Test Retest Abbreviations: SIE, simulated injured (knee) extension; SIF, simulated injured (knee) flexion; SNE, simulated noninjured (knee) extension; F, simulated noninjured (knee) flexion. * Difference between test and retest. Appendix 2. Actual knee range of motion data measured on patients. Values in degrees. PE PF AE AF Test Retest Error* Test Retest Error* Test Retest Error* Test Retest Error* J Orthop Sports Phys Ther Volume 34 Number 6 June 2004

9 Appendix 2. Actual knee range of motion data measured on patients. Values in degrees. (cont d.) PE PF AE AF Test Retest Error* Test Retest Error* Test Retest Error* Test Retest Error* Abbreviations: PE, passive extension; PF, passive flexion; AE, active extension; AF, active flexion. * Difference between test and retest. J Orthop Sports Phys Ther Volume 34 Number 6 June