Assessing short-term individual consistency using IRT-based statistics

Transcription

1 Pscológca (2010), 31, Assessng short-term ndvdual consstency usng IRT-based statstcs Pere J. Ferrando * Rovra Vrgl Unversty, Span Ths artcle proposes a procedure, based on a global statstc, for assessng ntra-ndvdual consstency n a test-retest desgn wth a short-term retest nterval. The procedure s developed wthn the framework of parametrc tem response theory, and the statstc s a lkelhood-based measure that can be consdered as an extenson of the well-known lz person-ft ndex. The ratonale for usng and nterpretng the proposed statstc s dscussed, and an adapted standardzed resdual at the tem level s also proposed to obtan clues about the possble causes of the detected nconsstency. The procedure s llustrated wth a real-data example and a parallel smulaton n the personalty doman. In recent decades n appled psychometrcs nterest n the assessment of the ntra-ndvdual consstency of the responses over a set of test tems has been growng. Wthn the framework of parametrc tem response theory (IRT), whch s the one consdered here, most of the research on ths topc has focused on developng and evaluatng global statstcs (.e. scalar measures based on the complete pattern of tem responses) that measure the extent to whch the answerng behavor of a respondent s consstent wth the psychometrc model whch was ftted to the data. These statstcs were ntally known as approprateness measures (Levne & Rubn, 1979), but now they are usually referred to as person-ft ndces (e.g. Meer & Stsma, 1995, 2001). Most of these model-based statstcs can be classfed n two broad classes (see Meer & Stsma, 1995). The measures n the frst class are based on a resdual that reflects the dfferences between the observed tem scores and the scores expected from the model. The measures n the second * Ths research was partally supported by a grant from the Spansh Mnstry of Educaton and Scence (PSI /PSIC). Correspondence: Pere Joan Ferrando. Unversdad 'Rovra Vrgl'. Departamento de Pscología. Carretera Valls s/n Tarragona (Span). E-mal: pereoan.ferrando@urv.cat

2 320 P.J. Ferrando class are based on the lkelhood functon. In ths second class, two of the bestknown and most commonly used person-ft ndces are the lo and lz measures ntally proposed by Levne and Rubn (1979). Person-ft ndces were ntally ntended for the ablty and apttude doman and manly for practcal purposes. However, over tme they have also been used n the personalty doman (Ferrando & Chco, 2001, Rese, 1995, Rese & Flannery, 1996, Rese & Waller, 1993, Meer et al., 2008). The assessment of ntra-ndvdual consstency n personalty s more than a practcal ssue, because n ths doman consstency s a topc of central theoretcal relevance (e.g. Tellegen, 1988). In personalty measurement three types of ntra-ndvdual consstency are generally dstngushed dependng on the temporal framework n whch the assessment takes place (e.g. Fske & Rce, 1955, Lumsden 1977, Watson, 2004). The frst type s momentary consstency, whch s assessed by analyzng the responses of the ndvdual over the set of tems durng a sngle test admnstraton. The second type s short-term consstency, usually assessed by usng a two-wave or a mult-wave retest desgn wth a short retest nterval (generally from a few days to a few weeks). Fnally, the thrd type s long-term consstency, whch s concerned wth trat stablty, and whch s generally assessed by usng two-wave or mult-wave desgns wth long retest perods (e.g. Conley, 1984). Cattell (1986) made a more specfc dfferentaton and dstngushed between dependablty (a retest nterval shorter than two months) and stablty (a retest nterval of two months or more). Now, accordng to these dstnctons, standard person-ft ndces appear to be useful measures for assessng the frst type of consstency at the ndvdual level (Rese, 1995, Rese & Flannery, 1996, Rese & Waller, 1993). However, statstcs for measurng the other two types also seem to be of potental nterest. In partcular, ths paper s concerned wth the assessment of short-term ndvdual consstency. In personalty measurement, ths assessment s of both theoretcal and practcal nterest. Theoretcally, the degree of short-term consstency can be consdered partly as a property of the trat (Cattell, 1986), and, for a personalty theorst the dfferences of trats n ther degree of consstency s mportant nformaton. In expermental and clncal settngs, short-term desgns are commonly used to assess the effects of expermental condtons or treatments. Also, n the selecton doman, short-term desgns are used to gauge the effects of test-coachng and practce. The classcal approach for assessng short-term consstency n personalty s to assume that the ndvduals are perfectly stable over tme: hence, all nconsstency s due solely to measurement error (see e.g.

3 Short-term consstency 321 Watson, 2004). However, ths ratonale s surely too smple. Frst, even wth short retest ntervals, ndvdual transent fluctuatons such as mood changes, cogntve energy varatons, mental atttude changes, etc are expected to occur (Lumsden 1977, Schmdt & Hunter, 1996). Second, changes n the assessment condtons (e.g. nstructons, pressures or motvatng condtons) mght gve rase to true or smulated temporary changes n the trat levels (e.g. Schuldberg, 1990, Zckar & Drasgow, 1996). Thrd, even f condtons reman the same over dfferent admnstratons, the mere fact of havng responded at Tme 1 (pror exposure) mght lead to systematc changes at the retest (e.g. Knowles, 1988). Fnally, we also need to consder the retest effects, understood here as the tendency for ndvduals to duplcate ther former responses (Gullksen, 1950). Ths tendency mght be due to memory effects, or to ncdental tem features whch are unrelated to the trat, but whch tend to elct the same response on each occason (Thorndke, 1951). So, overall, t seems more realstc to assume that short-term nconsstency arses as a result of a complex tem respondent nteracton process (e.g. Ferrando, Lorenzo-Seva & Molna, 2001, Schuldberg, 1990). In spte of ths assumpton, however, most of the research on the causes of short-term nconsstent respondng has focused solely on tem characterstcs (see Ferrando et al for a revew). The present paper develops and proposes an IRT-based statstc for assessng short-term consstency at the ndvdual level. Ths statstc s lkelhood-based, and can be consdered as an extenson of the lz person-ft ndex mentoned above. It s ntended for bnary tems whch are calbrated wth a undmensonal IRT model. Of the exstng parametrc IRT models, bnary undmensonal models are the smplest and best known. So, they seem to be an approprate startng framework for developng new measures. The number of questonnares based on bnary tems s stll substantal n personalty, so the potental nterest of the statstc n appled research seems clear. The statstc proposed here s expected to be of both theoretcal and practcal nterest. At the practcal level, t can be used for flaggng ndvduals for whom the estmated trat level at Tme 1 mght be napproprate for makng vald nferences n the short term. A second practcal use s for detectng outlers (.e. nconsstent ndvduals) n longtudnal studes. It s well known that the estmated parameters and ft results n structural equaton models are easly dstorted f outlers are present (e.g. Bollen & Armnger, 1991). At a more theoretcal level, the statstc can provde addtonal nformaton about the nconsstency of the response behavor of the ndvdual beyond that provded n a sngle test

4 322 P.J. Ferrando admnstraton, and can be useful for assessng dfferent types of ndvdual change as well as retest effects. Ratonale and Descrpton of the lz-ch Statstc Consder a personalty test that measures a sngle trat θ and whch s made up of 1,, n tems scored as 0 or 1. The tem responses are assumed to behave accordng to a specfc parametrc IRT model, and are ndependent for fxed trat level (local ndependence). Let P (θ) be the tem response functon correspondng to the IRT model, and let x =(x 1..x n ) be the response pattern of ndvdual. Then, the log lkelhood of x s n L θ ) = x ln( P ( θ )) + (1 x ) ln(1 P ( θ )) (1) ( Furthermore, the mean and varance of L(θ ) are (Drasgow, Levne & Wllams, 1985) E n L( θ )) = P ( θ ) ln( P ( θ )) + (1 P ( θ )) ln(1 P ( θ )) (2) ( and n P ( ) θ Var ( L( θ )) = P ( θ )(1 P ( θ )) ln (3) 1 P ( θ ) 2 Levne and Rubn (1979) defned the lo person-ft ndex as the log lkelhood (1) computed usng the maxmum lkelhood (ML) estmate of θ. The ratonale for ths choce s that when a pattern s nconsstent, no value of θ makes the lkelhood of ths pattern large. So, even when evaluated at the maxmum, the value of (1) s stll relatvely small. A lmtaton of lo that can be noted by nspecton of (2) s that ts value depends generally on the trat level. Furthermore, the ndex has no mmedate theoretcal reference dstrbuton for assessng the values

5 Short-term consstency 323 obtaned. To overcome these problems, Drasgow et al. (1985) proposed a standardzed verson of lo, whch they named lz. It s gven by l z lo E ( lo ) = (4) [ Var ( l )] 1 / 2 o where the mean and varance are obtaned from (2) and (3) computed usng the ML estmate of θ. The standardzaton provdes a known scale and s ntended to reduce the dependency of the values on θ. Furthermore, because the terms n (1) are ndependent for fxed θ, as the number of tems ncreases the dstrbuton of lz s expected to approach the standard normal. Ths approxmaton requres some condtons to be fulflled (see below). As for nterpretaton, typcally only the left tal of the dstrbuton s consdered: large negatve values ndcate potental person msft. Large postve values would ndcate an over-consstent, determnstc (.e. Guttman-type) respondng, however they are seldom nterpreted. Consder now that the test s admnstered to the same respondents at two ponts of tme wth a gven retest nterval. The present development s based on the addtonal assumpton that the local ndependence prncple also holds for repeated measurements of the same tem, or, more specfcally, that for fxed θ the condtonal dstrbutons of the responses to the same tem n two repeated admnstratons are ndependent of each other. If ths s so, the condtonal probablty of a response change to tem s gven by Pch θ ) = 2 P ( θ )( 1 P ( θ )) (5) (.e. the probablty of the 1-0 pattern of change plus the probablty of the 0-1 pattern (Ferrando et al., 2001, Nowakowska, 1983). We shall now defne the ndcator varable y to denote response change to tem as 1, f response of tem was changed y = 0, otherwse at Tme2 (6) and y =(y 1..y n ) as the change response pattern of ndvdual. Under the assumptons stated so far, the log lkelhood of the change response pattern y s

6 324 P.J. Ferrando L n θ ) = y ln( Pch ( θ )) + ( 1 y ) ln( 1 Pch ( θ )). (7) ch ( By usng the same ratonale as Levne and Rubn s, the short-term consstency counterpart of the lz person-ft ndex, whch we shall name lzch, s now defned as l E ( l ) = (8) o ch o ch l z ch / o ch [ Var ( l )] 1 2 where lo-ch s the value of (7) evaluated at the ML estmate of θ, and the mean and varance are gven respectvely by E = n ( l ) ( ˆ ) ln( ( ˆ )) ( ( ˆ 0 ch Pch θ Pch θ + 1 Pch θ )) ln( 1 Pch ( θˆ )) (9) Var ( l 0 ch ) n = Pch Pch ( θˆ ) ( θ ˆ )( 1 Pch ( θˆ )) ln (10) 1 Pch ˆ ( θ ) 2 and θˆ s the ML estmate of θ. Because the terms n (7) are ndependent for fxed θ, under the null hypothess that all respondents follow the postulated model, the dstrbuton of lz-ch s expected to approach the standard normal as the number of tems ncreases. Ths expectaton, however, s based on the same two strong assumptons that Drasgow et al. used when dervng the dstrbuton of lz. Frst, the tem parameters are assumed to be fxed and known. Second, the ML estmate of θ s expected to concde wth the true value. To see these ponts, note that the Pch(θ) terms n (7) are treated as constants when dervng the mean and varance (9) and (10). And ths treatment s only strctly correct when the two above assumptons are met. The lz-ch statstc s ntended to be used n a retest desgn n whch the tems are calbrated at Tme 1, and the ndvduals are also scored at Tme 1

7 Short-term consstency 325 by usng the tem estmates as fxed and known values. It s also assumed that the sample at Tme 1 s large and representatve enough for the assumpton of known tem parameters to be reasonable. However, the problem of usng ndvdual trat estmates n place of the true levels stll remans. In the context of lz, some correctons have been proposed (e.g. Snders, 2001). At the moment, however, we shall not stll consder these correctons for reasons whch are dscussed below. In summary, lz-ch s ntended to be used n a stuaton n whch a large and representatve group of respondents s avalable at Tme 1. The IRT model s calbrated n ths group and the ft of the model s checked. Next, ndvdual trat estmates are obtaned at Tme 1 by takng the tem parameter estmates as fxed and known values. Some or all of the ndvduals of the group are then retested after a short-term retest nterval, and, for these ndvduals, the lz-ch values are obtaned based on the tem and person estmates at Tme 1, and the observed responses at Tme 1 and at Tme 2. These values are used to flag those respondents that are potentally nconsstent n the short term. As for possble cut scores, the standard normal dstrbuton of lz-ch requres the fulfllment of some condtons that can never be fulflled wth real data. As well as the condtons dscussed above (known tem and person parameters), the model s assumed to be correct and the data s expected to contan no truly nconsstent respondents (.e. the null hypothess stated above). Models, however, are at best reasonable approxmatons, and the data s expected to contan a certan unknown proporton of nconsstent respondents. So, t seems unrealstc to expect normal dstrbutons wth real data, and, n the wrter s opnon, t s napproprate to suggest cut scores based on theoretcal nomnal levels. Future ntensve research may make headway n ths respect, ncludng the correctons dscussed above. For the moment t seems better to consder the standard normal dstrbuton only as a useful reference. Interpretaton Issues and Further Extensons Inspecton of equatons (7) to (10) shows that lz-ch values are negatve when the respondent changes hs/her tem responses more often than expected. On the other hand, values are postve when the respondent tends to duplcate the responses gven at Tme 1 more often than would be expected gven the IRT model and hs/her trat estmate. Ths result suggests than n the lz-ch case both tals of the dstrbuton should be examned, as large and postve values would probably ndcate retest

8 326 P.J. Ferrando effects of the type dscussed above. Furthermore, examnaton of the lz-ch dstrbuton over respondents mght also be useful for assessng retest effects. If the retest nterval s too short for avodng memory effects, the respondents as a group would tend to behave more consstently than expected. As a result, the mean of lz-ch n ths group would be expected to shft toward postve values. As mentoned above, one of the ams of lz-ch s to provde more nformaton about consstency than can be obtaned wth lz. In prncple, t seems clear that both ndces assess dfferent aspects of consstency. The lz ndex assesses consstency on a same-tme/dfferent-tem bass, whereas lzch manly assesses consstency on a dfferent-tme/same-tem bass (although the consstency contrbuton s then summed over tems). In spte of ths dstncton, however, f the assumptons on whch both ndces are based are fulflled, then lz and lz-ch are probablstcally related. To see ths pont, consder a sngle tem. If the response to ths tem at Tme 1 s consstent, then lz wll be postve (.e. above the mean). If t s, accordng to the assumptons, the probablty of also obtanng a postve lz-ch value s greater than the probablty of obtanng a negatve value. On the other hand, f the response at Tme 1 s nconsstent, lz wll be negatve. And the probablty of also obtanng a negatve lz-ch value s greater than the probablty of obtanng a postve value. From these results t follows that n a group of consstent respondents, lz and lz-ch are expected to be postvely correlated. Ths result s theoretcally sound. It seems reasonable to expect that those respondents who are the most consstent when respondng n a sngle test stuaton also tend to be the most consstent under repetton. Indeed, the ndex proposed here wll be most useful when the respondent behaves nconsstently and there are dscrepances between lz and lz-ch. As a frst example, consder a stuaton reported n the lterature (Jorm, Duncan-Jones & Scott, 1989). The ndvdual responds consstently at Tme 1, but at Tme 2 hs/her motvaton decreases and he/she responds more randomly or mechancally. In ths case lz would probably ndcate consstency, but lz-ch would not. As a second example consder the thetashft model (Zckar & Drasgow, 1996). Assume that the admnstraton condtons are neutral on the frst occason but that there s a strong motvaton for fakng good on the second occason. Assume further that, under ths pressure, the ndvdual responds as f hs/her trat level s more adapted than t really s. In ths case lz would probablty ndcate consstency, perhaps even when t s computed on both occasons. However, because the lz-ch values are obtaned based on the person estmates at the

9 Short-term consstency 327 frst occason, lz-ch would be expected to flag ths respondent as nconsstent gven the temporary changes n hs/her trat level. Whle the nformaton obtaned wth lz-ch seems useful, once a response pattern has been detected as potentally nconsstent, t also seems to be useful to collect more nformaton about the specfc tem responses n whch the nconsstent respondng s more pronounced. Informaton of ths type can be obtaned by usng ndvdual tem statstcs. In partcular, I propose to assess nconsstency at the ndvdual tem level by adaptng the standardzed resdual statstc proposed by Wrght (Wrght & Stone, 1979) n the context of the Rasch model. In the framework used here (see equatons 5 and 6), the standardzed change resdual for respondent on tem, denoted by Zch s obtaned as y Pch ( θˆ ) =. (11) Z ch / [ Pch ( θˆ )( 1 Pch ( θˆ ))] 1 2 The Zch measures the tem response contrbuton to the short-term nconsstency of ndvdual. So, a large postve value would be obtaned f the ndvdual changed a response to an tem n whch a change was very unlkely gven the model and trat estmate. The am of ths paper s not to derve the specfc relatons between (8) and (11). However, we note that lzch n (8) can be re-expressed as a weghted sum of the dscrepancy terms n the numerator of (11) (e.g. Snders, 2001). So, t seems clear that the ndvdual tem nconsstences drectly contrbute to the global nconsstency as measured by lz-ch. As for possble cut scores, t should be stressed that the scalng n (11) (.e. zero mean and unt varance) s only made so as to make the resduals more nterpretable. However, t cannot be sad that the Zch values are dstrbuted as a standard normal varable because, n fact, Zch s smply a transformed bnary varable. So, no attempt s made here to provde crtcal values based on exact probabltes. What s expected, however, s that the Zch values wll be regular enough to dentfy outlyng responses relably, and ths pont s consdered n the llustratve example below. Fnally, as a reference, we may consder that a Zch value of +3 would be obtaned f a respondent changed a response to an tem for whch the expected probablty of change was as low as 0.10.

10 328 P.J. Ferrando An Illustratve Example The statstcs and procedure dscussed so far are llustrated usng data collected by the author n personalty research. A 60-tem Neurotcsm scale was admnstered twce n the same condtons to a sample of undergraduate students wth a 4-week retest nterval. The analyses that follow were based on the 436 respondents who were present at both admnstratons. As dscussed above, tems were calbrated and respondents were scored usng the Tme 1 data. The tems of the scale were domnance-based, of the type whch are generally well ftted by the two-parameter IRT model (2PM, see e.g. Ferrando, 1994). Items were calbrated accordng to the 2PM n the normal-ogve metrc by usng BILOG MG-3 (Zmowsk et al. 2003), and the undmensonalty assumpton and the global model-data ft were assessed wth NOHARM (Fraser & McDonald, 1988). The ft of the model, both at the global level and at the ndvdual-tem level was reasonably good (detals of the goodness-of-ft results are avalable from the author). The tem locatons ranged from to 4.15, wth a mean of The average of the tem dscrmnatons was The tem parameter estmates obtaned n the calbraton stage were used n a real-parameter smulaton study, whch was parallel to the man emprcal study. Ths parallel study smulated the responses of 436 ndvduals at two ponts of tme to a test that behaved accordng to the 2PM, wth tem parameter values equal to those obtaned n the calbraton of the emprcal data. In the smulated data all of the assumptons used n the dervaton of lz-ch were met: The model was totally correct, the tem responses were locally ndependent under repetton, and the sample dd not contan truly nconsstent respondents. So, the smulated results were used to check the predctons about the behavor of lz-ch and to assess the dscrepances wth respect to the real-data results. Fgure 1 shows the dstrbuton of lz-ch wth the smulated data (thck dashed curve) and wth the emprcal data (thck sold curve). The dstrbutons were obtaned by usng Gaussan kernel densty estmaton (Slverman, 1986), whch essentally provdes smoothed mproved hstograms. The dstrbuton based on the smulated data agreed qute well wth the expected standard normal dstrbuton. The mean and the standard devatons were 0.05 and The adherence of the lz-ch values to the correspondng theoretcal dstrbuton was further assessed by usng the Kolmogorov-Smrnov statstc. The Kolmogorov dstance was 0.05, wth an assocated probablty of 0.13.

11 Short-term consstency 329 Note:.. : Real data; : Smulated data Fgure 1. Dstrbuton of lz-ch wth the real data and the smulated data. At frst sght, the dfference between the curves n fgure 1 does not seem to be too great. However, the dstrbuton based on real data, (a) s clearly shfted to the rght (the mean was 1.38), (b) s somewhat asymmetrcal wth a heaver left tal, and (c) tends to have more dsperson than the standard normal dstrbuton (the standard devaton was 1.17). Prevous studes wth lz show that when ndvdual trat estmates are used n place of the true levels the varance usually decreases (Snders, 2001, van Krmpen-Stoop & Meer, 1999). So, the true effect (c) mght be even larger. Results (b) and (c) are to be expected. The dstrbuton of lz-ch s obtaned by assumng that there are no nconsstent respondents n the group whch s assessed. However, real data s expected to contan a certan, unknown proporton of nconsstent respondents. These are precsely the

12 330 P.J. Ferrando ones we wsh to dentfy and who are expected to be found n the tals (partcularly the left tal) of the dstrbutons, thus ncreasng the dsperson. As dscussed above, result (a) would be expected f respondents tended to behave more consstently as a group than the model predcts. To further assess ths ssue, the expected number of response changes was obtaned for each respondent by evaluatng (5) at the estmated trat level, and summng the values over the 60 tems. The values obtaned were then compared to the observed number of response changes. The medan of the expected number of changes over respondents was 20, whereas the medan of the observed values was 10. The respondents clearly tended to behave more consstently n the short term than the model and the trat estmates would lead us to expect. Overall, these results suggest that the 4-week nterval was nsuffcent to avod retest (probably memory) effects. The next factor to be assessed was the extent to whch the proposed standardzaton (8) acheved ndependence from the trat levels estmated at Tme 1. For the smulated data, the product-moment correlaton between the θ estmates and lz-ch was For the real data t was Inspecton of the scatterplots dd not reveal a trend of any sort. So, the ntal results suggest that the values of lz-ch are essentally ndependent from the trat levels. Indeed, far more research s needed on ths ssue. We turn now to the relatons between lz-ch and the lz values estmated at Tme 1. For the smulated data the scatterplot suggested a lnear relaton wthout outlers, and the correlaton was Ths result agrees wth the theoretcal expectatons dscussed above. As for the real data, the scatterplot s shown n fgure 2 Unlke the smulated case, fgure 2 reveals the presence of outlers that affect the product-moment correlaton, whch n ths case was only These outlers are marked n fgure 2 and are presumably respondents who behaved consstently at Tme 1 (acceptable lz values) but nconsstently n the short-term. As dscussed above, these are precsely the most nterestng cases. The lz-ch values for the ndvduals flagged n fgure 2 were (respondent 56), (respondent 88) and (respondent 357). These hgh negatve values suggest that these respondents changed responses at Tme 2 more often than expected. To further assess the type of nconsstency that was potentally detected by lz-ch, the standardzed change resduals (11) were assessed for each tem. For respondents 56 and 88, t was found that nconsstences were located n a small group of tems (15, 17, and 34 for respondent 56, and 28, 41, 46 and 51 for respondent 88). For these tems, the Zch values were far above 4 n all cases. In contrast,

13 Short-term consstency 331 the Zch values for respondent 357 were systematcally hgh (around 2) for most tems but wthout the extremely hgh values obtaned n the two prevous cases. In appled research, further nformaton should be collected to determne the cause of the unexpected changes of respondents 56 and 88. The behavor of respondent 357, on the other hand, rather suggests some type of systematc change n the trat levels. Fgure 2. Relaton between lz at Tme 1 and lz-ch. Real data. DISCUSSIO In personalty measurement, short-term consstency s an mportant topc that has been the obect of consderable research. Earler studes used a classcal test theory framework, whereas most modern research tends to be based on IRT. In both cases, however, the focus has generally been on the tem characterstcs that tend to elct nconsstent respondng (e.g. Ferrando et al. 2001). Furthermore, the respondents have only been studed at the group level, not at the ndvdual level (e.g. Fscher, 1995). As far as the wrter

14 332 P.J. Ferrando knows, no procedures at the ndvdual level of the type dscussed here have been proposed so far. The present procedure s based on a lkelhood-type ndex that can be consdered as the short-term counterpart of the well known lz statstc. So, lzch can be assumed to have the same advantages and shortcomngs as lz. Its advantages are that t has a clear ratonale, s easy to nterpret and s expected to have a relatvely good power for detectng nconsstent respondents. Its dsadvantages are that t only adheres to the standard normal reference dstrbuton under strong assumptons that can never be fulflled wth real data. So, at present t does not seem approprate to use the statstc for conductng strct tests of ft; rather t can be used at the descrptve level as a useful screenng tool. Once a respondent has been flagged as potentally nconsstent, the standardzed change resduals also proposed n ths paper can be used to further assess the type of nconsstency detected by the global ndex, and perhaps to explan the reasons behnd the nconsstences. Further extensons can also be consdered. One revewer suggested that the ndvdual estmates obtaned separately at Tme 1 and at Tme 2 should be used for assessng specfc sources of nconsstency. For example, for the theta-shft case, lz-ch could be modfed so that t would take nto account the estmated ndvdual change. If so, the statstc would detect nconsstency beyond that due to the temporal change n the trat level. The results obtaned n the emprcal study supported the expectatons about the behavor of lz-ch. In the smulated data, based on a 60-tem test wth moderate dscrmnaton and a wde range of tem dffcultes, the dstrbuton of lz-ch approached the theoretcal reference dstrbuton qute well. In fact the departure from ths dstrbuton was non-sgnfcant accordng to the Kolmogorov-Smrnov test. Furthermore, the values of the statstc were vrtually ndependent from the estmated trat level. Wth the real data, the departure from the reference dstrbuton was clear. However, the dscrepances could be reasonably explaned, and they provded useful nformaton. Fnally, the man purpose of ths paper s to provde a useful tool for appled researchers. In ths respect, the procedure s relatvely smple, and the proposed statstcs can be easly programmed. At present, the wrter s developng a user-frendly program that wll be avalable at no cost.

15 Short-term consstency 333 RESUME Evaluacón de la consstenca ndvdual a corto plazo medante estadístcos basados en la TRI. Se propone un procedmento, basado en un estadístco global para evaluar la consstenca ntra-ndvdual en un dseño retest basado en un ntervalo de retest corto. El procedmento se desarrolla en el marco de los modelos paramétrcos de TRI, y el estadístco, que se basa en la funcón de verosmltud, puede ser consderado como una extensón del estadístco lz. Se dscute la ustfcacón para el uso e nterpretacón del estadístco y se propone además un estadístco estandarzado a nvel de ítem para obtener claves acerca de las posbles causas de la nconsstenca detectada medante el índce global. El procedmento se lustra con un eemplo real y una smulacón paralela en el domno de la personaldad. REFERE CES Bollen, K.A. & Armnger, G. (1991). Observatonal resduals n factor analyss and structural equaton models. In P.V. Marsden (ed.) Socologcal Methodology 1991 (pp ). New York: Basl Blackwell. Cattell, R.B. (1986). The psychometrc propertes of tests: consstency, valdty and effcency. In R.B. Cattell and R.C. Johnson (eds.) Functonal Psychologcal Testng (pp 54-78). New York: Brunner/Mazel. Conley, J.J. (1984). The herarchy of consstency: a revew and model of longtudnal fndngs on adult ndvdual dfferences n ntellgence, personalty and self-opnon. Personalty and Indvdual Dfferences, 5, Drasgow, F., Levne, M.V. & Wllams, E.A. (1985). Approprateness measurement wth polychotomous tem response models and standardzed ndces. Brtsh Journal of Mathematcal and Statstcal Psychology, 38, Ferrando, P.J. & Chco, E. (2001).Detectng dssmulaton n personalty test scores: A comparson between person-ft ndces and detecton scales. Educatonal and Psychologcal Measurement, 61, Ferrando, P.J.(1994). Fttng tem response models to the EPI-A mpulsvty subscale. Educatonal and Psychologcal Measurement. 54, Ferrando, P.J., Lorenzo-Seva, U., & Molna, G. (2001). An tem response theory analyss of response stablty n personalty measurement. Appled Psychologcal Measurement. 25, Fscher, G.H. (1995). Some neglected problems n IRT. Psychometrka, 60, Fske, D.W. & Rce, L. (1955). Intra-ndvdual response varablty. Psychologcal Bulletn, 52, Fraser, C. & McDonald, R.P. (1988). NOHARM: least squares tem factor analyss. Multvarate Behavoral Research, 23, Gullksen, H. (1950). Theory of mental tests. New York: Wley. Jorm, A.F., Duncan-Jones, P. & Scout, R. (1989). An analyss of the re-test artfact n longtudnal studes of psychatrc symptoms and personalty. Psychologcal Medcne, 19, Knowles, E.S. (1988). Item context effects on personalty scales: measurng changes the measure. Journal of Personalty and Socal Psychology, 55,

16 334 P.J. Ferrando Levne, M.V. & Rubn, D.B. (1979). Measurng the approprateness of multple choce test scores. Journal of Educatonal Statstcs, 4, Lumsden, J. (1977). Person relablty. Appled Psychologcal Measurement, 1, Meer, R.R.& Stsma, K. (1995). Detecton of aberrant tem scores patterns: A revew and new developments. Appled Measurement n Educaton. 8, Meer, R.R.& Stsma, K. (2001). Methodology revew: Evaluatong person ft. Appled Psychologcal Measurement. 25, Meer, R.R., Egbernk, I.J.K., Emons, W.H.M. & Stsma, K. (2008). Detecton and valdaton of unscalable tem score patterns usng tem response theory: An llustraton wth Harter s self-percepton profle for chldren. Journal of Personalty Assessment, 90, Nowakowska, M. (1983). Quanttatve pschology: some chosen problems and new deas. Amsterdam: North-Holland. Rese, S.P. & Waller, N.G. (1993). Tratedness and the assessment of response pattern scalablty. Journal of Personalty and Socal Psychology, 65, Rese, S.P. (1995). Scorng method and the detecton of person msft n a personalty assessment context. Appled Psychologcal Measurement, 19, Rese, S.P. & Flannery, W.P. (1996). Assessng person-ft on measures of typcal performance. Appled Measurement n Educaton, 9, Schmdt, F.L. & Hunter, J.E. (1996). Measurement error n psychologcal research: Lessons from 26 research scenaros. Psychologcal Methods, 1, Schuldberg, D. (1990). Varetes of nconsstency across test occasons: Effects of computerzed test admnstraton and repeated testng. Journal of Personalty Assessment, 55, Slverman, B.W. (1986). Densty estmaton for statstcs and data analyss. London: Chapman & Hall. Snders, T.A.B. (2001). Asymptotc null dstrbuton of person ft statstcs wth estmated person parameter. Psychometrka, 66, Tellegen, A. (1988). The analyss of consstency n personalty assessment. Journal of Personalty, 56, Thorndke, R.L. (1951). Relablty. In E.F. Lndqust (ed.) Educatonal Measurement (pp ). Washngton: Amercan Councl on Educaton. van Krmpen-Stoop, E.M.L.A. & Meer, R.R. (1999). The null dstrbuton of person-ft statstcs for conventonal and adaptve tests. Appled Psychologcal Measurement, 23, Watson, D. (2004). Stablty versus change, dependablty versus error: Issues n the assessment of personalty over tme. Journal of Research n Personalty, 38, Wrght, B. & Stone, C.H. (1979). Best test desgn. Chcago: Mesa press. Zckar, M. J. & Drasgow, F. (1996). Detectng fakng on a personalty nstrument usng aproprateness measurement. Appled Psychologcal Measurement, 20, Zmowsk, M., Murak, E., Mslevy, R. J., & Bock, R. D. (2003). BILOG-MG 3: Item analyss and test scorng wth bnary logstc models. Chcago: Scentfc Software. (Manuscrpt receved: 10 March 2009; accepted: 14 Aprl 2009)