Iatrogenic Specification Error: A Cautionary Tale of Cleaning Data

Transcription

1 DISCUSSION PAPER SERIES IZA DP No Iatrogeni Speifiation Error: A Cautionary Tale of Cleaning Data Christopher R. Bollinger Amitabh Chandra Marh 2004 Forshungsinstitut zur Zukunft der Arbeit Institute for the Study of Labor

2 Iatrogeni Speifiation Error: A Cautionary Tale of Cleaning Data Christopher R. Bollinger University of Kentuky Amitabh Chandra Dartmouth College, NBER and IZA Bonn Disussion Paper No Marh 2004 IZA P.O. Box Bonn Germany Phone: Fax: iza@iza.org Any opinions expressed here are those of the author(s) and not those of the institute. Researh disseminated by IZA may inlude views on poliy, but the institute itself takes no institutional poliy positions. The Institute for the Study of Labor (IZA) in Bonn is a loal and virtual international researh enter and a plae of ommuniation between siene, politis and business. IZA is an independent nonprofit ompany supported by Deutshe Post World Net. The enter is assoiated with the University of Bonn and offers a stimulating researh environment through its researh networks, researh support, and visitors and dotoral programs. IZA engages in (i) original and internationally ompetitive researh in all fields of labor eonomis, (ii) development of poliy onepts, and (iii) dissemination of researh results and onepts to the interested publi. IZA Disussion Papers often represent preliminary work and are irulated to enourage disussion. Citation of suh a paper should aount for its provisional harater. A revised version may be available on the IZA website ( or diretly from the author.

3 IZA Disussion Paper No Marh 2004 ABSTRACT Iatrogeni Speifiation Error: A Cautionary Tale of Cleaning Data In empirial researh it is ommon pratie to use sensible rules of thumb for leaning data. Measurement error is often the justifiation for removing (trimming) or reoding (winsorizing) observations whose values lie outside a speified range. We onsider a general measurement error proess that nests many plausible models. Analyti results demonstrate that winsorizing and trimming are only solutions for a narrow lass of measurement error proesses. Indeed, for the measurement error proesses found in most soial-siene data, suh proedures an indue or exaerbate bias, and even inflate the variane estimates. We term this soure of bias Iatrogeni (or eonometriian indued) error. Monte Carlo simulations and empirial results from the Census PUMS data and 2001 CPS data demonstrate the fragility of trimming and winsorizing as solutions to measurement error in the dependent variable. Even on asymptoti variane and RMSE riteria, we are unable to find generalizable justifiations for ommonly used leaning proedures. JEL Classifiation: C1, J1 Keywords: measurement error models, trimming, winsorizing Corresponding author: Amitabh Chandra Department of Eonomis Dartmouth College 6106 Rokefeller Hall Hanover, NH USA amitabh.handra@dartmouth.edu

4 1 Introdution Empirial researhers frequently use simple rules of thumb to lean data on the basis of the dependent variable. As an example, researhers analyzing survey reports of wages and salaries often remove observations whose value for the hourly wage is below the minimum wage or above some prespei ed uto : sample exlusions based on wages an be found in Katz and Murphy (1992), Card and Krueger (1992), Bound and Freeman (1992), Juhn, Murphy, and Piere (1993), and Buhinsky (1994). We ite these authors to illustrate the endorsement of this pratie by leading sholars in the eld. As we demonstrate in this paper, the intuitively appealing strategy of disarding ertain observations is not ostless and an introdue spei ation error in ases where no error previously existed. Given the fat that the inonsisteny is exaerbated by the analyst s ations, we borrow a term from the medial literature and term this form of bias iatrogeni spei ation error. In the medial literature an iatrogeni event is an adverse reation to a well-intentioned treatment initiated by a physiian, and we believe that parameter inonsisteny that is aused by the analysts well intentioned ations shares the same features of physiian indued ompliations. Given the widespread aeptane of this pratie, the topi of robust estimation has reeived the attention of both eonomists and statistiians. In one the earliest formal examinations, Stigler (1977) poses an interesting question: how muh have methods suh as trimming, winsorizing, the Edgeworth average, or Tukey s Biweight, redued the bias in the laboratory estimation of physial onstants suh as the speed of light or the density of the earth? Stigler onludes that the 10 perent trimmed mean, the smallest trimming amount onsidered in his study, is the most reliable estimator. In this he ehoes the famous mathematiian Legendre who reommended deleting those observations with errors too large to be admissible. Stigler looks at the role of measurement error in the physial sienes; the error proess may be vastly di erent in the soial-sienes where eonomi agents may have strategi or ultural inentives to in ate or de ate their reports. In the eonometris literature, Angrist and Krueger (2000) apply trimming and winsorizing tehniques to the mathed employer-employee data from Mellow and Sider (1983). When they trim both the employer and employee wage data, they nd that the orrelation between the two measures improves. Interestingly, this result does not hold for reports of hours worked. On the basis of this nding they onlude that a small amount of trimming ould be bene ial. Their presription, whih summarizes the intuition and urrent pratie of most analysts, may be summarized as: Loosely speaking, winsorizing the data is desirable if the extreme values are exaggerated versions of the true values, but the true values still lie in the tails. Trunating the sample is more desirable if the extremes are mistakes that bear no resemblane to the true values. (p.1349) We examine this pratie in detail here, using wages and earnings as a motivating example, though the results are likely to apply to other errors of measurement in survey data. We posit a general model of response error in the dependent variable of a linear regression model and haraterize the e et of di erent leaning tehniques on the estimated oe ients. We demonstrate, both analytially as well as through the use of simulations, that in general there is no reason to believe that removing obvious errors in the 1

5 dependent variable redues bias. This is similar in spirit, to the nding of Hyslop and Imbens (2001), who examine instrumental variables approahes to solving the measurement error problem and nd that they only apply to very spei measurement error proesses. Our work is most losely related to that of MaDonald and Robinson (1985), who onsider Bayesian estimation of an error omponents model in panel data when one of the error omponents is measurement error. They expliitly show that trimming an be thought of as an extremely dogmati prior belief. Our paper di ers in that we expliitly onsider a general error proess and we do not onsider a panel setting. Moreover we disuss an optimal trimming approah and other lassial approahes to estimation. We demonstrate that the results in Stigler (1977) do not neessarily arry over in a regression framework. Indeed, trimming or winsorizing an bias oe ient estimates by as muh as perent, and in many ases either indues bias that did not previously exist, or exaerbates the bias due to measurement error. The intuition for our result is simple: assuming that the researher trims or winsorizes the data based on a lower bound of or an upper bound of C, it will be shown that leaning reates seletion-bias, and this is generally worse than the e ets of measurement-error in the dependent variable. Our paper is organized as follows: Setion 2 desribes identi ation with general measurement error in the dependent variable. We use the linear projetion of the mismeasured dependent variable onto the ovariates to derive analytial results. In Setion 3, we generalize the use of this projetion to onsider three spei models of measurement error (additive white noise, linear transformation and the ontaminated data proess) that are found in soial-siene data. Setion 4 examines the theoretial impliations of trimming the data on bias as well as the asymptoti varianes of the oe ients. We prove that only in highly speialized ases, unlikely to be found in soial-siene data, does leaning redue bias. In these ases, we demonstrate that the information neessary to redue bias leads to a simpler orretion that requires fewer assumptions. This setion also demonstrates that trimming will not neessarily redue standard-errors. Setion 5 presents simulation results for the ases onsidered analytially. We generate quasi-simulated data from the 1990 US Deennial Census to study the properties of winsorizing and trimming in a multivariate ontext. Finally, we present an empirial example from the Marh 2001 Current Population Survey (CPS). These simulations and examples support the results of the earlier two setions. Setion 6 provides onluding omments. The Appendix to this paper provides detailed mathematial proofs and also onsiders the e ets of winsorizing on bias and e ieny. 2 General Measurement Error in the Dependent Variable To evaluate the widespread pratie of leaning data as desribed above we onsider a general model for measurement error proesses in the dependent variable. To keep the analysis simple, we fous on a linear regression model as the underlying strutural model of interest to the researher. Assume that the 2

6 relationship between the true dependent variable and the ovariate is desribed by: y i = x 0 i + u i ; (1) Our maintained assumption is that the analyst is interested in estimates of : 1 We assume a general proess that relates the true value y i to the observed value y i : y i = h (y i ; " i ) ; (2) There are six assumptions that are made for identi ation of the vetor and its assoiated ovariane matrix: 2 A1 : E [u i jx i ] = 0 A2 : x i is a vetor random variable with mean 0 and full rank seond moment matrix V x A3 : Random Sampling A4 : h(:; :) has nitely many disontinuities A5 : " i is independent of (y i ; x i ; u i ) A6 : Cov (y i ; y i ) > 0 Regardless of the proess in equation 2, one summary of the joint distribution of y i and y i linear projetion of y i on y i : Here, = E [y i ] ; = Cov(yi;y i ) V (y i ) is the population y i = + y i + e i : (3) ;and E [e i ] = E [e i yi ] = 0: The linear projetion is not a statement about the data generating proess, but rather a summary measure of the joint distribution of (y i ; yi ) : The atual measurement proess, as de ned by h (yi ; ") may be substantially more ompliated. Assumption A6 insures that > 0: The researher is only able to observe (y i ; x i ). Substituting equation 1 into equation 3 yields: y i = + x 0 i + u i + e i : (4) Assumption A5 insures that Cov (x i ; u i + e i ) = 0 and E [u i + e i ] = 0: This de nes the population linear projetion of y i on x i : where b = ; a = ;and i = u i + e i : y i = a + x 0 ib + i (5) 1 If the analyst is not interested in per se, but other features of the joint distribution between y and x suh as ov(y; x) or var(yjx), then it is possible that our results do not apply. Further analysis is required to understand the appliability of trimming, winsorizing or even resaling for this lass of problems. We thank an anonymous referee for suggesting this aveat. 2 The mean independene assumption is stronger than neessary for identi ation of the vetor, but allows for a simpler analysis below. The zero mean for x i is the usual normalization. The fourth assumption is neessary for moments to be well de ned, and A6 simply ensures that the measurement error proess is not so perverse that y i is uninformative about yi (ovariane of zero), or that y i and yi are negatively related. Indeed, the neessary ondition would simply be that the sign of the ovariane were known and that the ovariane is not zero. The fth assumption is the strongest one. It implies that the measurement error proess is independent of x i and u i exept through yi ; and insures that f(y ijyi ) = f(y ijyi ; x i; u i ): 3

7 Therefore, the OLS regression of y i on x i yields a onsistent estimate of b whih is proportional to : The parameters of interest are identi ed up to an unknown saling onstant. This would imply that estimates of ratios of the parameters are onsistent. In some settings, identi ation up to sale is onsidered su ient. For example, in wage regressions the oe ients on years of eduation and years of labor market experiene an be ombined to onsistently identify the relative return of experiene to eduation; a fat that might be su ient for estimation of shooling hoies. In general however, we assume the researher is interested in reovering the parameters. This suggests two important identi ation approahes: obtain information about the saling onstant, or obtain information about one of the elements in : While it may be possible to obtain some onsistent estimate of one element in from auxiliary regressions or eonomi theory, the use of validation data may permit estimation of : Bound and Krueger (1992) and Bollinger (1998) have examined the struture of response error when y is the natural log of annual labor market earnings using Soial Seurity Inome data mathed to the Current Population Survey. They nd a point estimate for is This estimate ould be used in log wage models to resale slope oe ients to aount for measurement error. 3 3 Spei Measurement Error Models The above analysis holds for general examples of measurement error. In this setion, we present three speial ases of the above model. These ases are hosen beause they are ommonly examined or supported in the literature or lead to results in the ontext of this paper whih are of interest. maintained, additional assumptions are also imposed. 3.1 Additive White Noise: Assumptions 1-6 are The lassial measurement error proess is often assumed: y i = y i + " i; and E [" i ] = 0: Indeed, the error term in regression models is often motivated as measurement error. The parameters of the linear-projetion of y on y are = 1, = a = 0, and the least squares estimates are onsistent for the parameters of interest : In this model, if y i were hourly wages, it would be possible to have observations less than the minimum wage (or for that matter even negative observations) and observations above whatever threshold is deemed as a maximum. While it may be true that observations outside the aeptable region are measured with error, observations within the aeptable region are also measured with error. However, as is well known, lassial measurement error does not lead to any bias the estimated standard-errors are in ated but all statistial tests remain valid. Researhers will often point out that standard errors are too large beause of the additional measurement error. Standard errors are meant to apture the variation in estimates due to 3 The results in Bound and Krueger (1991) and Bollinger (1998) rely on estimates from the the 1977 and 1978 CPS-SSA mathed les. It is possible that the struture of measurement-error has hanged over time, thereby reduing the appliability of the resaling option sine it hinges ritially on knowledge of the orret : Estimation of is further ompliated by the fat that low earning repondents in the SSA data may be reporting their CPS earnings orretly. Examining these hypotheses is an important avenue for future researh. 4

8 di erenes aross samples. As long as the data generating proess does not hange, the sampling variation of the estimating oe ients will depend on the variation in both the strutural model, as well as the variation in the error model. Hene, estimates of the standard error are not biased, but rather re et the variation aross samples for this data generating proess. 3.2 Linear Measurement Error A seond ase is where the data generating proess is linear: y i = d + gy i + " i: Here, the parameters in the linear-projetion of y on y are = g and = d; and the model an either have > 1 or < 1. data generating proess an lead to observations outside the aeptable range. The Beause of the values of and the distribution of " i ; even if < 1, it is quite possible to have both observations that are too high and observations that are too low. Empirial work by Bollinger (1998) and Bound and Krueger (1991) supports the possibility that < 1: For example, using non-parametri regression on the 1978 CPS-SSA mathed data, Bollinger (1998) estimates that is equal to 0.91 for men and 0.97 for women. He estimates the interepts to be $1,364 and $211 respetively. Cognitive psyhologists have noted that this model, with < 1, will arise when respondents exhibit regression to the mean. If survey respondents give answers that try to make them appear average, then those below the mean report higher values, on average, while those above the mean report lower values, on average. Similarly, the hot dek proedure used by Census to impute earnings an also lead to a regression to the mean (Hirsh and Shumaher, 2001). study has found any variable with > Contaminated Data To our knowledge, no A third example is a simple ontaminated sample: y i = (y i ) 1 [" 1i > ] + (d + " 2i ) 1 [" 1i < ]. The term 1[:] is the indiator funtion and (" 1i ; " 2i ) are mean zero and mutually independent. This model produes a mixture: with some probability p = Pr [" 1i > ], we observe the true variable yi, while with probability (1 p) we observe only noise: (d + " 3i ) : This leads to a model where we have some orretly measured observations and some observations where the observed y has no relationship to the atual y. In this model = p and = d (1 p) : Again, some observations may fall outside a given range, depending on the distribution of " 2i and the value of d. An important impliation of this model is that estimates of the slope parameter an be obtained if an estimate of p is available. Horowitz and Manski (1995) note that the expetation of y given x annot be bounded unless information about d is available. Our analysis does not ontradit this, but rather points out that in a linear model, the slopes an be identi ed up to the ontamination rate. In many ases researhers have a priori bounds for the ontamination rate. The bounds on the ontamination rate will yield trivial bounds for the slope oe ients. h i bj elements of, j, are bounded by p ; bj p. If p< p < p, then the 5

9 4 E et of Cleaning We assume that the researher trunates above and below the mean. The leaning approahes we onsider are de ned by fy i ; x i j y i Cg (6) for known onstants (; C) suh that < E [y] < C. We ompare the slopes obtained from a least squares projetion of the leaned y i on x i to those obtained from the unleaned data regressed on the ovariate (that is, relative to b the biased estimate of from the unensored data). Sine the hoie for the researher is to lean or not lean, this is the relevant omparison. As noted in the setion above, the slope b may be larger or smaller in magnitude than the true slope : 4.1 Analyti Results: Trimmed Data We rst derive analyti results under the additional assumption of joint-normality. A7 : (y i ; x i ) are jointly normal. As Goldberger (1981) demonstrates, the slope vetor from the least squares projetion of y i on x i in the trunated sample (where observations above C and below are disarded) is given by: Proposition 1 Under assumptions 1-7, the trunated slope b is attenuated relative to the slope b from the least squares projetion in the full sample. b = 1 (1 ) 2 b (7) with and = V (y ij y i C) 2 = b2 2 x : (9) (8) Proof: Goldberger (1981). As Goldberger notes, 0 1 (1 ) Clearly, if 1, then the attenuation bias of the measurement error is exaerbated by the attenuation bias of the sample trunation. Hene, only if the researher is ertain that > 1 an trunation alleviate bias from measurement error. For this ase, the optimal level is determined by nding (; C) suh that = 1. With two unknown terms and only one restrition, there are 1 (1 ) many solutions. Too little trunation will fail to fully orret for the bias, while too muh will overorret. Therefore, seleting the trimming bounds on the basis of a priori values for the supports of y (as proxied 4 Sine y i is normally distributed, 2 the variane of the doubly trunated distribution an be expressed (see Madalla, 1983) as: E[yi ] E[yi ] C E[yi ] C E[yi ] 6 V (y V (y i j y i C) = i ) 5 4 E[yi ] C E[yi ] 7 C E[yi ] E[yi ] C E[yi ] E[yi ] 5 6

10 by the 1 and 99 perentiles of the wage distribution, or trimming at the minimum wage) will not neessarily orrespond to the optimal trimming rule. Other solutions exist as well. For example, if the analyst hooses = E [y] rule. and C = E [y] + ; only the term needs to be found in order to devise an optimal trimming It is impliitly desribed in the next proposition: Proposition 2 Under assumptions 1-7 and > 1; an optimal trimming rule of the form fy i ; x i j y i Cg with = E [y] Proof: see Appendix. and C = E [y] + may be derived impliitly as: 0 1 A = 1 2 : (10) Beause the solution involves the df of the standard normal distribution, there is no losed form expression. The optimal leaning depends on the variane of the observed y; the orrelation between y and x; and : The right hand side of (10) is inreasing in and the left hand side of (10) is dereasing in. Therefore, as inreases, the trunation points must move loser to the mean and the data must be trunated more heavily. In order to use this approah a number of highly restritive assumptions must be met. First, the data must be jointly normally distributed. Any disrete variables in x i will violate this assumption. Seond, the measurement error proess must result in a projetion equation for y i on y i where > 1: Finally, spei information on must be obtained in order to arrive at a trunation rule. Ad ho leaning approahes that ignore the strong nature of these assumptions may be of little value in reduing the bias in b. The variane of the measurement error is often used as a measure of the severity of the error. Sine b = 1 (1 ) b under trimming; the derivative of 2 1 (1 ) with respet to 2 2 " reveals how the trunation bias is e eted by the measurement error. This result motivates the next proposition: Proposition 3 Under assumptions 1-7, the absolute value of the di erene between elements of b and b beomes larger sine, 1 (1 ) < 2 " The above proposition is may appear to be prima-faia ounterintuitive, but the intuition behind it is simple. As the measurement error beomes more severe (as measured by it variane), trimming is more likely to result in deleting observations based on the regression error instead of the measurement error, thereby ausing sample-seletion bias Does Trimming Redue Standard Errors? A seond reason sometimes ited for trimming is the redution of standard errors. To examine this proedure more rigorously we begin by noting that the trunation will introdue heteroskedastiity by reduing the variane of errors in the tails of the distribution. Therefore, the asymptoti variane of the estimated slope from 7

11 h i the trunated data an be derived from the expression: AV bb = Q 1 E (y i x 0 i b ) 2 x i x 0 i j y i C Q 1 ; where Q = E x i x T i j y i C : In the appendix we demonstrate that this expression for asymptoti variane may be written as the sum of two terms: AV bb = 1 +Q 1 E 1 (1 ) 2 h (x 0 ib m (x i )) 2 x i x 0 ij y i C 2 Q 1 (11) i Q 1 : The size of the leading term is indeterminate relative to its full-sample OLS ounterpart. 5 in ontrast to similar omparisons for the mean (and onsequently the results in Stigler (1977)). This fat is For the trimmed mean, the term < and therefore trimming neessarily redues the variane in the leading term. Here, the omparison is not so straight forward. The seond term is due to heteroskedastiity from the trimming and is neessarily positive de nite. Hene, even if the leading term is smaller than the variane of the OLS estimate on the full sample (in a positive-de nite sense), the seond term may reverse, or at least mitigate that di erene. Finally, an often overlooked reason for why trimming may not redue standard-errors is the e et of trunation on sample size. The nite sample variane of b b is given by AV bb V bb = N C The term C E[yi] V (y i) E[yi] V (y i) E[yi] V (y i) E[yi] V (y i) : (12) < 1; measures the proportion of the sample disarded from the trunation rule, and inreases in this proportion will raise estimates of the nite sample variane. In onlusion, we nd that omparisons between the variane of the trunated estimates and the variane of the full sample estimates is ompliated and depends on the underlying parameters of the joint distribution. It is not possible to sign this di erene, even under normality. no e et of trimming on standard-errors. 4.2 Analyti Results: Winsorized Data In fat, the simulations below show little or Rather than trunation, winsorized data are ensored at the points and C: removed, but values of y i outside of the region (; C) are transformed as: Here, no observations are y w i = C if y i C y i if < y i < C y i : (13) 5 The leading term is omparable to the asymptoti variane expression for the OLS estimate in the full sample: 1 2 E x i x T 1 i : Furthermore, V (yi ) 1 1 (1 ) : However, the di erene E x i x T 1 i E x i x T i j y i C 1 is neessarily positive semi-de nite sine the variane of xi in the trunated sample annot be larger than the variane of the full sample (a su ient ondition here is joint normality (see Goldberger(1981)). Hene, a omparison of the leading terms is indeterminate. 8

12 In the appendix we show that the analytial results from winsorizing are similar to those obtained for trimming. The empirial results suggest winsorizing has less of an impat on the slope oe ients (relative to OLS) than trunation. Again, if > 1;an optimal hoie of winsorizing points is available, but of ourse is unknown. Asertaining the e et of winsorizing on the size of the standard errors is oneptually similar to the e et of trimming on standard-errors, but algebraially more di ult. We provide a disussion of this point in the appendix. One advantage winsorizing has over trimming is that the penalty of lost data does not e et the expression for the nite sample variane. But overall, winsorizing and trimming have similar e ets on both slope estimates and asymptoti variane. 4.3 Cleaning Data in the General Case The results derived in the previous setion rely heavily upon normality. However, as Goldberger (1981) demonstrates, without normality some oe ients may be attenuated by trunation, while others may be in ated. Clearly, as a theoretial matter, trunation or winsorizing annot be relied upon to adjust slope oe ients for the bias in general. onditions. 5 Results In ontrast, the results from Setion 2 were derived under muh weaker To suggest more general results, we present a set of Monte Carlo simulations. We draw data from the 1990 PUMS and estimate the returns to shooling, treating estimates from the full PUMS le as population parameters. The results of di erent leaning proedures are gauged against these parameters. This allows for a ompliated measurement model, with relationships similar to those found in typial eonomi data Evidene from U.S. Deennial Census Data We begin with evidene from quasi-simulated data drawn from the PUMS samples of the 1990 US Deennial Census. The advantage of these data is that they provide a omplex multivariate distribution for analysis. We study the problem of estimating the returns to shooling, using a standard Minerian spei ation (that is, lnwage = Shooling + 2 Experiene+ 3 Experiene Blak +u) to desribe the relationship between hourly wages, years of shooling, rae and potential experiene. We rst selet prime-aged men who 6 Our working paper [Bollinger and Chandra (2003)] provides more detailed Monte Carlo simulations for the univariate and multivariate ase, and data from the normal, uniform, and log-normal distributions. Of note are the results for the realisti ase of = 0:9: We note that leaning proedures are always dominated by not leaning the data. Even though the bias from not leaning the data is a little over 10 perent, the bias from trimming is uniformly greater. 1% winsorizing or the use of median regression are neutral rules with respet to point-estimates, but both proedures are dominated by not leaning the data on the basis of a RMSE riteria. When > 1; a 1 perent trimming rule learly dominates not leaning the data. Before this onlusion is embraed too quikly by pratitioners, we raise two important aveats: First, even though 1% trimming works, 5% trimming is muh worse that not leaning the data; the optimal trimming rule is therefore not a known onstant and small perturbations from the optimal trunation will generate large biases relative to not leaning the data. In fat, the best rule for the ase of > 1 would be to use 5% winsorizing. Seond, we reiterate the di ulty in being able to justify a behavioral model for why would exeed one. 9

13 are working full time in a non-agriultural industry. We remove individuals who earn less than the minimum wage. The resulting 346,900 observations onsitute a sample that losely simulates the ideal population distribution assumed by many researhers. Column 1 of Table 1 presents mean and standard deviation parameters for this pseudo-population. Blak men omprise 8.3 perent of the population, the mean years of potential experiene is 17.67, and the mean years of shooling is The regression parameters generated by an OLS regression on all 346,900 observations are reported in Column 2 of Table 1. For our simulations, we randomly draw samples of size 1000 from the psuedo-population of 346,900 observations. Ordinary least squares performed on these data (without any leaning) are reported in the third olumn of the table. In Columns 4-8, we report the e et of alternative leaning proedures when no measurement error has been added to the PUMS data. 7 The idea of leaning data with no error might strike the reader as a peuliar exerise. Our motivation for doing so is to demonstrate that leaning proedures are not benign and an introdue signi ant bias when they are not required; alternatively, if the degree of ontamination is low, the iatrogeni error from leaning data may be substantial. In this situation, the leaning proedures do not generally perform better than not leaning the data. In general, the RMSE from the leaning proedures (inluding median regression) is greater than that from doing nothing. Whereas a 1% trimming rule improves the estimation of the oe ients on experiene and experiene squared, it is inferior to not leaning the data as regards the estimation of the oe ients on shooling and rae. Together these results on rm those from the univariate ase: No leaning proedure is neutral when applied to already lean data. Measurement error is added to the data in Table 2. We selet two values for the variane of this error using the results of Bound and Krueger (1991), who note that V ar(lny ) = and with orresponding error varianes are and This implies that the variane of the error is 18% and 22% of the total variation in lny. Rogers, Brown and Dunan (1993) nd even higher implied estimates of the variane of the measurement error. Therefore, to study empirially relevant ases we simulate measurement error whose variane is 0:1V ar(wage) and 0:3V ar(wage). In order to keep the reported number of results manageable we only report results from the latter simulation, but note that the results from the former are quantitatively similar [see Bollinger and Chandra (2003) for details]. In the ase of additive white noise we nd that trimming is one again dominated by not leaning the data. A ase an be made for a 1% winsorizing rule over not leaning the data, but it is important to note that signi ant bias is introdued when the ensoring rate is inreased to 5%. For this ase, least-squares is found to be superior to median regression. When = 0:9 there is no leaning proedure that stritly dominates OLS. A 1% winsorizing rule provides superior estimates on a RMSE riteria for many oe ients but simultaneously raises the bias on others. For example, the oe ients on Exp, Exp-Sq and Blak all have lower RMSE when a 1% winsorizing rule is applied, but the oe ient on shooling has a larger RMSE at the same time. When = 1:1 winsorizing at 1% and 5% are preferred to doing nothing. In this ase, trimming proedures dominate not leaning the data on a 7 In the last olumn of the table, we report results from performing Median Regression. While not expliitly studied in our paper, we inlude these estimates beause several readers of our paper argued that it may be viewed as an alternative to trimming and winsorizing. 10

14 RMSE riteria, but an be worse in terms of the bias omponent Comparing resaling approahes to trimming approahes. As noted in previous setions, another identi ation approah is to resale the estimates. The optimal trimming rule derived in Setion 4 requires both information about and normality: Clearly, if < 1, trimming or winsorizing will be dominated by the resaling approah. Even when > 1, the amount of trimming or winsorizing neessary depends upon the variane of the errors (see Bollinger and Chandra (2003)): Examining the rst Column of Table 2, shows that knowledge of alone will be su ient to arrive at onsistent estimates. Even if is not known, the resaling results an be used to perform sensitivity analysis. For example, researhers might ask: how sensitive to di erent values of are the onlusions we draw from our OLS estimates? The robustness of these onlusions may be examined either by plaing bounds on ; as suggested in Manski (1995), or alternatively by asking what values of support the onlusions typially drawn (an approah suggested in a similar ontext by Bollinger (2003), and Bollinger (2001)). Further, researhers may not have detailed information about but may have information about the likely range of : It is di ult to use that information for trimming and winsorizing, but it an be trivially used in a resaling approah. 5.2 Empirial example from the CPS We also examine data leaning approahes using the Marh 2001 Current Population Survey. There are two measures of hourly wage that a researher ould exploit in these data. One is the hourly wage onstruted from the reported annual earnings, weeks worked and usual hours worked. The CPS also asks the atual hourly wage for workers who are paid hourly. Most researhers do not use this variable, as the resulting sample is smaller and only represents hourly wage workers. In this ontext, the two measures provide an interesting omparison to examine the impliations of trimming as is typially pratied. Our sample onsists of males, working full time, year round in non-agriultural positions who are not self-employed. In Marh 2001, we nd 2,626 men who are full time, year round non-agriultural hourly wage workers. The rst olumn of Table 3 presents the log wage regression on the reported hourly wage, while the seond olumn uses the onstruted hourly wage. We restrited the onstruted hourly wage sample to ontain only those workers who also reported an hourly wage, hene any di erene between the two olumns re ets only di erenes in the measurement of the hourly wage, rather than sample di erenes. One perspetive with these results is that the rst olumn represents the "true oe ients," while the results in olumn two are biased due to response error. Interestingly, most of the oe ients in olumn 2 are larger in magnitude than their orresponding oe ients in olumn 1. The exeption to this is the oe ient on Bahelors degree. This is a ase where trimming might be useful. However, the fat that the oe ient on Bahelors is the exeption demonstrates that it is di ult to nd perfet ases. If olumn 1 represents the "true oe ients," then olumns 3 and 4 represent attempts to orret olumn 11

15 2. Trimming at about 1/2 of the minimum wage is a ommon pratie (see Angrist and Krueger, 2000). Column 3 represents this approah. Another logial orretion is to trim at the minimum wage; olumn 4 represents this approah. Comparing olumn 3 with olumns 1 and 2, we nd that the oe ients on experiene and experiene squared and blak are largely una eted by the trimming, and are still "too large." The oe ient on less than high shool has atually inreased in magnitude, and made the bias worse. The oe ients on assoiates degree and graduate degree have delined in magnitude, reduing the bias, but not eliminating it. The oe ient on bahelors degree has dereased in magnitude inreasing the bias in this oe ient. Trimming at the minimum wage, represented by olumn 4, improves some oe ients but not others. The oe ients on experiene and experiene squared are now both smaller in magnitude and loser to the "ideal" olumn 1. The oe ient on less than high shool has now delined in magnitude, but it still somewhat larger than the oe ient in olumn 1. The oe ient on assoiates degree has delined and is biased relative to the target in olumn 1; it now underestimates the magnitude. The oe ient on graduate degree has not hanged any further and still overstates the target in olumn 1. The oe ient on blak has delined in magnitude and is now loser to the oe ient in olumn 1, but is still larger in magnitude. The oe ient on bahelors degree has delined further in magnitude inreasing the bias still further. The onlusion we take from this is that there is no lear advantage to trimming. While it ertainly may redue the bias for some estimates, it is simultaneously making other estimates worse. Sine it is rare to have a target (as we do in this ase), it is only through serendipity that one will pik the right trimming rule even if the researher is only interested in one spei oe ient. A seond perspetive on the estimates in olumns 1 and 2 is that they both ontain measurement error in the dependent variable. One would expet that trimmed versions of the two regressions would onverge to some set of orret estimates. Columns 5 and 6 are trimmed versions of olumn 1. As one might expet, the reported wage has fewer observations below the trimming points than the onstruted wage. In olumn 5 there is very little hange in the oe ients. In olumn 6 there is little hange the oe ients on experiene, experiene squared, or less than High Shool. The oe ient on Bahelors degree inreases in magnitude. This is in sharp ontrast to trimming the onstruted wage, where the oe ient dereased in magnitude. The oe ients on graduate degree and Blak both inrease in magnitude slightly. We onlude that it appears only serendipitous if any oe ients onverge with trimming. If one is interested in the return to a ollege degree, trimming is likely to be undesirable. While if one is interested in the Blak-white wage gap, trimming at an even higher threshold may be desirable. The e ets of trimming are unlear sine we annot even predit whih diretion the slope oe ients will hange when we trim. Without apriori information, it is di ult or impossible to know if trimming has redued bias, inreased bias, or some of both. 12

16 6 Conlusions The ommon pratie of leaning data by removing observations where the dependent variable is larger or smaller than some threshold is used in the hope of reduing the impat of measurement error. While this sounds sensible, it may make matters worse. Analytial results using normality demonstrate that leaning strategies using trunation or winsorizing work only the ase where measurement error in the dependent variable results in an upward bias on the magnitude of oe ients. This ase is not empirially supported by investigations of response error in earnings data. Under ertain irumstanes it may be possible to ahieve an optimal leaning strategy, but if the information neessary for that result were available, a simpler approah based on resaling the estimated oe ients would work too. Our empirial results demonstrate that a 1 perent winsorizing rule does not alter the results in a meaningful manner, but we note that this poliy is generally dominated by doing nothing to the data. Still, winsorizing is learly better than trunation. We aution, however, that small inrements to this rule (for example to 5 perent) an dramatially inrease bias and render the leaning undesirable on MSE grounds. Two important extentions we hope to address in future work is leaning based on ovariates, and leaning based up panel data where the researher has multiple observations on the dependent variable. In both ases, it may be possible to develop leaning proeedures that exploit other information. 13

17 7 Appendix Derivation of equation10 To solve for the expression in 10, begin by noting that E [y] = x and = x u + 2 ": Additionally we simplify the analysis by only onsidering a symmetri trunation sheme where = E [y] and C = E [y] + : so that only need be found. Consider rst the expression for in this ase: 2 = E[y] 4 E[y] C E[y] C x x +2 2 u +2 " E[y] C E[y] E[y] C E[y] C E[y] E[y] substituting the symmetri expressions for ; C yields v(y) = = 1 A : Next, noting that is observable and is assumed to be known solve and : = 1 1 : 1 (1 ) Substituting for and solving, yields the impliit relationship expressed in equation (10): 0 1 A = 1 : Proof of Proposition = 1 for in terms of To show that the bias gets worse with more variane in ", di erentiate the bias term with respet to the variane of the 2 " 1 (1 ) 2 1 (1 ) 2 2 (1 2 = 1 (1 ) 2 2 " This term is negative if and only if the numerator is negative. Considering only the numerator and grouping similar 2 " : " 14

18 As noted in Goldberger both 2 and are bounded in the unit interval. Inspetion of the de nition of 2 learly < 0: Now, onsider the de nition of : inspetion reveals that this has 2 " the trunation points standardized by the mean and variane of y: Hene inreasing 2 " is equivalent to inreasing and dereasing C for a trunated standard normal random variable. Sine is also the ratio of the variane of the trunated standard normal to the variane of the untrunated standard normal (see Goldberger), inreasing and dereasing C will result in a lower variane for the trunated distribution and thus a lower. Hene, 2 " established. < 0. Combined with the bounds on 2 and, the result is Does Trimming Redue Standard Errors Under assumptions A1-A7, the onditional distribution of y i jx i is N x 0 i b; V (y i) 1 2 : We an use the results in Goldberger (1981) to obtain an expression for the trunated seond moment matrix as: h The term E E Q = E [x i x 0 ij y i C] = E [x i x 0 i] (1 ) E [x i x 0 i] bb 0 E [x i x 0 i] : i (y i x 0 i b ) 2 x i x 0 j y i C i is onsidered by using the law of iterated expetations. The an be deomposed into the variation of y i around the onditional mean in h (y i x 0 i b ) 2 jx i ; y i C the trunated distribution, and the squared di erene between the onditional mean and the linear projetion term x T i b : Doing so yields: 1 h E i (y i x 0 ib ) 2 jx i ; y i C = 1 (1 ) (x 0 ib m (x i )) 2 ; where m (x i ) is the onditional mean of y i given x i in the trunated sample. Combining these terms produes the equation in the paper: AV bb = 1 +Q 1 E 1 (1 ) 2 2 Q 1 h (x 0 ib m (x i )) 2 x i x 0 ij y i C i Q 1 : Analyti Results: Winsorized Data When data are winsorized, no observations are removed, but values of y i outside of the region (; C) are transformed as follows: y w i = C if y i C y i if < y i < C y i : As in the setion on trimming, let b represent the vetor of full-sample (unleaned) oe ients. Under bivariate normality, the winsorized oe ient vetor, b is given by : 2 0 C E b [yi ] E [yi ] 6 = : C E[yi] V (y i) E[yi] V (y i) C E[yi] V (y i) E[yi] V (y i) 1 A b: 15

19 Does Winsorizing Redue Standard Errors? The e ets of Winsorizing on the variane are derived similarly to the results for trimming. We start by 2 2 noting that the AV b = E [x i x 0 i ] 1 E yi W x 0 i b xi x 0 i E [x i x 0 i ] 1 : Here, the term E yi W x 0 i b jxi an be broken into three terms: h E yi W x i b i 2 E [yi ] jxi = x 0 ib 2 C E [yi ] + 1 C x 0 C E [yi ] E [yi ] + Combined with results from the previous setion, we obtain ib 2 h i E (y i x 0 ib ) 2 jx i ; y i C AV b = C E [yi ] E [yi ] 1 1 (1 ) 2 E [x i x 0 1 i] QE [xi x 0 1 i] h +E [x i x 0 1 i] E x 0 ib m W (x i ) i 2 xi x 0 ij < y i < C E [x i x 0 1 i] E [yi ] h i + E [x i x 0 1 i] E ( x i b ) 2 x i x 0 i jy i E [x i x 0 1 i] C E [yi ] h i + 1 E [x i x 0 1 i] E (C x i b ) 2 x i x 0 i jy i C E [x i x 0 i] 1 : Again, the omparison is di ult. Here, the rst term will neessarily be smaller than the OLS expression. However, the seond, third and fourth terms are all positive de nite. As in the trimming ase, the impat on standard errors depends upon the parameters of the model. 16

20 Referenes [1] Angrist, Joshua D. and Alan B. Krueger, Empirial Strategies in Labor Eonomis, in Orley Ashenfelter and David Card (Eds.) Handbook of Labor Eonomis, Vol 3A (Elsevier Siene). [2] Blak, Dan A., Mark C. Berger and Frank A. Sott, Bounding Parameter Estimates with Nonlassial Measurement Error, Journal of the Amerian Statistial Assoiation 95: [3] Bollinger, Christopher R., Bounding Mean Regressions When A Binary Regressor is Mismeasured, Journal of Eonometris 73: [4] Bollinger, Christopher R., 2003 Measurement Error in Human Capital and the Blak -White Wage Di erential, Review of Eonomis and Statistis85: [5] Bollinger, Christopher and Martin H. David Modeling Food Stamp Partiipation in the Presene of Reporting Errors, Journal of the Amerian Statistial Assoiation 92: [6] Bollinger, Christopher, Measurement Error in the Current Population Survey: A Nonparametri Look, Journal of Labor Eonomis 16(3): [7] Bollinger, Christopher and Amitabh Chandra "Iatrogeni Spei ation Error" NBER Tehnial Working Paper 289, Cambridge, MA. [8] Bound, John and Alan B. Krueger, The Extent of Measurement Error in Longitudinal Earnings Data: Do Two Wrongs Make a Right? Journal of Labor Eonomis 9: [9] Bound, John and Rihard Freeman, What Went Wrong? The Erosion of Relative Earnings and Employment Among Blak Men in the 1980s, Quarterly Journal of Eonomis 107(1), February: [10] Bound, John, Charles Brown, Greg J. Dunan and Willard L. Rodgers, Evidene on the Validity of Cross-Setional and Longitudinal Labor Market Data, Journal of Labor Eonomis 12: [11] Buhinsky, Mohe, Changes in the U.S. Wage Struture : Appliation of Quantile Regression, Eonometria 62: [12] Card, David and Alan B. Krueger, 1992a. Shool Quality and Blak-White Relative Earnings: A Diret Assessment, Quarterly Journal of Eonomis 107, February: [13] Fuller, Wayne A Measurement Error Models. John Wiley and Sons. (New York, NY). [14] Goldberger, Arthur S., 1981, Linear Regression after Seletion, Journal of Eonometris 15(3):

21 [15] Hirsh, Barry T. and Edward J. Shumaher, Math Bias in Wage Gap Estimates Due to Earnings Imputations, unpublished manusript. [16] Horowitz, Joel L. and Charles F. Manski, Identi ation and Robustness with Contaminated and Corrupted Data, Eonometria 63(2): [17] Hyslop, Dean R. and Guido W. Imbens, Bias From Classial and Other Forms of Measurement Error, Journal of Business and Eonomi Statistis 19(4): [18] Juhn, Chinhui, Kevin M. Murphy and Brooks Piere, Wage Inequality and the Rise in the Returns to Skill, Journal of Politial Eonomy 101: [19] Katz, Lawrene and Kevin M. Murphy, Changes in Relative Wages , Quarterly Journal of Eonomis 107(1): [20] MaDonald, Glenn M. and Robinson Chris, Cautionary Tales About Arbitrary Deletion of Observations; or, Throwing the Variane out with the Bathwater, Journal of Labor Eonomis 3(2): [21] Maddala, G. S., Limited Dependent and Qualitative Variables in Eonometris (Cambridge University Press). [22] Manski, Charles F., Identi ation Problems in the Soial Sienes (Harvard University Press). [23] Mellow, Wesley and Hal Sider, Auray of Response in Labor Market Surveys: Evidene and Impliations, Journal of Labor Eonomis 1: [24] Rodgers, Willard L., Charles C. Brown and Greg J. Dunan, Errors in Survey Reports of Earnings, Hours Worked and Hourly Wages, Journal of the Amerian Statistial Assoiation 88: [25] Stigler, Stephen M., Do Robust Estimators work with Real Data? Annals of Statistis 5(6):

22 Table 1: Effet of Cleaning Proedures on Unorrupted Data, Evidene from the Returns to Shooling in 1990 PUMS Data Mean Population b No Cleaning Trim 1% Trim 5% Wins 1% Wins 5% Median Shooling SE (Yrs of Shooling) RMSE Potential Experiene SE (Shooling) RMSE Pot. Exp. Sq / SE (Pot. Exp) RMSE Blak (1= yes) SE (Blak) RMSE Constant SE Dependent variable is ln hourly wage. PUMS data are restrited to white (non-hispani) and blak men in the 1990 PUMS files of the Deennial Census who are aged during the ensus referene week. Nonworkers and repondents with hourly wages less than $3.35 in 1989 (the nominal value of the minimum wage) are deleted from the analysis. Column (1) reports means and standard deviations for this sample of 346,900 individuals, and olumn 2 reports the parameters from estimating the model: ln wage=b 0 +b 1 Shooling +b 2 Exp+b 3 Exp 2 +b 4 Blak + u on this sample. Reported estimates in other olumns are empirial sample moments from 1,000 repliations eah with a sample size of 1,000.