The Impact of Homework Time on Academic Achievement

Transcription

1 The Impact of Homework Time on Academic Achievement Steven McMullen The University of North Carolina at Chapel Hill December 2007 Abstract Several charter schools have shown great success in improving the academic performance of students from low-performing school districts. It has been argued, however, that their success may be due to student selection. This paper tests of two aspects of these charter schools reforms using nationally representative panel data on student behavior and academic performance. First, I examine a policy that increases the proportion of homework that students complete. Second, I examine the impact of increasing the amount of homework assigned. Previous studies have not been able to accurately estimate the impact of homework because of important omitted variables and measurement error, which strongly bias the estimated impact of homework time. This paper, however, uses an instrumental variables approach with student fixed effects to account for both time-varying and time-invariant unobserved characteristics and inputs. This approach produces estimates of the impact of homework time on academic achievement that are much higher than those of previous studies. Also, when compared to popular policy changes such as decreasing class size or increasing teachers wages, a policy of assigning more homework is found to be the most cost effective policy tool. Finally, these findings suggest that assigning additional homework primarily improves the achievement of low performing students and students in low performing schools. Thus, assigning more homework could help close the gap in achievement between high and low performing students. The author would like to thank Tom Mroz, David Blau, Donna Gilleskie, David Guilkey, and Helen Tauchen for their excellent assistance. Steven can be reached at mcmullen@unc.edu or Department of Economics, Gardner Hall CB# 3305, University of North Carolina, Chapel Hill, NC

2 I. Introduction The KIPP (Knowledge Is Power Program) charter schools have demonstrated much higher than normal achievement growth, often among at-risk populations (AEI 2005). Scholars disagree, however, about whether this success is due primarily to a superior schooling model, or to student selection (Rothstein et al. 2005). One of the core reforms that the KIPP schools 1 is their treatment of homework. They assign more homework than other public schools 2, and have strict polices meant to encourage students to complete all of the homework assigned. The academic literature pertaining to these reforms, despite much research on the topic, is not conclusive. Scholars that have approached this topic have done little to correct for the biases caused by omitted variables that likely have a large influence on students choices regarding study time. 3 In this paper, I examine two related issues. The first is the effect of students time spent doing homework on achievement test scores. The second is the impact of assigning additional homework on achievement. By testing the impact of these policies using nationally representative data, this study avoids the sample selection problems that accompany comparisons of charter school students test scores with those of their peers. The primary econometric challenge that this study overcomes is that there are likely a host of unobserved variables, such as student ability, that influence both how much time a student spends on homework and the students achievement test scores. By 1 Other successful charter schools have implemented similar reforms, such as Roxbury Prep School in Boston, MA. 2 Assigning homework seems to be less and less popular. The Brookings Institution press release (2003) writes that Since 2001, feature stories about onerous homework loads and parents fighting back have appeared in Time, Newsweek, and People magazines; the New York Times, Washington Post, Los Angeles Times, Raleigh News and Observer, and the Tampa Tribune; and the CBS Evening News and other media outlets. Their research indicates that the homework load for most US students is actually quite low. 3 Recent work done by economists such as Aksoy and Link (2000), and Stinebrickner and Stinebrickner (2007) have made progress, but their approaches can only address a subset of the endogeneity problems involved with measuring this effect. 2

3 combining instrumental variables estimation with an individual-fixed-effects specification, I control for both unobserved heterogeneity that is constant over time and time-varying heterogeneity that might influence the amount of time students spend on homework. This approach yields estimates of the impact of homework that are much larger than those of previous studies. The presence of time-varying unobserved factors is especially important to consider when estimating the impact of education inputs that are under the students control, since students can respond in each period to changing incentives. I find that one extra hour of mathematics homework per week improves mathematics achievement by standard deviations 4. This change is large enough to move a student from the 50 th percentile of math achievement to 59 th percentile 5 over the course of a school year. Additionally, this effect varies based on student characteristics and institutional factors. I find evidence that low achieving students and those in lowerperforming schools realize much higher returns to their homework time than other students. Likewise, a policy of assigning more homework disproportionately benefits these students. These findings lead to two important conclusions. First, that it is possible for students to overcome past poor performance or a low quality school by spending more time doing homework. Second, a policy that increases the amount of homework assigned is likely to reduce the gap between low and high achieving students. Finally, in order to compare policy instruments, I estimate the cost-effectiveness of increasing student achievement by assigning more homework, decreasing class size, 4 The impact is reported here in terms of the standard deviation of the mathematics achievement test score used as the measure of academic achievement in this study. This sample used for this study is nationally representative, and thus the impacts can be interpreted as increases in achievement relative to the performance of similar students in the US. 5 The impacts are relative to students who do not increase their homework time. 3

4 and increasing teachers salaries. For a given monetary investment, increasing the amount of assigned homework improves achievement almost three times as much as increasing teachers wages, and eleven times as much as decreasing class size. II. Background Previous studies within the education production function literature have documented a number of inputs that have an impact on students academic achievement. These include school funding (Altonji and Dunn 1996), class size, 6 teacher characteristics and training (Hanushek, et al 1998; Hanushek and Rivkin 2007), the amount of time spent in class or in school (Aksoy and Link 2000), and the performance of a student s peers (Sacerdote 2001; Zimmerman 2003; Hanushek and Rivkin 2006). Certain characteristics of the student and their family also are important, including parents permanent income (Blau 1999; Dahl and Lochner 2005) as well as the student s race and sex. The focus of this study is on another input: student time spent on homework. The theoretic literature predicts a positive impact of homework time on academic achievement (Betts 1996; Neilson 2005). Almost all of the empirical studies on this topic find evidence that homework time has a positive impact on academic achievement, although there is no consensus on the magnitude of the impact. The literature pertaining to this topic will be presented in two sections: first, a review of literature from outside the economics discipline and second, a review of recent work done by applied economists. 6 Decreasing class size has a small impact, except in early grades. For a review of the class size literature, see Hanushek (1998). 4

5 Within the fields of education, psychology, and sociology there has been much research on the impact of homework. Cooper et al. (2006) provides a good review of recent work in these fields. All of the published studies that Cooper et al. review that use multivariate regression analysis find similar results: increased homework causes a small increase in academic achievement. These studies, however, are limited to cross section analysis and do not try to take into account omitted inputs or characteristics. Shuman et al. (1985), Hill (1991), and Rau and Durand (2000) all find similar results when trying to document the impact of homework time on the performance of college students. Each study either finds the relationship hard to document or finds a small effect of homework time on college grades. Applied economists have only recently started to pay attention to the impact of homework time on achievement, starting with Julian Betts 1996 analysis of the Longitudinal Study of American Youth. Since then Aksoy and Link (2000), Eren and Henderson (2007), and Stinebrickner and Stinebrickner (2007) have all examined this question. Each of these studies makes a serious attempt to address unobserved inputs. Eren and Henderson, in the parametric portion of their paper, use a value added specification, 7 both Betts and Aksoy and Link use a specification with student fixed effects, and Stinebrickner and Stinebrickner use the students roommate s videogame ownership as an instrument, estimating the effect with two stage least squares. This study improves the estimation of the effect of homework on academic achievement in three ways. First, the primary shortcoming in this literature is that 7 The value added specification is a cross section regression with a lagged test score (or other dependent variable) included on the right hand side of the equation to control for past achievement, inputs, and student characteristics. 5

6 previous work fails to adequately account for unobserved inputs that influence both academic achievement and the amount of time spent on homework. None of these studies, with the exception of Stinebrickner and Stinebrickner 8 can account for unobserved influences that vary over time. To address this potential problem I use the amount of assigned homework and the student s locus of control 9 as instruments for student homework time, as well as a student fixed effects. Dealing with the endogeneity of students homework time in this manner greatly influences the estimated effects. Second, previous research has documented that the return to homework time varies based on a student s ability (Eren and Henderson 2007). In this study, I also document that the return to homework also varies with school quality. Finally, this paper documents that a policy of increasing the amount of homework assigned to students is a cost-effective alternative to traditional policy interventions, and that this policy would disproportionately aid low-performing students and those in low-performing schools. III. A Model of Homework Time Allocation and Academic Achievement This section presents a standard descriptive model of homework demand in the context of an education production function. This framework will be used in the next section to motivate an econometric strategy. The utility of student i at time t depends on leisure time (D i -H it ), future expected income (I i ), and individual preferences ( P X it ), where 8 Stinebrickner and Stinebrickner (2007) does a good job documenting the relationship between homework and achievement, but does so within a selected non-representative college age population. Their instrumental variable strategy, however, is not reproducible on a large scale. 9 The locus of control measures the extent to which students believe that they can influence their own future outcomes. For a detailed description, see appendix A. 6

7 D i is the disposable time available to the student to allocate between homework time (H it ) and leisure. P (1) U = u( D H, I, X ) it i it i it The students human capital (E it ) is also a function of past human capital investments (P i ), innate ability (A i ), school and district inputs (S it ), teacher inputs (T it ), individual characteristics ( C X it ), and a shock ( υ it ) that might include an external event (illness, divorce) or the chance of getting placed in a class which proves exceptionally difficult or easy for other reasons. (2) E (,,,,, C it = f Hit Ai Pi Sit Tit X it, υit) This education production function will dictate the return (in terms of academic achievement) of an hour of homework, which can vary by student ability, school quality, and other characteristics. Education, and thus homework, pays off by increasing future income, which is also a function of ability, labor market characteristics (L T ), such as the unemployment rate, and whether or not the student chooses to go to college (C i ). Thus labor market characteristics will influence homework choices if they impact the student s expectations about the labor market that he/she will enter upon graduation. (3) I = g( E, A, L, C ) i it i T i A student s demand for homework will depend on the elements of their utility function, the determinants of their future expected income, and the inputs into the education production function. (4) H = h( A, P, S, T, L, X, X, υ ) * P C it i i it it t it it it 7

8 Even within a class students will vary in the amount of homework that they do because they differ in levels of ability, levels of past achievement, preferences, and because they realize different shocks to their education which may induce them to study more or less often. IV. Empirical Approach Using a framework based on the theoretical model, this section explains the challenge of identifying the parameters of the education production function (equation 2), and presents an econometric approach for overcoming these challenges. 10 The simplest approach would be to estimate the following linear cross sectional econometric model: E = + H + S + T + X + + (5) it α0 α1 it α2 it α3 it α4 it λi ε it where E it is an achievement test score, which serves as a measure of human capital, the alpha parameters represent the impact of the inputs on academic achievement, λ i is an error term that is constant over time, and ε it is a time-varying error term. The error term λ i will include two unobserved determinants of a student s human capital: student ability (A i ) and past education inputs (P i ). The time varying error term ( ε it ) will include the unobserved education shock ( υ it ). Estimating this model may produce biased estimates of the impact of homework (the parameter α 1 ) for three reasons: 1. The input H it is partially determined by the student ability (A i ) and previous human capital investments (P i ), which also impact the students achievement. 10 For a more general discussion of the identification of education production function parameters see Todd and Wolpin

9 These inputs are unobserved, however, and their impact on student study time is uncertain. 2. The unobserved human capital shock υ it likely biases the parameter estimate downward, since a negative shock to a students education would likely both decrease the amount the student learns in the year and increase the amount of time spent on homework (Stinebrickner and Stinebrickner 2007) The amount of homework that students complete is measured with some error. This variable is recorded categorically, and the questions and categories change slightly across waves. Measurement error of this type could bias the parameter estimate downward. Because the magnitude of these impacts is unknown, and there are possible biases in each direction, it is unclear whether the OLS cross section estimates will be biased upward or downward. One method for addressing these identification problems is the use of instrumental variables estimation. The required instruments must affect student homework and be uncorrelated with the omitted variables. Using these variables to instrument for homework time isolates variation in the homework variable that is not correlated with the omitted variables. Estimating the effect of this exogenous variation in homework on achievement gives a consistent estimate of the impact of homework. Using the two stage least squares estimation method, first, the amount of time spent on homework is estimated using ordinary least squares, as shown in equation (6): 11 For example, if a student became seriously ill for one semester, their test scores would likely suffer, but at the same time, they might increase the amount of time spent on homework in an attempt to make up for missed school. This would create a spurious negative correlation between homework time and test scores. A similar process may be likely for other types of shocks to a students education, such as an poor teacherstudent match. For a larger discussion of this effect, see Stinebrickner and Stinebrickner

10 = (6) Hit β0 β1zit β2sit β3tit β4 X it β5lit φi ϕit In this specification, the school inputs, teacher inputs and observed student characteristics are assumed to be exogenously determined. The second step is to estimate the education production function as shown in equation (7), using the amount of homework predicted by the estimation of equation (6) ( H ˆ it ) instead of the observed homework amount (H it ). AT = α + α Hˆ + α S + α T + α X + λ + ε (7) it 0 1 it 2 it 3 it 4 it i it The omitted ability, past inputs, and education shock variables (A i, P i, υ it ) are still present in the error terms in equation (7), but they will not be correlated with the predicted homework variable ( H ˆ it ). The instrumental variables approach may not produce a consistent estimate of the impact of homework if there are endogenous variables other than the homework variable. If the omitted variables are also correlated with the other inputs in the education production function, then including these other inputs can bias the estimate of the effect of homework ( α 1 ). A fixed effects estimator (or within estimator) can be useful for eliminating certain types of omitted variables problems. To illustrate this, assume that H the observed value of each variable can be decomposed as follows: H = H + Hɶ + σ, where it i it it H i is permanent homework, or the average value of the homework variable across time for student i, H ɶ it is transitory homework, defined as the deviation from the student s H permanent homework for that year, and σ it is measurement error in the homework variable. Rewriting equation (5), and suppressing the measurement error term for all except the homework variable yeilds: 10

11 ~ (8) AT AT 0 1( H Hɶ H ) 2( S Sɶ ) 3( T Tɶ ) 4( X Xɶ ) + = α + α + + σ + α + + α + + α + + λ+ ε i it i it it i it i it i it i it The inclusion of student fixed effects in the OLS specification will absorb the permanent component of each input, as well as the time in-variant error term λ i. What is shown in equation (9) is the resulting fixed-effects specification. ~ (9) ( ɶ H) AT = γ + α H + σ + α Sɶ + α Tɶ + α Xɶ + ε it i 1 it it 2 it 3 it 4 it it This will eliminate bias caused by omitted variables under certain conditions. First, the omitted variables must not change over time. Second, the relationship between the omitted variables and achievement must be linear. Third, the permanent component of each variable may be correlated with the omitted variables, but the transitory component must not be. Thus even if there are multiple endogenous inputs in the education production function, the bias from time-invariant omitted variables, such as student ability (A i ) and previous education investments (P i ), will be eliminated. Moreover, if the other inputs, such as teacher experience, certification, wages, or class size, are selected by students and parents, this selection is likely based on permanent student characteristics. As such, any bias caused by self selection is likely eliminated by using student fixed effects. The homework parameter ( α 1 ) will be the same in equation (9) and in equation (5) only if the marginal impact of a change in permanent homework time is equal to the marginal impact of a change in transitory homework time. The research of Cunha et al. (2007) suggests, however, that at least for large changes in education inputs, that this assumption is unrealistic. They find that the return to education inputs is much higher if they are spread out over many years, instead of being concentrated in a single period. 11

12 Thus the transitory effect that is measured in this paper is likely to be somewhat smaller than the cumulative impact of a permanent change in student homework time. The fixed effects estimation will likely produce estimates of the impact of homework ( α 1 ) that are biased downward, for two reasons. First, the time varying omitted shock υ it may be correlated with the transitory portion of the homework and test score variables. If this is the case, the fixed effects will magnify the impact of transitory changes in homework due to this shock. Second, the measurement error will now account for an even larger portion of the variation left in the observed amount of homework. This means that the bias caused by measurement error will be more serious in the fixed-effects specification (equation 9) than in the first specification (equation 5). In order to address these problems, I use instrumental variables to estimate a specification that includes student fixed effects. To do so, the first stage equation (equation 10) includes a student fixed effect δ i, as does the second stage education production function. Hɶ = δ + β Zɶ + β Sɶ + β Tɶ + β Xɶ + β Lɶ + ϕ (10) it i 1 it 2 it 3 it 4 it 5 it it (11) ~ ˆ AT = γ + α Hɶ + α Sɶ + α Tɶ + α Xɶ + ε it i 1 it 2 it 3 it 4 it it In this case, only the transitory component of the excluded instruments ( Z ɶ ) is used to predict the amount of time a student spends on homework. This somewhat eases the task of finding valid instruments, since the transitory component of the instrument can provide exogenous variation even if the permanent component is correlated with the omitted inputs. it 12

13 Instruments The two variables used in this study as instruments are the amount of homework assigned by the student s teacher and the student s measured locus of control. The amount of homework assigned by the student s instructor is unlikely to have any direct causal impact on test scores, apart from the effect on the amount of homework completed. This variable may reflect the ability and motivation of the students, however, if teachers assign more homework to classes with gifted or more motivated students, such as advanced placement or honors courses. For this reason, this variable may not be a valid instrument, except when student fixed effects are included in the first stage regression. The fixed effects will control for any selection based on time-invariant characteristics, such as student ability. These exogeneity of the instruments is tested in section six. The second variable that is used to predict students homework is the locus of control. This variable measures the degree to which a student believes that they can impact their own future, and varies greatly within students over time. A student with an internal locus is more likely to believe that their future depends on choices that they make, while a student with an external locus is more likely to believe that external forces will dictate the events of their life. 12 Previous studies have correlated a strong internal locus of control with positive outcomes in situations that require independent decisionmaking, such as positive health behaviors (Steptoe and Wardle 2001), lower dropout rates in distance education (Parker 1999) and success in web-based coursework (Wang and 12 The locus of control used here is a combination of students answers to three standard questions which ask the student things which might influence their future. For a detailed description of the construction of the locus of control variable, see appendix A. 13

14 Newlin 2000). Students with an internal locus, on average, spend more time doing homework, since they are more likely to expect the time investment today to pay off in the future. It may not be immediately obvious that the student s locus of control will be unrelated to their academic performance except through homework. It is reasonable to assume, however that variations in this variable do not impact a student s ability to answer mathematical questions correctly. I will show that the students locus of control does predict the amount of time spent on homework, and does not, in a fixed effects specification, separately predict achievement test scores. V. Data The primary data used for the empirical work are the National Education Longitudinal Study of 1988, a nationally representative longitudinal survey of students who were in the 8 th grade in Follow up surveys were given in 1990 and 1992, with teacher and school counselor surveys in each wave, and parent surveys in the first and third waves. With each survey the students were given achievement exams in mathematics, science, English, and history. The exams are designed to allow comparison across waves, and to accurately test students at different achievement levels. Of the 17,580 students who have at least two recorded mathematics exam scores, 2295 students were not included in the sample used here because they did not respond to the questions about homework time or locus of control. Moreover, 5928 students were not included because the students teachers did not respond to questions about their class size, experience or assigned homework, and 1455 were not included because of missing data on teachers wages or the student s peers behavior. The resulting sample includes

15 Table 1: Summary Statistics Wave 1 Wave 2 Wave 3 Mean Std. Dev. Mean Std. Dev. Mean Std. Dev. Hours of Mathematics Homework per Week Hours of Assigned Homework per Week Mathematics Test Score Class Size (in units of 10 students) Teachers are Certified in Subject Matter Inexperienced Teachers Minimum Annual Teacher Wage Locus of Control Average Peers' Test Score Average Peers' HW Time State Unemployment Rate Minimum Wage Industry Mix State Public four year tuition State Financial Aid Per Student State Higher Ed. Appropriations Per Student Number of Students This Data is from the National Education Longitudinal Study of Wave 1 corresponds to students in the 8th grade, wave 2 to the 10th grade, and wave 3 to the 12th grade. The summary statistics for this sample do not differ substantially from the complete sample of collected data. All monetary variables are measured in thousands of dollars, and inflation adjusted to 2005 dollars. observations on 7902 students. The characteristics of these students do not differ substantially from the general sample. Summary statistics of the variables used in this paper are shown in table 1. In addition to the NELS data, data on state minimum wage laws from the department of labor is added, as is industry mix data from the 1988, 1990, and 1992 IPUMS Current Population Survey March supplement. The state level unemployment rate data came from Bureau of Labor Statistics. These data sets are used to create the labor market variables, which are merged with the NELS by the students state of residence. Finally, the higher education variables come from the Almanac of Higher Education ( ). For variable definitions, see appendix A. 15

16 Table 2: Education Production Function Estimation Results Estimation Method OLS 2SLS OLS 2SLS 2SLS 2SLS 2SLS Hours of Math Homework per Week (0.002) (0.084) (0.001) (0.072) (0.074) (0.087) (0.086) Class Size (in units of 10 students) (0.008) (0.029) (0.004) (0.010) (0.011) (0.012) (0.012) Certified Teacher (0.026) (0.057) (0.014) (0.034) (0.034) (0.037) (0.037) Inexperienced Teacher (0.029) (0.102) (0.015) (0.051) (0.052) (0.059) (0.058) Minimum Annual Teacher Pay (in thousands) (0.002) (0.005) (0.001) (0.003) (0.003) (0.003) (0.003) Student Fixed Effects no no yes Yes yes yes yes Peer Characteristics no no no No yes yes yes Labor Market Characteristics no no no No no yes yes Higher Edu. Market Characteristics no no no No no no yes Standard errors are shown in parentheses. The dependent variable for each regression is the mathematics achievement test score. All regressions include wave indicators. The sample includes observations on 7902 students. Columns one and two are pooled regressions with the standard errors clustered at the level of the individual. The first stage regressions are shown in table 3. VI. Results This section first summarizes the estimates of the education production function. Second, it examines the validity of the instrumental variables, and finally it compares the impact of various education policies. The Impact of Mathematics Homework on Mathematics Achievement Table 2 displays estimates from each of estimation techniques and specifications described in the previous section. 13 Each specification includes wave indicator variables, the average observed class size and three indicators of teacher quality: whether the student s teachers are certified in their subject, an indicator that the teacher has less than 4 years of experience, and the minimum teacher pay in the student s school. 13 For the full results for the main specifications, see appendix B. 16

17 The ordinary least squares estimate of the impact of an hour of mathematics homework per week, shown in column one, is a standard deviation increase in the student s mathematics test score, which corresponds to an improvement of about 1.5 percentile points. 14 This estimate may be biased if the students unobserved prior education inputs or innate abilities influence the amount of time spent on homework. Column two shows the two stage least squares estimates of the same specification. The amount of homework assigned and the student s locus of control are used as instruments excluded from the second stage to identify the amount of homework that students complete. The instrumental variables estimate is twenty-three times higher that the estimate from column one, at achievement test standard deviations, or and improvement 29 percentile points for each hour of homework. This estimate is likely much higher for three reasons. First, it corrects for the downward bias due to measurement error. Second, it corrects for the downward bias documented by Stinebrickner and Stinebrickner (2007) that results from students responding to education shocks. Third, the two stage least squares estimate could be biased upward if students with greater motivation or intelligence select more difficult courses that include more assigned homework, since the amount of homework assigned is being used as an instrument. I test the validity of the instruments later, and find that in this specification, the estimate of the impact of homework is likely biased upward due to selection of this type. The instrumental variables estimate may also be biased if students select other inputs based on unobservable characteristics. For example, parents of more able children 14 Starting from the 50 th percentile in Mathematics test scores within the sample. 17

18 may select better school districts with smaller class sizes and more qualified teachers. If this is the case, the other endogenous inputs can bias the estimate of the impact of an extra hour of homework. In order to control for these selection issues, a fixed effects specification is employed. If the selection is based on student characteristics that do not change over time, then the student-fixed effects estimate will eliminate the bias not only in the impact of hours of homework, but also in the other inputs. This estimator will also control for the linear effects of any differences in past inputs that are not included in these specifications. The third column in table 2 shows the estimates of the same specification as the one shown in the first column, but with student fixed effects included. These estimates are much smaller. The impact of an hour of homework with student fixed effects is 1/6 th the size of the OLS estimates presented in the first column. The fixed effects estimate, however, is likely biased downward for a couple of reasons. First, as discussed in section four, any problems with measurement error can be magnified by including student fixed effects. 15 Second, Stinebrickner and Stinebrickner (2007) demonstrate that students respond to negative education shocks by spending more time on homework. This means that if a student becomes ill one semester they may increase their homework time. If this is the case, it could cause the OLS estimates of the impact of homework to be biased downward. The fixed effects estimate will magnify this bias as well. 15 The hours of homework variable is measured with some error: the question given to students is asked slightly differently in each wave, and is coded categorically. Students are assigned the mean value of each category range. 18

19 The problems with the fixed effects estimates can be addressed by using instrumental variables in addition to the student fixed effects. The fourth column in table 2 shows the results of the student fixed effects specification estimated using two-stage least squares, where the excluded instruments are again the amount of homework assigned by the student s teachers and the student s locus of control. This estimate, while less precisely estimated than the fixed-effects estimate, is much higher, and gives a quantitatively and qualitatively different result. A student who completes an extra hour of homework each week is estimated to improve their mathematics test score by 0.22 standard deviations, an improvement of 8 percentile points for each hour studied. A standard deviation increase in mathematics homework time, roughly 4 hours per week, would increase the student s mathematics test score by 30 percentile points. Even if we accept the lower bound of the 95% confidence interval for this estimate, which is standard deviations, the return to homework is still 13 times higher than the fixed effects estimate. This lower bound estimate indicates that an hour of homework per week would move a student from the 50 th percentile to the 53 rd percentile in mathematics achievement. If the instruments provide exogenous variation in student homework time, the estimates in the fourth column should not change substantially when other inputs are included in the specification. Columns five through seven test this by adding a series of other factors which might influence homework time. In each case the estimate of the return to homework changes only a small amount. First, in order to control for other school-wide inputs and peer group influences, the specification shown in column five includes the average amount of homework done 19

20 by the students peers, 16 and the average mathematics test score of the student s peers. 17 The estimate of the impact of homework remains almost exactly the same. Second, in column six, four state level labor market variables are added to the specification: the unemployment rate, the unemployment rate squared, the minimum wage, and an industry mix index that categorizes industries based on the average education level of their workers. These variables have been shown to be important predictors of student homework time (McMullen 2007). Moreover, column seven adds three higher education characteristics: the average tuition in four-year public institutions by state, the financial aid per college student by state, and state-level higher education appropriations per student. The combined impact of the labor market and college market variables is a small increase in the estimated impact of homework, indicating that controlling for external economic influences does not substantially impact the estimate of the return to homework. In the final specification, shown in the seventh column, one hour of homework is estimated to increase as student s mathematics test score by standard deviations, or nine percentile points. Four additional hours of math homework per week, a little over a standard deviation change, could move a student from the 50 th percentile in mathematics to the 82 nd percentile. This estimate is especially large when considered in the context of the literature on school inputs. Cunha et al. (2007) demonstrate that inputs into the education 16 The students peers, in this case, are any students with observed test scores from the same school in the same year. 17 Manski (1993) and Hanushek et. al. (2003) argue that peer effects of this type can introduce a bias due to the reintroduction of a student s ability through the student s influence on her peers. The estimated impact of homework does not change when these variables are included, indicating that if these variables are problematic, they do not bias the estimate of the impact of homework. 20

21 production function are best considered as cumulative investments, with strong diminishing returns in any one period. The effect estimated in this paper, however, is a transitory effect. The best interpretation of these estimates is that they measure the impact of an additional hour of homework in one period only. If a student devotes extra time to homework each year over the course of their high school career, however, the cumulative result on achievement is likely to be larger than the sum of these transitory effects. Other recent studies that have estimated the impact of homework time on student achievement have not been able to account for both time-varying and invariant unobserved characteristics. These studies found much smaller impacts. 18 The results reported here indicate that the use of student fixed effects to control for omitted inputs and characteristics has consistently produced estimates that are too low. This is likely the case because of a downward bias that results from measurement error, as well as downward bias caused by students responding to negative or positive education shocks. It is also likely that the impact of unobserved past inputs and characteristics biased the result downward, although the direction of bias from this source is uncertain. In light of this, the correction provided by the instrumental variables estimate is especially valuable. 18 The estimates presented in this paper are larger in magnitude than any in Cooper et al. s 2006 review. Aksoy and Link also used the NELS data, and employed student fixed effects. See Aksoy and Link (2000) table three specification 1. Their estimate is divided by 13, which is the approximate standard deviation reported in table 1. They found results similar to those in table two column three: an hour of homework increases mathematics achievement by standard deviations. The results of this study indicate that the true effect is much larger. 21

22 Table 3: First Stage Equation Estimation Results: Determinants of Student Homework Time Corresponding Column in Table 2: Hours of Homework Assigned per Week* (0.023) (0.026) (0.026) (0.026) (0.026) Locus of Control* (0.053) (0.076) (0.075) (0.075) (0.075) Class Size (in units of 10 students) (0.034) (0.041) (0.040) (0.040) (0.040) Certified Teacher (0.061) (0.144) (0.144) (0.144) (0.144) Inexperienced Teacher (0.113) (0.155) (0.154) (0.154) (0.154) Minimum Teacher pay (0.005) (0.009) (0.009) (0.009) (0.010) Student Fixed Effects no yes yes yes yes Peer Characteristics no no yes yes yes Labor Market Characteristics no no no yes yes Higher Edu. Market Characteristics no no no no yes Sargan Statistic (Chi-Sq p-value) P-value for F-test of the joint impact of excluded instruments The asterisk denotes instruments excluded from the second stage regression. Standard errors are shown in parentheses. The dependent variable for each regression is the total hours of math homework reported by the student. All regressions include wave indicators. The sample includes observations on 7808 students. Column two is a pooled regression with the standard errors clustered at the level of the individual. The second stage regressions are shown in table 2. First Stage Results and Instrument Validity It is important to demonstrate that the excluded instruments strongly predict students homework time without an independent effect on test scores. Table 3 shows the first stage equations for each of the regressions from table 2 that were estimated with two-stage least squares. These results show that both hours of assigned homework and locus of control are important determinants of student homework time in each specification. Additionally, the last row in table 3 shows the p-value for an F-test on the joint significance of the excluded instruments. The null hypothesis, that both variables 22

23 have no impact on student s homework, would be rejected at less than a 2% confidence level for every specification. For these instruments to be valid, however, it is also necessary that they do not have an independent impact on test scores, once the impact of homework is taken into account. The Sargan test for the over-identification of all instruments is reported for each specification in the last row of table 2. The null hypothesis for this test is that the instruments are uncorrelated with the error term, and thus that they are correctly excluded from the second stage equation. The null hypothesis can not be rejected for any of the fixed effects specifications at any reasonable confidence level. This test indicates that both the student s locus of control and the amount of homework assigned are valid instruments in a fixed effects specification. Despite this evidence, it is possible that the amount of assigned homework or the locus of control is correlated with past achievement. To test this, I estimate the impact of a lagged test score on the amount of homework students were assigned, including some current teacher-related covariates on the right hand side. The results are shown in table 4. The estimates in the first column show that lagged test scores are strong predictors of each instrument when student fixed effects are not included. When the specification includes student fixed effects, however, the coefficient associated with the lagged test scores can not be distinguished from zero. This indicates that past academic achievement is not correlated with the instruments when fixed effects are included. It is also possible that both of the instruments are correlated with the same unobserved student or school characteristics and contain the same bias. If this were the case the Sargan test may still indicate that they are exogenous when they are not. In the 23

24 Dependent Variable Table 4: Testing Instrument Validity Indpendent Variable Variable Lagged Test Without Student Fixed Effects With student Fixed Effects Assigned HW Score (0.021) (0.118) Locus of Lagged Test Control Score (0.009) (0.037) Assigned HW Locus of Control (0.029) Standard errors are shown in parentheses. Each regression includes a wave indicator, and the specifications in row 2 include teacher pay, teacher experience and teacher certification variables. last row and last column of table 4 the locus of control is regressed on the amount of assigned homework in specification with student fixed effects and period intercepts. In this specification the student s locus of control has no effect on the amount of homework that the student is assigned. These variables, therefore, likely do not capture any common unobserved variables and do provide exogenous variation in homework time. Diminishing Returns To Homework Time So far the econometric model estimated has restricted the impact of homework to a linear effect, not allowing the possibility of diminishing returns to studying. This restriction is easily relaxed by estimating a second order polynomial in homework, which is done in table 5. The first two columns show the first stage equations predicting student homework time and student homework time squared. The instruments excluded from the second stage equation were the amount of assigned homework, the locus of control, and each of these instruments squared. 24

25 Dependent Variable Table 5: Decreasing Returns to Hours of Homework Hours of Math HW Hours of Math HW Squared Math Test Score Hours of Math Homework per Week (0.059) Hours of Math Homework Squared (0.004) Hours of HW Assigned per Week (0.051) (1.080) Hours of Homework Assigned Squared (0.004) (0.084) Locus of Control (0.075) (1.581) Locus of Control Squared (0.069) (1.457) Class Size (in units of 10 students) (0.040) (0.849) (0.008) Teacher Certified in Subject (0.144) (3.025) (0.023) Inexperienced Teacher (0.154) (3.232) (0.033) Minimum Teacher pay (0.010) (0.203) (0.002) Marginal Effect of HW Evaluated at the mean (0.043) Standard errors are in parentheses. The first two columns are the first stage equations; the third column is the second stage equation. Each regression includes student fixed effects, peer characteristics, labor market and higher education market characteristics, and wave indicators. For students currently not spending any time doing mathematics homework, the return on completing one hour of homework a week would be about a 10 percentile point improvement in mathematics test scores. This return decreases as students spend more time studying. The marginal effect of studying an additional hour if the student is currently studying the average amount (about 3.7 hours) is a 7.5 percentile point increase in test scores. These results also indicate that mathematics homework ceases to improve test scores if a student does more than about 11 hours per week. By this standard, the 25

26 majority of students could increase their test scores if they spent more time on their mathematics homework. Impact of Homework Time by Achievement Level and School Quality In this section, I will try to relax the restriction that students at different levels of achievement or in different types of schools receive the same return on homework time. To do this I estimating the specification from table 5 separately for high achieving and low achieving students, as well as high and low performing schools. Each specification includes a squared term, 19 and thus marginal effects are reported, evaluated at the mean reported homework time for the entire sample. First, students are separated into two groups of equal size based on their 8 th grade science test scores. The students in the lower achieving group experience much higher returns to studying than those in the high achievement group. The impact of an hour of homework for the low-achievement group is large enough to improve a student s mathematics achievement by 7.6 percentile points. An additional hour of homework by a high achieving student, however, only improves their achievement by 4 percentile points. This indicates that students who fall behind do have some ability to catch up to their peers simply by doing the same amount of homework. For example, consider a student in the low achievement group, who is currently spending one hour on homework each week. By studying three hours more per week, this student could move from the 25 th percentile to the 50 th percentile in a single year. A student in the high achievement group, who is currently studying one hour a week, would have to study seven additional 19 The effect is similar if the squared term is not included. The second order polynomial is used in this section because this specification results in more precise estimates. 26

27 Table 6: Impact of Homework Time by Student Achievement and School Quality Group 8th Grd. Sci. Exam School Quality Hours of Math Homework (0.088) (0.246) Bottom Half Math HW Squared (0.005) (0.017) Marginal Effect (0.057) (0.157) Hours of Math Homework (0.072) (0.063) Top Half Math HW Squared (0.004) (0.004) Marginal Effect (0.047) (0.041) The impact is measured by the coefficients associated with an hour of homework per week in an instrumental variables regression with student fixed effects, where the dependent variable is the Mathematics achievement test score. Student achievement is measured by the student's 8th grade science test scores. School quality is measured by the average mathematics test score of all other observed students in the same school. hours each week to make a similar move from the 50 th to the 75 th percentile. This evidence does not support the common assumption that more able students receive a higher return to homework time because their time is more productive (Neilson, 2005; Eren and Henderson, 2007). Second, students were split into groups of equal size based on the test score performance of their peers, as a general measure of school quality. Students in the lower performing schools experienced a return to homework time that was twice as strong as those in high performing schools. A student in a low performing school, who currently studies one hour each week that wanted to move from the 50 th percentile to the 75 th percentile, would have to study 1.8 additional hours per week, whereas a student in a high performing school would have to study four additional hours to improve the same amount. This may indicate that at low performing schools homework is a stronger determinant of success. Bishop (2007) reports that less material is presented in class time 27