For More Information

Transcription

1 THE ARTS CHILD POLICY CIVIL JUSTICE EDUCATION ENERGY AND ENVIRONMENT This PDF document was made available from as a public service of the RAND Corporation. Jump down to document6 HEALTH AND HEALTH CARE INTERNATIONAL AFFAIRS NATIONAL SECURITY POPULATION AND AGING PUBLIC SAFETY SCIENCE AND TECHNOLOGY SUBSTANCE ABUSE The RAND Corporation is a nonprofit research organization providing objective analysis and effective solutions that address the challenges facing the public and private sectors around the world. TERRORISM AND HOMELAND SECURITY TRANSPORTATION AND INFRASTRUCTURE WORKFORCE AND WORKPLACE Support RAND Browse Books & Publications Make a charitable contribution For More Information Visit RAND at Explore RAND Education View document details Limited Electronic Distribution Rights This document and trademark(s) contained herein are protected by law as indicated in a notice appearing later in this work. This electronic representation of RAND intellectual property is provided for non-commercial use only. Unauthorized posting of RAND PDFs to a non-rand Web site is prohibited. RAND PDFs are protected under copyright law. Permission is required from RAND to reproduce, or reuse in another form, any of our research documents for commercial use. For information on reprint and linking permissions, please see RAND Permissions.

2 This product is part of the RAND Corporation technical report series. Reports may include research findings on a specific topic that is limited in scope; present discussions of the methodology employed in research; provide literature reviews, survey instruments, modeling exercises, guidelines for practitioners and research professionals, and supporting documentation; or deliver preliminary findings. All RAND reports undergo rigorous peer review to ensure that they meet high standards for research quality and objectivity.

3 Retaining Students in Grade A Literature Review of the Effects of Retention on Students Academic and Nonacademic Outcomes Nailing Xia, Sheila Nataraj Kirby Prepared for the New York City Department of Education EDUCATION

4 The research described in this report was prepared for the New York City Department of Education and conducted within RAND Education, a unit of the RAND Corporation. The RAND Corporation is a nonprofit research organization providing objective analysis and effective solutions that address the challenges facing the public and private sectors around the world. RAND s publications do not necessarily reflect the opinions of its research clients and sponsors. R is a registered trademark. Copyright 2009 RAND Corporation Permission is given to duplicate this document for personal use only, as long as it is unaltered and complete. Copies may not be duplicated for commercial purposes. Unauthorized posting of RAND documents to a non-rand Web site is prohibited. RAND documents are protected under copyright law. For information on reprint and linking permissions, please visit the RAND permissions page ( permissions.html). Published 2009 by the RAND Corporation 1776 Main Street, P.O. Box 2138, Santa Monica, CA South Hayes Street, Arlington, VA Fifth Avenue, Suite 600, Pittsburgh, PA RAND URL: To order RAND documents or to obtain additional information, contact Distribution Services: Telephone: (310) ; Fax: (310) ; order@rand.org

5 PREFACE Many states and districts are moving toward test-based requirements for promotion at key transitional points in students schooling careers, thus ending the practice of social promotion promoting students without regard for how much they have learned. The rationale is that repetition of the grade will give students an additional year to master the academic content they failed to master the previous year, and, thus, students should be less at risk for failure when they go on to the next grade. Opponents of grade retention argue that prior research has shown that grade retention disproportionately affects low-income and minority children, and is associated with low self-esteem, problem behaviors, and an increased risk of dropping out of school. In , the New York City Department of Education (NYCDOE) implemented a new promotion and retention policy for 3rd-grade students in New York City (NYC) public schools. The policy was extended to grade 5 in , grade 7 in , and grade 8 in NYCDOE asked the RAND Corporation to conduct an independent longitudinal evaluation of the 5th-grade social promotion policy and to examine the outcomes for two cohorts of 3rd-grade students. This study conducted between March 2006 and August 2009 examined (1) policy implementation, factors affecting implementation, and implementation progress over time; (2) the impact of the policy on student academic and socioemotional outcomes; and (3) the links between implementation and desired outcomes. The study included a systematic and rigorous review of the literature on grade retention and its effects on a variety of short- and long-term student outcomes. This report presents the results of the literature review and is one of three reports documenting the findings of the evaluation of the NYC promotion policy. The second report (Marsh, Gershwin, et al., 2009) provides lessons learned from other districts and states implementing test-based promotion policies similar to that of NYC. The third report (McCombs, Kirby, and Mariano, 2009) provides findings from the overall evaluation. All three reports should interest policymakers, practitioners, and researchers involved in designing, implementing, or studying interventions to improve outcomes for low-performing students. This research was conducted by RAND Education, a unit of the RAND Corporation. iii

6

7 CONTENTS Preface... iii Figures and Tables...vii Summary... ix Acknowledgments...xiii Abbreviations...xv 1. Introduction...1 Grade Retention: Pros and Cons...1 Purpose of This Report...3 Organization of This Report Methodological Issues...5 Literature Search and Selection Criteria...5 Conceptual/Methodological Issues in Retention Studies...7 Types of Comparisons...7 Comparison Groups...9 Statistical Methods...9 Age at Retention...12 Length of Follow-Up...13 Teacher Ratings...13 Regression to the Mean Effects Characteristics of Retained Students Relationship Between Grade Retention and Student Outcomes...19 Academic Achievement...19 Studies Using a Regression Discontinuity Design...20 Socioemotional Outcomes...21 Behavioral Outcomes...24 Propensity to Drop Out of School...26 Propensity to Enroll in Postsecondary Education...26 Employment Outcomes Conclusions...29 Appendix. Summaries of Reviewed Studies...31 References v

8

9 FIGURES AND TABLES Table 2.1. Number of Studies, by Topic...6 Figure 2.1. Illustration of Comparison Strategies...8 vii

10

11 SUMMARY Grade retention refers to the practice of keeping students at the same grade level for an additional year. The rationale behind retention is that it gives low-achieving students an extra year to catch up to the grade-level standard. As part of an increasing emphasis on standards and accountability, many districts are making decisions about grade retention based on student scores on district or state standardized tests. While eliminating social promotion has considerable intuitive and political appeal, it has also raised important concerns, partly because prior studies have shown that students do not appear to benefit from being retained in grade and, indeed, that retention may increase their risk of dropping out of school. PURPOSE OF THIS REPORT In , the New York City Department of Education (NYCDOE) implemented a new promotion and retention policy for 3rd-grade students in New York City (NYC) public schools. The policy was extended to grade 5 in , grade 7 in , and grade 8 in NYCDOE asked the RAND Corporation to conduct an independent longitudinal evaluation of the 5th-grade social promotion policy and to examine the outcomes for two cohorts of 3rd-grade students. This study conducted between March 2006 and August 2009 examined (1) policy implementation, factors affecting implementation, and implementation progress over time; (2) the impact of the policy on student academic and socioemotional outcomes; and (3) the links between implementation and desired outcomes. As part of the overall study, we conducted a systematic and rigorous search of the literature on grade retention in particular, what was known about the characteristics of retained students and the short- and longer-term effects on student outcomes (both academic and nonacademic). This report presents the results of this literature review and helps place the findings of the larger evaluation in the context of previous studies. METHODS A systematic search of the literature was conducted to identify relevant studies published since Three selection criteria were used for inclusion: relevance, methodological rigor, and time of publication. To be included, a study had to examine K 12 grade retention, and the results needed to address at least one of the following issues: (1) characteristics of retained students and (2) effects of grade retention on any of six student outcomes, i.e., academic achievement, socioemotional ix

12 outcomes, behavioral outcomes, propensity to drop out of school, postsecondary education, and employment outcomes. The study had to be either a work of empirical research that used wellestablished statistical methods or a systematic and rigorous review of past research. Empirical studies were required to use a credible comparison group or statistical method to control for selection bias. Studies based mainly on descriptive statistics were excluded. For a review essay to be included, it had to use a clear analytical method for synthesizing past research, such as a systematic literature search process, selection criteria for inclusion, or statistical procedures for combining the results of past studies (for example, meta-analysis). Only studies published between 1980 and 2008 were included in this review. Based on the criteria of relevance and publication date, a screening of titles and abstracts yielded a total of 178 studies. Of the 178 studies, 54 failed to meet the methodology criterion, 29 were not particularly relevant to the topics covered here, two were not yet published, and two failed to meet both the relevance and methodology criteria. As a result, 91 met all three criteria and were included in this review. Among the 91 selected studies, 87 were empirical, three were metaanalyses, and one was a systematic review of past research. OVERALL FINDINGS Our review of these 91 studies produced the following conclusions: Relative to students who are promoted, retained students are more likely to be male, minority, younger than their peers, of low socioeconomic status, and living in poor households and singleparent families. They are also more likely to have poorer academic performance prior to retention; significantly lower social skills and poorer emotional adjustment; more problem behaviors, such as inattention and absenteeism; more school transfers; poorer health; and disabilities. Parents of retained students are more likely to have lower IQ scores and lower levels of cognitive functioning, lower educational levels, lower occupational levels, less commitment to parenting responsibilities for their children s education, lower expectations of their children s educational attainment, and less involvement in school. In general, retention does not appear to benefit students academically. In most of the studies included here, we find negative relationships between retention and subsequent academic achievement. On the other hand, a few studies have found academic improvement in the immediate years after retention. Even so, these gains are often short-lived and tend to fade over time. Findings from the few studies using rigorous methods to adjust for selection bias have been mixed as well with some showing short-term gains and others reporting gains that disappeared over time. x

13 Retained students have a significantly increased risk of eventually dropping out of school. Compared with their peers, retained students also appear less likely to pursue postsecondary education and more likely to have poorer employment outcomes in terms of earnings (although only a few studies have looked at this outcome). Findings on social, emotional, attitudinal, and behavioral outcomes among the retained students compared with their promoted peers appear mixed, with some studies reporting positive outcomes and others finding insignificant or even negative results. xi

14

15 ACKNOWLEDGMENTS We are grateful to our RAND colleagues and members of the NYCDOE evaluation team for their help in identifying articles for this review and for helpful discussions regarding findings. We thank Laura Hamilton for useful comments on an earlier draft, our reviewers Vi-Nhuan Le, Jeffery Marshall, and Jennifer Steele for their thorough and thoughtful reviews of this report, and Lauren Skrabala for her careful and patient editing. xiii

16

17 ABBREVIATIONS ANCOVA ANOVA BCPS BMG BPI CAT CPS CTBS ECLS-K ERIC FCAT GED GPA ITBS IV MANOVA NELS NHES NYC NYCDOE OLS R-CBM RD analysis of covariance analysis of variance Baltimore City Public Schools below modal grade (for age) Behavior Problem Index California Achievement Test Chicago Public Schools Comprehensive Tests of Basic Skills Early Childhood Longitudinal Study Kindergarten cohort Education Resources Information Center Florida Comprehensive Assessment Test General Education Development grade point average Iowa Tests of Basic Skills instrumental variables multivariate analysis of variance National Education Longitudinal Study National Household Education Surveys Program New York City New York City Department of Education ordinary least squares Reading Curriculum-Based Measurement regression discontinuity xv

18 SES TAAS socioeconomic status Texas Assessment of Academic Skills xvi

19 1. INTRODUCTION Retention as a remedy for poor academic performance has been hotly debated and extensively studied for decades. In an era emphasizing educational accountability, policies of determining grade retention based on students scores on standardized tests have been adopted by many states and school districts in an effort to end social promotion. 1 GRADE RETENTION: PROS AND CONS Grade retention, also known as nonpromotion, being retained, flunking, repeating a grade, and being held back, is the practice of keeping students at the same grade level for an additional year, usually because of poor academic performance or emotional immaturity. 2 The rationale behind retention is that it gives low-achieving students an extra year to catch up to the grade-level standard. The assumption is that by catching up on prerequisite skills, students should be less at risk for failure when they go on to the next grade (Shepard and Smith, 1990, p. 84). Critics of grade retention contend that it fails to benefit children academically in the long run, hurts children s self-esteem, leads to behavioral problems often associated with being over-age for grade, has a correlative relationship with dropping out of school, and incurs significant financial costs of having children repeat a grade (Anderson, Jimerson, and Whipple, 2005; Bowman, 2005; Byrd, Weitzman, and Auinger, 1997; Eide and Goldhaber, 2005; Tanner and Galis, 1997; Xia and Glennie, 2005). In contrast, social promotion refers to the practice of promoting students with their class or completion group whether or not they have obtained the requisite skills for the next grade, as judged by standardized or classroom assessments. The practice is justified on the grounds that a child, if not promoted along with his or her peers, may incur psychological and emotional damages, such as low self-esteem or a low sense of self-worth. Opponents of social promotion argue that such practice creates many problems: It can frustrate unprepared but promoted students by placing them in grades in which they are not ready for the work; it sends a message to students that they can get 1 As of 2005, 18 states had policies that specified an assessment to be used in determining student eligibility for promotion or retention, and several others authorized local authorities to establish promotion policies or consider specific criteria for promotion (Zinth, 2005). However, our survey of state Web sites in 2006 and 2008 indicated that states and districts changed the content of policies over time and sometimes decided to end their policies. Thus, at any given time, it is surprisingly difficult to identify the number of states and districts implementing test-based promotion policies. 2 Children in kindergarten or 1st grade are typically recommended for retention by teachers on the grounds of emotional immaturity for exhibiting problem behaviors, such as acting out, anxiety, or inattention. 1

20 by without working hard, adversely affecting student motivation and work effort; it forces teachers to deal with underprepared students while trying to teach the prepared students; it gives parents a false sense of their children s progress; it leads employers to conclude that diplomas are meaningless; and it dumps poorly educated students into a society in which they are not prepared to perform (Hartke, 1999; Thompson and Cunningham, 2000). As part of an increasing emphasis on standards and accountability, grade retention has often been advocated and adopted in conjunction with the use of testing to end social promotion. A test-based promotion policy typically uses standardized tests as the main criterion to make high-stakes decisions about whether a student should be promoted to the next grade. Such promotion policies are very different from the traditional form of teacher-initiated retention, under which retention decisions are typically based on the assessments of teachers and parents. While teachers may use test scores as the basis for retaining students in grade, their decisions are influenced by many additional sources of information, such as student attendance, grades, and behavior, as well as intangible factors such as their own attitudes toward retention and their perceptions of the students. As a result, teacher-initiated retention decisions are likely to be more subjective (Allensworth, 2005; Greene and Winters, 2006). Test-based promotion policies are often implemented with additional short-term remedial supports, such as after-school programs, summer schools, early identification, and interventions targeted toward at-risk students. The premise of such policies is that the threat of retention, along with additional intervention programs, will both motivate and help students meet grade standards. Supporters argue that the threat of retention will provide incentives for students to work harder, for parents to monitor their child s progress, and for teachers to focus on the development of basic skills among low-achieving students, all of which should lead to increases in student achievement (Allensworth, 2005; Jacob, Stone and Roderick, 2004; Roderick and Engel, 2001; Roderick, Nagaoka, and Allensworth, 2005; Roderick, Bryk, et al., 1999). However, critics of test-based promotion policies believe that standardized tests tend to narrowly direct teaching efforts and school resources toward raising student test scores at the expense of teaching other important skills. They contend that such tests result in measurement error and often fail to accurately define a student s achievement in a subject area. Moreover, a single measure may not adequately describe a child s progress, and students may score abnormally lower than their true ability as a result of transient idiosyncratic factors during the testing day (Hartke, 1999; Heubert and Hauser, 1999; Lorence and Dworkin, 2006; Roderick and Nagaoka, 2005; Roderick, Nagaoka, and Allensworth, 2005; Thompson and Cunningham, 2000). 2

21 PURPOSE OF THIS REPORT In , the New York City Department of Education (NYCDOE) implemented a new promotion and retention policy for 3rd-grade students in New York City (NYC) public schools. The policy was extended to grade 5 in , grade 7 in , and grade 8 in NYCDOE asked the RAND Corporation to conduct an independent longitudinal evaluation of the 5th-grade social promotion policy and to examine the outcomes for two cohorts of 3rd-grade students who were subject to the 3rd-grade promotion policy. This study conducted between March 2006 and August 2009 examined (1) policy implementation, factors affecting implementation, and implementation progress over time; (2) the impact of the policy on student academic and socioemotional outcomes; and (3) the links between implementation and desired outcomes. As a first step toward understanding what was known about the effects of grade retention on students academic and socioemotional outcomes, we conducted a systematic and rigorous search of the literature on grade retention in particular, what was known about the characteristics of retained students and the short- and longer-term effects on student outcomes (both academic and nonacademic). A preliminary review found that the literature on promotion policy implementation was sparse and not very rigorous. Most of this literature tended to address more generally the elements of a successful promotion policy, often using exemplars rather than generalizable and rigorous research. As such, we do not include it here. This report focuses on the effects of grade retention and helps place the findings of the larger evaluation with respect to students academic and socioemotional outcomes in the context of prior studies. 3 This is one of three reports (the second is a report on lessons learned from other states and districts who implemented promotion policies similar to that of NYC and the third a final report on the study) documenting the results of the overall RAND evaluation. ORGANIZATION OF THIS REPORT The remainder of this report is organized as follows. Chapter Two presents a description of the methods used to conduct the literature review, and discusses the conceptual and methodological characteristics of the reviewed studies. The report then synthesizes research findings on the characteristics of retained students (Chapter Three) and the effects of grade retention on six different outcomes including academic achievement, socioemotional outcomes, behavioral outcomes, propensity to drop out of school, postsecondary education, and employment outcomes 3 We also reviewed the few studies that examined the effect of supportive interventions on student outcomes. While these are not reviewed here, they are presented as part of the final report (McCombs, Kirby, and Mariano, 2009). 3

22 (Chapter Four). Chapter Five briefly summarizes our findings from the literature. For each of the studies reviewed here, the Appendix provides a summary, including the research questions, data and methods, and findings. 4

23 2. METHODOLOGICAL ISSUES LITERATURE SEARCH AND SELECTION CRITERIA A systematic search of the literature was conducted to identify relevant studies published since Key words used in the search included grade retention, grade repetition, academic retention, in-grade retention, student retention, early retention, school retention, grade failure, academic failure, non-promotion, retained, flunked, failed, promotion, social promotion, student promotion, promoted, and other synonyms. Electronic databases searched include Resources in Education, Education Resources Information Center (ERIC), Google Scholar, PsycINFO, Education Abstracts, Academic Search Premier, Education Full Text, Ingenta Connect, JSTOR, Sociological Abstracts, and Social Science Abstracts. Additional studies were identified through a review of the bibliographies of the retrieved publications. Three selection criteria were used for inclusion: relevance, methodological rigor, and date of publication. The research was required to examine grade retention in K 12 grades, and results had to address at least one of the following issues: characteristics of retained students and effects of grade retention on at least one of six student outcomes (academic achievement, socioemotional outcomes, behavioral outcomes, propensity to drop out of school, postsecondary education, and employment outcomes). The study had to be either an empirical study that used well-established statistical methods or a systematic and rigorous review of past research. Empirical studies were required to use a credible comparison group or statistical method to control for selection bias. Studies based mainly on descriptive statistics were excluded. For a review essay to be included, it had to use a clear analytical method for synthesizing past research, such as a systematic literature search process, selection criteria for inclusion, or statistical procedures for combining the results of past studies (for example, meta-analysis). Only studies published between 1980 and 2008 were included in this review. Based on the criteria of relevance and publication date, a screening of titles and abstracts yielded a total of 178 studies. Of the 178 studies, 54 failed to meet the methodology criterion, 29 were not particularly relevant to the topics covered here, two were not yet published, and two failed to meet both the relevance and methodology criteria. As a result, 91 met all three criteria and were included in this review. Among the 91 selected studies, 87 were empirical, three were metaanalyses, and one was a systematic review of past research. 1 Table 2.1 summarizes the number of studies per topic area. 1 The Appendix provides details of each study reviewed here. 5

24 Topic Table 2.1. Number of Studies, by Topic Total Number of Studies Number of Empirical Studies Number of Reviews Characteristics of retained students Demographics Family and parental characteristics Cognitive and academic functioning Socioemotional and behavioral development Health Effects of grade retention Academic achievement Socioemotional outcomes Behavioral outcomes Propensity to drop out of school Enrollment in postsecondary education Employment outcomes While the three meta-analyses and one review of past research met our selection criteria, they all had limitations. The meta-analysis by Holmes and Matthews (1984) focused on elementary and junior high school grades, and included published studies as well as dissertations and master s theses. 2 The publication dates of these studies range from 1929 to 1981, with the majority conducted between 1960 and The meta-analysis by Holmes (1989) was largely an update of the earlier meta-analysis by Holmes and Matthews (1984) and synthesized findings from 63 studies published between 1929 and Both meta-analyses were based on studies mostly conducted more than 30 years ago, which were dated and often lacked statistical controls for selection bias. Jimerson (2001) reviewed 20 research articles published between 1990 and Most of these studies suffered from small sample size, with 14 of the 20 studies having fewer than 100 retained or promoted students. Only three studies had sample sizes larger than 1,000 in both retained and comparison groups. The meta-analysis did not weight the study effects by sample size. 2 The 44 studies reviewed by Holmes and Matthews (1984) consisted of 18 published studies, 14 dissertations, and 12 master s theses. 3 The 63 studies reviewed by Holmes (1989) consisted of 20 published studies, 22 dissertations, 18 master s these, and three unpublished papers. Forty-four of these studies were included in an earlier metaanalysis by Holmes and Matthews (1984). 6

25 Jimerson, Anderson, and Whipple (2002) conducted a systematic review of dropout literature that examined grade retention with both associative and predictive models. This review included 17 studies published between 1970 and Unlike the three meta-analyses, it did not use statistical methods to synthesize findings. Rather, the authors provided a narrative discussion of the findings from the 17 studies. CONCEPTUAL/METHODOLOGICAL ISSUES IN RETENTION STUDIES Studies included in this review used a variety of analytical approaches. They differed on issues such as types of comparisons, comparison groups, and statistical methods, which may affect findings. Types of Comparisons Three types of comparison strategies were used in the retention literature: same-age comparison, same-grade comparison, and across-year comparison. Same-age comparison compares retained students with their peers who are of the same age but promoted to higher grade levels. Same-grade comparison uses performance of retained and promoted students measured at the same grade as the basis of comparison. There are two variations of same-grade comparisons. In the first variation, outcomes of retained students measured in a given year are compared with outcomes of the sameage promoted peers measured a year earlier. Essentially, it involves comparing data from different years for the retained and promoted students (Holmes, 1989; Karweit, 1999; Roderick and Nagaoka, 2005). The second variation of same-grade comparison uses younger, nonretained students as the comparison group. In other words, the comparison groups are in the same grade with but are younger than the retained students at the time the comparison is made (Alexander, Entwisle, and Dauber, 2003; Hong and Yu, 2008; Tanner and Galis, 1997). An example helps illustrate the differences between same-age and same-grade comparisons. Suppose a student is retained in 5th grade for one year. At the end of his retention year, a sameage comparison would compare his current performance with that of his former classmates, who are now completing their 6th-grade year (i.e., comparing B with C, as illustrated in Figure 2.1). The first variation of same-grade comparison would compare the retained student s performance at the end of the repeating year with the outcomes of the 6th graders measured one year earlier when they were his classmates (i.e., comparing B with A). Both outcomes are measured at the end of the 5th grade, but in two different years. The second variation of same-grade comparison would compare the retained student s performance at the end of the retention year with that of his current 5thgrade classmates who are one year younger than the retained student, assuming that both are on grade when they enter 5th grade (i.e., comparing B with F). 7

26 Figure 2.1. Illustration of Comparison Strategies Grade 5 Grade 6 Grade 7 Year 1 A Retained Promoted Year 2 B F C Retained Promoted Year 3 Retained G Promoted D H E Cohort 1, indicated by squares, enters 5th grade in year 1. Cohort 2, indicated by circles, enters 5th grade in year 2. SOURCE: Adapted from Alexander, Entwisle, and Dauber (2003). Hong and Yu (2008, p. 409) provide a succinct summary of the issues that arise with respect to both same-age and same-grade comparisons: The essential problem here is how to identify an appropriate comparison group for the retained group. Previous studies have compared the outcomes of the retained children either with their new classmates who were experiencing the grade for the first time (i.e., same-grade comparison) or with their same-age peers who had been promoted to the next grade (i.e., same-age comparison). Obviously, in same-grade comparisons, most retained students cannot be matched with their new classmates on age. Hence, the comparison group provides little counterfactual information about how the retained students would fare had they been promoted. Even in same-age comparisons, the retained group and the promoted group are still vastly different on average in many prior characteristics. Adjustment by means of a linear model or multivariate matching sharply constrains the number of background variables that can be controlled [Little, 1985; R. Stone, 1993]. In particular, most promoted children have little or no risk of ever being retained. When the two groups are barely comparable, statistical adjustment for a limited number of background variables cannot be relied upon to remove bias. They argue for using propensity score matching methods to define a credible comparison group. Studies evaluating Chicago s 1996 policy and Florida s 2002 policy to end social promotion used across-year comparisons. Students in a given grade who were subject to a test-based promotion 8

27 policy were compared with students enrolled in that grade in the years before the promotion policy was enacted. Comparison Groups Within each type of comparison strategy, three kinds of comparison groups can be used: a nonmatched group of regularly promoted students, a comparison group of low-achieving but promoted students, and a matched comparison group. The nonmatched group of regularly promoted students is often randomly selected from all promoted students. The low-achieving but promoted comparison group consists of promoted students with a same level of low achievement as retained students. These low-achieving but promoted students were often recommended for retention by teachers or had similar grades or achievement test scores as retained students. The matched comparison group uses some kind of statistical procedure (for example, propensity score matching) to ensure that the comparison group is statistically comparable to the retained group on a number of selected variables, such as gender, race, age, grade, socioeconomic status (SES), prior academic achievement, and family background. Statistical Methods Studies included in this review used a variety of statistical methods. The most frequently-used statistical method is to control for confounding factors by including covariates (such as demographics, family background, school factors, and prior achievement) in the model. Many studies also used hierarchical models to take into account the nested nature of educational data, in which students are clustered within classrooms, schools, and districts. As pointed out by Raudenbush (1988), hierarchical models allow parameters at a lower level of aggregation to vary as a function of parameters at the next-higher level, which helps address the aggregation bias problem. Moreover, such models enable specification of appropriate error structures, including random intercepts and random coefficients, thus addressing the misestimated precision problem associated with nested data structures (Raudenbush, 1988). Of the studies that used hierarchical models, the majority employed hierarchical linear models, with many using a two-level model (student and school level) and others using a three-level model (time, student, and school level) (Allensworth, 2004; Guevremont, Roos, and Brownell, 2007; Hong and Yu, 2008; McCoy and Reynolds, 1999; Nagaoka and Roderick, 2004; Reynolds, 1992; Roderick and Nagaoka, 2005; Silberglitt, Appleton, et al., 2006; Silberglitt, Jimerson, et al., 2006). Several studies used hierarchical logistic models for dichotomous dependent variables (for example, whether a student drops out of school) 9

28 (Allensworth, 2005; Hong and Yu, 2008; Goldschmidt and Wang, 1999; Ou and Reynolds, 2008; Reynolds, 1992; Willson and Hughes, 2006). 4 Despite the advantages of addressing issues associated with nested data, hierarchical models are not designed to solve the omitted variable bias problem. In other words, the estimated treatment effect (i.e., difference in outcomes between retained students and comparison groups) may be due to factors other than retention or variables included in the model. To control for omitted variable bias, several studies used propensity score methods (Hong and Raudenbush, 2005; Hong and Yu, 2007, 2008; Wu, West, and Hughes, 2008). D Agostino (1998) offers a useful summary of how and why propensity scoring works: In observational studies, investigators have no control over the treatment assignment. The treated and non-treated (that is, control) groups may have large differences on their observed covariates, and these differences can lead to biased estimates of treatment effects. Even traditional covariance analysis adjustments may be inadequate to eliminate this bias. The propensity score, defined as the conditional probability of being treated given the covariates, can be used to balance the covariates in the two groups, and therefore reduce this bias. In order to estimate the propensity score, one must model the distribution of the treatment indicator variable given the observed covariates. Once estimated the propensity score can be used to reduce bias through matching, stratification (subclassification), regression adjustment, or some combination of all three. (p. 2265) [B]y definition the propensity score is the conditional probability of treatment given the observed covariates e(x)= pr(z=1 X), which implies that Z and X are conditionally independent given e(x). Thus, subjects in treatment and control groups with equal (or nearly equal) propensity scores will tend to have the same (or nearly the same) distributions on their background covariates. Exact adjustments made using the propensity score will, on average, remove all of the bias in the background covariates. Therefore bias-removing adjustments can be made using the propensity scores rather than all of the background covariates individually. (p. 2267) A few studies used an instrumental variables (IV) approach to estimate the causal relationship between grade retention and student outcomes (Eide and Showalter, 2001; Greene and Winters, 2004, 2007; Jacob and Lefgren, 2002; Matsudaira, 2008). The IV approach addresses selection bias by including in the estimation model an exogenous variable, or instrument, that predicts retention but is otherwise unrelated to the student outcome of interest. For the IV approach to work properly, the instrument needs to satisfy two conditions: (1) the instrument is strongly correlated with the independent variable of interest (in this case, grade retention), and (2) the only reason for any association between the instrument and the outcome variable is the association between this 4 Hierarchical models are not mutually exclusive with propensity score, instrumental variables, and regression discontinuity estimation strategies. The former accounts statistically for the nested structure of educational data, while the latter use natural experiments to estimate causal effects. 10

29 instrument and the independent variable of interest (Angrist and Lavy, 1999; Angrist, Imbens, and Rubin, 1996; Staiger and Stock, 1997). The standard IV estimation strategy is to use two-stage least squares regression: In the first stage, the probability of being retained is predicted as a function of the instrument plus a set of independent variables that are also included as covariates in the second stage, and the estimated probability of being retained replaces the observed retention indicator as an independent variable in the second stage. Because retention is a function of academic performance, finding a suitable instrument requires reliance on exogenous shifts or discontinuities in retention policies. For instance, Eide and Showalter (2001) used the IV approach as their primary estimation strategy to examine the effect of grade retention on dropping out of school and labor market earnings in adulthood. The instrument used in their estimation was the number of days between a child s birthday in the first year that the child is eligible to enter kindergarten and the statutorily determined kindergarten entry date that varies across states. According to the authors, birthdays across days of the calendar year typically follow a uniform distribution, and the kindergarten entry date across states are presumably exogenous; thus, this instrument is likely exogenous from outcomes, including dropping out of school and labor market earnings. They further argue that the instrument is plausibly related to the probability of being retained, because younger children are more likely to be held back for a year to allow for academic and social development. Another example of an instrument in retention studies is from the evaluations of the test-based promotion policy in Florida, in which Greene and Winters (2004, 2007) used a dichotomous variable of whether or not students were subject to retention policy as the instrument to predict the probably of being retained. In addition, Jacob and Lefgren (2002) and Matsudaira (2008) employed an IV approach within a regression discontinuity (RD) framework. 5 More recently, researchers have begun using an RD design in an attempt to estimate the causal relationship between grade retention and student outcomes. An RD design (Imbens and Lemieux, 2008; Shadish, Cook, and Campbell, 2002) compares the relationship between a treatment variable and an outcome variable, where assignment to the treatment is determined by a value on an observable assignment variable, such as a score on an academic assessment. Treatment status (such as enrollment in mandatory summer school) is determined by whether a student scores above or below a particular cut score on the assignment variable. Those who score below the threshold are exposed to the intervention, while those who score above are not (a case of sharp discontinuity). It might also be the case that some who are above the threshold also get treated while some below the threshold do not (a case of fuzzy discontinuity), although there is an 5 Chapter Four provides more details on studies using an RD design. 11

30 increased probability of being treated for those below the cut point. This approach exploits the fact that subjects included in the comparison and the intervention groups differ only with respect to the assignment variable (and with respect to any other variable correlated with it); thus, one controls for the confounding factors by contrasting marginal participants and marginal nonparticipants, where the term marginal refers to those units not too far from the threshold for selection (Battistin and Rettore, 2002). The difference in outcomes between these two groups then measures the mean causal impact of the treatment. However, this method yields a local treatment effect, generalizable only to those scoring near the cutoff on the assignment variable. RD models are most often implemented in a two-stage least squares framework (Imbens and Lemieux, 2008) and, therefore, are related to IV models. Cook (2008) points out that the strengths of the RD design are that it can be used in a particular set of circumstances in which a randomized experiment might not be feasible, and in these circumstances, the RD design is superior to all known quasi-experimental methods. As such, in Chapter Four, while we summarize findings across all the reviewed studies, we also provide more detailed findings from the studies that used an RD design. Age at Retention Conventional wisdom holds that students retained at a younger age tend to benefit from an additional year in the same grade. Children in early grades (typically, kindergarten or 1st grade) are often retained on the grounds of behavioral problems stemming from socioemotional immaturity, and an additional year, according to the popular belief, is a gift of time for children to reach the maturity level required for academic success (Hong and Yu, 2008; Jimerson, Carlson, et al., 1997; Mantzicopoulos and Morrison, 1992; Pianta, Tietbohl, and Bennett, 1997; Shepard and Smith, 1989). However, our review, which encompasses 11 empirical studies on the academic effects of retention in kindergarten or 1st grade, does not support this notion. In general, the majority of research shows that, contrary to popular belief, retention during kindergarten or 1st grade usually fails to improve academic performance and often has negative effects on student achievement in the long run (Baenen, 1988; Balitewicz, 1998; Hong and Raudenbush, 2005; Hong and Yu, 2007; Johnson, Merrell, and Stover, 1990; Setencich, 1994; Wu, West, and Hughes, 2008; Meisels and Liaw, 1993). In addition, we found four studies that examined whether the timing of grade retention matters in terms of academic effects. Two studies reported no statistically significant difference between students retained in early grades and those retained in later grades (Baenen, 1988; Silberglitt, Appleton, et al., 2006). Two showed that retention in later grades was associated with more negative outcomes than retention in early grades (Hagborg et al., 1991; Meisels and Liaw, 1993). 12

31 Length of Follow-Up In our review, the length of follow-up appears to matter in evaluating the academic effects of retention. Studies that followed students for three years or more tended to find dissipating effects on academic outcomes over time (Baenen, 1988; Hong and Yu, 2007; Lorence and Dworkin, 2006; Lorence et al., 2002; Mantzicopoulos and Morrison, 1992; Peterson, DeGracie, and Ayabe, 1987). Other studies even reported that short-term effects disappeared just two years after the retention decision (Nagaoka and Roderick, 2004; Roderick and Nagaoka, 2005). While the majority of studies reported that short-term academic gains among retained students completely disappeared several years later and retained students eventually fell behind again (Baenen, 1988; Mantzicopoulos and Morrison, 1992; Nagaoka and Roderick, 2004; Peterson, DeGracie, and Ayabe, 1987; Roderick and Nagaoka, 2005), studies that followed Texas 3rd-grade students for more than six years found positive effects that persisted, but decreased in magnitude, over time (Lorence and Dworkin, 2006; Lorence et al., 2002). Teacher Ratings Studies that examined socioemotional, attitudinal, and behavioral outcomes used several different methods to measure these outcomes, including student self-reported measures, parental reports, and teacher ratings. 6 Studies that use teacher ratings to measure student socioemotional or attitudinal outcomes are subject to threats to validity because teacher ratings may be influenced by teachers attitudes toward grade retention. For example, teachers who support retention policies may be more likely to report negative attitudes among retained students. Regression to the Mean Effects Regression to the mean refers to the phenomenon that low-performing students will likely achieve somewhat higher test scores when retested, while students with very high scores tend to score lower on a second test. Since students can have good and bad testing days, those who score abnormally low (or high) are very likely to have scores that underestimate (or overestimate) their true ability. They will naturally do better (or worse) and score closer to the mean on a second test. If the retention decision is based primarily on a single test, retained students are very likely to be those who had a bad testing day and tended to score higher the next year. As a result, findings of positive academic effects could be due to regression artifacts. To control for the regression to the mean effects, researchers examining Chicago s test-based promotion policy used growth-curve models to derive student achievement growth after retention 6 Parental reports were used only to measure behavioral outcomes, while student self-reported information and teacher ratings were used to measure socioemotional, attitudinal, and behavioral outcomes. 13

32 and compared the average estimated achievement growth of retained and comparison groups. They reported lower achievement growth among 6th-grade students and little evidence of greater achievement growth among 3rd-grade students two years after the retention decision (Nagaoka and Roderick, 2004; Roderick and Nagaoka, 2005). Lorence and Dworkin (2006) addressed the regression artifacts problem by conducting time-reverse analysis, in which the preretention test scores (i.e., reading test scores for 3rd-grade students in spring 1994) were used as the dependent variable, and postretention scores were entered as independent variables. Since the coefficients in the time-reverse analysis were approximately similar in magnitude, the authors concluded that the positive effect of grade retention on reading achievement among 3rd-grade students is not due to a statistical artifact of the lower performing retained 3rd graders regressing to the population mean (pp ). In the following chapters, we discuss findings on characteristics of retained students (Chapter Three) and six student outcomes: academic achievement, socioemotional outcomes, behavioral outcomes, propensity to drop out of school, postsecondary education, and employment outcomes (Chapter Four). 14

33 3. CHARACTERISTICS OF RETAINED STUDENTS The literature is fairly consistent regarding the characteristics of retained students. In general, grade retention is associated with gender, race, SES, age for grade, student mobility, family and parental characteristics, cognitive abilities, prior academic achievement, prior behavioral and socioemotional development, disabilities, and student health. We summarize findings by five types of student characteristics: demographics, family and parental characteristics, cognitive and academic functioning, socioemotional and behavioral development, and health. 1 With respect to student demographics, studies showed that retained students were more likely to be male, minority, of lower SES, and younger than their peers in the same grade. Specifically, boys were found to be much more likely to be retained than girls (Alexander, Entwisle, and Dauber, 2003; Byrd and Weitzman, 1994; Corman, 2003; Dauber, Alexander, and Entwisle, 1993; El- Hassan, 1998; Fine and Davis, 2003; Frymier, 1997; Gottfredson, Fink, and Graham, 1994; Guevremont, Roos, and Brownell, 2007; Hong and Raudenbush, 2005; Hong and Yu, 2007; Karweit, 1999; Lorence and Dworkin, 2006; Lorence et al., 2002; Mantzicopoulos et al., 1989; Meisels and Liaw, 1993). Most studies reported African-American students as being at an increased risk for retention in comparison with their white peers (Blair, 2001; Dauber, Alexander, and Entwisle, 1993; Frymier, 1997; Hauser, Pager, and Simmons, 2000; Hong and Yu, 2007; Lorence et al., 2002; Roderick, Bryk, et al., 1999), while some found that Hispanics were also more likely to repeat a grade than whites (Frymier, 1997; Hauser, Pager, and Simmons, 2000). Students in the lowest-income quartile were over 40 percent more likely to be retained than those in the highest quartile (Hauser, Frederick, and Andrew, 2007), and among high school graduates, those in the lowest quartile of SES were twice as likely to be have been retained than high-ses graduates (Fine and Davis, 2003). Retained students also tended to have more school transfers (El-Hassan, 1998; Guevremont, Roos, and Brownell, 2007; McCoy and Reynolds, 1999; Reynolds, 1992). The studies reported a significant association between grade retention and family characteristics. Retained students were more likely to come from poor households (Byrd and Weitzman, 1994; 1 While some studies of student characteristics assess the same aspects of student outcomes as summarized in the next chapter on effects of grade retention (e.g., cognitive and academic functioning, and socioemotional and behavioral development), they have fundamental differences in terms of the research questions being addressed. Studies on characteristics of retained students examine factors that precede retention, and typically measure student characteristics before students are retained. In contrast, studies on the relationship between retention and student outcomes address the research question of whether retention has a causal effect on student outcomes. As a result, the emphasis is on student outcomes measured after the retention year, which are typically treated as dependent variables. 15