Protocol: Effects of College Access Programs on College Readiness and Enrollment

Transcription

1 Protocol: Effects of College Access Programs on College Readiness and Enrollment Eleanor L. Harvill, Rebecca A. Maynard, Hoa T. H. Nguyen, Claire Robertson-Kraft, Namrata Tognatta, Rachél Fester Submitted to the Coordinating Group of: Crime and Justice Education Disability International Development Social Welfare Other: Plans to co-register: No Yes Cochrane Other Maybe Date Submitted: 3 February 2011 Date Revision Submitted: 6 October 2011 Approval Date: 29 November 2011 Publication Date: 1 September The Campbell Collaboration

2 BACKGROUND The Problem, Condition or Issue Over the past decade, national high school graduation and college enrollment rates have increased slightly, yet approximately 30% of students are still not graduating high school and, among those that do graduate, only 70% are enrolling in higher education (Snyder & Dillow, 2010). Additionally, recent research has demonstrated that, even when students earn a high school diploma, a large proportion have not developed the skills necessary to succeed academically in college (Callan, Finney, Kirst, Usdan, & Venezia, 2006; Greene & Winters, 2005). Of the 75% of high school graduates who enroll in two or four year colleges, only about 35% earn a bachelor s degree (Carnevale & Fry, 2000), and almost half of students in four year institutions and slightly less than two thirds of students in two year institutions require remedial coursework (Kirst & Bracco, 2004). Perhaps most concerning is the fact that, although college enrollment rates have increased, students of lower socioeconomic status or minority backgrounds, and students whose parents did not attend college, are still considerably less likely than their peers to graduate high school, pursue post-secondary education, and persist upon entry (College Board, 2010). Practitioners and policymakers offer a variety of explanations for enduring gaps in educational opportunity. One common explanation is that inequities result from the fact that financial aid programs do not provide sufficient funds to compensate for rising college costs (Fitzgerald, 2004; St. John, Musoba, & Simmons, 2003). Although financial aid is undoubtedly important, this review focuses on lack of academic preparation, information about college access, and social supports, which also have been identified as impediments (Ellwood & Kane, 2000; Perna, 2004). To provide a framework for our systematic review, we begin with a review of the literature examining the predictors of college readiness and enrollment. These predictors are situated within two theoretical approaches economic and sociological most often used to explain students decision-making processes. We adopt an integrated conceptual model, discussed in Perna (2006), to frame our examination of precollege outreach programs impact on college readiness and enrollment outcomes. Economists employ a rational model for explaining college enrollment decisions. This approach assumes that individuals act in ways that maximize their net benefits (Becker, 1993; Paulsen, 2001). Accordingly, high school students will weigh the benefits of higher education increased monetary and other societal advantages against both direct and indirect costs. Patterns of college enrollment cannot be explained through a straightforward cost-benefit analysis, however, as individuals have varying levels of academic preparation and resources necessary for success in higher education. A significant body of literature has demonstrated 2 The Campbell Collaboration

3 that low-ses and minority students are much less likely to graduate high school with the requisite skill set for success in higher education (ACT, 2004; NCES, 2003); as such, differential enrollment across groups can be explained, in part, by differential levels of academic preparation, or college readiness (Adelman, 1999). On the supply side, the rising costs of higher education make it more challenging for low income students, in particular, to enroll in college (Heller, 1997). Students perceptions and the reality of financial aid contribute to their predispositions towards (or away from) college, as well as their choice to enroll (Bound & Turner, 2007; Monks, 2009). Though a purely economic approach provides insight into the decision-making process, it largely ignores the nature of the information received by potential college enrollees and their families, and the broader influences affecting their ultimate decision (Avery & Hoxby, 2004; DesJardins & Toutkoushian, 2005). Sociological-cultural approaches address these shortcomings by considering individuals cultural capital, the system of attributes such as language skills and cultural knowledge (Bourdieu, 1986; McDonough, 1997), as well as their access to the social capital necessary to build important social networks (Coleman, 1988). Research documents how sociological factors, such as student, peer, and parental expectations (Goyette, 2008; Kim, Kevin, & Teresa, 2008; Wildhagen, 2009) and receipt of appropriate counseling and other social supports in high school influence patterns of college enrollment (McDonough, 2005; Perna et al., 2008). Perna (2006) offers a conceptual model, which integrates aspects of both economic and sociological approaches. Though the economic approach remains at the core of the model, it is situated within multiple contexts influencing the decision-making process. Students educational decisions are affected by their individual habitus, or system of values and beliefs, which shapes their views and interpretations, as well as by the social and cultural capital made available through their specific school and community context (Paulsen & St. John, 2002). Outcomes are further influenced by components of the higher education system, as well as broader changes in the social, economic, and policy landscape. College Access Interventions For decades, college access programs in the United States have aimed to improve college enrollment rates, particularly for underrepresented populations. Beginning with the TRIO programs in the 1960s, the federal government has sponsored many of these policies and, in recent years, has considerably expanded its role. Efforts at the federal level have been supplemented by numerous programs run by state and local governments, districts, foundations, universities, and private businesses. For this review, we define college access programs to be pre-college interventions that explicitly identify increasing college readiness and/or college enrollment as a primary goal of the program. This review examines two broad categories of college access programs: (1) prepackaged whole school reform efforts; and (2) supplementary services provided at the student level. 3 The Campbell Collaboration

4 How the Intervention Might Work Whole school reform efforts alter the organizational structure of the school and tend to focus on two key predictors of college enrollment students academic preparation and social supports (Martinez & Klopott, 2005). Supplementary pre-college outreach programs provide services, typically outside of school hours, that encourage students to become interested in attending college and increase parental engagement in the process. Despite commonalities between and among program types, individual initiatives vary greatly in approach, duration, and intensity (Gandara, 2001; Perna, 2002). Though there are many ways to organize a discussion of pre-college outreach programs, we use the taxonomy presented in Gandara (2001). This typology is based on programs primary source of funding private nonprofit, university based, state government sponsored, federal government sponsored, community based, and K-12 school systems. Within each program type, components are classified along the following dimensions: (1) Counseling provides students with access to information on the college enrollment process (2) Academic enrichment focuses on improving academic preparation so students have the requisite skill set (3) Parent involvement involves parents in the college enrollment process (4) Personal enrichment and social integration broadens students understanding of available opportunities through activities such as speakers and fieldtrips (5) Mentoring provides students with one-on-one mentoring throughout the college enrollment process (6) Scholarships offers financial aid for students in need These components, either alone or in combination, address the key predictors of college enrollment. For example, academic enrichment directly targets inadequate academic preparation, counseling and parent involvement address shortcomings in families cultural and social capital, and personal enrichment targets students low educational aspirations (Gandara, 2001). 1 Though much is known about the predictors of both students academic behaviors and preparedness for college, we have limited information about which program components are most effective in improving college readiness and raising college enrollment rates. 1 Programs with different funding sources are responding to very different needs of their primary stake-holders (Gandara, 2001). When programs receive funding from multiple sources, we follow Gandara (2001) in classifying them based on which sector started the program and/or maintains primary responsibility for overseeing day to day operations. A list of primary funding sources for a number of large-scale programs is available in Gandara (2001, pp ). 4 The Campbell Collaboration

5 Outcomes of Primary Interest This review investigates the impact of college access programs on college readiness and enrollment. We focus on these outcomes because they are the most aligned with the goals of the interventions we are considering. Also, as the interventions are fielded to pre-college students, we expect that most studies will report either pre-college or early-college outcomes. We consider four measures of college readiness: completed coursework, mathematics achievement test scores, language arts achievement test scores, and high school graduation. Our preferred measure of college readiness is an indicator of completed high school coursework (Adelman, 2006; America Diploma Project, 2004; Rose & Betts, 2001). The academic intensity of high school coursework is the best pre-college predictor of college completion (Adelman, 2006). The remaining measures are less preferred, but more commonly reported. Even though current state assessments are typically not well aligned with the rigor of college entrance requirements (Martinez & Klopott, 2005), students math and language arts achievement scores can still provide some insight into their college readiness. Though a high school diploma does not guarantee college readiness (America Diploma Project, 2004), it is one of the three critical steps that determines the likelihood that a disadvantaged student will enter college (Cabrera & La Nasa, 2001). As part of our coding process, we will note the full range of outcomes reported in the candidate studies. If there are multiple high quality studies reporting a particular outcome measure, we will consider expanding our analysis to include that measure. Outcomes measuring completion of milestones in the college application process, like SAT completion, and those measuring persistence in and completion of college would particularly enrich our analysis. Why it is Important to do the Review Though several published reports have produced comprehensive inventories of college access programs (Gandara, 2001; Perna, 2002; Tierney, Bailey, Constantine, Finkelstein, & Hurd, 2009), no systematic review and meta-analysis of the evidence regarding their effectiveness exists. 2 This systematic review will fill that knowledge gap by systematically gathering, reviewing, and synthesizing the findings on the effectiveness of programs designed to improve college readiness and enrollment for disadvantaged populations. In so doing, it will provide guidance for policymakers and practitioners implementing college 2 For the What Works Clearinghouse Practice Guide on Helping Students Navigate the Path to College, a panel of experts identified promising practices high schools engage in to increase college access (Tierney et al., 2009). A comprehensive search of the literature and review of studies by the WWC was performed to determine the level of evidence supporting the specific recommendations of the panel (Tierney et al., 2009). In contrast, this paper reports the effects of college access programs systematically, without limiting attention to particular strategies for increasing access. 5 The Campbell Collaboration

6 access programs and will identify important gaps in the scientific evidence base that warrant further research. OBJECTIVES The purpose of this review is to summarize the evidence regarding the effectiveness of college access programs on college readiness and college enrollment. Specifically, this review explores the following questions: 1. What is the corpus of evidence to judge the effectiveness of programs aimed at increasing college readiness and enrollment? 2. What does that evidence base tell us about the effectiveness of various intervention strategies? More specifically: a. What types of programs have been rigorously evaluated? b. How representative are these programs of the range of programs that exist? c. What are the estimated average impacts of programs on college readiness outcomes within the following four domains: math achievement, language arts achievement, completed coursework, and high school graduation? d. What are the estimated average impacts of programs on college enrollment rates? e. What are the estimated average impacts of programs targeted at students in different grades or of different ages? f. What are the estimated average impacts of programs for males and for females? g. What are the estimated average impacts of programs targeted at different racial/ethnic groups? h. What are the estimated average impacts of programs targeted at students with different levels of family income or socioeconomic status? i. Do the estimated average impacts of programs vary across demographic groups, including race/ethnicity, gender, and socioeconomic status? j. Do the estimated average impacts of programs vary depending on entering achievement levels? METHODOLOGY Criteria for Inclusion and Exclusion of Studies in the Review The following twelve criteria will determine whether a study will be included in the review database for purposes of estimating program impacts: 6 The Campbell Collaboration

7 Study design: The review will include randomized controlled trials and well-controlled quasi-experimental designs. Regression discontinuity designs will be evaluated on a case-bycase basis. 3 Interventions to be included: The review will include only interventions that explicitly identify increasing college readiness and/or college enrollment as a primary goal of the program. These interventions can include services and activities such as academic instruction, academic advisement, counseling, mentoring, transmission of information about how to apply for college or apply for financial aid, provision of financial aid, and other services meant to mitigate factors impeding college readiness or college enrollment. Counterfactual treatment: The review will include only studies where the counterfactual condition (i.e., the control group) received normal college counseling and support services. A study will not be included if the counterfactual condition was another college access program. Furthermore, studies in which the estimated effect of the treatment is confounded with another factor (such as the school attended) will not be included. Target population: The review will include only studies of interventions that target students between grade 6 and 12, or students of comparable ages who have not yet graduated from high school or earned a General Education Development certificate. 4 The review will not include studies in which less than 75% of the sample falls within the target population for this review. Key outcomes: The review will include only studies that measure one or more of the following outcomes: (1) college readiness; and (2) college enrollment. College readiness outcomes may fall within one or more of four domains: (1) math achievement, (2) language arts achievement, (3) completed coursework, and (4) high school graduation. 5 Math and language arts achievement should be measured separately using standardized tests administered to the entire sample or to representative subsets of the sample. 6 Completed coursework may be measured in one of two ways: (1) by a composite measure of academic performance based on the high school transcript data, such as 3 Regression discontinuity designs estimate the impact of the intervention for individuals near the eligibility cutscore, not the average treatment effect for the entire population of interest. This caveat will be taken into consideration when determining whether to include an RDD study in the meta-analysis. It is worth noting that preliminary search results did not identify any RDD studies of college access programs. 4 If an insufficient number of high quality studies of interventions for middle school students are available, the reviewers will restrict attention to studies of high school students. 5 Studies that only report a composite measure of college readiness, such as the one developed by Greene & Winters (2005) or Adelman s Academic Resources variable (2006), and do not separately report outcomes within one of the five domains will be coded, and the findings will be reported. However, the impact findings will not be included in the meta-analysis. If a large number of studies of this sort are found, we will consider also reporting out results for a composite academic readiness measure. 6 The requirement that the tests be administered to the entire sample or to representative subsets of the sample is intended to insure that we avoid the selection issues raised when one observes scores only for students who are intending to apply for college. 7 The Campbell Collaboration

8 Adelman s Academic Intensity measure (Adelman, 2006); or (2) (if the composite variable is unavailable) by a measure of the highest level of math completed (Rose & Betts, 2001). The preferred measure of high school graduation is the binary outcome indicating whether the student received a standard high school diploma within four years of starting high school (Bozick & Lauff, 2007; DesJardins & Lindsay, 2008; Eccles, 2008). College enrollment is measured by whether a student enrolled in any two-year or four-year college following the date corresponding to on-time high school graduation (Johnson, 2008; Park, 2008; Snyder, Dillow, Hoffman, & National Center for Education Statistics, 2008). The review will include up to two measures of enrollment per study: (1) within two years following the on-time graduation date; and (2) by the time of the most recent followup (if that is later than the on-time date). 7 Given the outcome measures described above, neither reliability nor over-alignment of the intervention and measures should be concerns for this review. Requiring standardized tests to measure achievement should ensure adequate reliability of those measures. In general, we would not expect there to be concerns related to the reliability or over-alignment of the other measures, since they relate to observed behaviors. Still, the data-coding instrument, included in Attachment A, instructs reviewers to note any concerns they may have regarding overalignment and/or reliability of outcome measures. At times, studies report impact estimates for key outcomes based on subsamples defined by criteria that are endogenous to the intervention. For example, studies might provide the impact of the program on enrollment rates for students who were admitted to at least one college, ignoring the fact that the program potentially affects the probability that the student applies to college, as well as the probability of acceptance conditional on application. If outcomes are reported for a non-representative subset of the study sample (such as for high school graduates or for students admitted to college), these outcomes will be included in the review only if it is possible to transform them to measures that pertain to the full study sample (or a representative subset of the study sample). For example, if results of college enrollment are reported based on a sample of college applicants and the study reports the proportions of treatment and control group members who applied to college, we would transform the reported conditional means into unconditional means by dividing the conditional means by the proportions that applied to college. Setting and timing of interventions: The review will include only studies reporting on interventions fielded in the past twenty years, since 1990; those that were conducted in the United States or in developed countries with a similar secondary and higher educational systems; and those written in English. The nature and prevalence of college programs is sufficiently different in the period pre-1990 as to raise questions about the current relevance 7 When studies report 2- and 4-year college enrollment separately, we will construct a measure of overall college enrollment by summing the proportions in the two categories. In addition, we will record the measures separately. 8 The Campbell Collaboration

9 of evidence regarding effectiveness. Similarly, studies of programs conducted in countries with dissimilar educational systems are not likely to have relevance to the U.S. and other represented countries. We exclude studies reported in languages other than English simply because we do not have the capacity to translate such studies. Data collection: The review will include only studies for which there is reason to believe that there is no systematic reporting bias in the outcome measures between the program and the control groups. For example, we would exclude a study in which the data for the program group were collected by the program staff, while independent data collectors conducted surveys with the control group, unless the study provided convincing evidence that this difference in data collection method did not lead to differential survey response bias. Appropriate analytic methods and adequate reporting: The review will exclude studies for which it is not possible to construct unbiased estimates of the effects of the interventions on one or more of the outcomes of interest for the full (intention-to-treat) follow-up sample or for sample subgroups defined by characteristics that are independent of the intervention. 8 Attrition: The level of attrition of a randomized controlled trial will be judged based on both the overall level of attrition and the differential level of attrition. Overall attrition is defined as the proportion of individuals lost from the study between randomization and the postintervention measurement. Differential attrition is the absolute value of the difference between attrition from the treatment and the control groups. We will use the What Works Clearinghouse Version 2 Standards (U.S. Department of Education, 2009, p. 33) to judge whether the study has an acceptable level of attrition, given these two measures. These standards allow for the fact that the expected bias in impact estimates is affected by both the overall attrition rate and the differential attrition rate. Roughly, these standards accept overall attrition rates up to 40 percent, if the difference in attrition rates between the treatment and control groups is 2 percentage points or fewer; in contrast, they only accept overall attrition rates in the neighborhood of 10 percent or less if the difference in attrition rates between the treatment and control groups rises to 10 percent or more. A randomized control trial with a high level of attrition will be considered a quasiexperimental design and included in the review provided it meets the quality standards established for quasi-experimental designs by demonstrating baseline equivalence for the analytic sample. A quasi-experimental design will not need to meet attrition standards; rather it must demonstrate baseline equivalence for the analytic sample (see below). 8 Cluster randomized trials analyzed at the individual level will be included and their estimates will be adjusted to correct for clustering. Studies will only be excluded if it not possible to use standard analytic tools to correct effect sizes and standard errors. 9 The Campbell Collaboration

10 Baseline equivalence: Quasi-experimental designs and randomized controlled trials with high levels of attrition must establish equivalence of initial measures for the treatment and comparison groups in the analytic sample to be included in the review. The study will only be included in the review if average pre-intervention continuous measures differ between the intervention and comparison groups by less than 0.25 of the population standard deviation. 9 A similar standard will be imposed on dichotomous pre-intervention standards. 10 Baseline equivalence must be established for demographic characteristics of the two groups and for achievement measures. Concealment: For this review, we will not require allocation concealment. While in many studies concealment is important for avoiding reporting bias, studies of educational interventions rarely use blind or double-blind designs for practical reasons. Search Strategy for Identification of Relevant Studies The search for relevant research will include two phases: a preliminary search phase and a final search phase. We have completed a preliminary search for relevant research, which included a keyword search of electronic databases and additional searches to identify the grey literature. The final search phase will extend the preliminary search by using additional electronic databases, searching select databases using controlled language, and adding strategies for identifying research in the grey literature. The entire search strategy is presented below. The presentation does not separate the preliminary search phase from the final search phase for ease of exposition. The search strategy is followed by a description of our bibliographic management strategy. Last, we present the results of our preliminary search. Electronic Database Search Strategy We will search the following databases for articles published between 1 January 1990 and 1 September 2011 in English. This timeline was determined by the date the search and by our exclusion of interventions that were fielded prior to For all databases, we will perform free-text searches. For the subject specific databases ERIC, PsycInfo, and Sociology Abstracts, 11 we will also perform controlled language searches. 9 This requirement is consistent with accepted quality standards for quasi-experimental designs, such as those developed by the What Works Clearinghouse (U.S. Department of Education, 2009, p. 15). 10 To assess the baseline equivalence of dichotomous variables, the reviewer calculates the log-odds ratio effect size for the pre-intervention measure, translates this into a continuous effect size measure, and then compares this result to Both of these calculations are performed using simple algebraic transformations of formulae found in Lipsey & Wilson (Lipsey & Wilson, 2001, p. 202). (Note that the calculation of the odds-ratio effect size does not take the size of the sample into account. Therefore, the reviewer is instructed not to assess baseline equivalence for uncommon demographic categories.) 11 We believe that the controlled vocabulary available in the EconLit database will not be particularly helpful in identifying relevant studies. 10 The Campbell Collaboration

11 General bibliographic databases/full-text journals EBSCO MegaFile JSTOR Project Muse Subject specific bibliographic databases: ERIC EconLit PsycInfo Sociology Abstracts Dissertation and theses databases ProQuest Dissertations and Theses Grey literature databases PolicyFile SSRN elibrary Within each database, our free-text searches will be of the form: (topic keyword) AND (intervention keyword) AND (evaluation keyword). All such queries generated by the following list of keywords will be performed: TABLE 1: SAMPLE SEARCH STRATEGY Topic Intervention Evaluation pre-colleg* precolleg* college AND transition* college access college enrollment college readiness college preparation college AND outreach college AND bridge college going college attendance program intervention evaluation experiment effect* Controlled language searches will be performed for the ERIC, PsycInfo, and Sociology Abstract databases. To identify relevant descriptors, we will look up the records of relevant studies identified in the preliminary search and record the descriptors in a word document. 11 The Campbell Collaboration

12 We will use the database thesaurus to identify additional potentially relevant descriptors. The final set of descriptors will be recorded and presented in the review paper. Search Strategy for Other Sources Before beginning this phase of the process, we will complete the review of the studies identified through the database search and leverage those results to find yet more studies. Conference abstracts or proceedings: We will browse the online programs of the following conferences to identify potentially relevant studies. For the smaller conferences, we will scan the entire program. For the larger conferences, we will use the association s divisions and descriptors to limit our search. American Educational Research Association (AERA) Online Paper Repository ( contains abstracts for papers presented at the AERA Annual Meetings from 2005 to the present. For the 2010 and 2011 conferences, the full-text of papers can be downloaded if the authors chose to make the text available. Association for Public Policy Analysis and Management (APPAM) Annual Fall Research Conference Archives ( Society for Research on Educational Effectiveness Spring Research Conference Programs ( These programs include links to abstracts. Online Repositories: We will browse the following online sources of research to find relevant research. The links are specific to the pages within the site that list relevant research. These pages will be read and relevant information will be downloaded and recorded. The US Department of Education s web site includes reports of research funded by the department ( In addition, we will search for all IES grants ( ) within in the Post-Secondary & Adult Education program area to identify relevant research projects currently underway. The National College Access Network maintains a searchable database of publications, research reports, websites, and other resources related to improving college access and success for underserved populations. We will browse the entries in the database by topic, focusing on two topics: college access programs and college readiness. ( The Campbell Collaboration

13 Citation indexes: We will use the ISI Citation Indexes to search for articles that cite or are cited by existing inventories of college access programs (e.g. Gandara, 2001; Perna, 2002; Tierney, Bailey, Constantine, Finkelstein, & Hurd, 2009). We will also use the ISI Citation Indexes to search for articles that cite or are cited by any studies that pass our quality screening. Web Searching: From the studies that pass our quality screening and from the existing inventories of college access programs, we will compile a list of the names of college access programs. We will then search the web for evaluations of these programs, using the name of the program followed by the word study as the search term: e.g. Gear Up study. These searches will be performed on both Google.com and Yahoo.com. Solicit Feedback from Subject Experts: After we have completed the searches described above, we will screen the identified studies and present the search results to experts in the field, again asking if they know of any work that is missing from the review. Management of References and Document Retrieval To manage bibliographic records for this review, we have created a RefWorks database. In addition to storing citation information, this program allows us to easily identify the source of the citation, track the outcome of the review process, and attach copies of the study and the coding instrument. The following procedure for managing citations was developed and refined during our preliminary search process, the results of which are described below. For the search results generated by a particular search of a particular database, the number of hits was recorded in an excel spreadsheet, and the results were saved to RefWorks with a tag indicating how the citation was found. 12 Immediately after adding these citations, we searched for and removed all duplicate citations in RefWorks. We were then able to use the tagging system in RefWorks to determine the number of unique citations found by each database. A similar process was used to add citations found through hand-searching. For each unique citation, we reviewed the title and abstract. If the paper seemed potentially relevant, we obtained the paper for review. To obtain the study for review, we will use the full-text feature of the referring database, if available in the University of Pennsylvania Library system. Otherwise, we will use the PennText Article Finder tool on the library s web site to find published articles and the library catalogue to find books. If the study is unpublished, we will search online to find a full-text paper. We will contact the study author via to request copies of unpublished dissertations and papers that are not available online. If these resources are not sufficient, we plan to use interlibrary loan to request the document. 12 The University of Pennsylvania Library s online resources allow us to directly export the references from the electronic database to RefWorks. After performing a search in one of the electronic databases, we choose select all and then save to RefWorks. 13 The Campbell Collaboration

14 Preliminary Search Results The preliminary search focused on identifying documents written or published between 1 January 1990 and 1 February The preliminary search utilized the following databases: Dissertation Abstracts, EBSCO Megafile, Econlit, ERIC, JSTOR, and Project Muse. Free-text searches using the strategies described above were performed. We feel that these strategies adequately covered the range of potentially relevant studies. We did not perform any controlled language searches and believe that these could identify additional studies in subject specific databases. This preliminary search process generated 2068 initial hits from the electronic databases alone, 1151 of which were unduplicated. A search for relevant studies from the U.S. Department of Education, the National College Access Network, and Google generated an additional 24 citations, bringing the total studies for initial review for relevance to Review of abstracts for the 1175 identified 192 studies that appeared potentially relevant and, thus, warranted a full-text review. Results from this search appear in the table below, broken down by electronic database and other searches. TABLE 2: SUMMARY OF PRELIMINARY SEARCH RESULTS (NUMBER OF STUDIES) Search Source Citations Identified Unique Citations Obtained for Review Electronic Databases EBSCO Megafile JSTOR Econlit ERIC Dissertation Abstracts Project Muse US Department of Education Hand Searches Total The Campbell Collaboration

15 Description of Methods Used in the Component Studies Both quasi-experimental and randomized controlled trials contribute to our knowledge about program effectiveness. A recent evaluation of the EXCEL program (Bergin, Cooks, & Bergin, 2007) is an example of a seemingly well-conducted randomized controlled trial in this area. Acceptable quasi-experimental designs generally range from analyses of existing survey or administrative data to studies that collected data from both program participants and a prospectively identified comparison group. For example, a recent evaluation of Talent Search (Constantine, Seftor, Martin, Silva, & Myers, 2006) demonstrates the propensity score matching approach to program evaluation using existing data sources, while a recent evaluation of GEAR UP (Standing, Judkins, Keller, & Shimshak, 2008) illustrates application of quasi-experimental methods in the context of prospective identification of the comparison group. The methods of these three studies are described below. 13 The recent EXCEL evaluation (Bergin et al., 2007) is a randomized field trial that stratified the sample based on achievement level, gender, and ethnic group before randomly assigning applicants to treatment and control groups. The authors collected the data for this evaluation and analyzed it using ANOVA to examine the overall impact of the program and the sensitivity of the program impacts to initial achievement level. In addition, raw tabulations are presented for binary outcomes and means and standard deviations for continuous outcomes. We can use these data to compute log-odds ratios and effect sizes. The Talent Search study (Constantine et al., 2006) is a quasi-experimental design that analyzed administrative records to evaluate the impact of the Talent Search program using a comparison group derived from propensity score matching. First, the propensity score was calculated. The first step in this method involved using logistic regression to estimate the probability that a student with a set of observed characteristics would participate in the program. Then, caliper matching was used to identify the comparison group. Based on the similarity of their estimated propensities to have participated, comparison students for each student who participated in the program were selected from those not in the participant sample. Finally, weights were constructed for each comparison individual to adjust for the number of treatment individuals for whom she or he served as a comparison, the number of comparison individuals with which each of those treatment individuals was associated, and the relative size of the treatment and control groups. The impact estimates were then generated using the treatment sample and the matched comparison group from ordinary least squares, weighting the contributions of comparison individuals. The regressions included covariates to increase the precision of the estimates and to control for any other measured differences between the treatment and comparison groups (Constantine et al., 2006, p. 81). 13 For a discussion of how the different analytical approaches illustrated by these studies will be synthesized, see the Statistical procedures and conventions section. 15 The Campbell Collaboration

16 The recent GEAR UP evaluation (Standing et al., 2008) is a quasi-experimental design study that first selected 20 projects receiving GEAR UP funding. It then selected two matched middle-schools within each project area, and, from each of the matched schools, drew a random sample of seventh graders. The comparison school was selected to match the participating school through a structured process that entailed the following steps: (1) a set of schools near the participating school was identified; (2) the similarity of each of these schools to the participating school was measured on a range of observable characteristics; and (3) the school most similar to the participating school on average was selected (Standing et al., 2008, pp. A-6 and A-7). Data analysis was conducted using a variety of methods, including hierarchical linear models and replicated counterfactual projection. Criteria for Determination of Independent Findings To guarantee that the pooled impact estimates are constructed from a set of independent measures, we will separately estimate the impacts for each outcome and domain. Each study may contribute only one impact estimate to each pooled impact estimate. 14 The final review will include findings for up to five outcome domains: (1) math achievement; (2) language arts achievement; (3) high school course completion; (4) high school graduation; and (5) college enrollment. As noted, we will include only those math and language arts achievement outcomes that pass the standards described above (i.e., the measure is a standardized test administered to the entire sample, and the study demonstrates comparability of the treatment and control groups either by meeting the differential and overall attrition standards for randomized controlled trials or by presenting evidence of baseline equivalence for the analytic sample). If we encounter cases where there are multiple measures that meet these criteria within a study, we will give priority to the test given closest to the expected date of graduation, and, if there is more than one such measure, we will select the test that is more commonly administered. For measures of completed coursework, we will only include measures based on the student s high school transcript as of the on-time date for graduation or later, which is four or more years after the student started high school. If multiple measures of completed coursework from this transcript are reported, we will give priority to the most recent transcript and to a composite measure that reflects the academic level of courses completed, such as Adelman s Academic Intensity measure (Adelman, 2006). Second priority would be to use the highest math course completed (i.e., AP calculus; Pre-calculus; Algebra 2; Geometry; and Algebra). If neither of these measures is available, we would accept a 14 Some reports include the results of multiple studies based on independent samples and, therefore, may contribute multiple measures to a meta-analysis. For example, the recent Talent Search evaluation (Constantine et al., 2006) separately reports the impact of the Talent Search program in three different states. Since these samples are non-overlapping and the authors do not present pooled impact estimates, we would treat the results from each state as a separate study. 16 The Campbell Collaboration

17 composite measure of completed coursework that reflects the number of units completed, such as a binary indicator of completion of the New Basics curriculum described in A Nation at Risk (National Commission on Excellence in Education, 1984). High school graduation is defined above to be the binary variable indicating that a student received a standard high school diploma within four years of starting high school. This definition should insure that a study reports at most one measure for this domain. College enrollment is defined above to be the binary variable indicating whether a student enrolled in any college (two-year or four-year) after the on-time date for high school graduation. When there are multiple measures of college enrollment taken at different points in time, we will focus on up to two measures: (1) the earliest date following the on-time date for graduation and (2) the most recent time period. Details of Study Coding Categories All retrieved studies will be coded using a data-coding instrument, included in this document as Appendix A. The first section of the coding instrument records bibliographic information and screens the studies for relevance. Relevance screening includes verifying that the intervention is a college access program, the study design is eligible, the study setting and timing are appropriate, and that the study reports one or more relevant outcome measures. 15 If a study fails any of the relevance criteria, a summary statement explaining the failure is written and no further information on the study is recorded. The second section of the data-coding instrument screens the study for quality, according to the criteria established above. This section records how participants are assigned to treatment and control groups, describes the counterfactual condition, notes the existence of any factor potentially confounded with treatment, assesses the quality of available outcome measures, calculates both overall and differential attrition for randomized controlled trials, and assesses the baseline equivalence of the treatment and control groups for quasiexperimental designs. Again, if a study fails any of the quality criteria, a summary statement explaining the failure is written and no further information on the study is recorded. If quality screening raises concerns but does not clearly exclude the study, a statement of the concern is recorded and the remainder of the instrument is completed. The final section of the data-coding instrument documents a range of moderating variables, including demographic details of the target population and the availability of program impacts for a variety of subgroups. To describe program implementation, we follow the taxonomy presented in Gandara (2001) in categorizing programs by their primary funding source and by the components of the programs. Programs are therefore categorized as belonging to one of the following categories based on the primary source of funding: (1) 15 A detailed description of acceptable outcome measures is given in Section a. Criteria for inclusion and exclusion of studies in the review and Section b. Criteria for determination of independent findings above. 17 The Campbell Collaboration

18 private nonprofit; (2) university based; (3) state government sponsored; (4) federal government sponsored; (5) community based; and (6) K-12 school systems (Gandara, 2001). 16 College access programs include one or more of the following components: (1) counseling; (2) academic enrichment; (3) parent involvement; (4) personal enrichment and social integration; (5) mentoring; and (6) scholarships (Gandara, 2001). See Attachment A for a complete list of all retained moderating variables. See the next section for a description of the moderator analysis to be included in the review. The data-coding instrument instructs reviewers to record any additional outcomes other than the key outcomes to be included in the review. After all studies have been coded, these additional outcomes will be revisited with particular attention to outcomes of secondary interest, such as measures related to completion of milestones in the college application process or success in the first year of college. Should there be a sufficient number of studies reporting a particular measure of interest, an additional meta-analysis will be conducted on this outcome. Statistical Procedures and Conventions Statistical software: Microsoft Excel will be used to code and store the data extracted from the studies and to transform extracted impact estimates reported in natural units into standardized mean differences for continuous measures and log odds ratio effect sizes for dichotomous measures. Comprehensive Meta-Analysis (Borenstein, Hedges, Higgins, & Rothstein, 2005) will be used to calculate pooled effect sizes for each of the four college readiness domains and for college enrollment. Effect size calculations: For the math achievement and language arts achievement domains, outcome measures are generally continuous. These study impact estimates will be reported in natural units (where possible) as well as converted to standardized mean differences. Both the standardized mean differences and the standard error associated with each standardized mean difference will be calculated as described by Lipsey & Wilson (2001, p. 49). The pooled effect size is given by the weighted average of these effect sizes, where the weights are given by the inverse of the squared effect size standard error (Lipsey & Wilson, pp ). Analysis of impacts on completed coursework, high school graduation, and college enrollment will be reported in both percentage point differences as well as log-odds ratios, since these are typically reported as dichotomous outcomes. 17 Study impacts reported in 16 Programs with different funding sources are responding to very different needs of their primary stake-holders (Gandara, 2001). When programs receive funding from multiple sources, we follow Gandara (2001) in classifying them based on which sector started the program and/or maintains primary responsibility for overseeing day to day operations. A list of primary funding sources a number of large-scale programs is available in Gandara (2001, pp ). 17 Note that these calculations assume that the outcomes are reported as binary measures. It is possible that some of the outcomes will be reported as multi-level categorical variables. If any such outcome measures are encountered, we will develop standards for converting such categorical measures into binary measures during the piloting the data-coding instrument. The standards adopted would be specified in the final report. 18 The Campbell Collaboration

19 percentages will be converted to log-odds ratios, and the variance of each log-odds ratio will be calculated as described by Wang and Bushman (1999, pp ). The pooled impact estimate is given by the weighted average of these log-odds ratios or by the weighted average of the difference in natural units, where the weights are given by the inverse of the variance (Wang & Bushman, 1999, pp ). If a study reports regression adjusted impact estimates, these impact estimates and their standard errors will be used in the meta-analysis. No adjustments will be made to account for missing data. Presentation of results: The review will report findings in both natural units and in effect sizes or log-odds ratios (as described above). However, we also will present a forest plot for each college readiness domain and college enrollment separately. Each plot will include an impact estimate and 95% confidence interval for all studies included in the review for that outcome measure. At the bottom of each plot, the pooled impact estimate and its 95% confidence interval will be displayed. Findings will be discussed in the narrative portion of the final report. Subgroup analysis: In cases where there are a sufficient number of studies, we will conduct moderator analyses to test the appropriateness of pooling impact estimates over the range of college access programs. In addition, we potentially would also conduct exploratory analyses of the moderating effects of factors such as the following: (1) characteristics of the target population; (2) program funding source; (3) program components; and (4) program setting. Sensitivity analysis: If there are a sufficient number of studies for multiple study designs, we will investigate the relationship between estimated program impacts and study design characteristics such as the following: (1) how the treatment and control groups were formed; (2) how the outcomes were measured; and (3) whether any concerns regarding quality were raised. Treatment of qualitative research: Reviewers will maintain a full reference list of studies, identified through the search process, that present qualitative analysis of college access programs. However, no attempt will be made to synthesize the findings from that research. 19 The Campbell Collaboration

20 REFERENCES ACT. (2004). ACT national data release. Iowa City, IA: Author. Adelman, C. (1999). Answers in the toolbox: Academic intensity, attendance patterns, and bachelor s degree attainment. Washington, DC: U.S. Department of Education, Office of Educational Research and Improvement. Adelman, C. (2006). The toolbox revisited: Paths to degree completion from high school through college. Washington, DC: US Department of Education. America Diploma Project (2004). Ready or not: creating a high school diploma that counts. Retrieved on 15 March 2010 from Avery, C., & Hoxby, C. M. (2004). Do and should financial aid packages affect students college choices? In C. M. Hoxby (Ed.), College choices: The economics of where to go, when to go, and how to pay for it (pp ). Chicago, IL: University of Chicago Press. Becker, G. S. (1993). Human capital: A theoretical and empirical analysis with special reference to education (3 rd ed.). Chicago, IL: University of Chicago Press. Bergin, D., Cooks, H., & Bergin, C. (2007). Effects of a college access program for youth underrepresented in higher education: A randomized experiment. Research in Higher Education, 48(6), Borenstein, M., Hedges, L., Higgins, J., & Rothstein, H. (2005). Comprehensive Metaanalysis. Version 2. Englewood, NJ: Biostat. Bound, J., & Turner, S. (2007). Cohort crowding: How resources affect collegiate attainment. Journal of Public Economics, 91(5-6), Bourdieu, P. (1986). The forms of capital. In J. G. Richardson (Ed.), Handbook of theory and research for the sociology of education (pp ). New York, NY: Greenwood Press. Bozick, R., & Lauff, E. (2007). Education Longitudinal Study of 2002 (ELS:2002): A first look at the initial postsecondary experiences of the high school sophomore class of 2002 (NCES ). Washington, DC: US Department of Education, Institute of Education Sciences, National Center for Education Statistics. Bragg, D. D., Eunyoung, K., & Barnett, E. A. (2006). Creating access and success: Academic pathways reaching underserved students. New Directions for Community Colleges 2006, (135), Cabrera, A. F., & La Nasa, S. M. (2001). On the path to college: Three critical tasks facing america's disadvantaged. Research in Higher Education, 42(2), Callan, P. M., Finney, J. E., Kirst, M. W., Usdan, M. D., & Venezia, A. (2006). Claiming common ground: State policymaking for improving college readiness and success. San Jose, CA: The Institute for Educational Leadership, The National Center for Public Policy and Higher Education, and The Stanford Institute for Higher Education Research. 20 The Campbell Collaboration