Effect of Twenty-first Century Scholars Program on College Aspirations and Completion. Robert K. Toutkoushian* Donald Hossler**

Transcription

1 Effect of Twenty-first Century Scholars Program on College Aspirations and Completion Robert K. Toutkoushian* Donald Hossler** Stephen L. DesJardins*** Brian McCall*** Manuel Gonzalez Canche* Draft: March 1, 2013 *University of Georgia **Indiana University ***University of Michigan. For presentation at the meetings of the Association for Education Finance and Policy, New Orleans, LA, March 14-16, This project was funded by the Spencer Foundation grant # , with additional financial support from Indiana University, the University of Georgia, and the University of Michigan. 1

2 Effect of Twenty-first Century Scholars Program on College Aspirations and Completion Introduction Research on the effects of financial aid on postsecondary access, retention at ones institution of origin, and persistence to degree completion has garnered the attention of policy makers, academic researchers, and institutional researchers since the 1970 s (see for example Hoenack, 1971; Jackson, 1978). The focus of financial aid research has changed over time, shifting from early studies of the effects of aid on the overall demand for postsecondary education or college choice, to more focused studies on a range of topics including federal needbased grants (e.g., Goldrick-Rab, Harris, & Trostel, 2009; Mundel, 2008), student loans (Heller, 2008; Hossler, Ziskin., Gross, Kim, & Cekic, 2009; St. John, 2006), state-based grant programs (e.g., Mundel, 2008; Hossler, et al, 2009; Perna, Rowan-Kenyon, Bell, Thomas, & Chunyan, 2008; Toutkoushian & Hillman, 2012), institutional merit-aid (e.g. Singell & Stater, 2006; Doyle, 2010; Hossler, et al., 2009), the application process for aid programs (e.g., Bettinger, Long, Oreopoulos, & Sanbonmatsu,2009; Dynarksi & Scott-Clayton, 2006), and early information (e.g., Bettinger, et al., 2009; Hossler, Schmit, & Vesper, 1999; Perna, 2006). One of the most visible state financial aid programs is Indiana s Twenty-first Century Scholars (TFCS) program. The TFCS program was created in It is a last dollar grant program that provides recipients with up to four years of tuition at in-state public institutions or the equivalent amount of money to attend an in-state private institution. The TFCS Program has often been called one of the most progressive need-based state scholarship programs in the country. This is because the means test is simple (based solely on eligibility for the federally funded free and reduced lunch program) and participation is relatively easy (willingness in 2

3 middle school to sign a document indicating the student wants to attend college, will refrain from drug use and stay out of legal problems while in school). To successfully complete the program, students must maintain a C average in high school, graduate from an Indiana high school, apply for financial aid to college, and apply to at least one postsecondary institution. Students who meet all of these requirements are then referred to as affirmed scholars. The State of Indiana then offers affirmed scholars a need-based grant that in combination with the federally-funded Pell Grant program and other state aid assures students that they will not have to pay anything for tuition and fees at a public institution or the student will be provided a similar amount of money to attend a private college or university within the state. In addition to scholarship funding, the TFCS program also provides enrolled students with academic support during high school (free tutoring and mentoring) as well as assistance finding part-time work through one of 16 designated TFCS Centers around the state. At the college level, TFCS recipients can receive academic support provided by TFCS centers located on each campus. Against the backdrop of the current public policy environment it is hard to imagine a time period where the utility of research on Indiana s TFCS program would be more relevant. The economic downturn beginning in 2008 has had an impact on the number of students who are eligible for and making use of federal financial aid (Supiano, 2009). In addition, there is growing concern about the level of loan debt incurred by students (Martin & Lehren, 2012; Liberto, 2012). Not surprisingly, there are calls for major changes in both the Pell Grant Program and federal student loan programs (Bill and Melinda Gates Foundation, 2012; The College Board, n.d.). At the same time, many states have had to reduce their state financial aid awards because of declines in state revenue and they are reexamining the structure of their state grant programs. These factors are converging and there appears to be more openness than at any 3

4 time since the passage of the Higher Education Act of 1965 (Pub. L. No ) to considering major changes in the focus and structure of both federal and state aid programs. Along with the Hope Scholarship in the State of Georgia, the TFCS grant program in the State of Indiana is one of the most studied state financial aid programs in the United States. In this current policy environment, TFCS makes an ideal financial aid program to study because it features include many of the key elements of reforms for federal and state financial aid programs that are being considered by public policy makers. It is regarded as a relatively generous needbased grant program so research on TFCS can help answer questions on the impact of needbased aid on both postsecondary access and persistence. The means test for the TFCS program is also simple; students need only be eligible for free and reduced lunch to be eligible to sign-up for the scholarship program in middle school, so research on TFCS can also shed light on the effects of aid simplification on postsecondary participation and subsequent success and completion. Finally, research on TFCS also helps policy makers and scholars to better understand the impact of early information on postsecondary access and persistence. This is so because if low-income students sign-up to participate in the TFCS grant program they are told at a relatively young age that the price of attending college will not be a barrier for them and students and their families are regularly sent information that provides early college knowledge (Conley, 2008; Perna, 2004) through a student funded program entitled Learn More ( and through the State of Indiana s extensive regional Gear-Up network that runs throughout the state. Thus, research on the TFCS program can provide useful information to public policy makers as they consider the efficacy of current federal and state financial aid programs and as they consider the possibility of major changes in these programs. 4

5 The Twenty-first Century Scholars grant program has already received considerable attention from researchers (see for example St. John and colleagues (2000; 2001, 2001, 2002, 2004)). To date these studies provide a rich resource to draw upon, and illustrate the importance of creating state longitudinal databases for examining issues such as this. Overall these studies concluded that students who completed the TFCS program were much more likely than non- TFCS students to enroll in college and persist once enrolled. The regression-based techniques used in these studies, however, did not take into account the different ways in which student self-selection may have impacted the key findings. The earlier studies primarily focused on only those students who successfully completed the TFCS program, meaning among other things that they graduated from high school, applied for financial aid, and applied to at least one institution. Because the majority of students who began the TFCS program did not successfully complete it, however, the findings cannot be generalized to the broader group of students who initially enrolled in the TFCS program. Similarly, the students who initially signed up for the program may be different from those who did not, and if the reasons for this decision are correlated with the outcome in question, then the estimated impact of TFCS participation will be biased. In recent years, a clear consensus in the research community has emerged that high quality policy relevant research on student financial aid must employ data that permit the use of techniques that control for problems associated with selfselection. In this study, we focus our attention on the impacts of participation in Indiana s TFCS program on students postsecondary intentions and outcomes. We created a student unit record database of more than 60,000 students in Indiana, where the students were followed over a 13- year period from 9 th grade through college completion. The database includes information from 5

6 a student survey in 9 th grade, whether the student was enrolled in the TFCS program, as well as socioeconomic data on the communities in which they lived, and their college enrollment and completion patterns. In contrast to prior studies, we focus on the impact of the program on the group of students who initially signed up for the TFCS program, rather than only the students who completed the program. The length of the longitudinal dataset also allows us to examine whether participation in TFCS has an effect on a student s likelihood of graduating from college, which has not been previously studied. We employed several empirical approaches to obtain better estimates of the causal effect of TFCS participation on students. The first set of models relied on OLS to obtain naïve estimates of the impact of TFCS participation on selected outcomes. We then used instrumental variables (the distance between each student s home and the assigned TFCS center as the main instrumental variable) as a way to reduce the possible bias due to self-selection into the program. Finally, we applied propensity score matching as a way to compare students in the TFCS program with other students who had similar values on factors that were observable to us. Although each of these approaches has its limitations, the resulting analysis should provide policy makers with useful information on the effects of TFCS participation on student aspirations and outcomes. Theoretical Frameworks & Previous Research Theoretical Frameworks The main theoretical framework that we use for our study is that of higher education as an investment in human capital. According to this theory, education is a means by which individuals can increase their skills and qualifications that are subsequently rewarded in the labor 6

7 market (Mincer, 1958; Schultz, 1961; Becker, 1975). Students estimate the direct costs (net tuition) and indirect costs (foregone earnings) they will incur from going to college, and compare these to the expected financial benefits they will receive in terms of higher earnings in the future. The expected net benefits (E(NB)) may be written as the expected gross benefits from higher future earnings (E(B)) minus the expected direct and indirect costs (E(IDC)) from going to college: (1) E(NB) = E(B) E(C) = E(B) E(T - F + IDC) where direct costs equal tuition and fees (T) minus financial aid (F). The student s demand for going to college will then be a function of the expected net benefits from going to college, the student s ability to pay, and the utility from consumptive aspects of going to college. One might posit that a student s (latent) demand for attending college (a*) is a function of the expected net benefit from attending college, the student s ability to pay for college (Y), and perhaps other personal and family characteristics (P) that have exogenous effects on demand, as in: * (2) a = f ( E( NB), Y, P) The student will then aspire to attend college if the latent demand in (2) exceeds his or her threshold value. From this model, a student from a low-income family may decide to not pursue a college education because either the net benefits from attending college or the student s ability to pay for college is not sufficiently high to cross the threshold. In other words, even if the student has 7

8 positive expected net benefits from attending college, he or she may not attend if they cannot afford to pay the price up front and do not have access to loans. Financial aid programs may therefore affect a student s demand for attending college in two ways: by raising the expected net benefits of attending college, or by increasing the student s ability to pay for college. In comparative statics terms, increases in financial aid (F) should have a positive effect on the likelihood of a student going to college: * * * a a E( NB) a Y (3) = + > 0 ( ) F E NB F Y F provided each of the partial derivatives in (3) are positive. These restrictions mean that a student s demand for college increases with either net financial benefits ( a E(NB) > 0) or ability to pay ( a Y > 0), net financial benefits increase with financial aid ( E(NB) F > 0), and ability to pay increases with financial aid ( Y F > 0). A second strand of theory relates to the various stages that students go through in forming their aspirations for attending college. According to this framing, a student begins by forming expectations about whether or not to go college. After this stage, he or she makes decisions about the types of institutions to consider, and perhaps forms an initial choice set. The choice set becomes narrower as the student moves into the application stage, and finally makes the decision of whether to enroll and if so, where. Net price and hence financial aid becomes more important as the student moves through these phases. Indeed there is a robust line of research that connects the underlying constructs of human capital theory with studies of the net price of college attendance. Consistent with human capital theory, this line of research examines the extent to which reductions in the net price of attendance influences postsecondary participation, the postsecondary destinations of students, and student persistence. This line of inquiry has 8

9 consistently found that financial aid, which reduces the net cost of attendance, is positively related to college going, college choices, and persistence (Dynarski, 1999; Heller, 1999; St. John & Starkey, 1995). However, the impact of net price is not as large as might be anticipated because, as will be noted later in this literature review, a family s perceived ability (and willingness) to pay for college may affect whether students take the steps necessary to be prepared for college, and these may also be influenced by other factors such as whether the student takes a college prep curriculum, earns good grades, takes the SAT, visit campuses, and so on. Because this study examines both college access and completion, a number of other models related to college choice and student persistence provide us with additional conceptual framing. Models of college choice advanced by Hossler and Gallagher (1985) and more recently by Perna (2006), for example, help frame this study and provide additional guidance on the selection of variables. Similarly the Tinto (1993) and Bean (1980) models of student attrition also provide insights into this study of the impact of TFCS on student persistence in college. The Bean model, however, provides a better fit for this study since it includes finances as one of the factors that can influence student persistence. However, many of the same background characteristics that are posited to influence postsecondary access are also posited to have an effect on student persistence and completion. The one model that brings both postsecondary participation and persistence together is the nexus model of college advanced by St. John, Paulsen, and Starkey (1996). This model views the decision to attend college, which college to enter, choice of major, and the decision to persist until graduation as a series of choices influenced by student and family background, net price of attendance, as well as experiences while enrolled in postsecondary education. 9

10 Literature Review We break down the literature review into four subsections. We first address the general literature on college retention and success. We then turn to studies that have focused on the effects of financial aid on access and persistence, and in particular the role played by need-based grants. Finally, we review the studies that have been conducted to date on Indiana s TFCS program. College Retention and Success. Student persistence is among the most researched topics in higher education, arguably because it is essential to the mission of higher education. The student s route and timing of movement through the higher education system influence the probability of arriving at the desired destination (enrollment in college or degree attainment). Research based on the work of Tinto and Bean has shown the following variables to be associated with student persistence: (a) students background characteristics, (b) the initial commitments of students to the institution, (c) students commitments to the goal of earning a degree, and (d) the extent to which students experience academic and social integration (Braxton, Sullivan, & Johnson, 1997). The Tinto model, as operationalized by Pascarella and Terenzini (1979, 1980, 1983) and Terenzini and Pascarella (1980), uses measures of interaction with faculty and with students as indicators of academic and social integration. However, other scholars have also worked to extend, critique, and refine the empirical base supporting the Tinto and Bean models (Astin, 1993; Braxton & McClendon, ; Hurtado & Carter, 1997; Jalomo, 1995; Murguia, Padilla, & Pavel, 1991; Nora, Attinasi, & Matonak, 1990; Nora & Cabrera, 1996; Pascarella & Terenzini, 1991; Porter, 1990; Rendón, Jalomo, & Nora, 2000; Tierney, 1992). By emphasizing the importance of students cultural background and experience of campus contexts to the understanding of student persistence, Tierney (1992) introduced a 10

11 cogent critique of the earlier models. More recently, Braxton and his colleagues (Braxton, 2000; Braxton, Hirschy, & McClendon, 2004; Braxton, Sullivan, & Johnson, 1997) have focused on testing the propositions of the Tinto model and advancing new theories of student persistence. Since the Tinto and Bean models emerged, a number of additional theoretical and empirical studies have also informed the debate surrounding student persistence. For example, in addition to factors associated with the first models, other variables measuring both student characteristics and institutional practices have been associated with increased student persistence (Dadashova, Ziskin, & Hossler, 2010). Early on Astin (1975) found that working on campus was positively associated with student persistence, positing that work-study experiences bring students into more frequent contact with faculty and staff, thus integrating them more with the campus. As mentioned above, St. John and colleagues (e.g., St. John, Paulsen, & Carter, 2005; St. John, Paulsen, & Starkey, 1996) have focused on the role of financial aid in student persistence, finding in one study (St. John, 2003a) that the association of adequate financial aid with student persistence is strong and strongest among lower-income students. Several scholars have also called for a greater focus on the policies and practices of colleges and universities and how these influence student persistence (Berger & Milem, 2000; Braxton & McClendon, ; Hossler, 2005; Pascarella & Terenzini, 1991, 2005; Tinto & Pusser, 2006). Bean and Metzner (1985) found that academic advising was positively associated with the persistence of adult students. Kuh et al. (2005) identify seven institutional orientations (e.g., clear pathways to student success, an improvement-oriented ethos) associated with betterthan-predicted student persistence. In addition, Stage and Hossler (2000) and Bean and Eaton (2000), in a book edited by Braxton (2000), suggest that observable measures of student behavior should be used to advance our understanding of student persistence. The use of measures of 11

12 student behavior to study student persistence has also been advanced by Kuh, Kinzie, Buckley, Bridges, and Hayek (2006). This line of inquiry promises valuable insights into student persistence because it calls for research on institutions policies and practices and their effects on student persistence. Effects of Financial Aid on Students. The links between financial aid, persistence, and academic performance in higher education have been researched using different approaches. One direction, informed by cost-benefit analysis, has led researchers to investigate the effects of different types of aid (grants, loans, work-study) on students perceived ability or inability to pay (Cabrera, Stampen, & Hansen, 1990). The packaging of financial aid as it relates to college choice and persistence for underrepresented students has followed this direction (Nora, 1990; Nora & Horvath, 1989). Though financial aid alone may not remove barriers to success for students from low-income families (Stinebrickner & Stinebrickner, 2003), studies have shown that it can have an equalizing effect across racial and ethnic groups (Lichtenstein, 2002; Nora, 1990; St. John, Paulsen, & Carter, 2005) by removing financial barriers to access, encouraging preparation, and enabling students to focus more fully on academic concerns though the effects of aid likely vary among different underrepresented racial groups (Heller, 1997). We will also examine the effects of different types of financial aid packaging on college choice and forms of persistence. Effects of Need-Based Grants. Goldrick-Rab, et al. (2009) in their review of studies of the effect of financial aid on college going and completion conclude that the evidence is clear that grants have a positive impact on postsecondary access (matriculation). Perna (2010) and Deming and Dynarski (2008) reach a similar conclusion regarding the effects of grants on the postsecondary enrollment of low income students. In addition to having a positive effect on 12

13 postsecondary access, studies have shown that it can have an equalizing effect across racial and ethnic groups (Lichtenstein, 2002; Nora, 1990; St. John, Paulsen, & Carter, 2005) not only by removing financial barriers to access, but also by encouraging preparation, and enabling students to focus more fully on academic concerns though the effects of aid likely vary among different underrepresented racial groups (Heller, 1997). Overall, there is little disagreement that grants increase postsecondary participation among low income students. The empirical evidence on the effects of grants on persistence, however, is more mixed. In a comprehensive review of research that examined the effects of federal and state grants on student persistence Hossler, et al.(2009) conclude that the evidence of the effectiveness of grants in promoting persistence is more uneven although most studies find a positive relationship. However, the effects are not as pronounced as they are upon student access. Goldrick-Rab, et al. (2009) come to similar conclusions. They note that the factors that affect college retention and completion are varied and complex and that it is methodologically difficult to isolate the unique effects of grants on college student persistence. Deming and Dynarski (2009) also conclude that the preponderance of evidence suggests that grants help promote college persistence. It is worth adding one caveat to this overview of the effects of grants on college access and retention. The studies summarized in this section refer only to federal and state grants. Institutionally funded grants may have differential effects. For example, Singell and Stater (2006) conclude that merit-based grants have an impact on the postsecondary destinations of students, but that these grants do not directly increase a student s propensity to persist, rather the grants attract students who are more likely to persist once enrolled. Research on the TFCS Program. Ed St. John and colleagues (2000; 2001, 2001, 2002, 2004) conducted several studies of the effects of the TFCS program on postsecondary access and 13

14 retention. They examined the effects of the Twenty First Century Scholars Program on the likelihood that students would apply for financial aid, enroll in college in Indiana, and persist in college. To do this, the authors constructed a longitudinal database beginning with a large cohort of 9 th grade students in 1995, who were followed through their second year at in-state public colleges and universities in Indiana. The authors reported a number of significant findings, including the following: 1. TFCS had a large positive effect on college participation (85% of TFCS completers went to college, compared to 56% of non-tfcs students). 2. In comparison to non-tfcs students, TFCS completers were found to be four times as likely to enroll at in-state four-year public colleges, six times as likely to enroll at in-state two-year public colleges, six times as likely to enroll at in-state private colleges, and even twice as likely as non-tfcs to enroll at out-of-state colleges (even though the TFCS scholarship cannot be used at these institutions). 3. TFCS enrollees were 34% more likely than non-tfcs students to apply for financial aid. 4. The effects of TFCS completion on college enrollment were similar for both white and African American students. 5. Freshmen TFCS recipients enrolled in Indiana public institutions were more likely to reenroll as sophomores. Despite the attention brought to the TFCS program from these studies, they were not without limitations and shortcomings. Collectively, these limitations result in significant questions regarding the extent to which they shed light on the effects of the Twenty-first Century Scholars program on both access and success in college. First many of the studies focused solely on students who were affirmed scholars, and not the larger set of students who had initially 14

15 enrolled in the TFCS program. This is important because affirmed scholars are clearly more likely to go to college (they graduated high school, applied for aid, and applied to colleges) than students who started in the program but did not become affirmed scholars. A second limitation of past TFCS studies is that they did not take into account how the self-selection of students into the program may have affected their findings. There is a high likelihood that students who sign the pledge are already more likely than other students to go to college (self-selection). Ideally scholars and policy makers want to know whether any observed differences in student aspirations or outcomes are due to participation in the program and not the result of reasons that led students to participate in the first place. There were also some results presented in prior studies with regard to financial aid that require additional review. In the study that found that TFCS enrollees were 40% more likely than other students to apply for financial aid (St. John et al., 2004), the authors interpreted this as evidence that the TFCS program enticed more students to apply for financial aid. However, because the study did not control for family income, it is equally plausible that the large effect attributed to participation in the TFCS program could simply reflect the fact that low-income students, on average, were more likely than high-income students to apply for aid. With the data in hand, the authors could not disentangle these two possible effects. Similarly, in a study of college freshmen (St. John et al., 2003), the authors concluded that there was a positive association between the level of financial aid received and a student s likelihood of returning for the sophomore year. They inferred the impact on retention of having the TFCS scholarship by creating a composite variable as the TFCS scholarship plus financial aid from other sources, and finding that the new variable had a positive and significant effect on retention. However, they did not examine the unique effect of the TFCS scholarship holding constant other financial aid. 15

16 Therefore, there is no direct evidence in the study that the TFCS scholarship alone was associated with improved retention in higher education. Data and Methodology Dataset Construction The longitudinal dataset we compiled for this study drew upon information from several sources. One source is a survey conducted in of more than 60,000 9 th grade students in the State of Indiana, representing roughly two-thirds of the students in this grade for this particular year. The survey was conducted by LearnMore in Indiana, and was intended to capture data on the plans and aspirations of students in the state. This is the same survey instrument that was used in the previous studies of the TFCS program, but for a later year. The LearnMore survey included questions that captured demographic information on students (gender, race/ethnicity, family status, age), their post-graduation educational (e.g., attend 4-year college, etc.) and occupational aspirations, perceived barriers to going to college, and so on. Of primary importance to our study, the survey also collected information on whether the student reported being enrolled in the TFCS program. We matched these data on students with information from Claritas about the socioeconomic characteristics of the neighborhoods (blocks) in which these students lived. The matching was accomplished by mapping each student s home address into Census tracts at the block level using their home address, and then combining these data with Claritas data on the blocks within Census tracts. The Claritas dataset not only contains identifiers for each block and Census tract, but also other measures such as the household income and education level within each block. 16

17 The data were also matched to the National Student Clearinghouse (NSC) to obtain postsecondary enrollment and completion results for students in the 9 th grade sample. Adding the NSC data allowed us to (retrospectively) track each student beginning in Fall 2002 to ascertain whether and if so where the student enrolled in college, and we followed the students through 2010 to assess whether or not they had earned a degree. By using data from the National Student Clearinghouse, we can more reliably identify the postsecondary enrollment decisions of students who did not enroll at institutions within the state of Indiana, and can determine whether the students were able to graduate from in- or out-of-state institutions. Dependent Variables. We constructed several dependent variables for this study. The variables were chosen to span the range of decisions that students make with regard to preparing for postsecondary education and subsequently enrolling in and completing college. The first two dependent variables that we considered were whether the student expressed aspirations in 9 th grade to earn either a two- or a four-year degree. We posited that students who chose to enroll in TFCS would be more likely than other students to have postsecondary aspirations. It is not clear, however, whether the direction of causation should be that TFCS participation affects aspirations or vice-versa. Therefore, we interpret the results as reflecting an association between the two factors as opposed to making a claim about the causal ordering. We then created dependent variables to denote whether a student expressed interest in 9 th grade to pursue a college preparatory high school diploma. In Indiana, these options are referred to as a Core40 diploma or the more challenging Academic Honors diploma. In this model, we were interested in determining whether students who enrolled in the TFCS program were more likely than other students to express interest in one of these diplomas, because earning the 17

18 diplomas would help prepare the student to go to college and increase the student s chances of enrolling in and possibly graduating from college. Turning to postsecondary outcomes, we devised a dependent variable for whether a student enrolled in college beginning in Fall 2002, where the variable is equal to one if the student enrolled in any postsecondary institution (two- or four-year) and zero otherwise. The last dependent variable that we focused on in this study indicates whether the student graduated from college. This variable was set equal to one if he/she graduated from any institution (two- or fouryear degree) thereby capturing graduation even if a student transferred from one institution to another and earned their degree from the second (or more) institution attended. Explanatory Variables. The variables included as regressors can be grouped into several categories. The first category represents demographic factors for each student, including their gender, race/ethnicity (5 categories), age as of 9 th grade, whether they were a firstgeneration student (i.e., neither parent graduated from college), whether Spanish was spoken at home, and family status (e.g., whether student lived with both parents, one parent, etc.). The second group of explanatory variables represent academic factors such as the student s middle school GPA (derived from self-reported grades), and whether they planned on selecting specific academic programs during high school (such as Advanced Placement courses or pursuing an Academic Honors diploma, which is more academically demanding than other diploma options in the state). The second set of variables also includes a measure for whether the student reported being a member of Indiana s TFCS program. The third group of control variables represent the geographic location of each student. We represented geographic location by the Census tract in which a student resides. Statistical Models 18

19 Naïve Models. For each of the seven dependent variables, we estimated a series of linear probability regression models: (4.1) Y = α + δt + ε (4.2) Y = α + δt + X ' β + ε (4.3) Y = α + δt + X ' β + S' θ + ε where Y = outcome of interest, T = treatment variable (i.e., participation in TFCS program) with coefficient δ, X = set of control variables with coefficients β, S = fixed effects for geographic location with coefficient θ, and ε represents a random error term. Estimating these via OLS (a linear probability model) or a logit/probit technique for equations (4.1) to (4.3) gives rise to what is often called a naïve estimate for the impact of T on Y because it ignores the possible bias that is due to correlation between the error term and T. This correlation can be due to omitted factors that influence both selection into the treatment group T and the outcome Y. Despite the inherent bias, the estimated impact δ has some informational value because it illustrates the reduction in the mean difference in outcomes that is due to the observable factors in X. For each of these models, we adjusted the standard errors for clustering by using the 91 counties as the clustering level. Instrumental Variables. For the dependent variables representing high school degree options, college enrollment and graduation, we also employed two-stage least squares to help reduce the possible bias in the effect of TFCS on these outcomes. In this procedure, we identified variables that were arguably related to a student s selection into the TFCS program but not independently related to their likelihood of either going to college or earning a degree. The 19

20 main instrumental variable that we used was the distance from each student s home to their assigned TFCS center. The theoretical justification for this instrumental variable is that there is a negative relationship between distance and a student s likelihood of enrolling in the TFCS program because as the distance increases, it would be more difficult for students to go to the center and take advantage of the various support services that they offer to students in the program (Alfonso, 2006; Long & Kurlaender, 2009). We enriched the set of instruments for the analysis by interacting the distance variable with the student s race/ethnicity, first-generation status, and family status, and then tested for whether the resulting model was overidentified. Propensity Score Matching. The second statistical approach that we used to measure the effect of the treatment in this study is propensity score matching (PSM). The object of PSM is to match students in the treatment group (TFCS) with other students based on their propensities to be in the treatment group. At different levels of the propensity to receive treatment, treated and non-treated participants should be similar in terms of their observable variables (i.e., SES, academic background, and social capital). This characteristic is known as the balancing assumption x z e(x), where conditional on the propensity towards treatment (e(x)), treatment assignment (z) is independent of participants characteristics (x) which translates into treated and non-treated participants having similar (post-match) observed characteristics. If this assumption holds, we can then compare the average outcomes for the two groups to determine the average treatment effect as follows: (5) E{r 1 e(x), z = 1} E{r 0 e(x), z = 0} = E{r 1 r 0 e(x)}, 20

21 where treated (z = 1) and non-treated participants (z = 0) are selected and matched upon e(x) and their outcomes were compared accordingly (Rosenbaum & Rubin, 1983, Agodini & Dynarski, 2004). Given that the propensity score is a continuous variable ranging from 0 to 1, the probability of observing two units with exactly the same value of the propensity score is, in principle, zero. We solved this problem by using Mahalanobis-Metric Matching (MMM). Although other matching mechanism are available, several of which we used for a robustness check, the MMM approach provided us with the flexibility to mirror the specifications used in the IV approach (for more details about matching mechanisms see Becker & Ichino, 2002). PSM assumes that units are not competing to gain access to treatment or that any unobservable factors will not materially change the results from the analysis. To test for this assumption we relied on the Rosenbaum bounds sensitivity analysis (Becker & Caliendo, 2007; Keele, 2010) which tests whether the treatment effect changes as a function of an increase in the probability of receiving treatment. If there is an increase or decrease, then there may be unobservable characteristics that are driving the findings and the results from PSM should be viewed with caution. Results Descriptive Statistics Table 1 contains descriptive statistics for the key variables used in our analysis. The first rows represent the postsecondary aspirations and outcomes for students in the dataset. The subsequent variables are the explanatory variables used in the various models that we estimated Insert Table 1 Here

22 Beginning with postsecondary aspirations, we found that nearly two-thirds (65%) of the 9 th graders in our sample indicated that they would like to earn a four-year degree, and another 9% indicated a preference for earning a degree from a two-year institution. Therefore, nearly three out of every four 9 th graders expressed an intention to earn a college degree of some form. The data indicated that 57% of all students actually enrolled in college, with about two-thirds of them choosing a four-year institution. We also found that slightly less than 30% earned either a twoor four-year college degree, and thus there was a notable drop off from early aspirations to college enrollment and graduation for students in our sample. For the control variables, we found that about 7% of the students in the sample were enrolled in the TFCS program as of 9 th grade. As expected, about half of the students were male, and nearly four out of five students were white. Interestingly, more than half (56%) of the sample were first-generation college students, and 59% reported living with both parents. Both factors proved to be important for selection into the TFCS program. In Table 2, we broke down the means for the outcome variables of interest separately for TFCS enrollees and all other students. As of 9 th grade, TFCS enrollees had very similar postsecondary aspirations as did students who were not enrolled in the program, with TFCS enrollees being slightly more likely to indicate aspirations for a four-year degree and pursuing a Core 40 diploma. TFCS enrollees were found to enroll in college at higher rates than non-tfcs enrollees, although the difference was more highly concentrated in two-year institutions. Finally, we found that there was a six percentage point difference in terms of actually obtaining a postsecondary degree (24% versus 30%) Insert Table 2 Here

23 To begin our focus on selection into the TFCS program, we examined how the means for the explanatory variables in our study varied by TFCS enrollment status. Table 3 provides a breakdown of the means for TFCS enrollees and non-enrollees, respectively: Insert Table 3 Here The results in Table 3 clearly illustrate that there are substantial differences between the students who enroll in the TFCS program and other students in the sample. TFCS enrollees were much more likely than non-enrollees to be female, non-hispanic Black, live in a single-parent household with only their mother, and/or be a first-generation college student. Models for Postsecondary Aspirations We now turn to the empirical results for the models explaining whether or not a student had aspirations to pursue either a two-year (Table 4) or four-year (Table 5) degree. The first column reports the mean difference in initial aspirations by TFCS enrollment status. The second column adjusts for the difference in aspirations that can be attributed to observable factors such as race/ethnicity, age, family living status, first-generation status, and middle school GPA. Finally, the last column contains the results for the regression model after adding fixed effects at the Census block level Insert Tables 4 and 5 Here The results indicate that there was no difference between TFCS enrollees and nonenrollees in their aspirations to earn a two-year degree. In contrast, after controlling for observable factors and fixed effects, TFCS enrollees were approximately 4% more likely than non-enrollees to indicate an interest in pursuing a four-year degree. As noted earlier, because aspirations were possibly formed prior to the TFCS enrollment decision, this result reflects an association rather than causation per se. With regard to other variables, we found that males and 23

24 first-generation students, as well as students with lower middle school grades, were more likely than others to indicate an interest in earning a two-year degree. In contrast, the results showed that the following students were more likely to have aspirations to earn a four-year degree: females, younger students, non-hispanic Blacks, those living with both parents, non-first generation college students, and those with higher grades in middle school. Tables 6 and 7 provide results from the linear probability models where the dependent variables represent the student s intended high school diploma goal. In the first set of results (Table 6), the dependent variable is whether the student intended to pursue a Core 40 diploma, which represents a minimum set of course requirements typically needed to enroll in college. In the second set of regression models (Table 7), the more ambitious Academic Honors diploma is the dependent variable. Each table contains a fourth model where we used two-stage least squares to help take into account the selection of students into the TFCS program Insert Tables 6 and 7 Here Beginning with Table 6, we found that after taking into account selection through twostage least squares, the statistically significant difference between TFCS enrollees and other students in their intent to earn a Core 40 diploma was eliminated. Males, Asians, and Blacks were less likely than other students across all models to indicate an intent to earn a Core 40 diploma. Similarly, younger students, students who live with both parents, and students with lower grades in middle school were more likely than their peers to express an intention to choose a Core 40 diploma. Turning to the next set of results in Table 7, the models indicated that there were no differences between TFCS enrollees and other students in their interest in earning an Academic Honors diploma, with the results not being materially affected by selection of students into the 24

25 TFCS program. Females, students with at least one college-educated parent, students with higher middle school grades, and those living with both parents were more likely than other students to express an interest in pursuing an Academic Honors diploma. Next we focused on the effects of participating in the TFCS program on enrolling in college. Table 8 contains results from models where the dependent variable is whether a student enrolled in college (two- or four-year). As with the dependent variables for high school degree options, we added a fourth model using two-stage least squares to help control for student selfselection into the TFCS program Insert Table 8 Here We found that after controlling for a series of observable factors, fixed effects, and selfselection, students who were enrolled in the TFCS program were more likely than other students to be enrolled in college in pursuit of either a two-year or four-year degree. There were few differences in college enrollment by race/ethnicity or gender. At the same time, older students and first-generation college students were less likely than their peers to enroll in college. We also observed that students living with both parents and those with higher middle school grades were more likely to enroll in either a 2-year or 4-year college. In Table 9, we isolated college enrollment to only those students who enrolled in a fouryear institution Insert Table 9 Here We found that the negative effect of TFCS participation on four-year college enrollment was attributable to observable characteristics such as race/ethnicity, gender, family living status, firstgeneration status, and academic performance in middle school. The insignificance of TFCS participation on four-year college enrollment persisted after we added fixed effects to the model 25

26 and controlled for self-selection using two-stage least squares. Overall, the findings for the other variables in the model were similar for both enrollment measures, with the point estimates for living with two parents, being a first-generation college student, and middle school GPA being slightly larger for explaining four-year enrollment as opposed to enrollment in any postsecondary institution. In Table 10 we examine how TFCS participation and other factors influence whether or not a student graduates from either a two- or four-year institution. We present results from the same four models as shown before Insert Table 10 Here Beginning with the control variables in our study, we found that males were about 2-3 percent less likely than females to graduate from college. With regard to race/ethnicity, non- Hispanic black students were 5 to 8 percent less likely than non-hispanic white students to earn a college degree. Students who lived with both parents were nearly 11 percent more likely to graduate from college, and at the same time, first-generation students were 11 percent less likely to complete college. Finally, there was a strong and positive connection between a student s middle school grades and his/her likelihood of earning a postsecondary degree. With respect to the key variable of interest, we found that the large negative effect of TFCS enrollment on graduation was eliminated once we controlled for observable factors of students and fixed effects. Although the point estimate for the TFCS variable became positive once we controlled for self-selection through instrumental variables, the coefficient was not statistically significant at the 5% level (p-value = 0.057). The last table presents findings for the average treatment effect of TFCS participation using propensity score matching. The propensity scores were obtained by using the same sets of 26

27 regressors as we report for models (3) in the prior tables. The first column shows the average value of each dependent variable for students in the (TFCS) treatment group. The second column contains the average for matched students in the (non-tfcs) control group. The differences in averages are shown in the last column Insert Table 11 Here Overall, the results from the propensity score models are similar to what we found with the regression model for each dependent variable where we controlled for both observable factors and fixed effects (i.e., the third column of results in Tables 4-10). Students who enrolled in the TFCS program were about 5.7% more likely than their matches to aspire to a four-year degree, about 7% more likely to plan on earning a Core40 high school diploma, and about 3% more likely to plan on earning an Academic Honors diploma. We found no statistically significant differences between TFCS enrollees and their matches in terms of college enrollment or completion. Conclusions and Next Steps In this study, we used several different statistical approaches to focus on the effects of enrolling in the TFCS program on students. One of the key distinguishing features of our study in comparison to past studies of the TFCS program is that we focused our analysis on the effects of enrolling in, rather than completing, the TFCS program. The analysis presented here is also important because we attempted to demonstrate how taking into account student self-selection into the program can influence conclusions about the impact of the program on student s postsecondary aspirations, plans, and outcomes. 27