U s i n g C I R P S t u d e n t - L e v e l D a t a t o S t u d y I n p u t - A d j u s t e d D e g r e e A t t a i n m e n t

Transcription

1 U s i n g C I R P S t u d e n t - L e v e l D a t a t o S t u d y I n p u t - A d j u s t e d D e g r e e A t t a i n m e n t John H. Pryor, Director Cooperative Institutional Research Program Sylvia Hurtado, Professor and Director Higher Education Research Institute, UCLA September 2012

2 We thank graduate student Ray Franke for his expert assistance with data analyses. We also thank Sandy Baum and Steve Porter for their comments on a previous draft of this paper. A b s t r a c t In this paper we illustrate the importance of input factors in examining the completion of college with a degree, using CIRP Freshman Survey and National Student Clearinghouse data on students entering college in the cohort of Results are based on previous studies of cohorts on degree attainment, a new report on college completion, and analyses on students who are mobile completers, those students who leave their initial institution and complete a baccalaureate degree elsewhere. Some colleges underperform based on their entering-student characteristics, and while low degree attainment is evident, some colleges are doing better than expected. The study concludes with needs for future research and policy and procedure recommendations concerning the importance of using input-adjusted graduation rates. 2

3 I n t r o d u c t i o n Questions about how to assess institutional quality have enjoyed resurgence in the past few years, mostly because of the rising cost of college and the economic reality of the job market. Outcomes measures, in particular graduation rates, tell an important part of the story about the educational experience. Although poor completion rates are well known in academic circles, still the general public, families of students attending college, and the students themselves assume that a student entering college will graduate four years later. This is not the case. About 57 percent of first-time, full-time students who enter a specific college graduate within six years, and far fewer, only 36 percent, are able to complete the task in four years 1. Institutional degree completion rates, therefore, impart important information about the probability of students achieving their educational goals at specific baccalaureate-granting colleges or universities. While a conventional view has been that institutions with higher graduation rates are of higher quality, this view fails to take into account how these rates are influenced by the makeup of each entering first-year class. Nor do graduation rates currently reported to the federal government take into account differences in institutional resources. The purpose of this paper is to discuss the complexities of assessing completion rates as an indicator of institutional quality. We discuss the importance of using adjusted graduation rates that take into account both unique characteristics of the student body and also variation in institutional resources. This is best accomplished through the use of extensive information at college entry, federally reported information about institutional resources, and tracking individual students to degree completion at the same or a different college. We contend that the information should be used to help develop indicators that are fair to the student (i.e., effectively meet Student Right to Know legislation) and fair to institutions that serve large numbers of first-generation, low-income, racially diverse students, and/or students who require additional preparation that they did not receive in high school. Most important, if the goal is to raise the level of education of the U.S. population, 2 each institution must do its part and such indicators should be used to focus on institutional improvement to not only increase degree attainments but also achieve greater equity across populations of students. Recognizing the importance of input-adjusted outcomes Historically, the Cooperative Institutional Research Program (CIRP) student surveys were conceptualized to fill a knowledge gap concerning the impact that college has on students. Alexander Astin, as director of research at the American Council on Education (ACE), observed in the early 1960s that the state of knowledge about what worked in higher education research was severely limited. Most existing studies were based at only one institution, were not comparable across institutions and were limited to cross-sectional examinations of student experience. The CIRP Freshman Survey was launched in 1966 to remedy this situation. The project subsequently moved from ACE to the Higher Education Research Institute in 1973 to continue national assessment of college impact. 3

4 Of primary interest for this paper, however, were Astin s observations about the connection between the outcome measures that were typically used in higher education to describe institutional excellence, such as Ph.D. production, and the input characteristics students brought with them into higher education. In examining the factors involved in producing students who go on to earn a Ph.D., a typical study of the time connected large Ph.D. numbers coming from certain undergraduate colleges with larger libraries, smaller student-faculty ratios, and more faculty who themselves had Ph.D.s than the less productive colleges. Astin, however, observed that these more productive colleges also enrolled more National Merit Scholars. In a subsequent study that Astin designed, he was able to show that when you took student inputs into account, some of the so-called highly productive institutions were actually underproducing Ph.D.s, whereas some of those with more modest outputs were actually producing more than one would expect from their student inputs. 3 Astin made three important observations: 1) Outcome-only measures tell us very little about the impact a college has on such measures; 2) multiple input measures have varying value in predicting outcome measures, and thus a single input measure is not sufficient; and 3) inputs and outputs tell only part of the picture, and in order to understand how to influence outcomes in the college environment with an eye toward institutional improvement, one also needs to collect information on the environment. 4 He then described his I-E-O model, which recognized that in order to understand what leads toward the outcomes of college (the O of the model), we need to account for both the environment that the students operate in during college (the E of the model) and, importantly, the characteristics and experiences that the student brings into college (the I or input of the model). The Cooperative Institutional Research Program represented a major forward move in educational research. CIRP started with the Freshman Survey, a common instrument designed to be administered at multiple institutions. This then ensured that the same exact questions would be asked at each school administering the survey and asked of the full cohort of entering first-time, full-time students. CIRP was also designed to be longitudinal, following the same students over time. The Follow-up Surveys (FUS) captured both the E and the O of the model. They were administered after two and four years of college to a random sample of students who had completed the CIRP Freshman Survey, thus creating a rich database that connected the same students at different points in time during their college career. Studies using the CIRP Freshman Survey and the FUS began to build a body of literature that examined the impact of college on various outcomes measures, linking those outcomes to both the environment of college and the input characteristics of the incoming students. 5 CIRP enriched the study of higher education in several ways, then, by introducing a comparable sampling frame, assessing the full cohort of college-entering students, weighting the data to 4

5 reflect the national student population, and then following up with students to provide a longitudinal assessment of change. With a large database of schools using a common instrument (307 institutions participated in the first survey in 1966), it was possible to compare results between schools. Further, it was then possible to compare schools with similar types of institutions: for example, benchmarking selective research universities with other selective research universities. This made the comparative data more relevant to participating institutions as they could choose to focus comparisons only across similar institutions. A stratified sampling scheme included institutions by selectivity, control, religious affiliation, college race (black colleges and universities), and two- and four-year colleges. Institutions were instructed to administer the survey to full cohorts of entering freshmen to 1) ensure representation of entering student cohorts, and 2) create a substantial baseline for follow-up studies. Weighting CIRP results up to the national population of incoming first-year students created the first national profile of such students. The weight was a two-stage weight: The first weight ensured that the institutional types were accurately represented, using selectivity, type and control. The second weight corrected for response biases in the mix of men and women, and weighted by sex up to national figures for incoming students. Normative reports were based on only institutions that have surveyed complete cohorts or a large percentage of all entering first-time, full time freshmen. Validity of self-reports in using CIRP survey data While the promise of CIRP was to allow large-scale comparisons between institutions of different types by the use of a common questionnaire used by hundreds of thousands of students, of necessity a written questionnaire allows less opportunity for clarification than a face-to-face interview. 6 Of the potential sources of error when using a survey, measurement error is one that results when a respondent is unable or unwilling to provide an accurate answer to the questions that are posed. A key premise in writing survey questions is that they are as clear as possible, ask about issues that the respondent can answer and minimize measurement error. A complete literature review of survey self-report is beyond the scope of this paper, so in this section we will only address several studies specific to the use of CIRP measures. One might argue that the easiest test of the validity of a self-report survey is to ask one well-defined question with an easily known answer, and to provide the means by which a respondent can readily report it in the correct fashion. For college students, one would be hard-pressed to find a more ideal test of this than to ask them to report their SAT scores. In 2010, more than 1.5 million high school students took the SAT. 7 The scores a student receives, in many ways, are a large part of what defines him or her as a college candidate, and arguably have high saliency for students who have been admitted to college. Thus, they are readily available to the respondent, highly memorable and easily communicated in a survey. However, even something as simple as SAT reporting and verifying can be complicated. 5

6 DeAngelo and Tran 8 examined the relationship between self-reported SAT scores on the 2004 CIRP Freshman Survey and actual SAT scores from those same students (n=128,944) obtained from the College Board. In this group, 79 percent took the SAT more than once, and thus had more than one possible answer for each question (SAT verbal and math scores). Thus, possible sources of confusion are 1) which score does the student report, and 2) which score does the College Board provide. In this case, the authors decided that the score the student was most likely to provide was the highest score. Since the College Board provided every score the student attained, and not just the last score or the highest score, there was a choice of potential matches. The authors chose to match the student self-report with the highest score in the College Board data. It should be noted that studies using registrar data to examine self-report SAT validity would need to clarify which score the registrar records as the actual SAT score, and some use the first reported, some the last reported, some the highest in one pair (verbal and math must be from the same session) and some the highest regardless of pairing. The authors considered a match as any self-reported score within 10 points of the actual score (a margin of error of 1.7 percent). When they used combined SAT scores for math and verbal, 78.9 percent of the scores matched, approximately four out of five. Of that 20 percent with scores that were off by more than 10 percentage points, half were within 20 points (a margin of error of 3.3 percent). There was a slight tendency of students with lower SAT scores to be less accurate overall, but without a particular bias in under- or overreporting. When there were, in small numbers, larger distortions in reported scores, they were more likely among students with higher actual scores who gave even higher self-report scores. Thus the validity of the self-report for SAT score would be fairly reliable once one clarifies the actual scores that are being compared and the acceptable margins for error. In a similar study with a smaller group of students taking the CIRP College Senior Survey, self-reported GPA was compared with actual GPA as collected from college registrars. 9 The authors found that 90 percent of the self-reported grade point averages were within 0.3 points of the registrar-reported GPA 10, and that when the error was over 0.3 points, it was fairly evenly distributed among over- and under-reporting. Of course, not all of the important issues in education are as easy as SAT and GPA. We are often more interested in more complex behaviors, attitudes and perceptions that can be more difficult to verify. Rhee and Hurtado 11 examined the connection between self-report on the CIRP Freshman Survey and standard measures of cognition, including the California Critical Thinking Skills Test (CCTST), as well as other surveybased constructs such as the Need for Cognition. The authors utilized a structural equation model with two separate samples, 289 American students at a public university in the United States and 302 Korean students at a small private university in Korea (only the U.S. results will be presented here). 6

7 One of the measures used was a scale created from three CIRP survey questions, which were self-ratings on intellectual self-confidence, academic ability and problem-solving ability (alpha =.85). These questions were taken from a bank of approximately 25 self-efficacy ratings that have the following instructions: Rate yourself on each of the following traits as compared with the average person your age. We want the most accurate estimate of how you see yourself. The response options were: Highest 10%, Above Average, Average, Below Average and Lowest 10%. The Rhee and Hurtado paper shows prediction equations for several cognitive measures. The standardized test (CCTST) and other cognitive measures were used with a random sample of public university students who were required to participate in experiments as part of a large psychology class pool of subjects. Standardized parameter estimates using structural equation modeling show that the CIRP selfratings construct of Academic Self-Concept in relation to the standardized test of students' critical thinking skills (CCTST) is.37**, while the relationship with SAT verbal is.25** and math is.32** in independent models. A combined equation model, controlling for SAT verbal and math, revealed that the estimate for Academic Self-Concept is.25 while the estimate for SAT verbal remained at.25, and math was substantially reduced to nonsignificance (.13). Not surprisingly, Academic Self-Concept is more strongly associated with other instruments based on student self-report, measuring students' Need for Cognition (.55**) and the Critical Dispositions Inventory (CCTDI) at.42** after controlling for SAT (verbal and math). Suffice it to say that student self-ratings in CIRP surveys are associated with students academic achievement and depict what many more standard assessments attempt to do in capturing students' assessment of skills and abilities. For other dispositions, behaviors and attitudes (self-ratings and reports of activities) that are covariates with achievement, there are simply no other measures to validate their veracity when students are the only source of the information, except in relationship to outcomes that matter (i.e., criterion validity). This is particularly the case with degree attainment in college, where countless studies have established a wide array of student activities, dispositions and perceptions of the environment that help to predict retention. CIRP survey measures should be used in concert with other outcome measures of students skills and abilities longitudinally, especially in cases where extensive standardized measures are absent. In another example, Anaya 12 used standardized tests scores with CIRP data to compare different methods of examining student learning. Using a matched set of CIRP Freshman Survey and College Student Survey data from seniors four years later, she also used GRE scores for 2,289 students obtained from the College Board. She then ran three sets of regressions: one using self-reported measures of self-change from CIRP as a dependent variable, one using residual change between SAT and GRE scores as a dependent variable, and the 7

8 other one using self-reported GPA as a dependent variable, all of which can be seen as summarizing student learning in college. The residual SAT math-gre quantitative gain scores were compared with the self-rated change in quantitative skills ( compared to when you entered college as a freshman, how would you now describe your analytical and problem solving skills ). The SAT verbal-gre residuals were compared with selfrated change in verbal ability ( compared to when you entered college as a freshman, how would you now describe your writing skills ). The independent variables in the model were identical in each regression, and included CIRP variables that measured pre-college factors, institutional factors, major and participation in various learning experiences such as working on a professor s research project, hours per week studying, and self-efficacy as measured by student self-ratings. Although the predictive powers of the models were strongest when GRE was used, followed by GPA and then followed by the self-reported gains, the substantive findings about the relative impact of the independent variables were remarkably similar. This indicates that a study using either one of the dependent measures would have come to the same conclusions about the importance of various student experiences on learning outcomes. Thus, we have seen with the CIRP surveys that self-reported SAT score is accurate when compared with actual SAT, that self-reported GPA is accurate when compared with actual GPA, that several self-ratings on the CIRP Freshman Survey are correlated with direct measures examining similar cognitive concepts, and that when direct and self-report measures are used interchangeably as dependent variables, the pattern of influence of independent measures is very similar. There have been other recent criticisms of survey self-report, 13 some of which go several decades back to the creator of the College Student Experiences Questionnaire, Bob Pace, who addressed many of the same issues currently being debated. 14 Pace, however, came to different conclusions about the veracity of some selfreport data, most notably the oft-asked question how often is often? While Pace and Friedlander conclude that qualifiers such as often and very often correspond well to questions that use numerical ranges depicting prevalence in combination with qualifiers (e.g., about once a week, and 1 or 2 times a month ), Porter s own analysis of their tables is that it signifies widespread discrepancies. Certainly we should recognize limitations and distortions that exist in memory and try to minimize their influence in self-report surveys. The CIRP surveys, in many cases, use qualifiers in response banks, but there are fewer response options to choose from with a greater degree of differentiation between the response options. When the surveys ask incoming first-year students to reflect on the past year and indicate how often they, for instance, were bored in class, students are given only three response options: frequently, occasionally or not at all. Fairly straightforward in themselves as qualifiers, the instructions also specify: if you engaged in an activity one or more times, but not frequently, mark occasionally and mark not at all if you have not performed the activity during the past year. The wording of these questions provides sufficient direction to respondents but not enough latitude to result in vagueness. 8

9 Another important recognition here is that some questions just cannot be compared against outside standards. Respondent opinions, perceptions, values and beliefs about themselves are important aspects of their everyday experiences and have value. Certainly the questions should be crafted with care by people familiar with all the potential sources of bias that can impact results. But just because perceptions and values are not easily verified does not mean that they are not important or reliable in predicting student achievement. There are scores of studies that examine the connection between perceived campus climate and outcome measures, such as graduation, that are backed up by observations and interview studies. This is why it is a common practice in research to also use multiple questions that examine the same trait from different perspectives to create constructs that attempt to describe a phenomenon. More sophisticated methods, such as the use of item response theory in creating constructs, also have moved the field forward. 15 We turn now to the business of creating adjusted graduation rates using both students self-reported and institutionally reported measures. Demonstrating the importance of student-level input variables While the study of who completes college has been prolific for several decades, the proportion of students graduating from college compared with those who enter college has remained fairly stable. Part of the problem has been the lack of relevant and suitably complex data to address the multiple issues involved in degree completion. In order to frame the question of how to increase degree completion, we created a unique data set that furthers previous reports at the Higher Education Research Institute by Astin, Tsui and Avalos 16, and by Astin and Oseguera. 17 In these previous studies, the authors combined data from the Cooperative Institutional Research Program s (CIRP) Freshman Survey with degree completion data obtained six years later from the registrars of the colleges and universities that had participated in the CIRP Freshman Survey in and in In both of these cases, the authors demonstrated that degree completion could be predicted fairly accurately just by knowing the characteristics of entering students. In an update of this work with a new study of degree attainment, 20 we merged student-level data from the 2004 CIRP Freshman Survey with four-, five- and six-year completion data from the National Student Clearinghouse (NCS). This has the immediate benefit of providing a richer source of data with greater breadth. Almost four times as many students are in this database (210,056, compared with 56,818 in the 2005 report), and we also have the ability to follow students who have left their initial institution for another. Traditional studies of graduation rates have typically concerned themselves only with students who either complete at the initial institution or leave, not accounting for those who finish elsewhere. A limitation of this data set is that it is composed of first-time, full-time students at four-year institutions, and does not reflect all students entering postsecondary institutions. Given that this population was 1.3 million students in 2004, it represents a significant population to be studied, and indeed has been the most-often examined in 9

10 federal reporting of graduation rates as required by IPEDS. Moreover, approximately 65 percent of all recent high school graduates begin as first-time freshmen. 21 These students have the best chances of completing college given their initial intentions, although the current study also shows there is variation across institutions in terms of first-time, full-time students initial intentions to stay at a particular four-year college that can affect graduation rates. As found in previous HERI studies, degree completion figures at three different intervals (four, five and six years after entering college) vary greatly as a function of student characteristics such as sex, first-generation status and race/ethnicity, but also institutional characteristics such as control (public versus private), type (college versus university) and religious affiliation. Our findings mirror other studies that have shown higher proportions of women, Asian and white students, and students with at least one parent having attended college. 22 Similarly, private colleges and universities are more likely to graduate students, as are, for the most part, universities as opposed to four-year colleges. The crux of the study is, however, a reassessing of institutional graduation rates that takes into account the significant influence of the input characteristics students bring with them to college. Institutions are aware that these characteristics can influence degree attainment, and many use rudimentary calculations to create an expected graduation rate based upon sex, race, SAT/ACT score and high school grades. By adding variables from the CIRP Freshman Survey, there is much more information concerning student input characteristics that include high school behaviors, self-efficacy ratings, college choice information and expectations of college, among others. Of the approximately 250 measures available from the survey, the previous study included 132 in a logistical regression model to predict graduation (variables were included if they had previously shown predictive power and were fairly consistently asked from year to year in the survey). Adding these variables into the model, when compared with the simple four-factor model described above, increases the ability to predict (using the Nagelkerke R-squared statistic) four-year graduation by 65 percent (from.203 to.335), by 54 percent for five-year graduation, and by 52 percent for six-year graduation. As with other studies, once these input variables are entered into the model, the importance of SAT/ACT is no longer significant, and adds only.003 to the Nagelkerke R-squared. Unadjusted graduation rates, therefore, when used as a measure of institutional quality can be misleading, since different types of institutions admit and enroll students with very different characteristics, past academic experiences and achievements, and goals. If institutions are to improve their degree completion rates they must first be able to accurately assess how effective they are in moving the students they enroll toward graduation. This has been referred to as talent development, while others sometimes use the term value added. 10

11 In fact, several illustrations from our 2011 report demonstrate the importance of taking into account the student input factors when judging an institution s graduation rate. In our results, we find that private universities graduated 64.0 percent of their students after four years compared with only 23.5 percent at public four-year colleges. However, in making a judgment about the institutional ability to move students to graduation in four years, we must take into account the types of students that enroll at these institutions. Public institutions, for instance, tend more toward higher admission rates than do private institutions. The expected graduation rate for four years, using CIRP Freshman Survey data and based upon the input characteristics of the students, is 19.3 percent for public four-year colleges and 67.7 percent for private universities. Thus, while the overall raw graduation rate is significantly lower, public institutions are actually overperforming when taking into account student input characteristics, while the private universities slightly underperform. We can take this demonstration further by using the model for public four-year colleges but replacing the input characteristics with those of the students entering private universities, essentially modeling what would happen if we sent the private university students to a public four-year college. If that were to happen, and the input characteristics changed, the public four-year college rate jumps by 140 percent to 56.4 percent. Clearly any judgment about institutional effectiveness is very misleading without taking into account the characteristics of the students who enroll at that particular institution type. While it is more likely that a student entering a private university will graduate, not every student has the chance to attend such a school, and many are limited in choices by finances and/or geography. Thus we have demonstrated that portraying unadjusted graduation rates as a measure of institutional quality and effectiveness, as required by the Students Right to Know and Campus Security Act, without first taking into account the types of students who enroll at an institution strongly favors the most selective institutions and tends to penalize institutions that enroll less-prepared students or those with broader access policies, even if these institutions have success in moving some of their students toward graduation who would not otherwise have had an opportunity. Moreover, the expectation that students will complete a degree in four years is now a norm only at the most selective colleges. A focus on improving only unadjusted graduation rates can have significant consequences for higher education. Such a focus could influence institutions to attempt to bring the same top students to campus as a means to improve the retention bottom line rather than to improve the college s efforts to enhance degree completion success among the students they already serve through changes in policy or educational programs. This focus can also discourage institutions from enrolling and working with underprepared students, a practice that works solidly against improving degree attainment rates among the adult population in the United States overall. In fact, in comparing our two degree attainment reports using the 11

12 entering classes of 1994 and 2004, growth in degree attainment in that period was influenced greatly by higher graduation rates among academically well-prepared students and lower rates than the previous decade among less-well-prepared students. Like academic preparation, certain input characteristics influence graduation rates more than others. These characteristics can be clustered into types that have differential implications if an institution were to actively manipulate input characteristics in order to increase graduation rates. For example, students considerations around financing college impact subsequent decisions to stay at the same college. Students who reported that cost was an important reason for choosing their college were more likely to graduate from the college they selected. Recent attempts to make college costs more transparent and understandable through the availability and promotion of net price calculators might also impact graduation rates if use of such tools were to instill a better sense of affordability. If, however, an institution were to focus more on admitting early-decision or early-action applications, another factor that influences degree attainment, they might increase graduation rates, but at a cost of having diminished social and economic diversity in the student body, as early applicants can be less dependent on financial aid. 23 As with any factor that influences graduation rates, such as academic preparation, the mission of the institution must be considered before changing any admission criteria. We also know that intending to work full time while in college has a negative impact on graduation, yet many institutions have such students as a major component of whom they serve. In our quest for higher graduation rates, we cannot neglect important segments of the student population. In order to actually improve degree completion rates at an institution, as well as the state and national level, the focus needs to be squarely on creating conditions for success for all students who begin college. At this crucial juncture, when the need for an educated citizenry is even more important, we must focus on improving the success of every student. Student and institutional characteristics as predictors of retention Up to this point, we have discussed how important it is to take into account the kinds of students that institutions enroll in order to understand whether they are doing better or worse than expected in advancing them to graduation. In this section, we refine the model in the previous study 24 to include student-level characteristics, peer norms based on full cohorts of entering freshmen at each institution, and also institutional characteristics. We first conducted an analysis of students who stayed at the institution and graduated within six years, compared with all students who left the institution ( stayer/leaver model). We extend our previous discussion to six years because more students are likely to complete in this time frame, and it gives public colleges that enroll large numbers of students from different socioeconomic groups more of an opportunity to work with students toward a degree completion. Drawing from the literature, we reduced the model from the previous report on degree attainment 25 to select the strongest measures that 12

13 have proven to be significant and that have theoretical justification according to recent modifications of the Tinto model that include student ability to finance college, push/pull factors associated with work and family, and predisposition for college involvement. 26 Most specifically, we utilized measures that have previously proven to have impact on national assessments of degree attainment for students who completed at the same college they entered as freshmen, and that reflect the notion that students arrive with cultural, social and academic capital that positions them well for retention and completion. 27 Peer norms are also influential in a variety of college outcomes, including retention. 28 Institutions are also stratified and constrained by resources that they can devote to each student in fulfilling their education mission, and such institutional finance effects have been identified in multilevel models of degree attainment. 29 Four models were employed to depict the following: who the students are that attend a particular institution in Model 1 (using the students entering characteristics from the Freshman Survey or the Level 1 in a hierarchical generalized linear model [HGLM] 30 ); adding information about the kinds of students each are educated with or peer norms in Model 2 (the students characteristics and select aggregate characteristics of their peers from the Freshman Survey or Level 2 HGLM model); adding information about the types of colleges they attend in Model 3 (the students, peer norms and select institutional characteristics in Level 2); and finally adding institutional resources in Model 4 to the Level 2 model. Multiple imputations were conducted to prevent unnecessary loss of cases, resulting in a sample of 194, 242 cases across 357 institutions for the initial model. HGLM does not produce statistics for model quality. Instead, we examined changes in Nagelkerke R-square for the models in a logistic regression prior to assessing HGLM results. A full model that perfectly predicts the outcome has a Nagelkerke likelihood of 1. Almost no model predicts that well, but results give some sense of model quality in that individual student characteristics improve over the baseline model probability of degree attainment by one quarter, and that the characteristics of their peers that shape the educational environment improves another 7 percent. Institutional characteristics (e,g., type, size, selectivity, control) and resources (tuition as a percent of total revenue and core expenditures per FTE) account for an additional 5 percent. Because not all institutions have the same resources, and low-income students often attend the lowest-resourced institutions, 31 we still contend that institutional characteristics and resources are important factors to consider in evaluating an institution s retention rate. Table 1. Pseudo R-Square Changes for Models 1-4 in Predicting Degree Attainment at the 2004 College Models Nagelkerke R-Square Student characteristics

14 Student characteristics + peer norms.258 Student characteristics + peer norms + institutional characteristics Student characteristics + peer norms + institutional characteristics + resources A second indicator of model quality is stability of the coefficients in the model, particularly in devising predicted rates for different populations and institutions. Building from the Completing College report six-year degree attainment models that included predicted rates using a large number of variables with only CIRP TFS data as input, we checked the four models prediction rates to understand sensitivity to populations from different institution types (across samples) as a first test of stability of the model across samples, and the sensitivity of the predicted rates when specific measures were excluded and new variables included (alternative models). Ideally, additional analyses would take place on different populations in the future to ensure accuracy and fairness in devising predicted rates for institutional goals. Table 2 shows the six-year predicted rates across model changes and across institution types. The reduced student input model (Model 1 in this analysis) shows the most differences from the larger model of covariates used in the report. Substantial differences are reduced, however, when institutional characteristics are taken into account (Model 3) and are even further reduced by the final model that contains institutional resources (Model 4). This indicates that one can choose to use a large number of student input factors (covariates from survey data) or employ a reduced model of student inputs supplemented by institutional characteristics and resources that are publicly available. We will return to the issue of easily obtaining some of the most salient predictors of retention for constructing prediction rates after reviewing results. Table 2. Model Changes and Predicted Degree Attainment Rates Across Types Institutions Report* Model 1 Diff. Model 2 Diff. Model 3 Diff. Model 4 Diff. Public Univ 68.9% Private Univ 82.4% Public 4yr 49.6% Private 4yr 65.7% Catholic 4yr 69.2% Oth Rel. 4yr 59.7% All 64.5% *DeAngelo, L., Franke, R., Hurtado, S., Pryor, J. H., & Tran, S. Completing college: Assessing graduation rates at four-year institutions. Los Angeles: Higher Education Research Institute, UCLA (2011). Results from the stayers/leavers model 14

15 Detailed log-odds and change in probabilities based on the measures in the study are available in the tables in the appendix. Because log-odds and change in probabilities are specific to the measurement categories of each particular variable, we focus here only on t-values as a standard way to identify the key predictors and generally summarize results. By far the most important predictor of retention and degree completion in six years at the institution of initial entry is student s self-reported high school GPA ( t =34.02). This is also substantiated in a recent study that indicates high school GPA is the strongest predictor of degree completion at public institutions of varying selectivity. 32 Its effect is also significantly stronger than SAT and converted ACT composite scores (t= 8.738). The next most positive predictors of graduation from the same institution are student-reported hours per week studying or doing homework as a high school senior (t= 9.68), which is a measure of student effort. The number of years of high school math is also an important indicator of degree attainment at the initial college (t= 7.90). The latter, however, is not as significant a predictor as the level of father s education (t=8.886). Most of these significant factors can be gleaned from a student s application if we wish to use these factors to adjust retention rates. That is, institutions have much more work to do when they admit students who have lower academic credentials (high school GPAs, test scores, years of mathematics), or who spend less time studying, or have lower levels of parental education. However, a strong set of survey measures emerge as important predictors that are taken primarily from the CIRP entering-student survey, and would not be available in the application process or readily collected by means other than a large-scale survey. Students who indicated at orientation (when the survey is administered) that they expected to transfer were significantly less likely to graduate from the same institution they entered as first-year students (t= ). In fact, it is the next highest predictor after high school GPA, taking into account all other student and institutional characteristics. This suggests that students enter colleges with significantly different levels of commitment, a factor that has long been hypothesized to play a role in student departure. 33 Even though the population is largely traditionally aged college students (98 percent started college immediately after high school) 34, the reported number of hours per week in household or child care duties (t= -9.64) and expecting to work full time while attending college t= -8.99) is associated with lower probabilities of attaining a degree in six years. If students reported that they frequently socialized with others from another racial/ethnic group (a proxy for coming from a high school or neighborhood with high diversity), they were less likely to finish at the same college in six years (t= ). A pattern associated with self-perceived emotional health was also evident. Reports of frequently feeling depressed (t= -8.14) or coming late to class (t= -6.09) were negative predictors, while higher self-ratings on emotional health were a positive predictor of retention to graduation (t= 5.070). Positive predictors also involve information about students predispositions and choices from the CIRP Freshman Survey. Students who considered the cost of the college before choosing (t= 8.18), planned to live on campus in the first year (t= 7.56), expected to change career goals (t= 6.44), expected to participate in student clubs/groups (t= 5.55) or engage in volunteer/community service (t= 5.32), or had higher self-ratings on drive to achieve (t= 5.07) were more likely to be retained to graduation. Some of the latter activities are 15

16 actually used in college admissions processes that consider both academic and personal accomplishment criteria in selection but are rarely quantified in institutional analyses of degree attainment. Clearly, it is important to consider a wider array of factors to help determine the probabilities of degree attainment. Financial indicators based on resources to pay for college in the first year were influential factors in predicting student retention at the same college. Students who reported that they took out large loans ($10,000 or more) were roughly 21 percent less likely to complete their studies at the same college than students who had not taken out any loans (t= ). The probabilities of completing at the same college increased as the amount of loans students assumed in the first year declined: Students who took out $6,000 9,999 in loans were 16 percent less likely, $3,000 5,999 were 13 percent less likely, and $3,000 (or less) were 8 percent less likely to finish at the same college than students who reported that they took out no loans. Further, students with parental incomes below $29,999 a year (the lowest income category) were 15 percent less likely to finish at the same college in six years than middle-income students, and were less likely to finish than students in any other income group. Conversely, students who relied primarily on family resources of more than $10,000 in the first year were nearly 18 percent more likely to finish college at the same institution than students who did not rely on family resources. How might this be factored into an institution s retention rate for adjustment? Institutions that attract a higher proportion of low-income students should be given credit for their role in providing opportunity to these students by using an adjusted graduation rate. Conversely, those institutions that enroll more students with ability to pay from family resources should be expected to graduate these students. However, regardless of family income, higher reliance on loans is not fair to the student given the lower probability of degree completion at their first college. This latter result is partially in the control of the institution and may not be part of the adjusted graduation rate, but affordability is a factor of both student and institutional resources: Students often take out larger loans because the institution cannot provide more direct aid or tuition discounts. It is important to note that some differences in terms of race/ethnicity were evident while others were explained by other factors in the model. African American and Latina/o students were not likely to have lower six-year completion rates than white students once other student characteristics were taken into account. However, later we present results from another analysis (completers vs. leavers) that show both African American and Latina/o students are at greater risk for leaving higher education altogether. The stayer model could not account for lower completion rates for multiracial students and American Indian students compared with white students. Nor could the model explain the relatively higher completion rates of Asian Americans compared with white and other students (see models in the appendix). This suggests that some adjustment on graduation rates should be made for the racial composition of the entering class of students. Further research will be conducted to determine factors that are important to specific student populations. 16

17 Several institutional factors are important to note in terms of their relationship to degree attainment. Model 2 revealed several peer norms that were not evident in subsequent models (3 and 4) after additional institutional controls were added, since they share variance with other institutional measures. The peer group income level (aggregated income of the first-year cohort) and the peer group s expectation to be involved in college were both positively associated with individual student retention to the sixth year. Conversely, the cohort s inclination to transfer adversely affected an individual student s retention to graduation, regardless of the individual s initial expectation to transfer. This effect remained significant in the final model (t= ), and it is a peer norm effect that has been noted in other studies using registrars data and CIRP Freshman Survey data. 35 This is an indication that some institutions have higher mobility rates than can be detected from students initial intentions. The percentage of part-time students relative to the undergraduate population also has a significant negative effect on individual retention at the institution (t= -3.57). Having a high proportion of part-time students accommodates students who need to work but may also contribute to student mobility. Institutional resources also have an independent effect on the dependent variable: Total (core) expenditures per FTE is a positive predictor of graduation from the same institution in six years (t= 3.82). Selectivity of the institution was not nearly as significant independent of other institutional factors in the HGLM model, although selectivity is highly correlated with institutional resources, and it is well known that students attending the most selective colleges are likely to finish their degrees. 36 This suggests that independent of student characteristics, institutions with more part-time students and/or those with lower resources per undergraduate student have distinct results that ought to be taken into account in formulating expectations for institutional graduation rates. Using multiple sources of data for constructing predicted rates We make extensive use of the CIRP Freshman Survey of 2004, a year in which 720 institutions administered the survey to entering students. A smaller sample of institutions (440), however, made the national norms based on surveying between 75 and 85 percent of their entering students, so not all freshmen will have taken the survey on a campus without concerted effort. Some measures can be obtained from other data sources. Table 3 table is ordered by significant variables in the analysis indicating potential sources of data, the nature of the survey question on the TFS, and its relative strength in predicting graduation from the same institution (t-value). For example, high school grades can be obtained from students transcripts, although electronic transcripts are still not widely available for use in admissions. The University of California system, which receives over 100,000 applications a year, relies on student self-reported grades in specific high school courses on the admissions application, and subsequently conducts random spot checks of transcripts for a select sample of students. The transcripts are received well after the admissions process is completed. Admissions applications are now electronic and so much more information is now available for analysis and 17