Longitudinal Research 1 Why Longitudinal Research is Critical in Studying College Impact: An Example of Between-College Effects Tricia Seifert a Kathleen Goodman a Sherri Edvalson a Jennifer Laskowski a Ernest Pascarella a Charlie Blaich b a The University of Iowa b Center of Inquiry in the Liberal Arts at Wabash College Please direct questions regarding this paper to: Tricia Seifert N491 Lindquist Center Iowa City, IA 52242 Tricia-Seifert@uiowa.edu This research is conducted through the generous support of the Center of Inquiry in the Liberal Arts at Wabash College.
Longitudinal Research 2 ABSTRACT College impact studies are one of the most frequent ways that researchers assess how colleges add value to student learning. The present study provides an example of how crosssectional design (the most frequent research design used in college impact studies) can lead to undetectable selection bias, which may confound inferences about college impact. Therefore, we argue for the importance of longitudinal panel (i.e., pretest-posttest) design as the preferred approach to best ascertain the value that the college experience adds to student learning.
Longitudinal Research 3
Longitudinal Research 4 Why Longitudinal Research is Critical in Studying College Impact: An Example of Between-College Effects Stakeholders are increasingly calling for the higher education community to be held accountable for student learning (U.S. Department of Education, 2006). The increased call for accountability provides an opportunity for educational research to step to the fore in aiding datadriven decision making. College impact studies are one of the most frequent ways that researchers assess how colleges add value to student learning. The present study provides an example of how cross-sectional design (the most frequent research design used in college impact studies) can lead to undetectable selection bias, which may confound inferences about college impact. Selection bias occurs when students across groups differ significantly at the time of matriculation on variables of interest. Although many cross-sectional studies attempt to control for confounding influences, without a baseline of where the student started before entering college (i.e., collecting a pretest score on the outcome of interest), it is difficult to accurately estimate the impact of college experience on the student s learning. Therefore, we argue for the importance of longitudinal panel (i.e., pretest-posttest) design as the preferred approach to best ascertain the value that the college experience adds to student learning. From past research, we provide a theoretical rationale for different types of educational research designs. We then describe the methods employed in the present study and discuss the results from our analyses. We conclude by detailing the implications of our findings for the design of educational research, particularly that which seeks to answer the stakeholders call of accountability by demonstrating the impact of college on student learning. Educational Research Design
Longitudinal Research 5 Two basic research designs exist for measuring college impact: cross-sectional and longitudinal panel studies. Cross-sectional studies collect data at one point in time from a group that can range in age, year in school, or type of institution attended. For example, in estimating the impact of attending a liberal arts college (versus another type of institution) on student learning, researchers refer to the added value of attending a liberal arts college compared to other institutional types as the difference on outcomes of interest between liberal arts college students and their peers attending other types of institutions. In estimating the impact of college on student learning, cross-sectional designs attempt to statistically control for all confounding influences that may affect the relationship between the independent variable and learning measure under examination. For example, intellectual ability from high school (e.g., measured by high school grades or a score on an objective instrument like the ACT or SAT) is often used as the covariate from which a baseline of student learning is estimated. Although the one-time nature of the data collection eliminates the possibility of sample attrition (Gall, Gall, & Borg, 2003), which is a clear benefit, the challenge with cross-sectional design is that the covariates selected are usually convenient, but imperfect and imprecise proxies for an actual baseline pretest (Pascarella, 2006). The result from this imprecision is a less accurate estimate of the relationship under investigation. Unlike cross-sectional design, longitudinal panel study design allows educational researchers the advantage of surveying the same sample of students at various times, as opposed to surveying certain students at one point and following up with different students at another (Gall, Gall, & Borg, 2003). The principal benefit of panel study design is that it enables researchers to better understand subjects pretest characteristics from the outset, allowing for more accurate statistical control throughout the duration of the study (Gall, Gall, & Borg). In
Longitudinal Research 6 college impact research, panel designs allow researchers to completely take into account where students started in measuring the impact (or the value-added) of college on their learning. Pascarella (2006) claims that gathering data in longitudinal pretest-posttest studies is more beneficial than statistical manipulation designed to render meaningful results from crosssectional data. Astin s (1993, 2003) input-environmental-outcome model is also based on this notion that longitudinal study design offers stronger and more reliable data than cross-sectional design, assuming that the inputs include pretest measures of outcomes. Although panel studies come with their own set of problems, like repeated measurement, subject drop-out, and the possibility of sample bias, they enable researchers to collect data with more internally valid results than any other research design, with the exception of randomized experiments (Gall, Gall, & Borg, 2003; Pascarella, 2006). Pretest-posttest design ultimately permits researchers to concentrate on observing and identifying the changes that occur in each subject in a more reliable manner than any other design, allowing researchers to make stronger inferences from the sample to the target population (Yee & Neimeier, 1996). By taking into account students precollege characteristics and pretest measures of desired educational outcomes, panel study design better equips higher education researchers to estimate the genuine impact of college on students (Pascarella & Terenzini, 1991, 2005). Longitudinal panel studies allow researchers to make more reliable observations of change in students and thus, more accurate estimates of the impact of college on student learning than cross-sectional design. The current study demonstrates the disadvantages of cross-sectional design and advantages of longitudinal panel design, first by assessing differences in student background characteristics across institutions and then by assessing differences in pretest
Longitudinal Research 7 measures of educational outcomes across institutions, while controlling for student background characteristics. The questions guiding our study are: Do student background characteristics such as gender, race/ethnicity, measures of socioeconomic status, and high school ability differ by institutional type? Do pretest measures of educational outcomes differ by institutional type, when statistically controlling for these student background characteristics? If differences exist between institutional types on the pretest measures of outcomes, controlling for student background characteristics, it indicates that the often used covariates of gender, race/ethnicity, and measures of high school academic ability are not sufficient proxies for a true baseline pretest. Any differences by institutional type on the pretest measures, controlling for student background characteristics, would support our assertion that it is critical to have both pretest and posttest measures on outcomes of interest in estimating the net impact of college on student learning. Methods The purpose of the Wabash National Study of Liberal Arts Education (WNSLAE study), which encompasses the present study, is to identify the institutional practices and conditions that promote the development of liberal arts outcomes, concurrently using both quantitative and qualitative methods. It seeks to discover the extent to which college experiences add value to students development. This focus calls for a quantitative research design that allows us to take into account students precollege characteristics, including individual background characteristics and pre-test measures of educational outcomes, in order to estimate the net effects of college on liberal arts outcomes. To this end, we employed a longitudinal pretest-posttest survey design.
Longitudinal Research 8 Because the purpose of the current analysis is to assess the value of the research design, we use our data to simulate cross-sectional analyses to determine whether students precollege characteristics and pretest scores on educational outcomes differ by institutional type. If the results of these cross-sectional analyses show significant differences across institutional types, they would demonstrate the importance of using the longitudinal pretest-posttest survey design. Sample Institutions. Because of the mixed method approach to the study, we faced competing sampling criteria. As a result, we purposefully chose a sample of 19 institutions, from the more than 60 that applied, based on their commitment to providing experiences believed to be associated with liberal arts education. In an effort to diversify our institutional sample, we then selected institutions representing a range of geographic regions, institutional types, institutional sizes and types of control, and student demographics. Individuals. In order to make the most of our longitudinal pretest-posttest design, we needed to ensure that we had enough students in our sample to complete the second and third waves of data collection. We used different sampling approaches within institutions to meet this need. Because the liberal arts colleges matriculate smaller populations, we could not randomly sample the same proportion of their students as those at the large institutions. At the liberal arts colleges, we surveyed the entire first-year cohort. At the larger institutions, we used a simple random sampling approach. However, at larger institutions in which students of color comprised less than ten percent of the first-year cohort, we oversampled students of color, in order to assure we had a large enough subsample of students of color for subsequent analyses. Each institution invited their students to participate in the WNSLAE study. Students were compensated $50 for each wave of outcomes assessments they completed.
Longitudinal Research 9 Data Collection We collected data at two times during fall 2006. Students who were interested in participating in the study completed either a paper or web-based registration form providing information on demographic characteristics and high school involvement. Those who completed the registration form were sent a letter and e-mail reminding them to attend an outcomes assessment session. At these sessions, we collected data on students orientations toward learning and career plans, and six outcomes associated with liberal arts education. 4,501 students completed the registration form and student survey of orientations toward learning and career. Based on a matrix sample, we randomly assigned students to an assessment group. Assessment Group A, consisting of 2,223 student completed the CAAP critical thinking test (ACT, 1990). Assessment Group B, consisting of 2,278 students, completed the Defining Issues Test-2 (DIT-2) (Rest, Narvaez, Thoma, & Bebeau, 1999). All students completed the other four assessments: Scales of Psychological Well-being (Ryff, 1989; Ryff & Keyes, 1995); the Miville-Guzman Universality Diversity Scale short form (Miville, Gelso, Pannu, Liu, Touradji, Holloway, & Fuertes, 1999); the Need for Cognition (Cacioppo, Petty, & Kao, 1984); and the Socially Responsible Leadership Scale revised version (Appel-Silbaugh, 2005; Dugan & Komives, 2006). Based on a review of the literature, we selected instruments we believed measured six outcomes theoretically and conceptually-related to liberal arts education (see King, Kendall-Brown, Lindsay, & VanHecke, 2007). These include: effective reasoning and problem solving; moral reasoning; well-being; intercultural effectiveness; inclination to inquire and lifelong learning; and leadership. Because of the variables included in the different models we estimated, the sample sizes for these analyses vary. We specify the N sizes of all models in Tables 1 and 2.
Longitudinal Research 10 Analyses We conducted two sets of analyses using logistic regression to estimate differences for dichotomous variables (Long, 1997) and Ordinary Least Squares (OLS) regression to estimate differences for continuous variables (Cohen, Cohen, West, & Aiken, 2003). In our first set of analyses, we assessed whether the background characteristics of students differed across liberal arts colleges, community colleges, regional universities, and research universities. In our second set of analyses, we assessed whether students pretest measures of educational outcomes differed across institutions, even when controlling for student background characteristics. Our intention was not to create a model predicting which students attend which institutions; we recognize that the reasons students enroll at specific institutions are based on many complex factors, including but not limited to self-selection, institutional recruitment, and enrollment management. We hypothesized that student background characteristics would differ across institutions for a variety of reasons. Our purpose was to determine whether the pretest measures on educational outcomes would differ across institutional types even after controlling for student background characteristics. If we found significant institutional type differences, this would provide empirical evidence demonstrating longitudinal panel designs are better suited to studying college impact than cross-sectional studies. Analyses of Student Background Characteristics To estimate differences related to demographic variables, we first regressed each student background characteristic variable on the institutional type variables, controlling for individual academic ability by including high school GPA and a measure of tested precollege academic ability based on ACT, SAT or COMPASS scores. The independent variables for all analyses
Longitudinal Research 11 were dichotomous variables representing community colleges, regional universities, and research universities. The reference group in all analyses was liberal arts colleges. Dependent variables. We used each student demographic characteristic as a dependent variable in separate regressions. The student background characteristics that we analyzed include: high school grade point average; high school achievement test (ACT/SAT/COMPASS); ethnicity/race (White vs. student of color); sex; high school involvement; and work for pay in high school. We also analyzed whether the student: has one or more dependent; has transfer credits; attended a predominantly White high school; is at his or her first choice college; has parents who have a four year degree or higher; or has parents whose annual income is greater than $200,000. Analyses of Pretest Measures of Educational Outcomes In our second set of analyses, we regressed each pretest measure of educational outcome on the institutional type variables, in two models. Model 1 included just the institutional types as independent variables dichotomous variables representing community colleges, regional universities, and research universities, with liberal arts colleges as the reference group in the regression equation. In Model 2, we added student demographic characteristics (as described in the Student Background Characteristic Analyses above) as controls in the regression. Dependent variables. The dependent variables in these analyses were pretests measures on outcomes often associated with college attendance (Pascarella & Terenzini, 1991, 2005). In these analyses our dependent variables included several scales related to the student s orientations toward careers and learning: importance of being politically and socially involved in the community; importance of achieving professional/career success; importance of contributing to science; importance of contributing to the arts; openness to diversity/challenge; academic
Longitudinal Research 12 motivation/learning for learning's sake; and positive attitude toward literacy. A single variables representing intention to earn more than a four-year degree was also included. Additionally, we analyzed six assessments representing liberal arts educational outcomes (King, Kendall Brown, Lindsay, & VanHecke, 2007): the Miville-Guzman Universality-Diversity scale; Need for Cognition scale; Socially Responsible Leadership Scale (release 2); the Scales of Psychological Well-being; the CAPP Critical Thinking score; and the Defining Issues Test (DIT-2) of moral reasoning and character. Results Differences in Background Characteristics Table 1 indicates the differences in background characteristics of students at community colleges, regional universities, and research universities, when compared to students at liberal arts colleges, in our sample. Students at community colleges, compared to students at liberal arts colleges, are more likely to be White, to have worked for pay in high school, to come from a predominantly White high school, and to have one or more dependent. They have lower high school GPAs, lower high school academic achievement test scores, and less involvement in high school than students at liberal arts colleges. They are less likely to have parents with four-year degrees or higher or parents with income greater than $200,000. They are more likely to have transferred credits to their current institution and it is less likely that they are at their first choice college. Students at regional universities, compared to students at liberal arts colleges, are more likely to have worked for pay in high school and to have lower high school GPAs and lower high school academic achievement test scores. Compared to students at liberal arts colleges, they are also more likely to be a student of color or female and to have transferred credits to their current
Longitudinal Research 13 institution. They are less likely to have a mother or father with a four-year college degree or higher and less likely to have parents with an annual income of $200,000. Additionally, they are likely to have come from a predominantly White high school. Students at research universities, compared to students at liberal arts colleges, are more likely to have been involved in high school, to be at the college of their first choice, to have a higher high school GPA, and a higher high school achievement test score. They are less likely to be White than students at liberal arts colleges and are more likely to have a father with a fouryear college degree or higher. These results demonstrate that there are differences between students that attend community colleges, regional universities, research universities, and liberal arts colleges. Student background characteristics such as demographics, high school involvement, and achievement, as well as parental education and income differ among students attending different institutional types for a variety of reasons related to college choice, recruitment, and enrollment management. This demonstrates the need to control for student background characteristics when estimating the effect of college on students. But is controlling for those characteristics sufficient? The results of our second analyses, described in the next section, suggest that it is not. Pretest Measures of Educational Outcomes Table 2 indicates that there are extensive differences in pretest measures of educational outcomes by institutional type, even after controlling for student background characteristics. For example, students at community colleges, regional institutions, and research universities tend to be less open to diversity and challenge than students at liberal arts colleges. Students at regional and research universities also show more interest in professional and career success and contributing to science than students at liberal arts colleges. Another example of differences
Longitudinal Research 14 across institutions is that community college students, when compared to liberal arts college students, score higher on the socially responsible leadership scale and several of the psychological well-being subscales. Students at community colleges and research universities scored higher on the CAAP Critical Thinking test, compared to students at liberal arts colleges, while students at community colleges and regional institutions scored lower than those at liberal arts colleges on the Defining Issues Test of moral reasoning and character. Only two outcomes showed no significant difference across any of the institutions. These included the self-acceptance subscale of the Scales of Psychological Well-being measure and the common purpose subscale of the Socially Responsible Leadership Scale. Overall, these results indicate that even when we control for student background characteristics, pretest measures of educational outcomes continue to differ by institutional type. Thus, even with an introduction of controls for student background characteristics, there remains a significant bias on pretest measures that could easily become part of differences by institutional type on a posttest measure of the outcome of interest. In order to accurately estimate the full impact of college on student learning, regression models need to include pretest measures of the outcomes of interest. Apart from random experiments, longitudinal panel studies are the only research design that allows for differences on pretest measures to be taken into account and thus, more accurately assesses the value added by college. Discussion The fact that we found institutional type differences on the pretest measures for all but two outcomes clearly demonstrates the distinct disadvantages of cross-sectional designs in accurately estimating the impact of institutional type on postsecondary student learning. Without the pretest to establish a student s level of learning at the time of matriculation, one may
Longitudinal Research 15 erroneously attribute differences on the outcomes, net of student background and demographic variables, to differential college impact by institutional type. Although it is quite common for cross-sectional studies of college impact to collect and statistically control for student background and demographic variables (e.g., sex, race, SES, ACT/SAT score), our analyses showed that even with such background and demographic influences controlled, statistically significant differences existed across institutional types on the pretest measure. The fact that cross-sectional designs do not include a pretest does not mean that unmeasured pretest bias will not be part of observed posttest differences. Thus, as illustrated in our example, the statistical controls employed in many cross-sectional designs would fail to adjust for significant selection bias across institutional types on the crucial covariate the pretest. Our results show a clear example of selection bias. In other words, based on the type of institution students attended, they differed in statistically significant ways on the baseline measure the pretest. The ultimate result of such significant selection bias on the pretest, for which cross-sectional designs cannot directly adjust, is the increased likelihood of a confounded estimate of the impact of institutional type on the outcome/posttest. This confounded estimate would likely result in a researcher erroneously reporting a differential impact (or value-added) of college by institutional type. The bottom line is that, absent the possibility of randomized experiments, only longitudinal panel designs permit the direct statistical adjustment for selection bias. Given external stakeholders increased focus on institutional accountability of student learning (Miller, 2007; Hersh, 2007; U.S. Department of Education, 2006), educational researchers have an opportunity to aid institutions and postsecondary education, in general, by conducting research in a manner that best facilitates data-driven decision making. Along with
Longitudinal Research 16 others (Pascarella, 2006), we recognize that longitudinal panel studies are not only time consuming but costly and difficult to conduct. However, in an accountability era for demonstrating the value that colleges add to student learning, we assert that longitudinal panel studies provide the most internally valid results and the most accurate estimate of the impact of college. There is no substitution for the gold standard that longitudinal pretest-posttest panel studies provide in assessing how college impacts students.
Longitudinal Research 17 References American College Testing Program (ACT). (1990). Report on the technical characteristics of CAAP; Pilot year 1: 1988-89. Iowa City, IA: Author. Appel-Silbaugh, C. (2005). SRLS Rev: The revision of SRLS. College Park, MD: National Clearinghouse for Leadership Programs. Astin, A.W. (1993). What matters in college: Four critical years revisited. San Francisco: Jossey-Bass. Astin, A.W. (2003). Studying how college affects students. About Campus, 8(3), 21-29. Cacioppo, J.T., Petty, R.E., & Kao, C.F. (1984). The efficient assessment of need for cognition. Journal of Personality Assessment, 48(3), 306-307. Cohen, J., Cohen, P., West, S.G., & Aiken, L.S. (2003). Applied multiple regression/correlation analysis for the behavioral sciences (3 rd ed.). Mahwah, NJ: Lawrence Erlbaum Associates. Dugan, J. P., Komives, S. R., & Associates. (2006). Multi-institutional study of leadership: A guidebook for participating campuses. College Park, MD: National Clearinghouse for Leadership Programs. Gall, M.D., Gall, J.P., & Borg, W.R. (2003). Educational research: An introduction (7 th edition). Boston, MA: Allyn and Bacon. Hersh, R. (2007). Going naked. [Electronic version] Peer Review, 9(2), 4-8. King, P. M., Kendall Brown, M., Linsay, N. K., & VanHecke, J. R. (2007). Liberal arts student learning outcomes: An integrated approach. About Campus, 12(4), 2-9.
Longitudinal Research 18 Long, J.S. (1997). Regression models for categorical and limited dependent variables. Thousand Oaks, CA: Sage Publications. Miller, R. (2007). Assessment in cycles of improvement: Faculty designs for essential learning outcomes. Washington, DC: Association of American Colleges and Universities. Miville, M.L., Gelso, C.J., Pannu, R., Liu, W., Touradji, P., Holloway, P., & Fuertes, J. (1999). Appreciating similarities and valuing differences: The Miville-Guzman Universality-Diversity scale. Journal of Counseling Psychology, 46(3), 291-307. Pascarella, E.T. (2006). How college affects students: Ten directions for future research. Journal of College Student Development, 47(5), 506-520. Pascarella, E.T. & Terenzini, P.T. (1991). How college affects students. San Francisco: Jossey-Bass. Pascarella, E. T. & Terenzini, P. T. (2005). How college affects students (Vol. 2): A third decade of research. San Francisco: Jossey-Bass. Rest, J. R., Narvaez, D., Thoma, S. J., & Bebeau, M. J. (1999). DIT2: Devising and testing a revised instrument of moral judgment. Journal of Educational Psychology, 91, 644-659. Ryff, C. D. (1989). Happiness is everything, or is it? Explorations on the meaning of psychological well-being. Journal of Personality and Social Psychology, 57, 1069-1081. Ryff, C. D., & Keyes, C. L. M. (1995). The structure of psychological well-being revisited. Journal of Personality and Social Psychology, 69, 719-727. U.S. Department of Education (2006). A test of leadership: Charting the future of U.S. higher education. [electronic version]. Accessed April 20, 2007 at http://www.ed.gov/about/bdscomm/list/hiedfuture/reports/final-report.pdf Yee, J. & Niemier, D. (1996). Advantages and disadvantages: Longitudinal vs. repeated
Longitudinal Research 19 cross-section surveys. Project Battelle 94(16). FHWA, HPM-40. [electronic version]. Accessed April 25, 2007 at http://ntl.bts.gov/lib/6000/6900/6910/bat.pdf
Longitudinal Research 20 Table 1: Significant Differences in Student Background Characteristics by Institutional Type, p <=.05 Community College Regional University Research University (Reference group = Liberal Arts Colleges) Background Characteristics, continuous N 1 Effect Size 2 High School GPA 4463-1.405-0.170 0.519 High School Ability (ACT/SAT/COMPASS) 4496-1.249-0.415 0.607 High School involvement 4427-0.652 0.188 High School work for pay 4427 0.439 0.124 Background Characteristics, dichotomous N Odds Ratio White (vs. student of color) 4463 7.303 0.787 0.500 Male (vs. female) 4463 0.794 No Transfer Credits 4463 0.574 0.796 At college of first choice 4463 0.712 1.438 Mom has a 4 yr degree or higher 4435 0.207 0.814 Dad has a 4 yr degree or higher 4388 0.363 0.826 1.198 Parent's Income Greater Than $200,000 4463 0.179 0.600 R has 1 or more dependent 4463 9.103 Predominantly White HS 4432 3.557 0.538 1 Sample size varies because of missing response items 2 Effect sizes represent a standard deviation change in Y (background characteristic) per 1 unit change in X (Institutional Type)
Longitudinal Research 21 Table 2: Significant Differences in Pre-test Measures of Educational Outcomes by Institutional Type, Controlling for Background Characteristics, p <=.05 Community College Regional University Research University (Reference group = Liberal Arts Colleges) Educational Outcome Pretest N 1 Effect Size 2 Intercultural Effectiveness: Openness to Diversity/Challenge Scale 4298-0.271-0.169-0.223 Miville-Guzman Universality-Diversity Scale- - Comfort with Diversity 4263 0.184-0.145 - Diversity Contacts 4263-0.170-0.166-0.182 - Relativistic Appreciation 4263-0.111-0.089 - Diverse Orientation - Total Score 4263-0.183-0.127 Leadership: Socially Responsible Leadership Scale (SRLS-R2)- - Consciousness of Self 4304 0.192 - Congruence 4302 0.198 - Commitment 4299 0.288 0.105 - Collaboration 4301 0.189 - Common Purpose 4300 - Controversy with Civility 4304-0.108-0.104 - Citizenship 4301-0.158 - Change 4302 0.158 Well Being: Ryff Scales of Psychological Well-being- -Autonomy 4295 0.183 - Environmental Mastery 4297 0.194 - Personal Growth 4296 0.155 - Self-Acceptance 4295 - Positive Relations 4296 0.202 - Purpose in Life 4297 0.203 0.089 Inclination to Inquire and Lifelong Learning: Need for Cognition Scale 4309-0.083-0.221 Positive Attitude Toward Literacy Scale 4295-0.237-0.289 Effective Reasoning and Problem Solving: CAAP Critical Thinking Scale 2135 0.270 0.116 Moral Reasoning and Character: Defining Issues Test (DIT-2) Postconventional Moral Reasoning 2086-0.209-0.116 Career Orientations: Importance of - -being politically/socially involved in comm 4306-0.170-0.141 -achieving professional/career success 4308 0.209 0.302 -contributing to science 4294 0.113 0.113 -contributing to the arts 4308-0.200-0.280 Academic Orientation: Academic Motivation Scale 4296-0.152-0.207 R intends to earn more than 4 yr degree 3 4311 0.249 1.890 1 Sample size varies due to missing response items and matrix sampling 2 Effect sizes represent a standard deviation change in Y (educational outcome pretest) per 1 unit change in X (Institutional Type) 3 Because this is a dichotomous variable, the statistics reported are Odds Ratios
Longitudinal Research 22