A Research Note. Norman B. Bryan Jr., Ph.D. Director of Institutional Research and Assessment Presbyterian College

Transcription

1 Looking Ahead to NSSE 2.0 By Knowing What to Keep from NSSE 1.1: Construct Validity and Invariance Properties of NSSE Benchmarks Across Institutional Types A Research Note Norman B. Bryan Jr., Ph.D. Director of Institutional Research and Assessment Presbyterian College Bruce W. Eagle, Ph.D. Professor of Management St. Cloud State University Bonnie M. Wright, Ph.D. Associate Vice President for Planning and Assessment Limestone College Marjorie L. Icenogle, Ph.D. Professor of Management University of South Alabama Presentation to the 2012 AIR Forum Association for Institutional Research New Orleans, Louisiana

2 Looking Ahead to NSSE 2.0 By Knowing What to Keep from NSSE 1.1: Construct Validity and Invariance Properties of NSSE Benchmarks Across Institutional Types A Research Note Abstract NSSE. Love it? Hate it? Puzzled by its recent attention? Happy it is criticized? There is much to like and dislike about NSSE, in addition to concern that expensive benchmark data will be obsolete given NSSE 2.0. This paper takes a critical look at the currently conceived NSSE benchmarks among two higher education types (liberal arts college and regional research university) and reveals benchmarks, and their respective items, that are adequately justified for use in institutional evaluation and change processes (Banta, Pike, & Hansen, 2009) and inclusion for NSSE 2.0. Via findings from structural equation methodology (SEM), three of the present NSSE benchmark s (level of academic challenge, studentfaculty interaction, and supportive campus environment) construct validity and invariance properties support continued use, however, two of the three remaining benchmarks (level of academic challenge and supportive campus environment) require revision. Introduction Received wisdom asserts that improvements in higher education are partly the result of institutional commitment to the assessment of student experiences, the plausibility of said wisdom or the discounting of external entity influence (e.g., accrediting body) notwithstanding, and further asserts that as institutions engage assessment efforts primarily for internal and moral-ethical reasons, institutions will more strongly demand instrumentation having psychometric properties attesting to the underlying educational theory on which it is based. Said wisdom is important for at least two reasons: 1) institutional capacity to use survey findings to describe, explain, and predict institutional and academic performance and make institutional adjustments relative to such, and 2) assessment offices ability to justify the cost of very expensive assessment instrumentation. Among the tripartite concern of higher education, access, affordability, and accountability, U.S News and World Report has had a pretty long run with accountability; but as a measure of institutional quality and effectiveness, although not without some degree of merit, surely the USNWR survey was not destined to be best that can be said of quality concerning higher education institutions. Into the milieu of measuring institutional quality and emerging from an impressive array of prior theory and research on student learning, arguably a better tool was made available NSSE and thus the accountability movement, with the backing of the Pew Foundation, provided a voice that was lacking: the student and her or his collegiate experience.

3 The hallmark of the National Survey of Student Engagement (NSSE) is the educational benchmark, which is central to engagement theory (Kuh, 2000). NSSE research proposes five benchmarks believed to influence important academic outcomes, but extant research, at best, supports only a moderate degree of efficacy for the benchmarks (Carini, Kuh, & Klein, 2006), thus raising doubts concerning the robustness of engagement theory, despite its historical and theoretical lineage. However, before such an assessment is accepted, it behooves educational researchers to test the theory with psychometricallysound constructs (Porter, 2011). Stated differently, engagement theory is well-developed, seems highly plausible, and has much intuitive appeal, yet reported efforts using NSSE to test relationships suggested by the theory are inadequate to effectively assess the theory s viability. Review of Relevant Research NSSE, subjected to numerous conceptual and statistical analyses (and much of it by NSSE-affiliated researchers [Olivas, 2011]), has not received much attention, until most recently, concerning benchmark construct validity and has received no reported attention concerning invariance properties. A question that lingers concerning NSSE benchmarks is whether or not they are valid when the object of interest is a single institution, that is, would extant findings at a higher level of analysis (e.g., institutional) be replicated at a lower level of analysis (e.g., individual)? An SEM study on deep learning constructs (Laird, Shoup, & Kuh, 2008) suggests such is the case, but recent research having this as a guiding question suggests that the benchmarks are questionable (e.g., Campbell & Cabrera, 2011) ; therefore, it is not surprising that numerous hypothesized academic outcomes are not as strongly supported as supposed (e.g., Carini, Kuh, & Klein, 2006). As Gordon, Ludlum, and Hoey (2009) likely would suggest, building and testing academic models before construct validity is assured can be highly problematic. That is, studies addressing relationships among various phenomena need to first assure that the phenomena under consideration are adequately defined, otherwise relationships, notwithstanding an array of statistical measures, actually may be something entirely different than what is supposed. Since 2000, when NSSE was widely available to institutions of higher education, and marketing of it largely based on its five benchmarks, little peer-reviewed research has been published concerning the construct validity of the instrument, whereas numerous studies founded on exploratory factor analyses are reported at the NSSE website. However, a wealth of critical information, from conceptual to empirical studies, was recently published in a 2011 edition of The Review of Higher Education. Whereas the adequacy of construct domain (Dowd, Sawatzy, & Korn, 2011; Nora, Crisp, & Matthews, 2011) and student ability to accurately respond to NSSE/CCSSE-type questions (Porter, 2011) are important considerations, this paper strictly concerns testing the adequacy of the currently conceived benchmarks, thus more central to this research is the work of Campbell and Cabrera (2011), which calls into question the NSSE benchmarks. The major differences being Campbell and Cabrera focused on a public, 2

4 research-extensive university instead of either a liberal arts or regional institution of higher education and their research focused only on non-transfer seniors, whereas this research concerns transfer and nontransfer freshman and seniors. However, the concern of Campbell and Cabrera is the same -- to what extent do NSSE benchmarks hold for the single institution with of exception: are the benchmarks invariant? Conducting research on NSSE findings of non-transfer seniors enrolled in a large public, researchextensive university, Campbell and Cabrera (2011) found the 42 manifest items and five benchmarks to reveal an insufficient fit (NNFI =.795 and CFI =.807) for its interdependent model conceptualization. Subsequent analyses indicated poor item fit and an unsuitable construct: enriching educational experiences. The researchers conclude that as a measure of effectiveness, NSSE is limited for researchextensive universities and then suggest, based on extant research the same may be said of NSSE in liberal arts institutions. Data and Methodology To help address current concerns in the NSSE literature, three liberal arts and a regional university with some graduate co-existence NSSE benchmark data are analyzed via SEM. Data The first private liberal arts college (PLA 1) analyzed serves as the foundation for this study. In order to generate a sufficient sample, NSSE 2005 and 2006 results were combined, which provides a usable sample size of 575, with 61% of the sample being female. The institution, at the time, did not have a graduate program, and the average age of participating students is 20 years old. The second private liberal arts institution (PLA 2), much like the first, has a useable sample size of 318 and an average respondent age of 20 years old and female representation is 65%. The 2009 NSSE survey is analyzed. The third private liberal arts college (PLA 3) has a large returning adult population, and its average participant age is 31 years old and female representation is 78%. The 2009 NSSE survey provides a useable sample of 342. The public, regional university s (PRU 1) usable sample size is 1220 for its 2007 NSSE survey, to which females represent 62% of the sample and the average age of the respondents is 22 years old. For all institutions, freshmen and seniors are included. Methodology The NSSE data are analyzed by structural equation methodology (i.e., confirmatory factor analysis [CFA]), which is a very sensitive, thus, conservative, statistical technique that typically follows 3

5 exploratory factor analysis in the determination of construct validity. Data were preprocessed in PRELIS before the covariance matrices were passed to LISREL for analyses. For Liberal Arts College 1 and Public University with Some Graduate Co-existence 1, additional invariance studies were conducted on the surviving and re-fitted benchmarks. For all institutions, the hypothesized five benchmark model was tested, as were two alternative benchmark models (see Table 1). Moreover, for all tests, a marker variable for each construct was set to 1 and the constructs were free to correlate. Figure 1, with two latent constructs, is an example of the CFA set-up used for this study. Insert Figure 1 and Table 1 About Here Findings and Discussion Tables 2 and 3 compare the three private liberal arts colleges and the public university on various fit measures and item properties. For all institutions, data measuring the hypothesized 42-item, 5- benchmark model, like previous research (Campbell & Cabrera, 2011), does not reveal an adequate fit of data with the conceptualized model. Of the very similar, yet poor fitting models, the two best were for the first private liberal arts college and the public regional university (NNFI:, ; CFI:, ; AGFI:.76,.79, and RMSEA:.073,.073, respectively). Item analysis on the first liberal arts college s SEM findings revealed very weak manifest indicators for active collaborative learning and enriching educational experiences, thus the items related to these benchmarks were dropped from additional analyses and the revised models for all schools are based on this assessment. Thus analyses proceeded with three of the hypothesized indicators (level of academic challenge, student-faculty interaction, and supportive campus environment), and subsequent model fitting produced two revised models (see Table 1), with the difference among the two models being that indicator envsurpt (on the supportive campus environment construct) is not included in the second revised model. Insert Tables 2 and 3 About Here For the first revised model, four items were retained for level of academic challenge (analyze, synthesz, evaluate, and applying), five for student-faculty interaction (facgrade, facplans, facideas, facfeed, and facother), and three for supportive campus environment (envsuprt, envnacad, and envsocal). The data for all schools fit the revised model very well (NNFI range: , CFI range: , AGFI range: 4

6 , and RMSEA range: ), however, the third liberal art college demonstrates less fit than the other institutions (i.e., AGFI =.88, and RMSEA =.082). Further model specification resulted in a second revised model in which the envsurpt indicator was omitted from analyses. The omission significantly improved the error rate for the third liberal art college (RMSEA.082 to.070), and chi-square difference tests on the nested models reveals that the second revised model is significantly better at the p <.05 level for all schools. Of concern in previous research (Campbell and Cabrera, 2011; LaNasa, Cabrera, & Trangsrud, 2009) were the benchmark correlations, but given the active and collaborative learning and enriching educational experiences were omitted from the revised models, correlations among the three benchmarks were not problematic. However, of the interrelationships, the correlations among level of active challenge and student-faculty interaction were the highest, but not sufficiently high to cause concern. In testing the first revised model, for the first liberal art college and the public regional university, the correlations between level of academic challenge and student-faculty interactions were.61 and.54, respectively. Table 3 indicates that the pattern of squared multiple correlations, measures of strength of the items to their respective theoretical constructs, is fairly consistent across all schools. For the level of academic challenge, three of the four schools have identical rankings for the first and second ranked items among the four indicators and three of the four schools have identical rankings for the third and fourth ranked indicator. For supportive campus environment, three of the four schools item rankings are identical, but for all four, the same item is ranked the highest. For student-faculty interaction, the SMC rankings are a bit more complex, but the same general pattern is operative. Three of the four schools patterns are very close, but the fourth school, albeit a bit different, ranks most of its highest items with those of the other three schools. Thus, for all schools there is a good bit of consistency exists in the dynamics of the underlying benchmarks. Yet a more sensitive analysis is available to address an instrument s ability to consistently measure different populations: invariance. Two sets of invariance analyses were conducted, one on the first liberal arts college and the second on the regional university. These institutions were chosen largely due to the number of participants involved; that is, the sample for each institution was adequate for conducting multi-sample analyses. For each set of invariance tests for the two schools, the samples were determined by sex, then by class. Table 4 presents the findings of the invariant tests. Invariant tests are designed to address the question: does an instrument measure different populations similarly? Typically these tests begin with testing for the equality of configural equivalence and proceed 5

7 to equivalence among the instrument s various structure, error, and variances and covariances. The more restrictions one places on the instrument in which model fit remains good, the greater the claim that an instrument is invariant. Insert Table 4 About Here For both the liberal art college and the regional university, a greater claim to configural invariance across sex many be made than one for said invariance across class. Yet for class invariance, the liberal art college seems to have a small advantage over the regional university. However, for both schools for both sex and class, the data do not support invariant factor loadings. At best, these data reveal that the invariant nature of either revised model of NSSE, while present, is weakly present. Conclusion Arguably, NSSE remains the best alternative to the college rankings provided by U.S. News and World Report, and its role in facilitating beneficial campus-wide discussions on student engagement and learning cannot be denied, notwithstanding its construct validity problems. NSSE is but the next step, which emerged from many prior steps, and in this paper s view a worthy step forward, in measuring processes that promote student learning. But the theory upon which it is based deserves a more adequate test of its viability than the present benchmarks permit; that is, the current benchmark configuration of NSSE, at the individual level for single institutions, does not adequately permit such a determination with a high degree of confidence. Whereas the NSSE benchmarks may have construct validity at a higher level of analysis, it is imperative that such be present at the single institution at the individual level, as well, as offices of institutional research and assessment are increasingly expected to justify the expense of nationally-based instruments that are used as a basis for organizational change processes, but more importantly, to only use instruments that asses student learning that reflect the best scholarship possible. That stated, even with is current problems, present in NSSE, with some modification, are benchmarks with adequate construct validity and a minimal, yet acceptable, degree of configural invariance across sex, namely level of academic challenge, student-faculty interaction, and supportive campus environment. These findings are consistent across two higher education institutional types: private liberal arts college and public regional university. 6

8 References Banta, T.W., Pike, G.R., & Hansen, M.J. (Spring 2009). The use of engagement data in accreditation, planning, and assessment. New Directions for Institutional Research, No. 141, pp Campbell, C.M., & Cabrera, A.F. (2011). How sound is NSSE? Investigating the psychometric properties of NSSE at a public, research-extensive institution. Review of Higher Education, 35(1), Carini, R.M., Kuh, G.D., & Klein, S.P. (2006). Student engagement and student learning: Testing the linkages. Research in Higher Education, 47, Dowd, A.C., Sawatzy, M., & Korn, R. (2011). Theoretical foundations and a research agenda to validate measures of intercultural effort. Review of Higher Education, 35(1), Gordon, J., Ludlum, J., & Hoey, J.J. (2008). Validating NSSE against student outcomes: Are they related? Research in Higher Education, 49, Laird, N.T.F., Shoup, R., & Kuh, G.D. (2008). Measuring deep approaches to learning using the National Survey of Student Engagement. Paper presented at the Annual Meeting of the Association of Institutional Research. Chicago, IL. LaNasa, S.M., Cabrera, A.F., & Trangsrud, H. (2009). The construct validity of student engagement: A confirmatory factor analysis approach. Research in Higher Education, 50, Nora, A., Crisp, G., & Matthews, C. (2011). A reconceptualization of CCSSE s benchmarks of student engagement. Review of Higher Education, 35(1), Olivas, M.A. (2011). If you build it, they will assess it (or an open letter to George Kuh, with love and respect). Review of Higher Education, 35(1), Kuh, G.D. (Spring 2009). The National Survey of Student Engagement: Conceptual and empirical foundations. New Directions for Institutional Research, No 141, pp Porter, S.R. (2011). Do college student surveys have any validity. Review of Higher Education, 35(1),

9 Figure 1 Sample CFA Programming Set-up θ δ1 Marker: x1 λ x11 = 1 θ δ2 x2 λ x21 ξ 1 θ δ3 x3 λ x31 θ δ4 Marker: x4 λ x42 = 1 φ 21 θ δ5 x5 λ x52 ξ 2 θ δ6 x6 λ x62 8

10 Table 1 NSSE Benchmark Models Model Hypothesized Model Indicators Revised Model 1 Indicators Benchmarks LAC ACL SFI EEE SCE workhard clquest facgrade itacadem envstu analyze clpresen facplans divrstud envfac synthesz classgrp facideas diffstu2 envadm evaluate occgrp facfeed intern04 envsuprt applying tutor facother volntr04 envnacad readasgn commproj resrch04 lrncom04 envsocal writemor oocideas forlng04 writemid stdabr04 writesml indstd04 acadpr01 snrx04 envschol cocurr01 envdivrs analyze facgrade envsuprt synthesz facplans envnacad evaluate facideas envsocal applying facfeed facother Revised analyze facgrade Model 2 synthesz facplans envnacad Indicators evaluate facideas envsocal applying facfeed facother Key: LAC = Level of Academic Challenge, ACL = Active Collaborative Learning, SFI = Student- Faculty Interaction, EEE = Enriching Educational Experiences, SCE = Supportive Campus Environment. 9

11 Table 2 Fit Indices of NSSE Benchmark Models Hypothesized Model (df = 809) Revised Model 1 (df = 51) Revised Model 2 (df = 41) School NNFI CFI AGFI RMSEA x 2 NNFI CFI AGFI RMSEA x 2 NNFI CFI AGFI RMSEA x 2 PLA , PLA , PLA , PRU ,

12 Table 3 Revised NSSE Models: Squared Multiple Correlation and Completely Standardized Solution SCHOOL PLA 1 LAC analyze synthesz evaluate applying SFI facgrade facplans facideas facfeed facother SCE envsurpt envnacad envsocal PLA 2 LAC analyze synthesz evaluate applying SFI facgrade facplans facideas facfeed facother SCE envsurpt envnacad envsocal Revised Model 1 Revised Model 2 Revised Model 1 Revised Model 2 SMC SC SMC SC SCHOOL SMC SC SMC SC PLA 3 LAC analyze synthesz evaluate applying SFI facgrade facplans facideas facfeed facother SCE envsurpt envnacad envsocal PRU 1 LAC analyze synthesz evaluate applying SFI facgrade facplans facideas facfeed facother SCE envsurpt envnacad envsocal

13 Table 4 NSEE Invariance Tests on Revised Models School Revised Model 1 Revised Model 2 & Test NNFI CFI GFI RMSEA x 2 df NNFI CFI GFI RMSEA x 2 df PLA 1 Sex. Factors Factor Loadings PLA 1 Class Factors Factor Loadings PRU 1 Sex Factors Factor Loadings PRU 1 Class Factors Factor Loadings