Contextualizing NSSE Effect Sizes: Empirical Analysis and Interpretation of Benchmark Comparisons

Transcription

1 Contextulizing NSSE Effect Sizes: Empiricl Anlysis nd Interprettion of Benchmrk Comprisons NSSE stff re frequently sked to help interpret effect sizes. Is.3 smll effect size? Is.5 relly lrge effect size? An effect size (ES) is ny mesure of the strength of reltionship between two vribles. In prctice ES sttistics re used to ssess comprisons involving correltions, percentges, men differences, probbilities, nd so on. In cses where lrge smple sizes mke it more likely tht difference even smll one will be sttisticlly significnt, ES sttistics re often thought of s mesure of prcticl significnce becuse they indicte the reltive mgnitude of the difference. Thus they re vluble in compring bstrct mesurement indices such s the NSSE benchmrks which re computed on 0 to 100 scle from sets of individul items using vrious response sets. NSSE s comprison reports use Cohen s d, the stndrdized difference between the institution s men nd the comprison group s men, clculted by dividing the men difference by the pooled stndrd devition. Thus, ES is discussed in this guide solely in terms of the Cohen s d sttistic. In his clssic book, Cohen (1988) reluctntly defined ES s "smll, d=.2," "medium, d =.5," nd "lrge, d =.8," preferring to be intentionlly vgue bout precise cut points nd decision rules. Cohen lso sid tht "there is certin risk inherent in offering conventionl opertionl definitions for those terms used in power nlysis in s diverse field of inquiry s behviorl science" (p. 25) nd urged reserchers to interpret ES bsed on the context of the dt. Nevertheless, Cohen s definition of smll, medium, nd lrge hs been widely ccepted nd incorported into mny socil science studies tht report ES. In the following sections, Cohen s cut points nd n empiriclly derived set of cut points re used to exmine the distribution of effect sizes from the NSSE 2007 Benchmrk Comprisons reports delivered to prticipting institutions (N=587). Following the nlysis, we offer recommendtions for interpreting the effect sizes of NSSE benchmrk comprisons. Frequency of Different Effect Sizes Bsed on Cohen s Generl Definition Tble 1 shows the percentges of 2007 institutions tht hd effect sizes within Cohen s cut point rnges on ech of the five NSSE benchmrks for first-yer (FY) nd senior (SR) students. Effect sizes in this tble re drwn from the individul institutions comprisons with the entire NSSE 2007 cohort. The tble shows tht the vst mjority of effect sizes on benchmrk reports were either trivil (less thn.20 in mgnitude) or smll (.20 to.49 in mgnitude). Very few institutions found medium or lrge effect sizes using Cohen s rule-ofthumb criteri. Tble 1 Distribution of NSSE Effect Sizes by Cohen s Generl Definition Effect Size Rnge Trivil (<.20) Smll (.20 to.49) Medium (.50 to.79) Lrge (.80 or greter) FY SR FY SR FY SR FY SR Level of Acdemic Chllenge 50% 62% 42% 30% 7% 7% 1% 1% Active & Collbortive Lerning 54% 56% 37% 36% 7% 7% 2% 1% Student Fculty Interction 60% 48% 34% 38% 6% 11% 1% 3% Enriching Eductionl Experiences 52% 40% 40% 37% 7% 15% 1% 8% Supportive Cmpus Environment 50% 46% 43% 44% 7% 9% 1% 1% Effect sizes were tken only from those NSSE 2007 institutions tht selected comprisons with the entire 2007 U.S. NSSE cohort (n=519). Becuse effects sizes cn be both positive nd negtive, bsolute vlues were used for the rnges. Contextulizing NSSE Effect Sizes. Pge 1 of 7

2 Effect Size Interprettion Bsed on NSSE Dt Cohen described smll effects s those tht re hrdly visible, medium effects s observble nd noticeble to the eye of the beholder, nd lrge effects s plinly evident or obvious. With respect to this rtionle, NSSE stff considered wys in which benchmrk differences would be observble in the dt, nd proposed scheme to interpret effect sizes bsed on the distribution of ctul benchmrk scores. To exmine this lterntive scheme, NSSE nlysts ssigned percentile rnkings to institutions 2007 benchmrk scores nd used them to model comprisons tht would resemble effect sizes of incresing mgnitude (illustrted in Figures 1 1d). To explin, suppose tht smll ES would resemble the difference between the benchmrk scores of students ttending institutions in the third qurtile (i.e., between the 50 th nd 75 th percentiles) nd those ttending institutions in the second qurtile (i.e., between the 25 th nd 50 th percentiles). These two sets of institutions re lbeled groups A nd B in Figure 1. Becuse groups A nd B re firly close within the distribution, the difference between the students ttending those institutions is expected to be smll. In the sme wy, medium ES (Figure 1b) would look like the difference between the benchmrk scores of students ttending institutions in the upper hlf (Group D) nd those ttending institutions in the lower hlf (Group C) of the distribution. A lrge ES (Figure 1c) would resemble the difference between students ttending institutions scoring in the top qurtile (Group F) nd those ttending institutions scoring in the bottom qurtile (Group E) of the distribution. And finlly, very lrge ES (Figure 1d) would be like the difference between students ttending institutions scoring in the lowest 10% (Group H) nd highest 10% (Group G) of the distribution. Figures 1 1d Illustrtion of Four Model Comprison Groups for Determining Empiriclly Bsed Effect Size Thresholds Bsed on the Distribution of NSSE Benchmrks 1. Smll Group A Group B 1b. Medium Group C 50% Group D 50% Distribution of Institutionl Benchmrks Distribution of Institutionl Benchmrks Smll ES ( xˉ b xˉ )/SD Medium ES ( xˉ d xˉ c )/SD 1c. Lrge Group E Group F 1d. Very lrge Group G 10% Group H 10% Distribution of Institutionl Benchmrks Distribution of Institutionl Benchmrks Lrge ES ( xˉ f xˉ e )/SD Very lrge ES ( xˉ h xˉ g )/SD Note: xˉ through xˉ h re the men benchmrk scores of the students ttending institutions within groups A through H. SD=Stndrd devition (pooled). Contextulizing NSSE Effect Sizes. Pge 2 of 7

3 Below ech figure is formul for clculting the Cohen s d effect size for the prticulr comprison being modeled in the illustrtion. For exmple, the formul under Figure 1 (Smll ES ( x b - x )/SD) conveys tht smll ES is pproximtely the stndrdized difference between the men benchmrk score of students ttending institutions in Group B (x b ) nd the men benchmrk score of students ttending institutions in Group A (x ). Tble 2 shows the effect sizes for these smll, medium, lrge, nd very lrge model comprisons for first-yer students nd seniors on ll five NSSE benchmrks from While the effect sizes in Tble 2 vry somewht between benchmrks nd between student clss levels, the rnges within the smll, medium, lrge, nd very lrge ctegories re consistent nd, with the exception of Enriching Eductionl Experiences for seniors, do not overlp. Tht is, the mximum smll ES is lower thn the minimum medium ES, the mximum medium ES is lower thn the minimum lrge ES, nd so on. Tble 2 Effect Sizes from the NSSE Benchmrk Percentile Group Comprisons First yer Seniors Smll b Medium Lrge Very lrge Smll Medium Lrge Very lrge Level of Acdemic Chllenge Active & Collbortive Lerning Student Fculty Interction Enriching Eductionl Experiences Supportive Cmpus Environment Minimum Mximum Only U.S. NSSE 2007 institutions were included in this nlysis (n=587). Precision weighted mens were used to determine the percentile rnking of ech institution s benchmrk, seprtely for first yer nd senior students. b Smll = ES difference between the scores of students ttending institutions in the third qurtile nd those of students ttending institutions in the second qurtile on prticulr benchmrk. Medium = ES difference between the scores of students ttending institutions in the upper hlf nd those of students ttending institutions in the lower hlf on prticulr benchmrk. Lrge = ES difference between the scores of students ttending institutions in the top qurtile nd those of students ttending institutions in the bottom qurtile on prticulr benchmrk. Very lrge = ES difference between the scores of students ttending institutions in the top 10% nd those of students ttending institutions in the bottom 10% on prticulr benchmrk. Tbles 1 nd 2 suggest tht slightly finer grined pproch to effect size interprettion thn Cohen s is pproprite for NSSE benchmrk comprisons. The consistency of ES vlues in Tble 2 mkes it possible to recommend new criteri for the interprettion of effect sizes in benchmrk comprisons. Therefore, Tble 3 proposes new set of reference vlues for interpreting effect sizes, bsed on the results in Tble 2. Like Cohen s, these new vlues should not be interpreted s precise cut points, but rther re to be viewed s corse set of thresholds or minimum vlues by which one might consider the mgnitude of n ES. These new reference vlues were selected fter n exmintion of the minimum vlues in Tble 2, which when rounded to the nerest tenth pproximted evenly-spced intervls between.1 nd.7. The simplicity of the proposed vlues (.1,.3,.5, nd.7) my hve intuitive nd functionl ppel for users of NSSE dt. Contextulizing NSSE Effect Sizes. Pge 3 of 7

4 Tble 3 Proposed Reference Vlues for the Interprettion of Effect Sizes from NSSE Benchmrk Comprisons Effect size Smll.1 Medium.3 Lrge.5 Very lrge.7 These vlues were bsed on NSSE benchmrk distributions nd re recommended for NSSE benchmrk comprisons, not for individul item men comprisons. Vlues re to be viewed s corse thresholds, not s precise cut points. Tble 4 reports the distribution of NSSE effect sizes bsed on these proposed reference vlues. As expected from our previous look t Tble 1, the mjority of effect sizes were trivil, smll, nd medium. Yet, this is finer distribution within ctegories from wht we sw in Tble 1 bsed on Cohen s definitions. In Tble 4 pproximtely one-qurter to one-third of ll effect sizes pper to be in the trivil rnge, more thn 40% re considered smll, nd the new medium rnge cptures bout 20 to of ll effect sizes. Lrge nd very lrge effect sizes re reltively rre. Tble 4 Distribution of NSSE 2007 Effect Sizes by the Proposed Reference Vlues Effect Size Rnge Trivil (0 to.09) Smll (.10 to.29) Medium (.30 to.49) Benchmrk FY SR FY SR FY SR FY SR FY SR Level of Acdemic Chllenge 27% 34% 43% 44% 22% 14% 6% 5% 2% 3% Active & Collbortive Lerning 29% 29% 44% 46% 18% 17% 6% 6% 3% 2% Student Fculty Interction 34% 45% 40% 15% 20% 5% 9% 2% 5% Enriching Eductionl Experiences 21% 46% 32% 22% 24% 5% 11% 2% 11% Supportive Cmpus Environment 27% 23% 41% 42% 24% 7% 7% 1% 3% Effect sizes were tken only from those institutions tht selected comprisons with ll 2007 U.S. institutions (n=519). Becuse effects sizes re both positive nd negtive, bsolute vlues were used for the rnges. Lrge (.50 to.69) Very Lrge (.70 or more) Cse Study Smple University The following provides n exmple of how informtion provided in this guide cn be pplied to rel results. Smple University (SU) endevors to be one of the most engging institutions in the US, with chllenging nd enriching cdemic experience, ctive nd collbortive students, n open nd helpful fculty, nd the most supportive infrstructure possible for student lerning. After work over severl yers on severl inititives, the provost sked the director of institutionl reserch to give progress report bsed on the ltest NSSE results. Tble 5 shows the five benchmrk scores for seniors ttending Smple University longside scores for the selected comprison group. The third column lists the effect sizes for the men comprisons. Of course SU is plesed with these results. Indeed, three of the five benchmrks re substntilly positive nd ffirming of their gols. The director of institutionl reserch noticed tht the Level of Acdemic Chllenge t SU is in fct quite strong, with very lrge effect t.72. Active nd Collbortive Lerning nd Supportive Cmpus Environment re lso well bove verge with medium effects of.44 Contextulizing NSSE Effect Sizes. Pge 4 of 7

5 nd.30 respectively. The effect size for Enriching Eductionl Experiences is lso on the positive side, but perhps smll in mgnitude t.23. The only benchmrk showing perhps no meningful or prcticl difference is Student-Fculty Interction t.08. Tble 5 Smple University Benchmrk Comprisons (Seniors) Benchmrk Benchmrk Scores Smple University Selected comprison group Effect size Level of Acdemic Chllenge Very lrge Active nd Collbortive Lerning Medium Student Fculty Interction Trivil Enriching Eductionl Experiences Smll Supportive Cmpus Environment Medium These results re tken from n ctul institution, yet we cknowledge tht they re unusully positive. They were intentionlly selected for the illustrtive purposes of this mnuscript. Although the recommended definitions in the ES chrt in Tble 3 re useful in interpreting the comprison results, more ctionble observtions often exist t the item level. Item frequencies cn mke benchmrk scores nd effect sizes more tngible nd observble. For exmple, Tble 6 reports Smple University s frequencies for the individul items corresponding to the benchmrk scores in Tble 5. The response options for ll the items were collpsed for quick review nd interprettion. Both SU nd the selected comprison group percentges re given, with the percentge differences listed in the right hnd column. With the exception of items ssocited with Student-Fculty Interction, nerly ll show positive differences when compred with the selected comprison group. A series of smll differences cn ccumulte into pprecible effect sizes when combined to form the benchmrk score. Among the Level of Acdemic Chllenge items, severl lrge percentge differences stnd out for Smple University. For exmple, 36% more SU students sid they red 10 or more ssigned books nd 27% more wrote t lest four mid-length ppers. SU seniors lso reported tht their coursework emphsized substntilly more nlysis, synthesis, evlution, nd ppliction. These differences in the individul item responses ccount for the very lrge effect size of.72 on this benchmrk. The medium Active nd Collbortive Lerning effect size of.44 is lso evident in the individul item frequencies. For exmple, compred to seniors ttending the selected comprison group institutions, 21% more SU seniors contributed to clss discussions frequently (often or very often), nd 17% more mde clss presenttions frequently. Likewise, the medium effect on the Supportive Cmpus Environment benchmrk is evident in the mostly positive response differences on the individul items, rnging up to 14%. The smll mgnitude of the Enriching Eductionl Experiences benchmrk is due to mixed results mong the items, with those showing positive differences for SU (such s foreign lnguge coursework, internships, nd co-curriculr ctivities) to some extent offset by those showing negtive differences for SU (such s independent studies nd culminting senior experiences). Other items show only modest differences. Still, the net result is positive effect size of.23. Contextulizing NSSE Effect Sizes. Pge 5 of 7

6 Tble 6 Smple University Item Frequencies by Benchmrk (Seniors) Item # Percent of students who... Smple University Selected Comprison Group Level of Acdemic Chllenge (ES=.72) 10. Sid the institution emphsizes studying nd cdemic work 3 82% 78% 4% 1r. Worked hrder thn you expected to meet n instructor's expecttions 1 66% 57% 10% 2b. Sid courses emphsized nlyzing ides, experiences, or theories 3 95% 84% 11% 2c. Sid courses emphsized synthesizing ides into new complex reltionships 3 90% 74% 16% 2d. Sid courses emphsized mking judgments bout the vlue of informtion 3 86% 71% 15% 2e. Sid courses emphsized pplying theories or concepts to new situtions 3 95% 79% 16% 3. Red more thn 10 ssigned books or book length pcks of redings 68% 32% 36% 3c. Wrote t lest one pper or report of 20 pges or more 62% 49% 12% 3d. Wrote more thn 4 ppers or reports between 5 nd 19 pges 72% 46% 27% 3e. Wrote more thn 10 ppers or reports of fewer thn 5 pges 37% 31% 6% 9. Spent more thn 10 hours/week prepring for clss (studying, etc.) 69% 55% 14% Active nd Collbortive Lerning (ES=.44) 1. Asked questions/contributed to clss discussions 1 90% 69% 21% 1b. Mde clss presenttion 1 76% 59% 17% 1g. Worked with other students on projects during clss 1 45% 47% 2% 1h. Worked with clssmtes outside of clss to prepre clss ssignments 1 69% 58% 11% 1j. Tutored or tught other students (pid or voluntry) 1 30% 22% 8% 1k. Did community bsed project s prt of regulr course 1 27% 17% 10% 1t. Discussed ides from redings or clsses with others outside of clss 1 64% 62% 1% Student Fculty Interction (ES=.08) 1n. Discussed grdes or ssignments with n instructor 1 57% 58% 1% 1o. Tlked bout creer plns with fculty member or dvisor 1 47% 40% 6% 1p. Discussed ides from clsses with fculty outside of clss 1 27% 27% 0% 1q. Received prompt written or orl feedbck from fculty 1 64% 62% 1% 1s. Worked with fculty members on ctivities other thn coursework 1 22% 21% 1% 7d. Worked on reserch project with fculty member outside of clss 19% 19% 0% Enriching Eductionl Experiences (ES=.23) 10c. Sid the institution substntilly encourges contcts mong diverse peers 3 49% 46% 3% 1l. Used n electronic medium to discuss or complete n ssignment 1 59% 60% 1% 1u. Hd serious converstions w/ students of nother rce or ethnicity 1 53% 53% 0% 1v. Hd serious converstions w/ students of other relig./politics/vlues 1 62% 55% 7% 7. Did prcticum, internship, field exp., clinicl ssgmt 65% 53% 12% 7b. Prticipted in community service or volunteer work 68% 59% 9% 7c. Prticipted in lerning community 27% 1% 7e. Completed foreign lnguge coursework 61% 41% 20% 7f. Completed study brod progrm 23% 14% 9% 7g. Prticipted in n independent study or self designed mjor 11% 18% 6% 7h. Completed culminting senior experience 23% 32% 9% 9d. Spent more thn 5 hours/week prticipting in co curriculr ctivities 37% 24% 13% Supportive Cmpus Environment (ES=.30) 10b. Sid the institution provides substntil support for cdemic success 3 82% 68% 14% 10d. Sid the institution substntilly helps students cope w/ non cd. mtters 3 30% 24% 6% 10e. Sid the institution provides substntil support for students' socil needs 3 40% 34% 6% 8. Positively rted their reltionships with other students 2 77% 82% 4% 8b. Positively rted their reltionships with fculty members 2 87% 78% 9% 8c. Positively rted their reltionships with dmin. personnel nd offices 2 64% 53% 11% 1 Combintion of students responding 'very often' or 'often' 2 Rted t lest 5 on 7 point scle 3 Combintion of students responding 'very much' or 'quite bit' Difference Contextulizing NSSE Effect Sizes. Pge 6 of 7

7 Finlly, Student-Fculty Interction shows trivil effect for SU in comprison with the selected comprison group, which is plinly evident by the meger percentge differences between the two groups. Tken together, the differences between Smple University nd the comprison group in these item frequencies provide rich explntion for the effect sizes seen in Tble 5. Observtions like this cn help dministrtors nd policy mkers cultivte specific ction plns to improve the undergrdute experience. Conclusion The purpose of this study ws to nlyze effect sizes in the context of ctul NSSE dt nd to guide the interprettion of the effect sizes on NSSE s Benchmrk Comprisons reports. These nlyses informed the development of new set of reference vlues for interpreting the benchmrk effect sizes. As prcticl mtter for NSSE users, t lest four pproches cn be tken with regrd to effect sizes. 1. First, it s not unresonble to continue using Cohen s purposefully vgue definition. The new reference vlues offered in Tble 3 only devite from Cohen in the lower vlues. Some my be convinced tht smll effect sizes re unworthy of further exmintion, nd thus should continue to look for vlues round.5 nd greter. 2. Second, for those willing to consider the new reference vlues proposed in Tble 3, the thresholds of.1,.3,.5, nd.7 could hve ppel for their simplicity nd functionlity. They re grounded in ctul NSSE findings nd my llow for richer interprettions of NSSE results. 3. Third, it s lso possible to ignore the new reference vlues nd to exmine the results in Tble 2 for more nunced interprettion of prticulr ES. Tble 2 revels different pttern of effect sizes for ech benchmrk, nd lso tht these differ between first-yer students nd seniors. Wht s more, effect sizes for the Enriching Eductionl Experiences benchmrk for seniors tend to be lrger in mgnitude thn for other benchmrks. 4. Finlly, the guide lso recommends n exmintion of individul item frequencies in combintion with ES interprettion. Individul items provide richer explntion for the mgnitude of the effect sizes, nd cn help dministrtors nd policy mkers interpret results in wys tht re context-specific nd ctionble. Be wre tht mny combintions of individul item results cn produce prticulr ES. For exmple, consider two institutions with the sme ES on prticulr benchmrk. The first my hve lrge percentge differences on just few of the benchmrk items nd no differences on the others, while the second could hve smll percentge differences on ll the items. Whtever the pproch, effect sizes cn be useful sttistic to help institutions interpret the strength or mgnitude of their benchmrk scores in reltion to their selected comprison groups. References Cohen, J. (1988). Sttisticl power nlysis for the behviorl sciences (2nd ed.). Hillsdle, NJ: Lwrence Erlbum Assocites. Indin University Center for Postsecondry Reserch 1900 Est Tenth Street, Suite 419 Bloomington, IN Phone: Fx: E-mil: nsse@indin.edu Web: Contextulizing NSSE Effect Sizes. Pge 7 of 7