An Evaluation Guide for STEM Graduate School Leaders EVALUATION ENGINEERING BIOLOGY PHYSICS MATH CHEMISTRY

Transcription

1 Measurig DIVERSITY A Evaluatio Guide for STEM Graduate School Leaders ENGINEERING CHEMISTRY BIOLOGY PHYSICS MATH data EVALUATION

2 About AAAS The America Associatio for the Advacemet of Sciece (AAAS) is the world s largest geeral scietific society, ad publisher of the joural, Sciece (www. sciecemag.org) as well as Sciece Traslatioal Medicie ( ad Sciece Sigalig ( AAAS was fouded i 1848, ad icludes some 262 affiliated societies ad academies of sciece, servig 10 millio idividuals. Sciece has the largest paid circulatio of ay peer-reviewed geeral sciece joural i the world, with a estimated total readership of 1 millio. The o-profit AAAS ( is ope to all ad fulfills its missio to advace sciece ad serve society through iitiatives i sciece policy; iteratioal programs; sciece educatio; ad more. For the latest research ews, log oto EurekAlert!, the premier sciece-ews Web site, a service of AAAS.

3 Measurig Diversity A Evaluatio Guide for STEM Graduate Program Leaders A Fial Product of a Evaluatio Capacity Buildig Project Fuded by the Natioal Sciece Foudatio Divisio of Research o Learig i Formal ad Iformal Settigs ad the Alliaces for Graduate Educatio ad the Professoriate Program For more iformatio about this project ad related evaluatio tools, see This material is based upo work supported by the Natioal Sciece Foudatio uder grat No. NSF The opiios, fidigs, iterpretatios, coclusios or recommedatios expressed i this material are those of its authors ad do ot represet the views of the AAAS Board of Directors, the Coucil of AAAS, AAAS membership or the Natioal Sciece Foudatio. Copyright America Associatio for the Advacemet of Sciece. All rights reserved. Read our privacy policy ad terms of use ( ISBN #:

4 Measurig Diversity A Evaluatio Guide for STEM Graduate Program Leaders Editors Yolada S. George ad Shirley M. Malcom America Associatio for the Advacemet of Sciece (AAAS) Patricia Campbell Campbell-Kibler, Associates, Ic. (CKA) ii For more iformatio, see

5 Cotributig Authors by Topics The Framework ad Cotext for Evaluatio of STEM Diversity Programs ad Idicators ad Evaluatio Desig: Yolada S. George, AAAS; Patricia Campbell, CKA; ad Lisa Frehill, Natioal Actio Coucil for Miorities i Egieerig, Ic. (NACME) Graduate School/Departmet Recruitmet Strategies ad Admissios Practices: Ae MacLachla, UC Berkeley; Leell Alle, Califoria Istitute of Techology, Jet Propulsio Lab; Hery Frierso, Uiversity of Florida; Richard Fry, Pew Hispaic Ceter; Christie Marlow, New Mexico State Uiversity; Lisa Frehill, NACME; ad Wedy Stock, Motaa State Uiversity Graduate School/Departmetal Retetio ad Admissios Patters: Patricia Campbell; Willie Pearso, Jr., Georgia Istitute of Techology; Jaso Kim, Systemic Research, Ic.; Eric Jolly, Sciece Museum of Miesota; Yolada S. George, AAAS; Lisa Frehill, NACME; ad Wedy Stock, Motaa State Uiversity Graduate School Support Programs ad Eviromet: Suzae Gage Braiard, Uiversity of Washigto; Elizabeth Kuz, formerly at the Uiversity of Califoria; Keeth Mato, Uiversity of Marylad, Baltimore Couty; Carlos Rodriguez, America Istitutes for Research; ad Sherry Watt, Uiversity of Iowa Evaluatio Reflectios o Faculty Metorig ad Socializatio: Catherie Millett, Educatioal Testig Service; Legad Burge, Tuskegee Uiversity; Lisa Frehill, NACME; Judy Jackso, New York Uiversity; ad Sue Rosser, Sa Fracisco State Uiversity Idicators for Assessig Preparatio for the Professoriate: Orlado Taylor, Chicago School of Professioal Psychology, Washigto, DC, Campus; Michael Radis, Pesylvaia State Uiversity; Cythia Wisto, Howard Uiversity; Mario Robles, Uiversity of Califoria, Irvie; ad Stacy Tice, Syracuse Uiversity Reportig Formats ad Natioal Data Sources: Lisa Frehill, NACME Measurig diversity: A Evaluatio guide for STEM Graduate school leaders iii

6 TABLE OF CONTENTS Measurig Diversity A Evaluatio Guide for STEM Graduate Program Leaders Preface... vi LETTER FROM AAAS... viii ACKNOWLEDGMENTS...x Chapter 1: The Framework ad Cotext for Evaluatio of STEM Diversity Programs... 1 Itroductio...2 figure 1: Framework for Istitutioal Chage...2 Tool Alert: Are You o Target?...4 The Cotext for STEM Diversity Programs...4 Chapter 2: Idicators ad Other Experimetal Desig Tips...9 usig Quatitative Data to Examie Treds Other Quatitative Idicators for Evaluatio Studies...11 compariso Groups from the Evaluatio Field: A Large-Scale Compariso Group Study: Evaluatio of the Louis Stokes Alliaces for Miority Participatio (LSAMP) Chapter 3: Evaluatio of Graduate Studet Recruitmet Strategies ad Admissios Practices recruitmet Strategies Tool Alert: Lookig at Erollmet Yield...20 Studet Admissios Practices Chapter 4: Chapter 5: Graduate Studet Retetio ad Ph.D. Completio Studies...23 Tool Alert: Computig Approximate Retetio Rates...24 Tool Alert: Trackig Studets Progress Towards Degree...25 Examiig Istitutioal ad Departmetal Eviromets...25 from the Evaluatio Field: A Retetio Study by the CGS Ph.D. Completio Project...27 Reflectios about Evaluatio Studies o Faculty Metorig Activities...29 Sources of Data o the Faculty Studet Metorig Relatioship figure 5-1: A Model of Studet Flow through Doctoral Programs to a Post-Doc Job faculty Backgroud Data...32 Studet Recruitmet ad Faculty Metorig...32 iv For more iformatio, see

7 Aspects of the Metorig Role...33 Table 5-1: Idicators for STEM Faculty Studet Metorig...33 faculty Metorig ad Retetio...35 career Placemet...36 Chapter 6: Evaluatio of Preparatio for the Professoriate idicators of Preparatio for the Professoriate...38 Chapter 7: Reportig Formats Orgaizig Data Table 7-1: Impacts of the AAAS Wome s Iteratioal Scietific Cooperatio (WISC ) Mii-grats Program...44 Displayig Data i Graphs...45 figure 7-1: A Example of a Lie Graph...45 figure 7-2: A Example of a Pie Graph figure 7-3: A Example of a Bar Graph...46 Example of a Two-Page Brief...47 Chapter 8: Natioal Sources of Data...49 Table 8-1: Overview of Natioal Data Sources with Key Variables...50 Public Data Sources...50 Table 8-2: U.S. Citize ad Permaet Residet Doctorate Recipiets, by Race/Ethicity ad Major Field of Study: Data Sources from Professioal Societies...54 Summary Epilogue Appedix A: Selected Graduate Research Studies...58 Appedix B: Sample AGEP Istitutio Report for Uiversity X Appedix C: Sample AGEP Ifo Brief...67 Key Resources Cited...83 Measurig diversity: A Evaluatio guide for STEM Graduate school leaders v

8 Measurig diversity: A Evaluatio guide for STEM Graduate school leaders Preface Six years ago, the America Associatio for the Advacemet of Sciece (AAAS) ad Campbell-Kibler Associates, Ic. (CKA) implemeted a evaluatio capacity project with gratees of the NSF Alliaces for Graduate Educatio ad the Professoriate (AGEP) ad fudig from NSF s Evaluative Research ad Evaluatio Capacity Buildig (EREC) ad Research o Learig ad Educatio (ROLE) uit. The goal of the project was to provide advice, tools, ad resources to assist admiistrators, faculty, ad staff with the evaluatio of sciece, techology, egieerig, ad mathematics (STEM) diversity iitiatives at the graduate school level. This project icluded: Developmet of a framework for examiig ad collectig evidece about chages i graduate school that would lead to icreases i the umber of uderrepreseted miority studets (URM) who eared Ph.D. degrees ad etered the professoriate. URM studets iclude studets who are Africa Americas, Hispaic Americas, America Idias, Alaska Natives, Native Hawaiias, or other Pacific Isladers. Evaluatio capacity buildig meetigs ad workshops that icorporated opportuities to braistorm ad etwork with evaluators ad researchers ivolved i graduate educatio research or evaluatio, particularly as related to URMs. Meetigs with graduate school leaders to discuss the challeges ad lessos leared about the implemetatio of high-quality, iovative, useful, ad credible STEM graduate educatio evaluatio studies, particularly as related to URM studets. Developig ad testig tools ad protocols for use i evaluatio of STEM graduate school diversity projects. Developmet of a series of four iteractive tools to help graduate school faculty ad admiistrators: Assess their progress toward their diversity goal. Determie erollmet yields (the proportio of applicats who are admitted ad go o to eroll) for differet demographic populatios. Easily compute approximate retetio rates for differet demographic populatios. Track studet progress to degree. Idetifyig quatitative ad qualitative idicators ad research questios that might be useful i evaluatio of STEM graduate school diversity projects. The result of this work is summarized i this guidebook. The guidebook icludes eight chapters ad a epilogue with iformatio writte by 25 graduate school admiistrators ad faculty ad evaluators ad researchers. The authors from the graduate schools are grat recipiets of the NSF AGEP, a major graduate educatio diversity iitiative focused o icreasig the umber of miority studets who ear a Ph.D. i STEM ad eter the professoriate. vi For more iformatio, see

9 Chapter 1 provides the framework ad cotext for evaluatio of STEM diversity programs. The mai messages i this chapter are that fuders, policymakers, ad accreditatio groups wat evidece that practices, programs, or itervetios are havig a positive impact o all studets ad that evaluatio eed ot be a burde. This chapter also icludes a set of questios to use to examie the diversity cotext of graduate schools ad STEM departmets. A summary of graduate school diversity strategies is also outlied, with metio of Appedix A, which has refereces ad Web liks to selected graduate educatio research studies. Chapter 2 provides geeral idicators ad other experimetal desig tips, icludig iformatio about a tool ad format that ca be used to examie chages i graduate studet data for admits, applicats, ew erollees, erollmet, ad degrees awarded. Appedices B ad C provide sample reports geerated usig the tool. Suggestios are also provided o how to set up compariso groups that show that programs, practices, or itervetios are havig a positive effect or resultig i positive outcomes for participats. Chapters 3, 4, 5, ad 6, respectively, cotai more specific idicators ad suggested evaluatio studies for examiig: Recruitmet strategies ad admissios practices; Retetio ad Ph.D. completio; Faculty metorig; ad Preparatio for the professoriate. Chapter 7 provides tips o reportig, orgaizig, ad displayig data. The last chapter, Chapter 8, poits out atioal data sources that ca be used to collect additioal iformatio about your istitutio or doctorate recipiets for istitutioal bechmarkig. The Epilogue summarizes the book s fidigs ad emphasizes the importace of cotiuig to use evaluatio as a tool to icrease diversity. More tools ad resources for use i evaluatio of STEM diversity graduate programs ca be foud o the project website ( This guide is ot iteded to serve as a how to guide for evaluatio. Rather, its goal is to lay out the framework that eeds to be established whe desigig evaluatio studies that will lead to icreased umbers of URM studets who ear a STEM Ph.D. ad eter the professoriate. For more specific iformatio about the why ad how of evaluatio, please see The 2010 User Friedly Hadbook for Project Evaluatio ( com/westat/pdf/ews/ufhb.pdf). I additio, aother useful evaluatio tool is the Framework for Evaluatig Impacts of Broadeig Participatio Projects: Measurig diversity: A Evaluatio guide for STEM Graduate school leaders vii

10 Measurig diversity: A Evaluatio guide for STEM Graduate school leaders Letter from AAAS Dear Colleagues: It is i the iterest of U.S. uiversities ad their STEM graduate schools to assess the quality of their programs so that they remai highly competitive i what is becomig a world market for STEM talet. High-quality graduate programs must work to become more efficiet ad eve more effective. At preset, estimates of the overall rate of Ph.D. completio i STEM fields rage from less tha four i te for Africa America studets i the physical scieces to aroud seve i te for iteratioal studets. For ay group the o-completers represet a tremedous loss of talet, especially cosiderig the profile of eterig studets ad the sizable ivestmet beig made i their educatio. The tremedous challeges that we face ow ad ito the future as a coutry ad a plaet ca oly be addressed by preparig the best mids to attack the biggest problems. This guide is iteded as a tool for assessig the stregths ad weakesses of graduate programs. While developed to support better outcomes for uderrepreseted miority studets (URMs), evaluatio, program improvemet, ad itervetios will likely yield positive outcomes for all studets. Wome ad URM graduate studets i STEM have bee cosidered the caaries i the coal mie ; weakesses i program structures have ofte affected them first ad most severely. I the past, whe differetial outcomes have appeared for studets from differet populatio groups, researchers have ofte attributed these to differeces i studet characteristics (e.g., levels of preparatio, quality of baccalaureate programs completed, etc.). Less attetio has bee paid to the impact of program characteristics ad their effectiveess i providig highly capable studets with what they eed to be successful i the graduate program ad beyod. This much eeded focus is what the NSF Alliaces for Graduate Educatio ad the Professoriate Program has provided, the idea that moitorig, evaluatio, ad chages i the structure of graduate programs ad faculty behavior, based o evidece, could lead to sigificat improvemets i program outcomes, especially for populatios that have ot see such levels of success. To get thigs right, what is required is a systematic approach to uderstadig the movemet of studets through graduate programs, from before the poit of etry to post-ph.d. completio. For example, what are the critical milestoes for the program, ad what are idicators of success? What happes at the decisio poits; is guidace provided to studets as to how they might respod to these? What does successful performace look like? Is each studet gettig whatever she or he eeds to succeed i the graduate program? Ad are the stregths that each studet brigs to the program beig tapped? Careful evaluatio of programs ad measures of outcomes ca tell admiistrators ad faculty where they stad. Every program wats to be kow for its quality, ad the success of its studets viii For more iformatio, see

11 is a importat measure of quality. From the graduate school level, STEM departmets build etworks of alumi ad research collaboratios across istitutios ad aroud the world. May of these etworks lead to developmet of R&D ceters that attract busiess vetures ad become egies for ecoomic developmet. I the U.S. we eed oly look at Research Triagle i North Carolia, Silico Valley i Califoria, ad Route 128 i Massachusetts to uderscore how the itegratio of research ad educatio i area uiversities ca lead to iovatio ad support busiess developmet. Give the importace of graduate program quality to departmet ad istitutioal reputatio, the ability to attract ad retai top faculty ad studets, as well as gai fudig ad ejoy research productivity, it is critical that high priority be give to assessig rather tha just assumig program effectiveess. Whe all studets are give the opportuity to succeed i graduate educatio i STEM, all beefit: Programs gai the talets ad diverse perspectives these studets brig to research. Society beefits from the productio of a diverse professoriate for a icreasigly diverse studet populatio. Istitutios gai isights ito overall program improvemet leadig to greater success for all, savig moey ad ehacig research eviromets. Evaluatig the effectiveess of graduate programs makes sese, eve for oes that are curretly viewed iterally ad exterally as high quality. Cotiuously improvig ad evolvig programs remai competitive, ad forward lookig programs stay i the lead. Wishig you success, Shirley M. Malcom, Director, AAAS Educatio ad Huma Resources Programs Ala Lesher CEO of AAAS & Publisher, Sciece Measurig diversity: A Evaluatio guide for STEM Graduate school leaders ix

12 Measurig diversity: A Evaluatio guide for STEM Graduate school leaders Ackowledgmets With fudig from the Natioal Sciece Foudatio, AAAS ad Campbell-Kibler Associates, Ic. were able to form a learig commuity with graduate school leaders, faculty, ad staff who were recipiets of the NSF AGEP grats, as well as graduate educatio evaluators ad researchers. Through meetigs, workshops, ad writig teams, we were able to brig together some of the best mids to poder ways of usig evaluatio to scale-up the umber of URM studets who ear STEM doctorates. Figurig out strategies ad resources for evaluatig STEM graduate educatio diversity was o easy task; testig data collectio tools was particularly challegig. What s more, the path to idetifyig tools, resources, ad evaluatio strategies was ofte cotetious, though always elighteig. We are especially grateful to the AGEP leaders for their hoest opiios o how best to implemet these types of evaluatio studies i graduate schools. Without such opeess ad wisdom garered from years o the frotlie, we would ot have bee able to develop ad test evaluatio tools ad resources that we thik will be useful for all graduate school programs. We appreciate the evaluators ad researchers who made presetatios, led focus groups, wrote papers ad materials, ad wet the extra mile with us to sort out the best evaluatio approaches. Their o-the-groud expertise implemetig multiple istitutio studies brought aother valuable perspective to the task of developig credible ad valid evaluatio approaches. At CKA, we would like to thak Tom Kibler for coutless hours of tool developmet, testig, ad refiemet ad Jeifer L. Weisma for data cleaup ad overall support. At AAAS, we wat to express our gratitude to Marty McGiho ad Doa Behar(Web ad publicatios) ad Kelly Hase, Chrissy-Rey Drapeau, ad Michael Dace (our techical team at Pogos Iteractive). We especially appreciate the support that the NSF educatio uit ad AGEP program provided for this very importat work. Noe of this work would have bee possible without these fuds. We would also like to thak our Program Officer, Elmima C. Johso, for her support, as well as Berice Aderso, NSF Seior Advisor, for her guidace durig the developmet of the guidebook. For his guidace ad wise isights, we would like to dedicate this publicatio to Roosevelt Johso, the NSF Program Officer who helped shape this evaluatio capacity iitiative durig its critical formative stages. His perspective ad guidace were ivaluable i helpig us develop useful tools ad resources. x For more iformatio, see

13 Fially, we would like to thak Marily Feichel (publicatio maagemet) ad Gail Peck, Peck Studios, Ic. (graphic desig) for fializig this guidebook. We do ot see this publicatio as a ed for the learig commuity, but rather as a ew stage, where STEM graduate educatio diversity leaders are equipped with the tools ad resources to form their ow evaluatio learig commuities. We sicerely hope that this guidebook ad the website tools will be useful i helpig graduate school admiistrators, faculty, ad staff scale-up the umber of URM studets who ear STEM doctorates ad eter the professoriate. Best Regards, Yolada S. George, AAAS Patricia B. Campbell, Campbell-Kibler Associates, Ic. Measurig diversity: A Evaluatio guide for STEM Graduate school leaders xi

14

15 Chapter 1 The Framework ad Cotext for Evaluatio of STEM Diversity Programs Chapter Highlights Fuders, policymakers, ad accreditatio groups wat evaluatio evidece that practices, programs, or itervetios are havig a positive impact. For istitutios ad departmets, data ca help i decisio makig ad show evidece of progress, ad it should be collected ad produced i a simple, cost-effective way. Data must be disaggregated by race/ ethicity, sex withi race/ethicity, disability, citizeship, ad STEM departmet whe appropriate. Usig disaggregated studet data should be a part of regular maagemet practices at the graduate school ad departmetal level.

16 Measurig diversity: A Evaluatio guide for STEM Graduate school leaders Chapter 1 The Framework ad Cotext for Evaluatio of STEM Diversity Programs Itroductio This guidebook is desiged to provide advice to vice presidets, provosts, deas, other admiistrators, ad faculty about the evaluatio of sciece, egieerig, techology ad mathematics (STEM) diversity programs at the graduate school level. Africa Americas, Hispaic Americas, America Idias, Alaska Natives, Native Hawaiias or other Pacific Isladers cotiue to ear a small percetage of the U.S. STEM college ad uiversity degrees, particularly doctorate degrees ( This uderrepresetatio of these racial/ethic groups has serious implicatios for the atio s ability to compete i a global ecoomy drive by iovatios i sciece ad techology. I geeral, STEM diversity graduate school programs are iteded to icrease the umber of uderrepreseted miority (URM) studets who pursue ad receive master s ad/or Ph.D. degrees. Figure 1-1 illustrates the way these differet factors work together to create sigificat istitutioal chage. Leadership ad the use of data are importat compoets of istitutioal chage. Figure 1-1 Framework for Istitutioal Chage Trustees, Fuders Achievig this goal will require ogoig assessmet ad evaluatio of istitutioal ad departmetal policies, practices, programs, ad itervetios related to: Istitutioal policies Recruitmet; Admissios ad selectio; Fiacial aid; Advisig ad metorig; Culture ad climate; ad Professioal socializatio. departmet chages i practices & culture Itervetios SHORT TERM STUDENT OUTCOMES LONGER TERM STUDENT OUTCOMES PROGRAM CHANGE FACULTY CHANGE 2 For more iformatio, see

17 Chapter 1: The Framework ad Cotext for Evaluatio of STEM Diversity Programs Sice the maagemet of graduate programs is usually distributed across multiple uits ad committees, all decisios makers should be ivolved i the collectio ad use of disaggregated studet data, icludig: Uiversity trustees; Graduate school admiistrators, such as vice presidets, provosts, ad deas; Admissios ad fiacial aid officers; Departmet chairs; Departmet admissios ad selectio committees; Departmetal graduate program directors or advisors; Faculty; ad Staff. To uderstad the experieces of all studets withi graduate programs, whe appropriate, data must be disaggregated by race/ethicity, sex withi race/ ethicity, disability, citizeship, ad STEM departmet. Usig disaggregated studet data should be a part of regular maagemet practices at the graduate school ad departmetal level; it should ot be viewed as added work. Evaluatio studies with disaggregated studet data ca help admiistrators, faculty, ad staff set goals related to their duties ad resposibilities (icludig studet advisig ad metorig) ad be more reflective about their decisiomakig process for all graduate studets. Also, data collected for evaluatio purposes ca be used to provide evidece where practices, programs, or itervetios are havig a positive impact, particularly for use i: Preparig reports for uiversity admiistrators ad trustees, policymakers, accreditatio groups, ad others; Makig the case for fudig, icludig iteral ad exteral fudig; ad Makig the case for recogitios ad awards for the istitutio, departmet, or idividuals. Coductig cost-effective evaluatios requires buildig a effective cross-campus evaluatio commuity ad mechaisms for ogoig plaig ad coordiatio. Decisios will have to be made about how best to coordiate a effective evaluatio of STEM diversity programs at the graduate school level across multiple offices ad committees, icludig addressig the followig questios: What uit will lead the effort ad which key perso(s) will be resposible? What admiistrator(s) will be team leaders? Ideally, this type of work should be located i the office of a Vice Presidet or Provost for Research. What are the goals, objectives, ad resposibilities for the evaluatio group? What are the appropriate evaluatio questios ad timelie? What data should be collected (quatitative ad qualitative idicators)? What studet data are already beig collected by uits or special project? Are these data disaggregated? What evaluatio studies are already i place? What studies are eeded? Measurig diversity: A Evaluatio guide for STEM Graduate school leaders 3

18 Measurig diversity: A Evaluatio guide for STEM Graduate school leaders How should the data be collected, aalyzed, ad displayed? Whe ad how ofte should data be collected ad reports prepared? What uits or committees will collect ad compile the data? How will the data ad reports be distributed? It is importat that all stakeholders, icludig graduate studets, have access to summary reports. Data collectio ad evaluatio should be as simple ad cost-effective as possible. The goal is to collect evidece periodically that is useful i decisio makig ad shows overall istitutioal or departmetal progress. Tool Alert Are You o Target? A importat first step for uiversities strivig to icrease the diversity of STEM graduate studets is to set a goal ad most uiversities have already doe so. To examie cotiual progress towards this goal, uiversities should assess progress aually. Tool #1 Gettig To Your Goal: Are You o Target? ( GettigToYourGoal.html) is a iteractive tool to help istitutios look at the progress they ve made toward their diversity goal ad see the umbers they will eed aually i order to achieve that goal withi their timeframe. The Cotext for STEM Diversity Programs The cotext for STEM diversity programs is liked to both the strategies ad resources i place for (a) icreasig the umber of URM studets i STEM graduate programs ad (b) determiig whether a culture of diversity has bee established at the istitutioal or departmetal level. To icrease the umber of URM studets completig a Ph.D. at istitutios, admiistrators, faculty, ad staff eed to periodically assess their policies, practices, ad programs ad implemet appropriate strategies. Diversity Strategies I a effort to icrease the umber of URM STEM doctorates, as well as the umber of miorities eterig the professoriate, govermet agecies ad private foudatios have implemeted grat-based programs, icludig the followig: Natioal Sciece Foudatio (NSF) Alliaces for Graduate Educatio ad the Professoriate (AGEP). AGEP was created i 1998 with the goal of icreasig the umber of URM doctorates, as well as the umber of miorities eterig the professoriate. Sice its iceptio, over 100 istitutios have bee ivolved i this effort ( id=503563&org=hrd&from=home). 4 For more iformatio, see

19 Chapter 1: The Framework ad Cotext for Evaluatio of STEM Diversity Programs NSF Louis Stokes Alliaces for Miority Participatio (LSAMP) program, Bridge to the Doctorate (LSAMP-BD) Activity. Created i 2003, the goal of this activity is to remove miority studets hesitacy about eterig graduate school ad the fear of creatig additioal fiacial idebtedess associated with graduate educatio ( id=13646&org=hrd&from=home). The Natioal Istitutes of Health (NIH), Divisio of Miority Opportuities i Research (MORE). These programs iclude research ad research traiig programs aimed at icreasig the umber of miority biomedical ad behavioral scietists ( Coucil of Graduate School (CGS) Ph.D. Completio Project. This grat-fuded project addresses the issues surroudig Ph.D. completio ad attritio. Supported by Pfizer Ic. ad the Ford Foudatio, the goal of this program is to create itervetio strategies ad pilot projects ad to evaluate the impact of these projects o doctoral completio rates ad attritio patters ( phdcompletio.org/idex.asp). These grat-based programs provide resources to implemet research-based strategies that have bee effective i the recruitmet, retetio, persistece, ad attaimet of STEM graduate degrees, especially for populatios uderrepreseted i STEM disciplies. Graduate school itervetio activities that have bee developed as a result of these programs iclude: Establishig udergraduate research programs with miority servig istitutios. Recruitig prospective studets via campus visits ad meetigs where udergraduate or graduate studets are presetig posters ad oral presetatios, icludig the aual meetigs of SACNAS (a society of scietists dedicated to fosterig the success of Hispaic/Chicao ad Native America scietists), the Aual Biomedical Research Coferece for Miority Studets (ABRCMS), ad the AAAS/NSF Historically Black Colleges ad Uiversities Udergraduate Program (HBCU-UP) Coferece ad the Emergig Researcher Natioal (ERN) Coferece. Reviewig ad moitorig istitutioal ad departmetal practices, icludig practices related to graduate studet admissios, fiacial aid, advisig ad metorig, academic requiremets, ad milestoes to the Ph.D. Providig fiacial aid packages that reduce the debt burde of graduate studets. Offerig professioal developmet programs for faculty, with a emphasis o strategies for recruitig ad retaiig URM studets ad effective graduate studet metorig. Offerig supplemetary academic support workshops or tutorig for graduate studets i writig, statistics, computer-modelig, ad other subjects. Providig activities that foster the social ad early itellectual itegratio of graduate studets ito the departmet ad research, icludig graduate studet bridge programs, early research metor idetificatio, ad strategies for family/work balace. Measurig diversity: A Evaluatio guide for STEM Graduate school leaders 5

20 Measurig diversity: A Evaluatio guide for STEM Graduate school leaders Providig graduate studet travel awards ad other icetives to icrease research productivity (poster or oral presetatios at professioal meetigs, publicatios, iteratioal travel ad research experieces, etc). Moitorig graduate studet progressio with attetio to early achievemet of Ph.D. milestoes. Some of the key research studies that have idetified strategies for icreasig the umber of uderrepreseted miorities i STEM graduate programs are listed i Appedix A. Istitutioal ad Departmetal Cotext for Diversity Legal Cotext Before decidig which strategies are appropriate for icreasig diversity at a particular istitutio, it is importat for leaders i the STEM commuity to gai a better uderstadig of the istitutioal ad departmetal cotext for diversity, icludig uderstadig the legal ladscape for fosterig diversity i both higher educatio ad i your state. May complex legal structures gover access ad diversity efforts, ad may legal issues are differet for faculty tha they are for studets. This legal cotext makes desigig successful diversity programs very challegig. To help istitutios uderstad these issue, AAAS, with participatio by the Associatio of America Uiversities (AAU), has published a valuable tool, Hadbook o Diversity ad the Law: Navigatig a Complex Ladscape to Foster Greater Faculty ad Studet Diversity i Higher Educatio. The hadbook provides extesive legal ad policy resources for academic ad legal leaders to help them collaborate to improve access ad broade the diversity of their faculties ad studet bodies, particularly i STEM. The hadbook ca be dowloaded, for free, at ( 1 Diversity Cotext at the Graduate School Level I additio to uderstadig the legal cotext for diversity, some key assessmet questios should be addressed to help faculty members ad admiistrators uderstad the diversity cotext at the graduate school level. These questios iclude: How are diversity goals ad objectives itegrated ito the regular operatios ad maagemet of the graduate school office ad departmets? I geeral, what policies, practices, activities, ad resources are i place to build leadership ad accoutability with admiistrators, faculty, ad staff i terms of fosterig ad maitaiig diversity i the graduate studet populatio, icludig Departmet chairs; Departmetal admissios committees; Departmetal academic faculty ad staff studet advisors; ad 6 For more iformatio, see

21 Chapter 1: The Framework ad Cotext for Evaluatio of STEM Diversity Programs Departmetal graduate studies coordiators. How is fiacial aid used to foster ad maitai diversity i the graduate school? Is fiacial aid offered as part of the admissios letter? How may years of fiacial aid is a ew erollee give? Does the admissios letter iclude iformatio about how graduate studets are supported to Ph.D. completio? What are the sources of fuds for studet recruitmet (e.g., campus visitatios, visits for iterviews, graduate studet days, etc.) ad retetio? Are grat overhead recovery fuds a source for recruitmet? How are graduate school departmets evaluated i terms of fosterig diversity i the graduate studet pool ad i the professoriate? How are faculty evaluated i terms of fosterig diversity i the graduate studet pool ad i the professoriate? Are there other geeral or targeted programs that the graduate school operates to recruit ad retai STEM graduate studets, ot just URM graduate studets? Has the graduate school coducted ay research or evaluatio studies o graduate studet recruitmet, admissios, retetio/ attritio, climate, degree attaimet, or post-ph.d. employmet withi the last five years? Does the graduate school coduct exit iterviews with: Those who leave graduate school (leavers)? Master s degree graduates? Ph.D. graduates? Is a aual report with disaggregated data about graduate school recruitmet, applicats/admits, ew graduate school erollmet, overall graduate school erollmet, graduate school retetio or attritio, umber advacig to doctoral cadidacy, or degrees awarded distributed to departmets? If yes, who gets the report? Are the data disaggregated by departmets? Are the data disaggregated i other ways? How are the reports used? How are they distributed? Diversity Cotext at the STEM Departmet Level Recogizig that STEM departmets vary by size ad resources, geeral questios to cosider for icreasig diversity at the departmet level iclude: How does the departmet recruit graduate studets? What are the sources of fuds for studet recruitmet (e.g., campus visitatios, visits for iterviews, graduate studet days, etc.) ad retetio? Are grat overhead recovery fuds a source for recruitmet? How is the graduate studet applicatio/admissios process maaged withi departmets? What are the selectio criteria? How is fiacial aid used to foster ad maitai diversity i the graduate school? How does the departmet decide o the fiacial package for ew Measurig diversity: A Evaluatio guide for STEM Graduate school leaders 7

22 Measurig diversity: A Evaluatio guide for STEM Graduate school leaders graduate studet admits? Is fiacial aid offered as part of the admissios letter? How may years of fiacial aid is a ew erollee give? Does the admissios letter iclude iformatio about how graduate studets are supported to Ph.D. completio? What types of academic support or programs exist for graduate studets withi the departmet? What is the process for providig academic advisig: Pre-assessmet exams i the first semester to determie courses eeded? Lab rotatios to make sure that the doctoral research lab ad dissertatio selectio is a good match, both persoally ad professioally? Advisor/metor committee? Aual assessmet of studet progress with attetio to time-to-degree (i.e., what are the remaiig requiremets)? How is graduate studet academic progress moitored? Does the departmet moitor graduate studet retetio/attritio? If, yes how? How are diversity cocers beig itegrated ito the regular operatios ad maagemet of the departmet? Does the departmet coduct exit iterviews with studets who leave the departmet? Those who leave graduate school (leavers)? Master s degree graduates? Ph.D. graduates? What are the departmetal challeges related to disaggregated studet data collectio ad reportig? What are the departmetal challeges related to trackig former studets? Graduate school leavers? Ph.D. graduates? Does the departmet collect data for departmetal purposes that are ot required by the graduate school? Has the departmet coducted ay research or evaluatio studies o graduate studet recruitmet, admissios, retetio/attritio, climate, degree attaimet, or post-ph.d. employmet withi the last five years? What resources would STEM departmets eed to maage the collectio, aalysis, ad reportig of disaggregated graduate studet data about recruitmet, admissios, erollmet, retetio/attritio, advacemet to doctoral cadidacy, degree attaimet, ad post-ph.d. employmet? 1 Keith, Jamie Lewis ad Daryl E. Chubi. (2011). Hadbook o Diversity ad the Law: Navigatig a Complex Ladscape to Foster Greater Faculty ad Studet Diversity i Higher Educatio. Washigto, DC: America Associatio for the Advacemet of Sciece. 8 For more iformatio, see

23 Chapter 2 Idicators ad Other Experimetal Desig Tips Chapter Highlights Both quatitative data (umbers) ad qualitative data should be collected i evaluatio studies of STEM graduate school programs. Quatitative data are very useful i determiig what is happeig, especially what is happeig over time, but they are less useful i determiig why thigs are (or are ot) happeig. For both the qualitative ad the quatitative data to be of value, they eed to be collected at multiple time poits ad iclude compariso groups made up of members who are similar to participats ad/or have ot participated i the program or activity. I a educatioal study, the two groups should be matched by educatioal characteristics.

24 Measurig diversity: A Evaluatio guide for STEM Graduate school leaders Chapter 2 Idicators ad Other Experimetal Desig Tips Both quatitative ad qualitative data should be collected i evaluatio studies of STEM graduate school programs. Quatitative data, umerical iformatio that ca be collected ad compiled ito tables, charts, or graphs, are very useful i determiig what is happeig, especially over time, but are less useful i determiig why thigs are (or are ot) happeig. Qualitative data, iformatio collected through surveys, iterviews, ad focus groups, rely o idividual resposes of program participats. As a result, this iformatio is more likely to iclude resposes to meaigful questios related to the graduate school or departmetal experieces of both studets erolled i a particular istitutio ad those who chose to leave. Through periodic surveys, iterviews, or focus groups, evaluators ca collect iformatio about whether studets were satisfied or dissatisfied with a variety of practices ad activities, icludig: recruitmet; admissios; fiacial aid; academic support workshops, activities, ad resources; ad faculty behavior ad attitudes, as well as studets attitudes toward these activities. But the dow side is that qualitative data are ofte more difficult ad time cosumig to collect ad to aalyze. Therefore, it is importat to determie how high-quality qualitative data ca be collected ad aalyzed without addig too great a burde to faculty ad staff. For both the qualitative ad the quatitative data to be of value, they eed to be collected at multiple time poits ad iclude a compariso group; how to assig compariso groups is discussed later i this chapter. I additio, appropriate admiistrators ad/or faculty members eed to have some discussio about the time periods for data collectio, particularly for studies related to leavers ad degree completers ad those examiig istitutioal ad departmetal culture ad climate. Usig Quatitative Data to Examie Treds For studies o studet etry ito a STEM graduate program to Ph.D. degree completio, disaggregated data ca be collected for multiple years to examie chages i umber ad percetage of: Applicats; Admits; New or first time erollees; New or first time erollees i master s programs; New or first time erollees i Ph.D. programs; Overall erollmet; All master s erollees; 10 For more iformatio, see

25 Chapter 2: Idicators ad Other Experimetal Desig Tips All Ph.D. erollees; Studets advacig to doctoral cadidacy; ad Studets completig master s or Ph.D. degrees. I additio, post-ph.d. employmet iformatio ca be collected, particularly as related to employmet i academic, govermet, busiess, ad other sectors. Whe possible, these data should be disaggregated by race/ethicity, sex withi race/ethicity, disability, citizeship, ad STEM fields. As part of a NSF evaluatio capacity buildig project with colleges ad uiversities fuded by the NSF AGEP Program, AAAS ad Campbell-Kibler Associates, Ic, collected logitudial disaggregated data, as described above. Data were collected ad compiled for 73 istitutios. I may cases, graduate schools were already collectig disaggregated data for: New or first time erollees; Overall erollmet; ad Studets completig master s or Ph.D. degrees. However, oly 11 istitutios could provide disaggregated data o post-ph.d. employmet for some of the years betwee 2000 ad The data collected from the AGEP istitutios were compiled ito a format for use by the idividual istitutios ad aggregated to report o the overall progress of AGEP istitutios. Appedices B ad C iclude sample reports of how the data were compiled ad reported. Appedix C also cotais iformatio o how the data were collected ad aalyzed. More AGEP Ifo Briefs are posted o the NSF AGEP website: Other Quatitative Idicators for Evaluatio Studies I additio to examiig data disaggregated by race/ethicity, sex withi race/ ethicity, disability, citizeship, ad STEM fields, data for studets ad faculty ca be disaggregated i a variety of other ways. A good source of iformatio about graduate school idicators for evaluatio studies is the NSF Survey of Eared Doctorates (SED): A samplig of idicators ofte used to disaggregate graduate studet data are listed below: Age. Marital or partership status ad umber of depedets. Type of udergraduate istitutio where the baccalaureate degree was eared. These data ca be grouped usig the Caregie Classificatio of Higher Educatio Istitutios ( or by Historically Black Colleges ad Uiversities (HBCUs), Hispaic-servig Istitutio (HSIs), wome s colleges, or colleges with a high erollmet of studets with disabilities. Measurig diversity: A Evaluatio guide for STEM Graduate school leaders 11

26 Measurig diversity: A Evaluatio guide for STEM Graduate school leaders Academic backgroud, icludig: overall udergraduate GPA; graduate school GPA; ad verbal, aalytical, ad/or quatitative scores o the GRE. Participatio i extracurricular programs durig the udergraduate years, icludig participatios i udergraduate research programs or i miority STEM udergraduate programs. Sources of graduate school fiacial support, icludig fellowships, scholarships, dissertatio grats, teachig assistatships, research assistatships, traieeships, iterships, loas, persoal savigs, family savigs, ad employer reimbursemet. Duratio ad cotiuity of fiacial support. Amout of educatio ad o-educatio debt at time of bachelor s degree completio. Amout of educatio ad o-educatio debt at time of Ph.D. completio. Highest educatioal attaimet of mother ad/or father, icludig less tha high school/secodary school, high school/secodary graduate, bachelor s degree, master s degree, professioal degree, or doctoral degree. Time-to-Ph.D. degree. Iformatio about classificatio o colleges ad uiversities ca also be located at the followig websites: Historically Black Colleges ad Uiversities (HBCUs): about/iits/list/whhbcu/edlite-list.html. Hispaic-Servig Istitutios (HSIs): Tribal colleges: Wome s colleges: Colleges ad uiversities with high erollmet of disabled studets: Gallaudet Uiversity ( or the Natioal Techical Istitute for the Deaf at Rochester Istitute of Techology ( The U.S. Departmet of Educatio also has a list of miority postsecodary istitutios: Compariso Groups I order to make a strog case that programs, practices, or itervetios are havig a positive effect or resultig i positive outcomes for participats, it is importat to idetify a evaluatio desig that icludes compariso groups. It is also importat to esure that sufficiet data ca be collected to permit appropriate ad covicig comparisos. I geeral, compariso groups are usually made up of members who are (a) similar to participats ad/or (b) have ot participated i the program or activity. For a educatioal study, depedig o its objectives, it is geerally importat that the two groups be matched o characteristics that are correlated with: 12 For more iformatio, see

27 Chapter 2: Idicators ad Other Experimetal Desig Tips Studets educatioal achievemet prior to graduate school etry; GRE scores; Type or rak of udergraduate istitutio where bachelor s degree was eared; Age whe eterig graduate school; Race/ethicity, sex, disability, or citizeship; STEM departmet or udergraduate major; Number of years i a graduate program; or Type, duratio, or timig of fiacial aid. I evaluatio studies about URM graduate studets, there is a tedecy to oly ask the target populatio critical questios about the educatioal program, practices, or itervetios. The problems with this evaluatio desig are as follows: 1 Members of the targeted group might be reluctat to share ay egative iformatio with evaluators out of cocer for potetial repercussios. 2 It is difficult to determie the extet to which data from such a small sample ca be used to pait a represetative picture of the larger cotext i which the graduate educatioal process occurs. Also, with small umbers, it might be difficult to keep iformatio from beig associated with idividuals. 3 Coclusios caot be used to idicate that problems are particular to race/ ethicity. I additio, it is difficult to separate out problems ot associated with race/ethicity but related to more geeral issues that affect all graduate studets regardless of race/ethicity, sex, disability, or citizeship. The example o the ext page illustrates how compariso groups were used for a large-scale group study. Measurig diversity: A Evaluatio guide for STEM Graduate school leaders 13

28 From the Evaluatio Field A Large-Scale Compariso Group Study: Evaluatio of the Louis Stokes Alliaces for Miority Participatio (LSAMP) I 2005, a team of researchers at the Urba Istitute i Washigto, DC, completed a evaluatio of the Louis Stokes Alliaces for Miority Participatio (LSAMP), a udergraduate-focused program that had bee fuded by NSF sice The purpose of the program is to icrease the represetatio of uderrepreseted miorities i the STEM disciplies. At the time the evaluatio report was writte, there were 34 alliaces ivolvig 450 differet istitutios across the coutry. As a large-scale evaluatio, may dimesios of the program required ivestigatio. This example discusses oly oe of these dimesios: the umber of LSAMP participats who completed their degrees compared to miority studets ot i the program. The example illustrates importat poits related to presetig data ad the issue of compariso groups. The full report is available olie at org/uploadedpdf/411301_lsamp_report_appe.pdf. Compariso groups are critical i large-scale evaluatios. Ulike a cotrol group, however, compariso groups ofte differ i importat ways from the subject treatmet group. The term cotrol group is used oly whe the people participatig i a program ad those i the compariso group are assiged to their respective groups via a radom process. To set up a cotrol group for a program such as LSAMP, all iterested studets would be required to apply, followed by a radom selectio process for participats. This approach is ot usually feasible with programs of this type. Rather, studets are selected via o-radom processes (e.g., selected due to high grades or as a result of a recommedatio by a professor). This meas that the participats i such programs, by defiitio, are a self-selected group. To a large extet, self-selected groups geerally differ from those who do ot choose to participate Measurig diversity: A Evaluatio guide for STEM Graduate school leaders i programs, makig a classical experimetal desig impossible. Istead, quasi-experimetal desigs are more commoly used for evaluatios. Quasiexperimetal desigs follow a similar framework as the gold stadard the radomized cotrol trial but deviate by lackig radom assigmet to treatmet (i.e., the program) ad cotrol (i.e., ot participatig i the program). The goal, the, becomes to idetify appropriate compariso groups. For the LSAMP evaluatio, two atural compariso groups emerged: URM studets who had ot participated i the LSAMP program ad U.S. studets of white, o-hispaic, ad Asia descet who were ot the origial targets of the program. By comparig the outcomes for LSAMP participats to URM o-participats, a geeral sese of the program impact ca be gaied but with the importat caveat that the participats are already kow to be a selfselected group (ad, hece, possibly already more persoally motivated, with higher GPAs ad stroger coectios to professors i their uiversities). That said, however, the compariso to the white ad Asia studets does provide a sese of the extet to which the playig field for the program participats was effectively leveled. That is, how did the graduate school outcomes compare for the LSAMP participats to those of studets who were ot eligible to participate i the program, presumably those studets who were already implicitly advataged withi the higher educatio system? Figure 2-1, take from the Urba Istitute report, illustrates fidigs from LSAMP participats ad those from the two compariso groups. The data are from two sources: (1) LSAMP participat data gathered from a survey of those who had graduated betwee 1992 ad 1997 ad (2) for the compariso groups, atioally represetative data available from NSF via the Natioal Survey of Recet College Graduates. 2 The graph i Figure 2-1 shows the overall rates of trasitio to graduate school as well as the specific rates of trasitio ito STEM fields, which were the oes of predomiat iterest to the LSAMP program. I this case, the two compariso groups are, ideed, radom samples of people who were recet bachelor s ad master s degree recipiets ad were cotemporaries of the LSAMP participats. 14 For more iformatio, see