Evaluation of the National Science Foundation's GK-12 Program

Evaluation of the National Science Foundation's GK-12 Program Final Report, Volume I and II: Technical Report and Appendices Cambridge, MA Bethesda, MD Chicago, IL Durham, NC Hadley, MA Lexington, MA Contract # GS-10F-0086K, Order # NSFDACS06D1412 November 2010 Prepared for National Science Foundation 4201 Wilson Blvd. Arlington, VA 22230 Authors Beth Gamse W. Carter Smith Amanda Parsad John Dreier Kristen Neishi Jennifer Carney Linda Caswell Eddie Breaux Tom McCall Jon Spader Abt Associates Inc. 55 Wheeler Street Cambridge, MA 02138-1168 Assisted by Hilary Rhodes Eileen Fahey Katheleen Linton Bulbul Kaul Ellen Bobronnikov Marc Moss Sean Smith Satyendra Patrabansh

Contents Executive Summary... i Chapter 1: Overview of the GK-12 Program and Evaluation... 1 Introduction... 1 GK-12 Program Goals... 2 Evolution of Program Goals... 2 Prior GK-12 Program Evaluation... 3 Program Participants... 4 Principal Investigators and Co-Principal Investigators... 5 GK-12 Fellows: Undergraduate and Graduate Students... 6 K-12 Educators... 6 Organization of This Report... 8 Chapter 2: Study Design, Samples, and Analytic Methods... 9 Overview... 9 Research Questions for the Evaluation of the GK-12 Program... 9 Study Design and Background... 10 Revised Study Plan and Sampling Strategies... 11 Summary of Data Sources... 14 Data Collection: Methods and Response Rates... 14 Data Collection... 14 Eligibility... 15 Response Rates... 15 Approach to Analyses... 19 Descriptive Analyses... 19 Comparative Analyses... 20 Final Analytic Sample... 21 Limitations to the Current Study Design... 21 Survey Response Rates... 21 Testing Multiple Hypotheses... 22 Statistical Significance... 22 Chapter 3: The GK-12 Program and Its Components... 23 Key Findings... 23 Institutional Environment of GK-12 Projects... 24 Characteristics of Participating Fellows... 24 Fellows Disciplinary Affiliations... 25 Fellows Degree Type... 26 Fellows Demographic Characteristics... 26 Fellows Prior Teaching and Research Experience... 26 GK-12 Project Goals and Selection Criteria... 28 GK-12 Project Goals for Fellows... 28 Selection Criteria for Fellows... 29 Experiences of Participating Fellows... 30 Reasons for Applying for a GK-12 Fellowship... 30 Abt Associates Inc. Contents

Fellows Perceptions of Their Faculty Advisor s Support for GK-12... 31 Fellows Reasons for Accepting the GK-12 Fellowship over Other Funding Sources.. 31 Training or Educational Activities Provided to Fellows... 33 Fellows School Assignments... 34 Activities Conducted as Part of Fellowship Experience... 36 Characteristics and Experiences of Participating Teachers... 39 Teachers Demographic Characteristics... 39 Teachers School and Grade-Level Assignments... 40 Teachers Educational Characteristics... 40 Goals of the GK-12 Program for K-12 Teachers... 42 Teachers Reasons for Participating in the GK-12 Program... 42 Selection of GK-12 Teachers... 43 GK-12 Teachers Level of Involvement... 44 Professional Development and Support Provided to Teachers... 45 Teachers Engagement in Professional Development and Training Activities... 46 Chapter Summary... 47 Chapter 4: Former Fellows Education and Employment Outcomes... 49 Key Findings... 49 Former Fellows Academic Progress... 50 Role of GK-12 Fellowship in Degree Completion and Academic Progression... 52 Post-Fellowship Education and Employment... 53 Fellows Employment Choices after Graduate School... 55 Fellows First Full-Time Position after Graduate School... 55 Former Fellows Current Employment Status... 57 College or University Employment... 59 Industry/Business, Government and K-12 Employment... 61 Disciplines Fellows Use in Their Current Work... 61 Former Fellows Current Job Responsibilities... 62 Types of Research in Which Former Fellows Engage... 63 Former Fellows Engagement in Research Involving STEM Education... 63 Fellows Current Job Activities... 65 Chapter Summary... 65 Chapter 5: How Do Current Fellows Compare To Their Peers?... 67 Overview... 67 Key Findings... 67 Reasons for GK-12 Application/Participation... 68 Educational Progress, Funding Support, and Teaching Interest... 69 Allocation of Time to Research and other Activities... 76 Chapter Summary... 80 Contents Abt Associates Inc.

Chapter 6: Self-Reported Effects of the GK-12 Program... 81 Key Findings... 81 Effects on Fellows... 82 Effects on Institutions... 91 Effects on Teachers... 94 Effects on Students... 100 Chapter Summary... 102 Chapter 7: Summary... 103 Summary of Findings... 103 General Conclusions and Limitations... 107 Comments... 108 References... 111 Appendix A: Sampling, Power, and Regression Models... A-1 Appendix B: Chapter 3 Backup Exhibits... B-1 Appendix C: Chapter 4 Backup Exhibits... C-1 Appendix D: Chapter 6 Backup Exhibits... D-1 Appendix E: Copies of Instruments... E-1 Abt Associates Inc. Contents

Contents Abt Associates Inc.

List of Exhibits Exhibit 1.1: GK-12 Award Characteristics, by Year of Program Implementation (1999-2007)... 4 Exhibit 1.2: GK-12 Principal Investigators Departmental Affiliations, 1998 2007a... 5 Exhibit 1.3: Number of GK-12 Fellowships, By Degree Level and Year of Program Implementation (2000-2007)... 7 Exhibit 2.1: Currently Enrolled Fellows and Matched Comparison Samples... 15 Exhibit 2.2a: Samples of Respondents for Descriptive Analyses... 17 Exhibit 2.2b: Samples of Respondents for Comparative Analysis... 18 Exhibit 2.3: Study Response Rates... 19 Exhibit 2.4: Final Analytic Sample, by Degree Level... 21 Exhibit 3.1: Graduate Program Discipline, by Fellow Type... 25 Exhibit 3.2: Gender and Race/Ethnicity, by Fellow Type... 27 Exhibit 3.3: Top Six Experiences Before Entering GK-12 Graduate School, a by Fellow Type... 28 Exhibit 3.4: Project Goals for Fellows, as Reported by PIs... 29 Exhibit 3.5: Factors Used to Select Most Recent GK-12 Fellows, as Reported by PIs... 30 Exhibit 3.6: Reasons for Applying for a GK-12 Fellowship, by Fellow Type... 31 Exhibit 3.7: Sources of Other Funding Declined, and Reasons for Accepting GK-12 Fellowship, by Fellow Type... 32 Exhibit 3.8: GK-12 Related Training and Educational Activities, by Fellow Type... 34 Exhibit 3.9: Primary GK-12 Assignment for Fellows... 35 Exhibit 3.10: Fellows School Assignments... 36 Exhibit 3.11: PI s Characterizations of Fellows GK-12 Activities... 37 Exhibit 3.12: Description of GK-12 Related Activities, by Fellow Type... 38 Exhibit 3.13: Content Source for GK-12 Lessons and Activities, by Fellow Type... 39 Exhibit 3.14: Teachers Grade Level(s) Currently Teaching... 40 Exhibit 3.15: Teachers Field of Study, by Highest Degree Obtained... 41 Exhibit 3.16: Teachers Fields of Study and Grades Currently Teaching... 42 Exhibit 3.17: Goals of the Program for K-12 Teachers, as Reported by PIs... 43 Exhibit 3.18: Teachers Reasons for Participating in the GK-12 Program... 43 Exhibit 3.19: Preferred Characteristics of GK-12 Teachers, Reported by PIs... 44 Exhibit 3.20: Teachers Frequency of Interaction with GK-12 Program Participants... 45 Exhibit 3.21: Types of Support Provided to GK-12 Teachers, as Reported by PIs... 46 Exhibit 3.22: Teachers Participation in GK-12 Professional Development Opportunities... 47 Abt Associates Inc. List of Exhibits

Exhibit 4.1: Current Status of Former Fellows, by Fellowship Degree Level... 51 Exhibit 4.2: Former Fellows Perceptions of GK-12 Experience s Influence on Academic Progress and Career Interests... 53 Exhibit 4.3: Former Master s Fellows Subsequent Graduate Study, by Program Type... 54 Exhibit 4.4: Former Fellows First Choice for Full-Time Post-Graduate Position, by Degree Level... 55 Exhibit 4.5: Former Fellows First Post-Graduate Positions, by Degree Level... 56 Exhibit 4.6: Former Fellows Perceptions of Preparation for Multiple Employment Sectors... 57 Exhibit 4.7: Former GK-12 Fellows Current Employment Status... 58 Exhibit 4.8: Former Fellows Current Employment Setting, by Degree Level... 59 Exhibit 4.9: Types of IHEs in Which Former Fellows are Currently Employed, by Degree Level... 60 Exhibit 4.10: Types of Positions Currently Held by Former Fellows in IHEs, by Degree Level... 60 Exhibit 4.11: Employment of Former Fellows in Government Positions... 61 Exhibit 4.12: Primary Discipline Former Fellows Use in Current Work... 62 Exhibit 4.13: Academic Responsibilities of Former Fellows in IHEs, by Degree Level... 63 Exhibit 4.14: Research Former Fellows are Responsible for in Current Work, by Degree Level... 64 Exhibit 4.15: Former Fellows Engagement in Research Associated with STEM Education... 64 Exhibit 5.1: Reasons to Apply for a GK-12 Fellowship... 70 Exhibit 5.2: Additional Sources of Financial Support, by GK-12 Status... 71 Exhibit 5.3: Expected Graduation and Desired Employment Outcomes, by GK-12 Status... 72 Exhibit 5.4: Communication, Teaching, Teamwork, and Research Activities, by GK- 12 Status... 73 Exhibit 5.5: Perceived Benefits to Graduate Students Ability to Engage in Communication, Teaching, Teamwork, and Research Development, by GK-12 Status... 75 Exhibit 5.6: Average Hours Per Week Spent on Academic and Research Activities, by GK-12 Status... 77 Exhibit 5.7: Frequency of Volunteering and Work Activities, by GK-12 Status... 78 Exhibit 5.8: Receipt of Funding and Other Honors/Awards, by GK-12 Status... 79 Exhibit 5.9: Career Goals, by GK-12 Status... 80 Exhibit 6.1: GK-12 Fellows Perceptions of the Benefits Associated with GK-12 Experience, by Fellow Type... 83 Exhibit 6.2: Fellows Perceptions of the Role of Scientists and K-12 Teachers, by Fellow Type... 84 Exhibit 6.3: Influence of GK-12 on Current Fellows Academic Progress... 85 Exhibit 6.4: Influence of GK-12 on Current Fellows Professional Interests... 86 Exhibit 6.5: Advisor Involvement in K-12 Outreach Activities... 87 Exhibit 6.6: Degree of Departmental Reward for Engaging in GK-12 Activities... 88 Exhibit 6.7: Types of Active Participation for Advisors in GK-12 Activities... 88 List of Exhibits Abt Associates Inc.

Exhibit 6.8: Advisors Reports of Perceived Effect of GK-12 on Improving Advisee(s) Abilities... 89 Exhibit 6.9: Advisors Reports of Perceived Effect of GK-12 on Participating Fellows... 90 Exhibit 6.10: GK-12 Effects on Training of All Graduate Students, Reported by PIs... 92 Exhibit 6.11: Ways Non-Fellows Participate in GK-12, Reported by PIs... 93 Exhibit 6.12: Advisors Perceived Benefits of GK-12 to Other Graduate Students... 93 Exhibit 6.13: Perceived Effect of GK-12 on Increasing Teachers Content Knowledge... 95 Exhibit 6.14: Opinions of Influence of GK-12 on Pedagogical Techniques... 96 Exhibit 6.15: Teachers Perceived Effect of GK-12 Participation... 97 Exhibit 6.16: Teachers Perceived Benefits and Drawbacks of Fellow in Classroom... 98 Exhibit 6.17: Teachers Perceived Greatest Challenge and/or Disappointment of GK-12 Experience... 99 Exhibit 6.18: Teachers Perceived Greatest Impact of GK-12 Experience... 100 Exhibit 6.19: Teachers Perceived Impact of GK-12 on Students Knowledge and Skills... 101 Exhibit 6.20: Impact of GK-12 on Students Interest in STEM... 102 Exhibit A.1: Distribution of the Population of Fellows A by Strata... A-1 Exhibit A.2: Sample Sizes Required to Detect Differences between Subgroups of PhD Fellows... A-2 Exhibit A.3: Sample Sizes Required to Detect Differences between Subgroups of Master s-level Fellows... A-2 Exhibit A.4: Distribution of the Sample of Fellows A (Former Fellows)... A-3 Exhibit A.5: Sample Size Requirements for Comparison Group... A-4 Exhibit A.6: Representativeness of the Sample of Faculty Advisors... A-7 Exhibit A.7: Estimated Distribution of K-12 Teachers Recruited for the Study... A-8 Exhibit A.8: Final Analytic Sample, by Degree Level... A-9 Exhibit B.1: Contributors to the Original Proposal and Design of GK-12 Projects... B-1 Exhibit B.2: GK-12 Institution or Organization Partners... B-1 Exhibit B.3: Degree of Faculty Support in PI s Department for GK-12 Project... B-2 Exhibit B.4: Degree of Faculty Support in other STEM Departments for GK-12 Project... B-2 Exhibit B.5: Degree of Faculty Support in Education Departments for GK-12 Project... B-3 Exhibit B.6: Degree of University Administration Support for GK-12 Project... B-3 Exhibit B.7: Institutional Resources for GK-12 Projects... B-4 Exhibit B.8: Outside Sources of Funding for GK-12 Projects, Reported by PIs... B-4 Exhibit B.9: Fellow s Discipline Aligned with Graduate Program, by Fellow Type... B-5 Exhibit B.10: Proportion of Former Master s Fellows Reporting Disciplines Most Closely Aligned with Their Graduate Program (Ranked First through Fifth)... B-6 Exhibit B.11: Proportion of Former Doctoral Fellows Reporting Disciplines Most Closely Aligned with Their Graduate Program (Ranked First through Fifth)... B-7 Exhibit B.12: Proportion of Current Master s Fellows Reporting Disciplines Most Closely Aligned with Their Graduate Program (Ranked First through Fifth)... B-8 Exhibit B.13: Proportion of Current Doctoral Fellows Reporting Disciplines Most Closely Aligned with Their Graduate Program (Ranked First through Fifth)... B-9 Exhibit B.14: Degree Types of GK-12 Fellows... B-10 Exhibit B.15: Similarity of Fellows GK-12 Activities across Educational Levels... B-10 Exhibit B.16: Initial Funding of GK-12 Fellowship, by Fellow Type... B-10 Abt Associates Inc. List of Exhibits

Exhibit B.17: Experiences Before Entering Graduate School at GK-12 Institution, by Fellow Type... B-11 Exhibit B.18: Faculty Advisors Initial Level of Support for GK-12, Reported by Fellows... B-12 Exhibit B.19: Sources of Research Fellowship, Traineeship, or Assistantship Turned Down to Accept the GK-12 Fellowship, by Fellow Type... B-13 Exhibit B.20: PI Provision of Formal Training or Education to Fellows... B-14 Exhibit B.21: Format of PI s Formal Training or Education to Fellows... B-14 Exhibit B.22: Leaders of Fellows Training or Educational Activities... B-15 Exhibit B.23: Topics Addressed in Fellows Training or Educational Activities... B-15 Exhibit B.24: Fellows GK-12 Assignments... B-16 Exhibit B.25: Years Taught by GK-12 Teachers at the K-12 Level... B-16 Exhibit B.26: Gender, Reported by Teachers... B-17 Exhibit B.27: Race/Ethnicity, Reported by Teachers... B-17 Exhibit B.28: Demographic Characteristics of Public School Teachers in Academic Year 2003-04... B-17 Exhibit B.29: GK-12 Teachers Current Subject Area of Teaching... B-18 Exhibit B.30: Elementary School Teachers Field of Study, by Highest Degree Obtained... B-18 Exhibit B.31: Middle School Teachers Field of Study, by Highest Degree Obtained... B-19 Exhibit B.32: High School Teachers Field of Study, by Highest Degree Obtained... B-20 Exhibit B.33: Years of Involvement in GK-12... B-21 Exhibit B.34: Number of Fellows with Whom Teachers Have Worked... B-21 Exhibit B.35: Grade Level and Subject Area of GK-12 Involvement, Reported by Teachers... B-22 Exhibit B.36: Frequency of Interaction with GK-12 Program Participants... B-22 Exhibit B.37: Teacher Training or Professional Development Topics for K-12 Teachers... B-23 Exhibit B.38: Leaders of K-12 Teacher Training or Professional Development Activities... B-23 Exhibit B.39: Teachers Participation in GK-12 Related Training... B-24 Exhibit B.40: Professional Development/Learning Sessions, by Type of Training... B-24 Exhibit B.41: Attendance in Professional Conferences Outside of GK-12... B-25 Exhibit B.42: Other Interactions with GK-12 Fellows/Faculty... B-25 Exhibit C.1: GK-12 Effect on Fellows Degree Completion Time, by Degree Type... C-1 Exhibit C.2: Fellows Perceptions of GK-12 s Influence on Their Academic Progress and Career Interests, by Degree Type... C-2 Exhibit C.3: Former Master s Fellows Subsequent Graduate Degree Status... C-3 Exhibit C.4: Enrollment in College Courses and Enrollment Status Since Most Recent Degree for 2001 and 2002 Master s Degree Recipients, by Major Degree Field... C-4 Exhibit C.5: Current Career Goals of Former Fellows Currently Enrolled in a Graduate Program... C-5 Exhibit C.6: GK-12 Former Fellows Perceived Influence of the GK-12 Fellowship on Their Interest in Selected Activities, by Fellow Type... C-6 Exhibit C.7: Fellows Enrollment Status... C-7 Exhibit C.8: Reasons for Not Having Held a Full-Time Position Since Graduate School... C-7 Exhibit C.9: Time Between Leaving GK-12 Institution to Obtaining Full-Time Position Aligned with Career Goals, by Degree Type... C-8 Exhibit C.10: GK-12 Fellows Alignment Between Career Goals and First Full-Time Position, by Degree Type... C-9 List of Exhibits Abt Associates Inc.

Exhibit C.11: Level of Agreement of GK-12 Experience as Fellows Competitive Edge in Full-Time Position Applications, by Degree Type... C-9 Exhibit C.12: Primary Reasons for Not Currently Working, by Degree Type... C-10 Exhibit C.13: Self-Employment Status, by Degree Type... C-10 Exhibit C.14: Type of Self-Employment... C-11 Exhibit C.15: Graduate Program Primary Discipline, by Type of Employer and by Degree Level... C-11 Exhibit C.16: Former Fellows Current Employment Setting, by Degree Level and Graduate Program Primary Discipline... C-12 Exhibit C.17: Former Fellows Current Employment Setting, by Degree Level and Primary Discipline Used in Current Work... C-13 Exhibit C.18: Employer Type of Private Foundation or Philanthropic Institution... C-13 Exhibit C.19: Employer Type of Medical or Health Care Organization... C-14 Exhibit C.20: Current Academic Appointment... C-14 Exhibit C.21: Current Academic Tenure Status... C-14 Exhibit C.22: Primary Function of Fellows Employed in Industry/Business or Self- Employed... C-15 Exhibit C.23: K-12 Academic Position at Preschool, Elementary, or Secondary School... C-15 Exhibit C.24: Primary Work Responsibilities, by Degree Type... C-16 Exhibit C.25: Frequency of Former Master s Fellows Engagement in Professional Activities... C-17 Exhibit C.26: Frequency of Former Doctoral Fellows Engagement in Professional Activities... C-18 Exhibit D.1: Advisors Views on Graduate Student Engagement in Educational Outreach Outside University... D-1 Exhibit D.2: Advisors Perception of GK-12 Effect on Fellows Degree Completion... D-1 Exhibit D.3: Departments Encouragement of Graduate Student Involvement in K-12 Outreach... D-1 Exhibit D.4: Non-GK-12 Fellows Participation in GK-12 Project... D-2 Exhibit D.5: GK-12 Benefit for Other (Non GK-12) Graduate Students... D-2 Exhibit D.6: How GK-12 Program Might Benefit Non-Fellows... D-2 Exhibit D.7: Intent to Continue GK-12 Activities after Grant Funding Ends (Active PIs Only)... D-3 Exhibit D.8: Elements of GK-12 Projects Active PIs Plan to Continue... D-3 Exhibit D.9: Continuation of GK-12 Activities After Grant Funding Ended (Inactive PIs Only)... D-4 Exhibit D.10: Sustained Elements of Inactive GK-12 Projects (Inactive PIs Only)... D-4 Exhibit D.11: Influence of GK-12 on Attainment of Advanced Degree... D-5 Exhibit D.12: Overall Impression of Working with Fellows... D-5 Exhibit D.13: Impact of GK-12 on Students, as Reported by Elementary-Only Level Teachers... D-6 Exhibit D.14: Impact of GK-12 on Students, as Reported by Secondary Level Teachers... D-7 Exhibit D.15: K-12 Students Interactions with Fellows... D-8 Exhibit D.16: Impact of GK-12 on K-12 Students Perceptions of Who Can be Mathematicians/ Scientists... D-8 Abt Associates Inc. List of Exhibits

List of Exhibits Abt Associates Inc.

Executive Summary Program Background The Graduate Science, Technology, Engineering, and Mathematics (STEM) Fellows (GK-12) program (formerly the Graduate Teaching Fellows in K-12 Education) was established in 1999. Operating within the National Science Foundation s (NSF) Division of Graduate Education, the GK- 12 Program is designed to improve graduate students communication, teaching, collaboration, and teamwork skills; provide professional development opportunities to K-12 teachers; enrich the learning of K-12 students; and foster stronger partnerships between institutions of higher education and local school districts. It also provides grants to institutions of higher education (IHEs) to fund STEM graduate students through GK-12 Fellowships. Since its inception through 2008, approximately 250 GK-12 awards were funded; some projects were funded for three years, and many received subsequent continuation funding that allowed them to operate for more than the initial three years of funding. Based upon data from the GK-12 Monitoring System, the GK-12 Program has supported well over 7,400 Fellows (within the projects included in this study), and those Fellows have worked in nearly 5,000 schools with over 9,000 K-12 teachers, ultimately impacting more than 600,000 K-12 students. The Evaluation of the GK-12 Fellowship Program In 2006, NSF contracted with Abt Associates Inc. and its subcontractors, Horizon Research and ICF (formerly ORC) Macro, to conduct a retrospective evaluation of the GK-12 program. The study is designed to provide detailed information about the experiences of GK-12 participants over the history of the program, to document the education and early career outcomes of former GK-12 Fellows, and to explore whether there are significant differences in the educational experiences of current GK-12 Fellows compared to other graduate students in their home departments. It also considers outcomes for the K-12 teachers who participate in the program. The current study has capitalized upon the fact that the GK-12 program has been operational for a decade, and many of the Fellows from early cohorts have since graduated and entered the workforce. This allows us to describe not only Fellows graduate school experiences, but their early career trajectories as well. We include in this study a quasi-experimental comparison group matched by institution, department, and degree type (master s, Ph.D.) to examine the experiences of Fellows and their graduate student peers, and to capture both the kinds of training received and the perceived benefits to professional preparation. Key research questions ask whether the program has been implemented as intended, and include: How do GK-12 Fellows describe the key elements of their GK-12 participation such as professional preparation and retention in graduate school? How do GK-12 Fellows and their comparison peers describe their graduate school experiences and post-graduate school plans? How do participating K-12 teachers describe the pedagogical and content resources provided by the GK-12 Program and the GK-12 Fellows? Abt Associates Inc. Executive Summary i

How do participating K-12 teachers perceive the influence of GK-12 projects on their students and schools? Evaluation Approach To address these research questions, we use both descriptive and comparative analyses of data from key GK-12 stakeholders. Specifically, data were collected during the winter and spring of 2009 via online surveys from the following respondent groups: Principal Investigators from all projects funded between FY 2000 and FY 2007 (n=144; response rate of 81 percent) Sample of GK-12 Fellows (former Fellows) from FY 2000 to FY 2004 projects (n=865; response rate of 51 percent) Sample of faculty advisors of these Fellows (n=352; response rate of 50 percent) Currently-funded GK-12 Fellows (current Fellows) at projects active as of spring 2009 (n=592; response rate of 89 percent) Sample of comparison non-gk-12 graduate students matched to current Fellows by the type of degree-candidacy (master s or Ph.D.) and institution (n=702; response rate of 61 percent) Sample of K-12 teachers affiliated with projects active as of spring 2008 (n=740; response rate of 53 percent) Major Findings The GK-12 Program is being implemented and experienced as intended. The projects are providing funded individuals with the range of activities outlined in the program solicitations related to communication, teaching, teamwork, and research skills. The process of selecting teachers is aligned with the program's intent: the majority of PIs reported that their top criteria for selecting participating teachers include selecting those who demonstrate a willingness to participate, those who work in high need/high poverty schools, and those who work in schools with administrative support. Participants' Experiences Project participants reported having had rich experiences in their GK-12 projects and classrooms. Specifically: According to PIs, GK-12 projects were designed to enhance Fellows ability to teach STEM research and concepts to K-12 students, communicate STEM concepts to the public, and work in teams. Further, the majority of PIs reported providing training to Fellows on precisely these same kinds of skills. The overwhelming majority of PIs reported that their projects are intended to improve participating teachers STEM content knowledge, skills/techniques for STEM instruction, and participation in STEM-related professional development. Nearly three-quarters of PIs reported that their projects prefer to select teachers from high needs or high poverty schools. ii Executive Summary Abt Associates Inc.

Most Fellows reported that they worked with teachers and students in regular K-12 school settings. Nearly two-thirds of Fellows were assigned to work in schools serving middle and/or secondary grade students as part of their GK-12 experience. The majority of Fellows reported that they engaged in multiple teaching activities with K-12 students, including presenting lessons or lectures, designing or modifying lessons or instructional modules, and leading small group activities. More current than former GK-12 Fellows reported that they rely upon current research from their graduate work or field of study for content of lessons and activities in K-12 classrooms (54 and 37 percent, respectively). Nearly all participating K-12 teachers took advantage of the professional development provided through GK-12, learning about STEM instructional strategies, training on math/science content and access to research about STEM. Nearly one-third of teachers reported that they had attended professional STEM-related conferences above and beyond those supported directly through their respective GK-12 projects. Degree Completion of Former Fellows and Early Post-Fellowship Outcomes What we learned from a sample of former Fellows suggests that the majority of master s degree candidates complete their graduate education in a timely fashion (of those who have since graduated), and that the doctoral candidates time to degree attainment is in line with national estimates. The vast majority of former master s degree Fellows had completed their degrees by Spring 2009, and 50 percent of the former master s students who had since completed their degrees had done so within 2.7 years of their initial enrollment as master s degree students. Approximately one-third of former Ph.D. Fellows were still enrolled in graduate school when the study fielded surveys in spring 2009, and over half (58 percent) of former Ph.D. degree Fellows had completed their degrees by spring 2009. For those graduated students, the median time to degree was 6.0 years from the date of initial enrollment as doctoral students. Forty percent of former master s Fellows subsequently enroll in another graduate program, primarily doctoral degree programs. Nearly three-quarters of former doctoral Fellows are employed in higher education, and slightly more than half are engaged in scientific or technical research. The proportions differ for former master s Fellows: nearly 40 percent are employed in business or industry settings, about one-quarter are employed in higher education (master s Fellows reported being employed in both teaching and administrative positions), and nearly 20 percent reported being employed in government agencies. Approximately one-third of former master s Fellows report that they are engaged in scientific or technical research. Perceived Professional Preparation of Former Fellows About two-thirds of the former Fellows agreed that their GK-12 experience gave them a competitive edge when applying for full-time positions after graduate school. The majority of former Fellow graduates quickly found employment that met their career goals; 67 percent of master s and 81 percent of doctoral Fellows were employed within three months of their graduation. Abt Associates Inc. Executive Summary iii

Most Fellows (77 percent of master s and 86 percent of doctoral Fellows) reported that their first full-time position was aligned with their career goals at the time of graduation. More than half of the Fellows reported being somewhat or very prepared for work in multiple employment settings. Nearly all former Fellows reported being prepared to work in a college or university (96 percent), and the majority also reported that their graduate work had prepared them for government work (87 percent) and K-12 district/school jobs (81 percent). Comparison of Current Fellows with Similar Graduate Students We conducted exploratory analyses comparing currently enrolled (as of spring 2009) GK-12 Fellows to other currently enrolled graduate students in the same departments. There are observed differences between GK-12 Fellows and their comparison peers on many dimensions of graduate school, including how they pay for graduate education, the nature of activities pursued during graduate school, and plans for the future. GK-12 Fellows at both master s degree and Ph.D. levels reported having received financial support for their education from fewer sources, and spending more time on activities related to communication, teaching, and teamwork than their counterparts; differences related to research activities were decidedly more mixed. Aspects of Graduate Education Fellows were significantly less likely to report that they rely on other sources of financial support for graduate education than were comparison students. Comparison students reported significantly less engagement in teaching STEM content or methods to K-12 students, and significantly greater engagement in teaching undergraduate students than Fellows. GK-12 Fellows reported that they conducted independent research and developed a research or technology agenda more frequently than their non GK-12 peers. Comparison students spent significantly more time than GK-12 Fellows preparing patent or copyright applications. GK-12 Fellows, on average, reported significantly more frequent involvement in communication and teamwork activities than non-gk-12 students; they reported more frequent experience working on a team, leading a team, facilitating group discussions, greater involvement in explaining STEM concepts to non-technical audiences, and presenting work at professional conferences. GK-12 Fellows were more likely to have received a teaching award at both master s and doctoral levels (9 and 19 percent for master s and doctoral Fellows, compared to 3 and 9 percent, respectively, for non-gk-12 Fellows). Master s level Fellows were more likely to have received a research award or honor than their comparison peers (25 and 15 percent, respectively). Perceived Benefits of Graduate Education Doctoral-level Fellows were more likely to agree that their graduate experiences had benefited their ability to teach undergraduate students despite reporting less frequent involvement in undergraduate teaching than were comparison doctoral students. iv Executive Summary Abt Associates Inc.

GK-12 Fellows were more likely to agree that their graduate training had benefited their ability to explain STEM concepts to non-technical audiences than were their non-gk-12 peers. Post-Graduation Career Goals GK-12 Fellows in both master s and Ph.D. programs were more likely than non-gk-12 students to report career goals that include teaching-related employment; there are no significant differences with respect to interest in research-related and other careers. Program's Perceived Effectiveness Overall, across all respondent groups, the GK-12 Fellowship is perceived as helping Fellows develop important skills related to communication, teaching, and teamwork; the evidence about the role of the Fellowship in helping Fellows develop research skills is more mixed. There is modest evidence to suggest that spillover of the GK-12 program into graduate coursework or requirements for other graduate students has occurred. From the teachers perspective, working with the Fellows provides multiple benefits, not only in terms of their own STEM content knowledge, access to STEM resources, and for their students, but in terms of providing benefits to the Fellows. A majority of Fellows indicated that the GK-12 experience benefited their ability to conduct various communication, teaching, and teamwork tasks, which align with NSF s GK-12 program goals. A majority of Advisors also indicated that the GK-12 program helps to develop Fellows communication, teaching, and teamwork skills. Specific highlights are presented below. Perceived Effects on Fellows, Higher Education Institutions, K-12 Teachers, and Students Findings on the effects of GK-12 on Fellows research skills were mixed. A majority of current Fellows indicated that their GK-12 experience benefited their research skills, while a majority of both former Fellows and Advisors disagreed that the GK-12 experience had benefited Fellows research skills. More than 90 percent of former and current Fellows reported that interaction between STEM K-12 teachers and researchers is mutually beneficial. Over four-fifths of PIs reported that they plan to sustain some elements of their GK-12 projects once funding has ceased; of the PIs who expect to continue GK-12-like activities, two-thirds or more indicated that they plan to continue to provide opportunities for graduate students to volunteer in K-12 settings, collaborate with partner organizations, and provide professional development to K-12 teachers. About half of the PIs reported that graduate training in their departments/institutions had changed as a result of GK-12, through such mechanisms as new outreach opportunities for graduate students, or changes in courses available to all students (among other mechanisms). About one-third of Advisors reported that GK-12 had benefitted other (non-gk-12) graduate students. A majority of teachers reported that GK-12 had a range of positive effects, including increasing their STEM content knowledge, their use of more effective pedagogical techniques, their access to STEM resources, their confidence and preparedness to teach, and on their students STEM knowledge and skills. Abt Associates Inc. Executive Summary v

The greatest benefit of the GK-12 program reported by teachers was Fellows pedagogical competence and practice. The majority of teachers reported that there were no drawbacks to the GK-12 program, and no teachers reported negative effects on students. vi Executive Summary Abt Associates Inc.

Chapter 1: Overview of the GK-12 Program and Evaluation The GK-12 The GK-12 Program: Scope, Program: Scope, Goals, Evolution Goals, Evolution The GK-12 Study The GK-12 Study Design, Sample, Design, Sample, and Methods and Methods GK-12 Program GK-12 Program Components and Components and Experiences Experiences Former Fellows' Former Fellows' Education and Education and Employment Employment Outcomes Outcomes How Do Current How Do Current Fellows Compare Fellows Compare To Their Peers? To Their Peers? Self-Reported Self-Reported Effects of the Effects of the GK-12 Program GK-12 Program Summary Summary Introduction The Graduate Teaching Fellows in K-12 Education (GK-12) program supports the education of graduate students in science, technology, engineering, and mathematics (STEM) disciplines, who in return, interact with teachers and students in K-12 schools and other educational settings. The program provides grants to institutions of higher education (IHEs) to fund STEM graduate students. The GK-12 Program is managed by the National Science Foundation s (NSF) Division of Graduate Education and is supported by all NSF Directorates and officers, GK-12 is designed to improve graduate students communication, teaching, collaboration, and teamwork skills; provide professional development to K-12 teachers; enrich the learning of K-12 students; and foster the development of partnerships between institutions of higher education and local school districts. Participating institutions of higher education are also encouraged to incorporate GK-12 like activities into the training of all of their STEM graduate students. In 2006 NSF contracted with Abt Associates, Inc. and its subcontractors, Horizon Research and ICF (formerly ORC) Macro, to conduct a nationwide retrospective evaluation of the GK-12 program. This evaluation study has three purposes: 1. To describe the experiences of past and present GK-12 participants, including Fellows and K-12 teachers; 2. To document the education and early career outcomes of past GK-12 Fellows; and 3. To compare the educational experiences of current GK-12 Fellows with other students in their home departments. This is the final report for this important evaluation. The report begins with an overview of GK-12 program goals, objectives, funding mechanisms, and participants. It also includes a brief summary of prior evaluation results. Data for this chapter come from GK-12 program solicitations, conversations with NSF program officers, and the GK-12 program Distance Monitoring System, which collects annual information from GK-12 projects and participants. Abt Associates Inc. Chapter 1: Overview of the GK-12 Program and Evaluation 1

GK-12 Program Goals GK-12 made its first awards in 1999. Since that time there have been 10 program solicitations. The goals of the program as outlined in the 2007 program solicitation 1 are to: 1. Support highly qualified graduate students in NSF-supported STEM disciplines through Fellowships to provide them with an opportunity to acquire additional skills that will broadly prepare them for professional and scientific careers in the 21st century; 2. Improve STEM instruction in K-12 schools; and 3. Provide institutions of higher education with an opportunity to make a permanent change in their graduate programs by incorporating GK-12 like activities in the training of their STEM graduate students. To accomplish these goals, GK-12 supports the development of partnerships between institutions of higher education and K-12 schools (or other educational organizations). Within the framework of these partnerships, STEM graduate students (or Fellows ) interact with K-12 students and teachers in a variety of instructional and research activities designed to increase the STEM content knowledge, interest, and learning of K-12 participants while developing the communication, teaching, collaboration, and teamwork skills of the graduate Fellows. To maximize skill development, GK-12 projects often provide training to Fellows either before or during their GK-12 involvement; they also often provide professional development to K-12 teachers on STEM-related topics. Faculty members in participating STEM departments including Principal Investigators are encouraged to incorporate activities developed for or inspired by GK-12 into the graduate training of all STEM graduate students. Evolution of Program Goals Since its inception, the GK-12 program has evolved to reflect both practical experiences from the field as well as heightened sensitivity to the importance of particular program features. Over the past decade, NSF s and the GK-12 community s understanding of the program goals has evolved, an evolution that is reflected in subtle modifications in the text of program solicitations over the years around two key emphases: 1. Deepening the emphasis of the program to provide professional training to graduate students through interactions with K-12 teachers and students, and 2. Increasing the focus of the program on STEM research (while still attending to pedagogy), both that of participating faculty members and of Fellows. Training graduate students. The GK-12 solicitations have placed increasing emphasis on the professional benefits to Fellows themselves of participating in GK-12, in addition to serving as educational resources for K-12 teachers and students. The 2000 solicitation added instruction for projects to provide appropriate training to Fellows. While solicitations have always stressed the development of communication and teaching skills for Fellows, the 2004 solicitation also 1 NSF Graduate Teaching Fellows in K-12 Education (GK-12) Program Solicitation NSF 07-555. http://www.nsf.gov/pubs/2007/nsf07555/nsf07555.htm. 2 Chapter 1: Overview of the GK-12 Program and Evaluation Abt Associates Inc.

emphasized the development of collaboration and teamwork skills. In 2005, the program description changed from a focus on supporting Fellowships and the associated training that would enable STEM graduate students to serve in K-12 schools as resources knowledgeable about both the content and applications of STEM to a focus on enabling Fellows to acquire additional skills that will broadly prepare them for professional and scientific careers in the 21 st century. Focusing on STEM research. Emphasis on the role of GK-12 Fellows and PIs as STEM researchers has increased. In 2006, NSF added text to the solicitation requiring grants to describe how GK-12 Fellows are bringing their research into schools, and in 2007 plans for incorporating Fellows research into the GK-12 project was added to the grant review criteria. The 2008 solicitation added the clarification that PIs must be actively conducting STEM research and in 2009, the program name was changed to Graduate STEM Fellows in K-12 Education (emphasis added). [Note that this evaluation began data collection after the 2008 and 2009 solicitations were released; the most recent program solicitation governing projects included in this evaluation was 2007.] Prior GK-12 Program Evaluation The GK-12 program was evaluated once prior to the current study. In 2000, NSF contracted with the American Institutes for Research (AIR) to conduct a program evaluation of GK-12. 2 Using a mixedmethods approach, AIR focused primarily on project implementation and early outcomes, using data collected during the 2002-03 school year. Evaluation data sources included a critical incident study focused on partnership-building and institutionalization of projects; surveys of PIs, Fellows, comparison graduate students, K-12 teachers, and school district staff; and site visits to 21 award sites. The earlier evaluation effort focused on award implementation understanding the factors associated with successful implementation and the impact of GK-12 awards on establishing and strengthening IHE / K-12 partnerships. Effective projects were found to be those that recruited and selected strong candidates for GK-12, implemented strategies for matching Fellows to teachers, provided training to Fellows, included K-12 staff in grant planning and implementation, and allowed teachers to have a high level of influence on Fellows work in the classroom. In terms of early program impacts, GK-12 participants reported gains in Fellows communication and instructional skills. Although the evaluation included a comparison group of non-gk-12 students that was matched to the Fellow sample by STEM discipline and having the same faculty advisor, the group was not matched on degree level, and as a result, the Fellow sample had significantly fewer Ph.D. students (39 percent) than the comparison sample (49 percent). 3 Therefore, the comparison group was used primarily to compare Fellows and non-fellows perceived supports within their graduate programs and anticipated time to degree, but not to measure skills or outcomes gained. The current study has been able to capitalize upon the fact that the GK-12 program has been operational for a decade, and many of the Fellows from early cohorts have since graduated and 2 3 American Institutes for Research, Final Report on the Evaluation of the National Science Foundation s GK- 12 Fellowship Program: Volume I (Draft), 2005. American Institutes for Research, Survey Analysis of the Impact of the GK-12 Fellowship Program: Summary Report, May 2004. Abt Associates Inc. Chapter 1: Overview of the GK-12 Program and Evaluation 3

entered the workforce. This allows us to describe not only Fellows graduate school experiences, but their early career trajectories as well. We include in this study a quasi-experimental comparison group matched by institution, department, and degree type (master s, Ph.D.) to examine the experiences of Fellows and their graduate student peers, and to capture both the kinds of training received and the benefits to professional preparation. Program Participants GK-12 awards are granted to academic institutions in the United States and its territories that award master s or doctoral degrees in STEM disciplines supported by the National Science Foundation. Since 1999, approximately 144 institutions have received a GK-12 award. These institutions then partner with K-12 educational organizations (schools, museums, agencies, etc.). GK-12 participants include STEM faculty members and graduate students, the IHE s Education faculty members, K-12 teachers and educators, and individuals providing administrative support to the grants. Project participants may be drawn from one or multiple STEM disciplines and/or departments or other academic units, and NSF encourages submission of proposals for projects organized around interdisciplinary or multidisciplinary themes. Since the GK-12 program began in 1999, the NSF has supported approximately 28 new GK-12 grants every year (Exhibit 1.1). In its first four years of operation (1999 2002) GK-12 awards were in the range of $200,000 to $500,000 per year for two to three years. In 2003, Track 2 continuation awards were introduced, which provided some grantees with up to an additional five years of support; the upper annual funding limit was also increased to $600,000. In 2006, a transition year, all new awards were granted for a period of five years, Track 1 grants could apply for another three years of funding, and Track 2 continuation awards were eliminated. Over the first ten years of the program s operation, GK-12 projects received between two to eight years of support from the NSF. Exhibit 1.1: GK-12 Award Characteristics, by Year of Program Implementation (1999-2007) Program Year Funding Year # New Awards Award Amount Grant Duration 1999 2000 31 $200,000 $500,000/year 2 3 years 2000 2001 25 $200,000 $500,000/year 2 3 years 2001 2002 23 $200,000 $500,000/year 2 3 years 2002 2003 36 $300,000 $500,000/year 2 3 years 2003 2004 34 Track 1: $300,000 $600,000/year Track 2: $500,000/year Track 1: 2 3 years Track 2: 5 years 2004 2005 20 Track 1: $300,000 $660,000/year Track 2: $500,000/year Track 1: 2 3 years Track 2: 5 years 2005 2006 31 Track 1: $300,000 $660,000/year Track 2: $500,000/year Track 1: 2 3 years Track 2: 5 years 2006 2007 25 $600,000/year New awards: 5 years Continuing: 3 years 2007 2008 26 $600,000/year 5 years Total, all years 251 Exhibit reads: In the program year 1999 and funding year 2000, 31 new GK-12 awards were made, ranging between $200,000 and $500,000 per year with a grant duration of 2 and 3 years. Source: GK-12 Distance Monitoring Data System. 4 Chapter 1: Overview of the GK-12 Program and Evaluation Abt Associates Inc.

The bulk of each GK-12 award supports GK-12 Fellows in the form of an annual Fellowship award that includes stipends and cost-of-education allowance. The annual Fellowship stipend was initially set at $18,000 in 1999, and has since been raised to $20,500 (in 2001), $21,500 (in 2002), and then to $30,000 (in 2004), where it remains. NSF also provides an annual cost-of-education allowance for tuition, health insurance, and normal fees of $10,500 per student (for 12 months). Direct support of graduate students is the primary financial benefit received by universities from GK-12 awards, as indirect costs on GK-12 awards are currently limited to eight percent of all Total Direct Costs, excluding Fellowship stipends, cost-of-education allowances (as of 2009), and equipment. Principal Investigators and Co-Principal Investigators Each GK-12 award is directed by a Principal Investigator (PI), who is required to be a STEM discipline faculty member at the lead institution. As stated previously, the 2008 solicitation added the clarification that PIs must be actively conducting STEM research. GK-12 PIs (and co-pis) come from a range of STEM disciplines (Exhibit 1.2); the most common broad STEM disciplines are: biological sciences, engineering, and chemistry. 4 The GK-12 PI is responsible for leading a team of IHE and K-12 participants to implement the proposed activities. PIs wear multiple hats; they are responsible for communication about the projects with the NSF; they lead the faculty members responsible for conceptualizing and implementing the project; they oversee the project s budget and coordinate with the GK-12 project coordinator or manager; they may also be the project s direct liaison to K-12 partners and/or advise GK-12 Fellows. Any disciplinary faculty or administrator at universities, or educators/administrators from K-12 schools, or partnership institutions may serve as Co-PI on a GK-12 grant. Co-PIs may help operate GK-12 awards, coordinate or lead activities, and/or mentor GK-12 Fellows. Exhibit 1.2: GK-12 Principal Investigators Departmental Affiliations, 1998 2007 a Discipline N (%) Biological science 108 (16) Engineering 82 (12) Chemistry 49 (7) Mathematics 41 (6) Geosciences 39 (6) Physics 33 (5) Behavioral science 14 (2) Social science 14 (2) Computer science 13 (2) Astronomy 3 (<1) Environmental science 3 (<1) Other b 279 (41) Total 678 (100%) Notes: a Reporting Principal Investigator for first year of funding. Does not account for changes in individual PIs departments or change in grant PIs. Base N=747; 69 respondents are missing this information. b Disciplines captured in the other category include: education, curriculum & instruction, and science education. Exhibit reads: Between 1998 and 2007, 108 GK-12 Principal Investigators (16 percent) indicated that their department was affiliated with biological science. Source: GK-12 Distance Monitoring Data System. 4 Note that data on PI discipline is from the DMS; we do not have more detailed information on the disciplinary categories represented in the other category. Abt Associates Inc. Chapter 1: Overview of the GK-12 Program and Evaluation 5