Evaluation of STEM Professional Development in Massachusetts CCSSO Meeting: Improving Evaluation of Professional Development May 5, 2005 Barbara Libby Director, Office of Math and Science Massachusetts Department of Education Eric Heller Director of Research & Evaluation UMass Donahue Institute PD Programs (STEM and other) Evaluated by Donahue Institute State Level: Title lib Math Science Partnership Programs Massachusetts Summer Content Institutes Massachusetts STEM Pipeline Title IIA Improving Teacher Quality Initiative Reading First Local Level: Early Reading First Teaching American History 2 Varying Roles vis-à-vis PD Evaluation Comprehensive statewide formative and summative program evaluation Cross-site data collection and analysis re: program implementation Technical assistance to support quality local evaluation Local evaluation planning and implementation 3
4 Basic Logic Model as Organizing Framework for Evaluation of PD PD and Support Activities Gains in Teacher Knowledge or Skills Desirable Classroom Practices Improved Student Outcomes Modified Logic Model Incorporating NCLB Teacher Qualifications Goal PD and Support Activities Gains in Teacher Knowledge or Skills Highly Qualified Teachers Desirable Classroom Practices Improved Student Outcomes 5 PD & Support Activities (Implementation Evaluation) Provider implementation reporting (e.g., end-of-course report) Syllabi / agenda Attendance records Standardized participant background survey (demographics, qualifications, district & school info, etc.) Participant feedback surveys 6
7 Gains in Teacher Knowledge/Skills (Short-term participant outcomes) Pre/post assessment of specific PD content directly aligned with learning objectives use of non-threatening, unique participant code to facilitate analysis, linking to other data sources easily scored Optional 3rd data point as follow-up e.g., towards end of subsequent school year potential measure of retention, deepening of knowledge gains poses logistical challenges for administration Major challenge: Identifying or developing quality assessment tools directly aligned with PD contents Changes in Teacher Qualifications (NCLB Highly Qualified) Self-report on Licensure status Passage of Massachusetts licensure exam (MTEL) Progress on/completion of PD plan Challenges Current lack of reliable, centralized data source Variation in quality, accessibility of district data sources Difficulties tracking participants post PD to survey status changes 8 Changes in Classroom Practices Structured observation Requires clear articulation of observable desired practices Requires valid and reliable observation tool Typically resource intensive Self-report Pre/post or retrospective survey or interviews Limited reliability, but still better than no data 9
10 Changes in Student Outcomes State standardized test results Other standardized assessments used by district PD content specific testing Major Challenges (too many to list, but include ) Limited sensitivity of assessment data Short-term program funding cycles vs. time required for changes in teaching to impact student results Research design issues, e.g., unit of analysis Rigorous Research Designs Desire for randomized experiments/quasi-experimental designs compromised by well-known field constraints Limited success in some Math Science Partnerships and one Teaching American History grant Most promising design scenario: Over enrollment of interested and eligible participants Random assignment to participant/non-participant groups OR If plan is for repetitive cycles, then baseline data collection among members of future cohorts as comparison group for current participants 11
The Commonwealth of Massachusetts Department of Education 350 Main Street, Malden, Massachusetts 02148-5023 Telephone: (781) 338-3000 Mathematics and Science Professional Development in Massachusetts A Sample of Two Programs I. Content Institute Program Across All Disciplines (started with Math and Science-PALMS) This summer more than 37 free graduate-level content institutes on topics ranging from mathematics to special education will be available for educators around the state. The institutes, offered by the Department of Education, will provide at least 45 hours of instruction and up to 20 hours of follow-up sessions in the fall and winter to support implementation of the content into classroom teaching and learning. Participants are required to attend all sessions, take part in preand post-assessments of content knowledge, and complete a project outside of class that demonstrates their learning. Teachers institute- related work should be included in their professional development plans and linked to school and district priorities. Since 1995, 11,000 educators have taken advantage of the summer courses to sharpen their instructional skills and knowledge in the academic subjects included in the state s learning standards. More than 900 K-12 teachers are expected to participate in this year s programs. All educators are eligible to participate. Priority is given to teachers from high-need districts and for those educators who need to become highly qualified in the subject area that they teach. Courses will be offered in mathematics, science, technology/engineering, history and social science, English language arts, the arts, and selected areas in special education. All of the institutes will have an instructional technology component such as Geometer's Sketchpad, graphing calculators, electronic probes, and online interactive software. Many institutes will use the Department's Virtual Education Space (VES) to share and discuss the content of the institute. It is critical that educators constantly work to improve their content knowledge and skills in the subject matter they teach, said Education Commissioner David P. Driscoll. We encourage all educators to take advantage of these free opportunities and to pursue other professional development activities as well. The content institutes will be offered from June through December 31, 2005. In partnership with colleagues from school districts, educational collaboratives, and charter schools, the courses will be taught by educators and specialists from colleges, universities and cultural institutions. Professional development points (PDPs) will be awarded which can be used toward becoming highly qualified and relicensed. Graduate credit is also available. Content institutes for science and technology/engineering will address topics such as renewable energy, climate change, engineering, physics, earth science, and chemistry. Offerings in mathematics will include linear equations, probability and statistics, measurement, fractions, algebra, and properties of geometric figures. Arts and humanities courses will include topics such as American history and government, the geography of Africa, explorers, literature, new technologies in the visual arts, art and architecture, and music.
A brochure is available at: http://www.doe.mass.edu/frameworks/cinstitute/. This brochure lists descriptions and registration information for the 2005 content institutes and other summer professional development offerings, including: Designing Assessments for the Standards-based Classroom and Advanced Placement Teacher Institutes. II. Title IIB: Massachusetts Mathematics and Science Partnership Program The Massachusetts MSP Title IIB Program has awarded 10 partnerships across the state for a three-year period, awards range from $50,000-300,000. The purpose of the MMSP is to increase the number of highly qualified teachers in the Commonwealth and to ensure that teachers continue to receive quality content-focused professional development, in order to improve classroom teaching and student achievement. MMSP partnerships must include a high need district and a mathematics, science, or engineering department from an institution of higher education. For program information: http://www.doe.mass.edu/omste/msp.html. Each MMSP partnership will develop and implement an effective and sustained course of study for inservice teachers of mathematics, science, and technology/engineering and integrate the course of studies into schools of education and/or arts and sciences at institutions of higher education. The University of Massachusetts Donahue Institute is the program s external evaluator. We have developed program -wide indicators and tools to collect baseline and ongoing data to analyze the impact of the partnership programs. A survey is collected from every participant that completes a course in order to collect data related to participants licensure, highly qualified status, type of district (high needs district have priority), the type of students they teach, etc. All participants are pre and post-tested to assess content knowledge gain. The Donahue Institute also provides technical assistance to each partnership regarding their evaluation/research design and they meet with the partners and the local evaluator of each program. We established a Title IIB Steering Committee with mathematicians, scientists, and math and science leaders in the state, they meet regularly with the Department of Education program staff and Donahue staff to advise and guide all aspects of the program. We also collaborate with the Title IIA Teacher Quality K-12 and Higher Education Programs and the two NSF MSP programs in our state. Last year in year 1, we convened a Math and Science Partnership Network meeting and invited representatives from all these programs to share successes and challenges. One of the partnerships titled Intensive Immersion Institutes in Mathematics for Grades 4-8 Teachers whose partners include: Fitchburg Public Schools, Gardner Public Schools, Leominster Public Schools, EduTron Corporation, Massachusetts Institute of Technology, Fitchburg State College and Quinsigamond Community College sponsored a grass roots conference that grew to over 200 participants including educators, students, district administrators, teachers and preservice teachers. Teachers who had participated in the partnership math courses planned the conference, shared their math work in the classroom and attended standing room only sessions on topics such as Singapore Math, New Degree and Course Changes at higher education institution partners, etc. Summaries of 3 of the 10 Funded Projects in 2005 1. Title: PROJECT SALEM - Educating Mathematicians Partners: *Salem State College, Lynn Public Schools, Salem Public Schools, and Haverhill Public Schools. Salem State College has developed and is starting to implement a Masters of Art in Teaching in Middle School Mathematics degree program. Teachers enrolled in this program through Title IIB funds will take 12 three-credit courses, two each summer and one each during the fall and spring Massachusetts Mathematics & Science Professional Development Programs 2
semesters. The Professional Standards for Middle School Mathematics Teachers will guide the development of these courses. Course proposals were developed by members of the Mathematics Department at Salem State College in consultation with distinguished members of the mathematics education community. The first courses were offered in Fall 2004. Courses make use of technology to enhance understanding and visualization of problems. This technology includes hand-held graphing calculators as well as CBL and CBR data collection devises. Dynamic geometry software such as Geometer's Sketchpad will also be used. All course material emphasizes topics important to the middle school curriculum or provides an understanding of why certain topics in middle school mathematics are needed for success in more advanced mathematics courses. Courses are taught by Salem State College faculty and are offered in Haverhill as well as on and off campus in Salem. Teachers enrolled in the program through Title IIB funding participate in an additional 20 hours of post-course mentoring and classroom implementation discussions and workshop. Non-grant funded teachers and other members of the public may also register for the courses developed for this degree and work toward their own MAT in Middle School Mathematics. For more project information, contact the program coordinator, Christopher Fauske. 2. Title: Intensive Immersion Institutes in Mathematics for Grades 4-8 Teachers Partners: Fitchburg Public Schools, Gardner Public Schools, Leominster Public Schools, *EduTron Corporation, Massachusetts Institute of Technology, Fitchburg State College and Quinsigamond Community College Based on the analyses of the students' and teachers' knowledge gaps in the partnering districts, three Intense Immersion Institute courses will be developed. Conceptual understanding, computational fluency, and problem solving will be emphasized with a balanced approach. These highly customized courses, aimed at deepening the mathematical understanding of grades 4-8 teachers, will be delivered annually. The math faculty from Fitchburg State College will work with the EduTron instruction team to integrate the courses of study into the FSC offerings in year 3. The partnership plans to reach about 120 teachers and impact (indirectly) more than 2000 students on a yearly basis. Assessment tools developed as an integral part of the course allow the partnership to simultaneously monitor student and teacher progress annually. In addition to the measurable content gain, the chemistry, dynamics, and positive peer pressure fostered in the intensive immersion experience will trigger qualitative changes in individual teachers to such an extent that some of them may become catalysts to transform their local math community into a learning machine! These transformations are expected to play a pivotal role in sustaining peer-based learning beyond the project span. The courses provide an intensive yet non-threatening environment for teachers to acquire solid content knowledge effectively. The outcomes should be: (1) more teachers will meet the NCLB Highly Qualified Requirements for Massachusetts Teachers, and (2) more students will perform at higher levels in MCAS. For more project information, contact the program coordinator, Andrew Chen, president, EduTron. 3.Title: The Coalition for Higher Standards Math Partnership Program Partners: *Lesley University; Mass Insight Education; University of Massachusetts at Boston; Malden Public Schools, Revere Public Schools, Saugus Public Schools, Peabody Public Schools, Gloucester Public Schools, and Canton Public Schools. This project focuses on improving teacher quality in Malden, Revere and Saugus, three neighboring "inner-ring" urban school districts of high need. Additionally, on a space available basis, this initiative also serves selected teachers from three other affiliated districts - Peabody, Gloucester, and Canton - which are members of the Coalition for Higher Standards, a statewide network of standards-focused school districts organized and administered by Mass Insight. Approximately 90 teachers from across these districts are currently enrolled in Year 2 of the program. The primary focus is on teachers in grades 4-8, with a small number of teachers from both lower and higher grades also enrolled in the program. The specific goal is to provide strong, intensive, and sustained professional development leading to improved instructional practice and Massachusetts Mathematics & Science Professional Development Programs 3
satisfying requirements for professional licensure and/or HOUSSE plans in Middle School Mathematics. Over the three years of this project, each teacher-participant will complete a program of study consisting of six courses, all of which emphasize strong mathematics content, consist of 48 hours of in-class instruction, award three Arts and Sciences graduate credits, and serve as models for effective classroom pedagogy. Thus, at the end of the three years, a teacher will have received a total of 18 graduate credits. In addition to the 48 class hours, each course includes 20 hours of school implementation in which content knowledge is applied to classroom practice. Lesley University has developed two 33-credit Masters Degree programs, one with a specialization in Middle School Mathematics and another with a specialization in Elementary Mathematics. Teacher-participants will be able to apply the 18 credits earned in this program toward either of these masters degrees. For those initially licensed teachers who already have a Masters degree, Lesley offers an approved 18-credit program leading to professional licensure. Also, for those teacher-participants who already have a Masters degree, Lesley is in the process of developing a program leading to a Certificate of Advanced Graduate Study (CAGS) with a specialization in Mathematics. For more information, contact the program coordinator, Matthew McClain, mmcclai2@lesley.edu Lesley University, 617-349-8359. Prepared by Massachusetts Department of Education Barbara Libby, Director Office for Mathematics, Science, and Technology/Engineering 350 Main Street Malden, MA 02148 Tel. 781-338-3460 Fax:781-338-3395 blibby@doe.mass.edu Massachusetts Mathematics & Science Professional Development Programs 4