Inquiry-Based Instruction in Iowa: A Report on the Implementation of Every Learner Inquires in Year 2

Transcription

1 Inquiry-Based Instruction in Iowa: A Report on the Implementation of Every Learner Inquires in Year 2 July 2008 Submitted to Kathy McKee Iowa Department of Education Prepared by Jonathan Margolin, Ph.D., Principal Investigator Megan Brown Shazia Miller 20 North Wacker Drive, Suite 1231 Chicago, IL _07/08

2 Contents Page Executive Summary... 1 Student Learning... 2 Teacher Practice... 3 Organizational Support... 3 Teacher Learning and Understanding... 4 Reactions to Professional Development Workshops... 4 Recommendations... 5 Introduction... 7 What Is Inquiry-Based Science Instruction?... 7 Program Design... 9 Program Goals Design of Professional Development Evaluation Overview Methods ELI Workshop Reaction Data Teacher Survey Student Survey Instructional Logs Classroom Observations Interviews With Principals and Teachers Section 1: Workshop Feedback Summary Workshop Attendance Workshop Topics and Activities Ratings of Quality and Utility Summary of Findings Section 2: Impact on Teacher Understanding Level of Understanding Self-Efficacy Summary of Teacher Learning and Understanding... 37

3 Section 3: Impact on Teaching Practice Inquiry Instruction Classroom Culture Learning Cycle Summary of Impact on Teaching Practice Section 4: Organizational Support Administrator Support Professional Learning School Resources Coherence Implementation Concerns Related to Organizational Support Summary of Organizational Support Section 5: Impacts on Student Learning Impact on Student Understanding of Scientific Concepts and Science Inquiry Impact on Student Engagement and Self-Efficacy Addressing the Needs of All Subgroups Summary of Impact on Student Learning Conclusions and Recommendations References Appendixes Appendix A. Survey Methodology Appendix B. Teacher Survey Frequency Tables Appendix C. Instructional Log and Observation Frequency Data Appendix D. Interview Protocols... 91

4 Executive Summary Every Learner Inquires (ELI), which runs from August 2006 through June 2010, is a statewide K 12 professional development effort to improve science learning in Iowa schools by promoting inquiry-based instruction. This program, currently involving teachers from four schools in four districts, provides a summer institute and five workshops per year; participants are expected to engage in job-embedded professional learning during the time between workshops. The goals of the initiative are to improve science learning for all students, build teacher leadership and content expertise in the area of science, provide teachers with the content and skills necessary to implement inquiry-based instruction, and establish a structure for sustained implementation. Under contract from the Iowa Department of Education, is conducting a mixed-method evaluation of the program. This report provides formative feedback on the outcomes during Year 2, based on findings from interviews with teachers and principals, a survey of teachers, observations of science lessons, and instructional logs. In addition, this report provides formative feedback on the professional development workshops offered during the winter and spring of Year 2. A separate report, to be delivered in spring 2009, will present findings from an analysis of student achievement data to describe student learning outcomes. The design of the ELI initiative calls for the training of teachers from four case study schools along with regional teams based in area education agencies (AEAs) who will expand the program statewide in Years 3 and 4. All science teachers in the four case study schools participate in the workshops, implement the instructional strategies presented therein, keep logs of their implementation, and use inservice time to support implementation. Eleven AEA leadership teams are participating in the program to build their capacity to deliver professional development in inquiry instruction; in the third and fourth years of the initiative, AEA teams are expected to begin to provide professional development for schools in their respective areas, building on the lessons learned from the case study schools. Each AEA team includes AEA science education staff and at least two teacher leaders drawn from schools within the jurisdiction of the given AEA. The focus of the ELI professional development is to promote inquiry-based instruction. This approach, articulated in the National Science Education Standards (National Research Council, 1996), provides students direct experience with the scientific inquiry process. Although the skills and understandings related to scientific inquiry vary according to grade levels, the Standards articulate the following essential features of inquiry instruction for all grade levels: Learners are engaged by scientifically oriented questions. Learners give priority to evidence, which allows them to develop and evaluate explanations that address scientifically oriented questions. Learners formulate explanations from evidence to address scientifically oriented questions. Learners evaluate their explanations in light of alternative explanations, particularly those reflecting scientific understanding. Learners communicate and justify their proposed explanations. Every Learner Inquires Year 2 Evaluation 1

5 Inquiry-based instruction is supported by a student-centered classroom culture in which teachers serve as facilitators of learning, promote discourse and discussion, encourage curiosity and questioning, and respect the diversity of their students. In light of the fact that teachers need guidance and structure to enact the Standards, the ELI program provides a pedagogical model for lesson design. This model, called the 5E Learning Cycle, articulates five phases of inquiry, and each phase corresponds to a different learning goal. During Year 2 of the ELI program, teachers were expected to implement two full learning cycles. The findings of the Year 2 evaluation are organized by questions aligned to five levels of program outcomes: student learning, teacher practice, organizational support, teacher learning and understanding, and reactions to professional development (Guskey, 2000). Student Learning 1. To what extent have students acquired abilities and fundamental understandings about scientific inquiry? 2. What is the impact of the ELI program on student enthusiasm and self-efficacy toward science learning? 3. Does the ELI program address the needs of all student subgroups? 4. Are the scientific understandings and abilities of students improving? Most teachers reported that inquiry instruction has improved student understanding of both scientific concepts and the scientific inquiry process, with the number who reported the latter outcome growing since last year. Several teachers stated that inquiry-based instruction has improved problem-solving skills in particular. Teachers and principals reported that inquiry instruction has improved the level of student engagement and enthusiasm in science; they mentioned that students appear to be more excited about attending science class than before. Elementary students themselves typically reported that they are enthusiastic toward science. However, high school students typically reported that they are not enthusiastic about science class. The overall level of enthusiasm toward science has not increased since Year 1. Regarding self-efficacy toward science, about half of elementary students and one fifth of high school students rated themselves as being capable to accomplish various tasks involving their science class. The ELI program appears to address the needs of all student subgroups. As reported last year, teachers and principals indicated inquiry instruction supports the learning of special education students and English language learners. Respondents described the program as reaching students by meeting students at their own level and teaching students through hands-on activities. The improvement of scientific understandings and abilities of students during Year 2 will be evaluated through an analysis of student achievement data to be completed in spring Every Learner Inquires Year 2 Evaluation 2

6 Teacher Practice 5. To what extent has teaching practice changed to reflect an inquiry-based approach? The evaluation focused on the extent to which teaching practice reflected inquiry-oriented instruction, the extent to which the classroom culture became more student centered, and the extent to which teachers adapted their lessons to the 5E Learning Cycle, a pedagogical model for lesson design. Teachers reported that these three aspects of inquiry are occurring frequently in their classrooms. Moreover, most teachers have completed at least three learning cycles. Although the frequency of the inquiry-related activities has not increased since Year 1, several teachers stated that their ability and comfort level have increased over the past year. However, the level of implementation appears less than comprehensive because it is largely focused on hands-on data collection and analysis. Other aspects of inquiry learning do not appear to be prevalent, such as designing studies, writing reports of findings, critically assessing procedures, or evaluating findings. Several aspects of a student-centered classroom culture are prevalent, but their implementation is also less than comprehensive. On the one hand, students frequently engaged in discussion, and teachers typically encouraged active participation of all students and acted as facilitators. On the other hand, lecturing remained very prevalent, teachers infrequently encouraged students to generate conjectures or find alternate solutions, and student questioning did not drive the focus of the lesson. Organizational Support 6. To what extent does the school administration advocate, facilitate, and support implementation? 7. To what extent are teachers participating in collaborative planning for inquiry? The evaluation focused on several aspects of organizational support, including the level of principal support for the program, material resources, coherence with other initiatives, and opportunities for job-embedded learning (also referred to as professional learning communities). In general, teachers reported low levels of support both from their principals and from the professional learning community in their schools. Both of these ratings were more negative than in Year 1. The involvement of principals was generally limited to providing resources and enabling teachers to attend professional development but did not extend to strongly advocating for the program or providing instructional leadership in the form of monitoring or feedback on instruction. Principals reported that they support the program by providing time for collaboration, but teachers typically did not report this. Although the ELI workshops themselves provided team meeting time, most teachers felt they lacked sufficient time to plan lessons or work with other teachers during the school day. Teachers did not regularly receive feedback on inquiry instruction, although they appreciated feedback when they did receive it. Several teachers cited this lack of support from principals and other teachers as a concern about implementation. Underscoring the importance of organizational support, teachers who reported higher levels of job-embedded learning tended also to report greater frequency of inquiry instruction. Every Learner Inquires Year 2 Evaluation 3

7 Most teachers expressed concern about the lack of time for planning and implementing inquirybased lessons. Teachers typically reported that their material resources were adequate. They expressed appreciation for science kits, mentioning that the kits make it easier for them to incorporate inquiry in their classrooms. In general, teachers reported strong levels of program coherence, with most teachers reporting that the program was consistent with their goals for their own professional learning. Some teachers, however, cited their difficulty with aligning inquiry instruction with existing curriculum requirements as a concern about implementation. Teacher Learning and Understanding 8. To what extent are teachers gaining the knowledge and skills to implement inquirybased science instruction? Teacher learning was evaluated with survey items focusing on understanding and self-efficacy. As a whole, teachers indicated that they understand inquiry instruction and the learning cycle and that the workshops were successful in explaining what both should look like in the classroom. Regarding self-efficacy toward different aspects of inquiry instruction, findings are mixed. About three fourths of teachers felt well prepared to implement inquiry-based instruction, and the proportion of such teachers increased from last year. About two thirds of teachers expressed confidence regarding their ability to align their curriculum units to the learning cycle. Just over half of teachers expressed confidence regarding evaluating student work in inquiry lessons; this proportion has increased somewhat since last year. Fewer than half of teachers felt prepared to differentiate instruction or align lessons to local standards, and these ratings did not change from Year 1. Elementary teachers felt more prepared than secondary teachers to implement inquiry instruction, differentiate instruction, and evaluate student work. In summary, it appears that most teachers understand what inquiry instruction looks like and have self-efficacy toward teaching lessons in this manner. However, teachers have less self-efficacy toward several related instructional tasks (evaluation, alignment to local standards, differentiated instruction) that are essential for full implementation. Reactions to Professional Development Workshops 9. What were the perceptions of the quality, utility, and effectiveness of the workshops? 10. In what ways should the professional development be improved? Attendance at the three winter/spring 2008 workshops varied by about 10 percentage points, with about 60 percent attendance in January, 70 percent in February, and 50 percent in April. The attendance rate of case study participants was 25 percent in January, and the same as the overall rate in the other two months. Quality of Presentation Overall, spring workshop participants considered the workshops to be of good quality. The overwhelming majority of respondents agreed that the content of the session was well organized, materials were clear, questions and concerns were addressed, and information was presented understandably. The February session was rated as particularly high regarding both quality and utility. Participants valued the opportunity to share and discuss real-world examples through Every Learner Inquires Year 2 Evaluation 4

8 modeling (as in the February session) and lesson study (in April). Many respondents wanted to see more of these types of activities at future workshops. Respondents would like to receive more information in advance of the workshop, including agendas, meeting details, and a list of which materials to bring. Utility of Workshops Overall, respondents found the workshops useful, particularly in regards to the relevance of information and for providing tools and strategies for the classroom. However, different groups of teachers viewed the workshops as too basic and too advanced, respectively. That is, some respondents who believed that they were already familiar with inquiry saw the sessions as repetitive of what they had already learned. Other teachers who were new to inquiry would have liked an even slower pace to allow themselves to understand inquiry instruction more fully. Respondents found team-based planning useful, and the time provided for this activity in the latter two sessions seemed about right to them. Finally, respondents wanted more guidance for implementation. AEA teams in particular sought more guidance for their efforts to provide professional development to schools in their respective regions during Year 3. Ratings for the utility of team planning time and clarity of next steps for implementation improved between January and the final two workshops. Effectiveness Participants perceived the workshops as being most effective at increasing their understanding of how students learn and of the learning cycle. However, several teachers reported not yet clearly understanding the differences among the stages of the learning cycle, and they stated that they would like continued instruction on this topic. The workshops were perceived as less effective at promoting the learning goals of identifying students preconceptions and differentiating instruction, which seem to be more challenging instructional tasks. Recommendations Because the Every Learner Inquires initiative will scale up during Years 3 and 4, it is important to be explicit with those continuing and considering participation about what the program is; what its goals and milestones are; and what is required from teachers, principals, schools, districts, AEA teams, and the state. The findings suggest that the explicitness would increase implementation and impact through clear goals and clear expectations about work, contributions, support, and progress. Program leaders should provide more explicit guidelines for local support of the ELI initiative. At the minimum, schools ought to provide teachers with protected time on a monthly to weekly basis to collaborate and plan for implementation. The main recommendation is to promote job-embedded professional learning by setting clearer expectations for how teachers should work together, what they should accomplish, and how principals should support them. To this end, the workshops should provide teachers with a framework in which to engage in this learning, such as lesson study, and should guide participants in setting up an embedded professional development system in their schools. The involvement of outside science consultants also would be helpful in the lesson study process, through coaching and modeling. To enhance the focus on changes in teaching practice, program leaders should map out the goals Every Learner Inquires Year 2 Evaluation 5

9 and milestones for teacher learning and implementation for the entire year. Participants should receive assignments between workshops to promote opportunities to practice what they have learned and to receive feedback. These expectations should be conveyed to schools continuing and considering participation in the initiative. Every Learner Inquires Year 2 Evaluation 6

10 Introduction Every Learner Inquires (ELI) is a statewide K 12 professional development effort to improve science learning in Iowa schools by promoting inquiry-based instruction. The program, which runs from August 2006 through June 2010, seeks to promote an inquiry-based approach to science instruction, as outlined in the National Science Education Standards (National Research Council, 1996). This program provides several workshops per year, and participants are expected to engage in job-embedded professional learning during the time between workshops. The goals of the initiative are to improve science learning for all students, build teacher leadership and content expertise in the area of science, provide teachers with the content and skills necessary to implement inquiry-based instruction, and establish a structure for sustained implementation. Under contract from the Iowa Department of Education, is conducting a mixed-method evaluation that examines the following anticipated outcomes of the program: Teacher reactions to professional development. Changes in teacher understanding of inquiry-based instruction. Changes in classroom instruction. School experiences in supporting the program. Student learning outcomes. The purpose of this report is to provide formative feedback on these outcomes during Year 2, based on findings from interviews, surveys, observations, and instructional logs. These findings are presented by section according to the five outcomes. A separate report, to be delivered in spring 2009, will present findings from an analysis of student achievement data to describe student learning outcomes. What Is Inquiry-Based Science Instruction? Inquiry-based science instruction is at the heart of the National Science Education Standards published by the National Research Council (1996) and of the Benchmarks for Science Literacy published by the American Association for the Advancement of Science (1993). The Standards define inquiry in the following manner: Inquiry is a multifaceted activity that involves making observations; posing questions; examining books and other sources of information to see what is already known; planning investigations; reviewing what is already known in light of experimental evidence; using tools to gather, analyze, and interpret data; proposing answers, explanations, and predictions; and communicating the results. Inquiry requires identification of assumptions, use of critical and logical thinking, and consideration of alternative explanations. (p. 23) The Standards for science content indicate that students should learn the process of scientific inquiry and are based on the assumption that students need direct experience with scientific inquiry to gain the understanding and skills necessary to engage in such activities. Moreover, they are based on the assumption that teachers need to introduce students to the logic and Every Learner Inquires Year 2 Evaluation 7

11 meaning of the process of inquiry, so that students can understand how inquiry activities allow them to arrive at scientific understandings. For these reasons, the Standards articulate both skills and understandings relating to inquiry that students should attain at different grade levels. Across all grade levels, the Standards articulate the following essential features of inquiry instruction: Learners are engaged by scientifically oriented questions. Learners give priority to evidence, which allows them to develop and evaluate explanations that address scientifically oriented questions. Learners formulate explanations from evidence to address scientifically oriented questions. Learners evaluate their explanations in light of alternative explanations, particularly those reflecting scientific understanding. Learners communicate and justify their proposed explanations. Not every teaching style is equally suited for affording students the opportunity to engage in inquiry. For these reasons, the Standards (in Chapter 3) describe what teachers should know and do in their planning, instruction, assessment, and collegial relationships in order to support inquiry instruction. These standards describe several aspects of a student-centered classroom culture that promote opportunities to engage in inquiry. For example, teachers should serve as facilitators of learning such that they promote discourse and discussion, encourage curiosity and questioning, and respect the diversity of their students. Also, teachers should promote learning communities in which intellectual rigor is respected, students collaborate on projects, and where students interests may determine the focus of classroom investigations. These student-centered teaching practices are considered to be necessary conditions to promote inquiry learning. In light of the fact that teachers need guidance and structure to enact the standards, the ELI program provides a pedagogical model for lesson design. This model, called the 5E Learning Cycle, articulates five phases of inquiry, with each phase corresponding to a different learning goal. The model aligns with a student-centered, constructivist approach to science education. To this end, the model describes the teacher and student behaviors at each stage that will promote high levels of student involvement with generating and discussing questions, ideas, and explanations. The five phases in this model are as follows: Engage. Students become interested in a problem or a phenomenon and consider what they currently know about it. Explore. Students conduct investigations or develop common experiences so they can compare results and share ideas. Explain. Students explain concepts and ideas in their own words and use data from their investigations to support their explanations. Students learn to use appropriate scientific terms. Elaborate. Students apply what they learned to a new situation and draw conclusions based on evidence. Every Learner Inquires Year 2 Evaluation 8

12 Evaluate. Students compare their conclusions to those of others and perhaps revise their explanations. Students generate new questions that lead them into a deeper exploration of the topic. Program Design Participation in the ELI program is organized by teams. There are two types of teams: case study school teams and area education agency (AEA) leadership teams. Four schools have been chosen for the first cohort, to act as case studies with the purpose of providing formative data about how schools implement and support the initiative. Case study schools agreed to implement the instructional strategies presented in the professional development, keep logs of their implementation, allocate inservice time to supporting implementation, and cooperate with data collection for the evaluation. All science teachers in each case study school are expected to attend the workshops and participate in regular team meetings to discuss assignments that are part of the workshops, as well as other topics related to program implementation. This model of school implementation includes several components that research suggests may promote implementation of inquiry instruction, elaborated as follows: Collective participation of teachers from the same school (Desimone, Porter, Garet, Yoon, & Birman, 2002). Case study schools are expected to involve the entire science teaching staff and building principal in the ELI initiative over a four-year period. As part of this commitment, all science teachers and the building principal are expected to participate in the professional development workshops. Job-embedded professional learning, such as observations, collaborative lesson planning, mentoring, or study groups (Desimone et al., 2002). As a condition of their participation, case study schools commit to conducting follow-up planning meetings at their school. The extent to which teachers engaged in job-embedded learning outside of these planning meetings was left of to the individual schools. Coherence with teachers own goals for professional learning and with other programs, standards, and requirements (Penuel, Fishman, Yamaguchi, & Gallagher, 2007). Although there are no requirements per se related to program coherence, case study schools agreed to commit inservice time to this program, and principal participation should encourage coherence with other programs in the school. The 11 AEA leadership teams also are participating in the full four-year cycle of professional development. These teams include AEA science education staff and at least two teacher leaders drawn from schools within the jurisdiction of the given AEA. The purpose of the participation of these AEA teams corresponds to the Teacher Learning goal of building leadership throughout the K 12 educational system in Iowa. These teams will participate in the ELI program in order to build their own capacity to deliver professional development in inquiry instruction. During the third and fourth years of the initiative, the AEA teams are expected to begin to provide professional development for schools in their respective areas, building on the lessons learned from the case study schools. Every Learner Inquires Year 2 Evaluation 9

13 Program Goals Several stated goals of the ELI program encompass student learning, teacher learning and practice, and organizational change. These goals were expressed in ELI documents as follows: Teacher Practice goal: Implement inquiry-based instruction. Organizational goal: Establish a structure that sustains the implementation of ELI. Teacher Learning goal: Build teacher leadership and content expertise within the system. Student Learning goal: Improve science learning for all K 12 students in the state of Iowa. The following paragraphs elaborate on these goals, the program s approach to accomplishing them, and the approach for evaluating their accomplishment. Teacher Practice Goal The Teacher Practice goal is to implement inquiry-based instruction in science classrooms at all school levels. The evaluation focused on the extent to which teaching practice reflected inquiryoriented instruction, the extent to which the classroom culture supported inquiry instruction, and the extent to which teachers adapted their lessons to the 5E Learning Cycle model. In addition, the evaluation included in its focus the extent to which teachers differentiated instruction in the context of inquiry lessons. Organizational Goal The evaluation focused on several aspects of the Organizational goal, including the level of principal support for the program, material resources, coherence with other initiatives, and opportunities for job-embedded learning (also referred to as professional learning communities). Principal support is the degree to which the principal articulates a vision for inquiry-oriented instruction and advocates and promotes participation in the ELI program as well as changes in teacher practice. Material resources include lab equipment, lab space, aligned curriculum materials, and kits or consumable laboratory supplies. The other two aspects under this goal (i.e., coherence and job-embedded learning) were described above in the discussion of team organization (see Program Design). Teacher Learning Goal In line with the changes in teacher practice outlined above, the evaluation addressed two aspects of teacher learning: understanding and self-efficacy. The evaluation focused on whether teachers had acquired a fundamental understanding of inquiry instruction and the learning cycle. In the context of this program, self-efficacy refers to whether teachers perceive that they are prepared to implement the instructional strategies and approaches that are the intended goals for teacher practice. Every Learner Inquires Year 2 Evaluation 10

14 Student Learning Goal The Student Learning goal is to improve science education for all learners in Grades K 12 in the state of Iowa. However, this goal may be understood in several different ways. One intended outcome is to increase the scientific understanding of students. A related outcome is to increase the ability of learners to engage in the scientific inquiry process. Other outcomes relate to improved attitudes of students toward science learning, and stronger engagement in learning science. The evaluation examined each of these types of student learning outcomes through two methodologies. This report presents data from a descriptive study of teacher and student opinions regarding these student learning outcomes. The evaluation also has included a statistical analysis of student science achievement over time, as measured by annual standardized tests. This latter study was recently completed for Year 1 of the program and was submitted as a separate report to the Iowa Department of Education in May 2008; the Year 2 achievement report is scheduled to be submitted in April Design of Professional Development The cycle of workshops during Year 2 was intended to build upon the previous year. As in every year of the program, professional development was delivered through a summer institute and whole-day workshops during the academic year. During the time between these semimonthly workshops, participants were assigned to collaborate on lesson design, reflect on their learning, and implement different aspects of inquiry instruction. Summer Institute and Workshops Most of the interactions between teams and project staff (i.e., the design team) occurred during professional development workshops. There was a four-day kick-off workshop in July and five one-day academic-year workshops that occurred in September, November, January, February, and April. The September, January, and February workshops were delivered in both elementary and secondary strands, with teachers choosing the strand that corresponded to their grade level. To facilitate travel, each of the academic-year training workshops was presented separately to participants in the eastern and western halves of the state. During the workshops, teachers participated in several types of interactive learning activities that were designed to increase their level of involvement. For example, during some workshops, participants experienced an inquiry-based science lesson from the perspective of a student. Following presentations from the instructors, participants would typically engage in a group or paired discussion. Participants engaged in written reflection several times per workshop. For example, they were asked to enter a line of learning in project journals, in which they described their current understanding of a particular topic. They also engaged in quick writes to answer a particular question, designed to engage the participants in a particular topic. Postworkshop Assignments Following each workshop, teachers had an assignment to complete during the interval before the next workshop. The purpose of these assignments was to reinforce the learning that had occurred Every Learner Inquires Year 2 Evaluation 11

15 during the workshop. The major assignment during Year 2 was to design and implement two full learning cycles. School- and Web-Based Meetings Case study teams met with varying frequency to discuss assigned readings or other assignments and to develop lesson plans. Some of the case study teams posted minutes from these meeting to a Web-based discussion board. Each case study and AEA team had its own discussion thread on this Web board. Its main purpose was to facilitate communication among team members. Workshop Topics Each workshop focused on a particular aspect of inquiry instruction. In addition, the workshops provided guidance on different techniques and fundamental understanding related to inquiry. Descriptions of these topics follow. How Students Learn. The professional development introduced research about how students learn in relation to three critical learning needs: (1) addressing preconceptions of learners, (2) building conceptual structures to organize factual information, and (3) fostering self-monitoring of learners. Facilitation Strategies. The workshops included Accountable Talk presentations, a collection of techniques for four group-facilitation purposes: supporting group discussion, supporting accountability to the learning community, supporting accountability to accurate knowledge, and supporting accountability to rigorous thinking. Strategies for Differentiation. The professional development included training on how to address the needs of all learners. Avenues for differentiation included allowing students to choose different assignments and providing background reading according to student reading level. Building Professional Learning Communities. The workshops presented information about characteristics of collaborative groups, in order to build the capacity of each of the teams to engage in collaborative learning activities. For example, during the November workshop, facilitators modeled a process for reflective dialogue about science lessons. Evaluation Overview The evaluation study of Year 2 of ELI focused on the four program goals. Several data-collection methods provided data on progress toward these goals. During site visits conducted by the evaluator, case study schools staffs participated in interviews and classroom observations. Case study teachers also completed implementation logs, and students in these schools completed a survey about their science classroom experiences. AEA teachers joined case study teachers in completing a survey about their experiences with science instruction and with the ELI program. The evaluation instruments addressed the specific evaluation questions related to the program Every Learner Inquires Year 2 Evaluation 12

16 goals. Table 1 aligns each evaluation question with the data source that addresses it; these data sources are described in the Methods section. In addition to the focus on the four program goals, this report provides formative feedback on the professional development workshops offered during the winter and spring of Year 2. (A previous report summarized reactions to the workshops offered during the fall of Year 2.) This feedback is based on surveys filled out by participants of the January, February, and April workshops. It addresses the quality, utility, and effectiveness of the workshops and provides suggestions for improving these workshops in the future. Every Learner Inquires Year 2 Evaluation 13

17 Evaluation Question 1. To what extent have students acquired the abilities and fundamental understandings about scientific inquiry? 2. What is the impact on student enthusiasm and self-efficacy toward science learning? Table 1. Crosswalk of Evaluation Questions to Data Sources Teacher Survey Teacher Interview x x Principal Interview Observation Teacher Logs Student Survey x x x x Reaction Form 3. Does the program address the needs of all student subgroups? x x x 4. Are the scientific understandings and abilities of students improving? 5. To what extent has teacher practice changed to reflect an inquirybased approach? 6. To what extent does the school administration advocate, facilitate, and support implementation? 7. To what extent are teachers participating in collaborative planning for inquiry? 8. To what extent are teachers gaining the knowledge and skills to implement inquiry-based science instruction? 9. What were the perceptions of the quality, utility, and effectiveness of the workshops? ITBS Data* x x x x x x x x x x x x x x x x x x x x x 10. In what ways should the professional development be improved? x x x *Iowa Tests of Basic Skills (ITBS) student achievement data for Year 2 will be collected in fall 2008 and is not included in this report. Every Learner Inquires Year 2 Evaluation 14

18 Methods ELI Workshop Reaction Data This report summarizes participant reactions to three ELI workshops presented in winter and spring of 2008, along with the attendance and topics addressed in these workshops. Workshop Feedback Forms Program leaders distributed and collected workshop feedback forms after each workshop. The purpose of the postworkshop feedback form was to assess participant satisfaction with the professional development sessions. The form asked participants to rate items aligned with four topics: (1) workshop quality, (2) workshop utility, (3) the amount of time spent on different formats of activities, and (4) the extent to which the workshop improved understanding or ability in regards to five learning goals. The form also presented five open-ended questions designed to elicit feedback regarding overall strengths and areas for improvement of the workshop. The facilitators distributed the postevent reaction forms to all participants at the end of the workshop. All participants had the opportunity to complete the feedback form, including AEA science consultant staff, teachers, and administrators. Evaluators completed descriptive analyses of the Likert scale items (multiple choice) from the feedback forms, and they identified themes and tones (positive or negative) in participant comments within and across workshops. Workshop Agendas Workshop agendas were analyzed to categorize the planned activities based on topic and method of delivery. Categories of workshop topic were derived inductively. The categories of delivery method were determined a priori to include the following: lecture, small or large group discussion, planning (including lessons and team planning), and modeling or demonstrations of lessons and instructional methods. Workshop Attendance Participants at each workshop were asked to sign in to indicate their attendance. The attendance was tracked across sessions of the workshops and disaggregated by team type (AEA or case study). Teacher Survey developed a teacher survey to address the four goals of the ELI program: teacher learning, implementation of inquiry instruction, organizational support, and student impact. For each construct described in the introduction, the survey included several items to reflect different details or elements of the construct. Psychometric procedures validated that these items could be combined into a single scale score. The following paragraphs describe how each construct was evaluated with the survey. A more extensive description of the instrument and its validation appears in Appendix A. Every Learner Inquires Year 2 Evaluation 15

19 Teacher Learning The survey included several items to evaluate the extent to which teachers gained the knowledge and skills to implement inquiry-based science instruction. Survey items asked teachers to rate their level of understanding of both the National Science Education Standards and the learning cycle, and the extent to which the workshops provided sufficient information on what inquiry instruction and the learning cycle look like in the classroom. In addition, teachers rated their level of preparation to implement inquiry learning, modify curriculum units to align with the learning cycle, differentiate instruction while doing inquiry, evaluate student work from inquirybased lessons, and align inquiry lessons to state standards. Organizational Support The survey included four groups of items that were used to create scales addressing professional learning communities, principal support, resources, and coherence. Each scale is described in more detail, as follows: Professional Learning Community. Eight items asked teachers to rate their level of agreement with statements about professional learning opportunities in their school, for example, Science teachers in this school regularly observe each other teaching classes in order to learn how to implement inquiry-based lessons. This scale was included on the Year 1 survey and was compared across years. Administrator Support. Nine items asked teachers to rate the extent to which administrators support inquiry learning. The stem To what extent does your principal, assistant principal, or department head: introduced the items. Examples of these items include understand inquiry-based instruction and monitor teacher implementation of inquiry learning. This scale was included on the Year 1 survey and was compared across years. Resources for Inquiry Instruction. Eight items asked teachers to rate their agreement with statements about whether they had adequate resources for inquiry instruction, for example, I have sufficient laboratory equipment to support inquiry learning. Coherence. Five items asked teachers to rate the extent to which the ELI program is consistent with other programs, reform efforts, standards, and assessments. The stem To what extent is Every Learner Inquires consistent with each of the following: introduced the items. Examples of the items include Other science professional development programs at your school or district and State and district standards and curriculum frameworks. Teacher Practice Several items examined the frequency with which teachers used certain strategies or with which students engaged in certain activities aligned with inquiry-based learning. When answering these items, teachers were directed to describe a particular class of theirs. There were three groups of items of this type: Every Learner Inquires Year 2 Evaluation 16

20 Inquiry Instruction Scale. Teachers rated the frequency with which students in their class took part in 15 types of science activities aligned with the essential features of inquiry learning. These items were introduced with the question, About how often do students in this class take part in the following types of science activities? Examples of items include Critically examine the scientific explanations of other students and Record, represent, or analyze data. This scale was included on the Year 1 survey and was compared across years. Classroom Culture Scale. Teachers rated the frequency with which they incorporate nine student-centered instructional practices in their science instruction. An example of an item of this type was Assign students to work in groups on projects. This scale was not included on the Year 1 survey. Learning Cycle Implementation. Three items referred to implementation of the learning cycle. Teachers reported the number of times they taught a complete learning cycle in science during Year 2. Another item asked if they experienced any barriers to teaching a full learning cycle (with response options of Yes or No), and if so, to describe the barriers in an open-ended fashion. Student Learning Four items asked teachers to rate the extent to which their implementation of inquiry learning has improved student learning outcomes. These outcomes were engaged learning, understanding of science concepts, enthusiasm, and understanding of inquiry. Teachers rated the extent of improvement with the response options of Not at all/slightly, Somewhat, Moderately, and Very much so. These items reflect different aspects of student learning outcomes and were each analyzed individually. They were included on the Year 1 survey, and responses were compared across years. Response Rate Of the 80 teachers who were contacted, 46 teachers completed the survey, for an overall response rate of 57 percent. The response rate did not vary by team type; 12 of 21 case study teachers and 34 of 59 AEA teachers completed the survey. Student Survey The student survey was administered during April 2008 to students who were taking a science class in the case study schools in Grades 3 and above (see Appendix A for a discussion of response rates along with a more extensive description of the instrument). Learning Point Associates developed the student survey based on items from previously published instruments. All items on the survey had a response scale of Yes, No, and Sometimes. Each of the following constructs was combined into a single scale score (all except the self-efficacy scale were included on the Year 1 student survey): Inquiry Instruction. Ten items from the Investigation scale of the What Is Happening in this Classroom (WIHIC) questionnaire (Fraser, McRobbie, & Fisher, 1996) were used to Every Learner Inquires Year 2 Evaluation 17

21 address the construct of inquiry learning. A sample item from this scale is I carry out investigations to test my ideas. Classroom Culture. Nine items from the Involvement scale of the WIHIC questionnaire were used to address the construct of classroom culture. A sample item from this scale is I talk with other students about how to solve problems. Enthusiasm Toward Science. Six items from the Test of Science-Related Attitudes (TOSRA) (Fraser, 1982) were used to measure a student s enthusiasm toward science. A sample item of this sort is I look forward to science lessons. Self-Efficacy for Science. Six items asked students to rate whether they can accomplish certain science classroom skills. Each of these items started with the phrase I can and was completed with phrases such as ask a scientific question and collect data to answer a question. A psychometric analysis comparing responses of elementary and high school students determined that these two groups responded differently to the survey items. For this reason, the scale scores from these two groups are reported separately. Instructional Logs Instructional logs were used to collect teachers self-reported data on their use of inquiryoriented instruction. The logs were revised from last year s evaluation to reflect feedback from members of the ELI design team. Overall, the log form required teachers to indicate whether different student or teacher activities were present during a particular lesson. For the purpose of this report, the term lesson denotes the instruction occurring in a single class period. Teachers in case study schools completed one log per lesson for 10 consecutive lessons during November The first section of the form, corresponding to student activities, was adapted from the Local Systemic Change Classroom Observation Protocol (Horizon Research, 2005). This section listed the following four categories of student activities (number of specific activities in each category in parentheses): Listened to a presentation. (3) Engaged in discussion/seminar. (2) Engaged in reading/reflection/written communication about science. (6) Engaged in problem solving/investigation. (12) Under each category were a number of specific activities. For example, the category of Engaged in reading/reflection/written communication about science included activities such as Read about science and Answered textbook/worksheet questions. The category of Engaged in problem solving/investigation included activities such as Worked with manipulatives and Recorded patterns, cycles, or trends. Teachers indicated which student activities occurred in the class session. Every Learner Inquires Year 2 Evaluation 18

22 The second section of the form listed teacher activities, each of which corresponded to one of the five phases of the learning cycle: Engage, Explore, Explain, Elaborate, and Evaluate. For example, the activity Piqued students curiosity and generated interest corresponded to the Engage stage; Provided time for students to puzzle through problems corresponded to the Explore stage; Requested justification (evidence) for students explanations corresponded to the Explain stage; Encouraged students to apply what they have learned to a new situation corresponded to the Elaborate stage; and Encouraged students to assess their own progress corresponded to the Evaluate stage. These examples were taken from a description of the 5E Learning Cycle (National Institutes of Health, 2005). Teachers checked each teacher activity that occurred during each lesson, along with the phase of the learning cycle to which the activity belonged. Response Rate The evaluation team sent packets of instructional logs to every classroom science teacher in each of the case study schools (a total of 20 teachers 1 ). A total of 9 teachers completed all 10 forms for a response rate of 45 percent. One of these teachers completed two sets of instructional logs, one for an upper elementary class and one for a lower elementary class. Analysis The items in the first two categories (regarding presentations and discussion) align with the classroom culture construct, and the second two categories (reading/writing and problem solving/investigation) align with the inquiry instruction construct. The category of the activity was considered to have occurred if any of its activities were reported to have occurred. For each teacher, evaluators calculated the percentage of lessons that the teacher used each activity and category of activity. These percentages were then averaged across the 10 teachers. Classroom Observations The evaluation team used an observation protocol that was adapted from previously used instruments. The first two sections of this form were identical to the instructional log in all respects except one: In the section on teacher activities, the observer rated the degree of emphasis each phase of the learning cycle received in the lesson (in the instructional log, teachers simply indicated whether a lesson addressed each phase of the learning cycle or not). The remaining sections of the observation form were identical to the Reformed Teaching Observation Protocol (RTOP) (Piburn & Sawada, 2000). According to the RTOP Training Guide (Sawada et al., 2000), the RTOP is an observational instrument that can be used to assess the degree to which mathematics or science instruction is reformed. It embodies the recommendations and standards for the teaching of mathematics and science that have been promulgated by professional societies of mathematicians, scientists and educators. (p. 1 ) 1 One case study teacher who was invited to participate in the teacher survey was not asked to complete the instructional logs on the grounds that this teacher did not teach a regular class. Every Learner Inquires Year 2 Evaluation 19