PBS TeacherLine Pre-Survey and Post-Survey Evaluation Report

Transcription

1 PBS TeacherLine Pre-Survey and Post-Survey Evaluation Report June 2010 Prepared by: Jonathan Margolin, Ph.D. Debbie Davidson-Gibbs Natalie Tucker Data analyst: Jiuping Chen 20 N. Wacker Drive, Suite 1231 Chicago, IL Copyright All rights reserved. 4698_06/10

2 Contents Page Executive Summary... i PBS TeacherLine Presurvey and Postsurvey: Evaluation Report... 1 Evaluation Purpose and Questions... 1 Data Sources... 2 Interviews... 4 Impact of Course on Knowledge and Practice... 6 Effect of PBS TeacherLine Courses on Knowledge of Content and Instructional Strategies... 6 Impact of PBS TeacherLine Courses on Teaching Practice... 7 Differences Among Courses in Level of Impact... 9 Course Quality Quality of Course Facilitation Quality of Course Content Discussions and Assignments Customer Satisfaction Repeat Learners Conclusions and Recommendations Reference Appendix A. Survey Tables PBS TeacherLine Presurvey Tables Postsurvey Response Tables Appendix B. Interview Protocol Appendix C. Recommendations for PBS TeacherLine Presurvey and Postsurvey... 29

3 Executive Summary The evaluation has three key goals. First, the goal of the evaluation is to evaluate the effectiveness of PBS TeacherLine courses on improving teachers knowledge, skills, and practice. In a broader sense, the purpose is to determine the value of courses. A second goal of the evaluation is to determine whether courses vary in their effectiveness and to try to understand the reasons for this variation. The broader purpose is to provide feedback that is relevant to the improvement of the quality and effectiveness of courses. A third goal of the evaluation is to describe the characteristics of repeat learners (i.e., teachers who participate in multiple PBS TeacherLine courses). The following evaluation questions address these goals and purposes: 1. To what extent has participation in Courses enhanced knowledge of subject content, pedagogy, and assessment? 2. To what extent has participation in Courses influenced learner practices related to curriculum, instruction, and assessment? 3. Which Courses, and which types of Courses, have the greatest impact on learner knowledge and practice, and for what reason(s)? 4. What is the level of satisfaction with course quality, and how can Courses be improved? 5. What are the characteristics of repeat learners in terms of professional development background, Courses taken, satisfaction with Courses over time, and changes in teaching practice? Question 1. To what extent has participation in Courses enhanced knowledge of subject content, pedagogy, and assessment? Learner ratings of their level of knowledge of content and instructional strategies shift strongly upward after completing a PBS TeacherLine course. Eighty-four percent of participants increased their ratings of their knowledge of both content and instructional strategies from presurvey to postsurvey. This finding excludes the two to three percent of participants who already rated themselves at the highest level at the time of the pretest. During interviews, learners were asked to describe the ways in which the course expanded content knowledge. The majority of participants (18 of 21) indicated that they expanded their content knowledge because they learned a lot and better understood alternative perspectives from their PBS TeacherLine course. As one participant said, The Web design was totally new for me so it developed and expanded all of my content knowledge there. With the math one, it just expanded my content knowledge as to newer online resources. Question 2. To what extent has participation in Courses influenced learner practices related to curriculum, instruction, and assessment? Nearly three quarters of those surveyed after taking a course reported incorporating content knowledge from a PBS TeacherLine course into their teaching practice. Seventy percent of respondents reported incorporating instructional strategies into their teaching practice based on PBS TeacherLine Evaluation: Executive Summary i

4 the course they had taken. Fewer than 10 percent responded that they had not incorporated content or strategies, with the remaining portion (about 15 percent) indicating that they were not currently in a teaching position. During interviews, all but one respondent indicated at least one way that their PBS TeacherLine course influenced their teaching practice. Major themes identified by these interviewees were as follows: Teachers learned and applied new teaching strategies in the classroom. Teachers learned how to integrate technology. Teachers used and applied resources and materials from the course, most prevalently, links to Web-based resources, as well as templates, models, or graphic organizers that they could use in the classroom. Teachers gained confidence in their teaching ability. Question 3. Which Courses, and which types of Courses, have the greatest impact on learner knowledge and practice, and for what reason(s)? There were no noticeable differences in learner ratings of impact as a function of curriculum area, either in respect to impact on knowledge or impact on instructional practice. Question 4. What is the level of satisfaction with course quality, and how can Courses be improved? Course satisfaction was measured in terms of satisfaction with the instructor, the course content, and participants interactions with other learners. Overall, the majority of respondents were satisfied with their instructor; when asked if they would recommend that their colleagues take a course taught by their facilitator, 92 percent responded in the affirmative. Most survey respondents agreed that their facilitator frequently engaged in best practices such as providing feedback on assignments, answering questions from learners, and reminding students about and clarifying assignments. These three practices also were mentioned frequently by interviewees as the ways in which instructors supported their learning. A minority of learners suggested that the facilitator could have been more supportive, such as by providing more feedback on assignments or more clarity in the form of examples and explanations. Some would have preferred greater flexibility about assignments and due dates. Learners expressed strongly favorable opinions of course content. For every item, more than 90 percent strongly agreed or agreed that the course met expectations, provided content or strategies that were applicable to instruction, and prepared them to improve their students academic performance. Moreover, all interviewees indicated that their course provided them with useful resources. Several learners also praised the quality of course materials, such as the readings, videos of lessons, and websites. However, a number of learners, especially in technology courses, suggested that the materials needed to be more up-to-date. PBS TeacherLine Evaluation: Executive Summary ii

5 Learners frequently mentioned the discussions with other participants as the most beneficial aspect of the course. Learners stated that they benefitted from the opportunity to share ideas and resources with learners from around the country. However, several learners disliked the requirements regarding the discussions (e.g., the number of posts required), which they considered onerous. In general, the most prevalent criticism (expressed by 40 percent of learners) was that the amount of work was not commensurate with the number credits awarded. A minority of learners stated that the assignments were either not explained clearly or were too time consuming given the number of credits offered for the course. Question 5. What are the characteristics of repeat learners in terms of professional development background, Courses taken, satisfaction with Courses over time, and changes in teaching practice? High repeaters (who took four or more courses), in comparison with nonrepeaters (who took only one PBS course) and low repeaters (who took two courses), more frequently stated that they were taking the course to earn graduate credit. Repeat learners were asked in interviews why they took more than one course. Learners most frequently mentioned the convenience of the online format as the reason for taking more than one PBS TeacherLine course. Similarly, the convenience of the online format was the most frequent response to the survey item asking what learners liked best about their course. A smaller number of interviewees mentioned only their positive experiences in previous PBS courses. Finally, a review of registration records indicates that repeat learners are typically completing most of their courses in the same curriculum area. The pattern that emerges from these three findings is that learners take multiple PBS TeacherLine courses to collect graduate credit in a specific curriculum area, and they value the flexibility of the online format. Although the online format was a strong selling point for the program, a minority of learners voiced criticism about the way certain courses were implemented. About one fifth of learners reported experiencing some technical difficulties. A minority stated that the layout of the course lacked clarity, especially in regard to the listing of assignments and due dates. Summary of Program Strengths Based on these findings, the following is a summary of program strengths: 1. Learners typically increased understanding of content and instructional strategies after completing their PBS TeacherLine course. One English as a second language (ESL) teacher in her 34th year of teaching stated, I have a much better understanding of special education labels and terms and of the way special education looks not only in a classroom but as a department. 2. PBS TeacherLine courses provide useful teaching strategies. Most teachers were able to apply what they learned directly to the classroom setting. One survey respondent stated, I liked that I could immediately take what I learned back to the classroom. The walkthroughs were very easy to follow. PBS TeacherLine Evaluation: Executive Summary iii

6 3. The courses typically provide useful resources and materials that are applicable to instruction, such as Web-based curriculum resources, templates, models, and graphic organizers. One survey respondent stated, I loved the wide variety of resources offered. I was introduced to new websites such as Reading Rockets. I had no idea PBS offers so much for teachers, students, and parents on their website. 4. PBS TeacherLine improves confidence in teaching practice. For example, a high school science teacher in her 14th year stated, It made me more confident in having students do things on the computer because I got better at figuring out what was going wrong. 5. Facilitators of PBS TeacherLine courses typically exhibit best practices such as providing feedback on assignments, promptly answering questions, and clarifying assignments. A second-grade teacher in her third year stated, The instructor was great. Very hands on. If I asked her a question, she responded immediately. She gave me feedback on my work all the time. However, a minority of respondents stated that their instructors should have provided more feedback on work or explanations of assignments. A minority also described their instructors as lacking appropriate flexibility. 6. Courses present content through a variety of modalities, such as readings, videos of lessons, and websites. A high school mathematics teacher in her 10th year remarked, The materials were great. They were very clear. It was all online. 7. Courses provide frequent opportunities for learners to share ideas and resources. Many learners considered these interactions to be a strength of the program, including an art teacher who stated, With the discussion board, you just learn so much. It s good to connect with other people all over the country. With PBS, you could be anywhere in the country and connecting and getting ideas, so I think that s just wonderful. 8. The online format of PBS TeacherLine courses makes the program highly convenient, allowing teachers to incorporate their coursework into their daily schedules. One survey respondent noted, It allowed me to teach during the day, spend family time, then do PBS work. However, a minority of learners noted that the online layout of their course could be improved. 9. Most teachers agreed that the course made an impact on their students learning, in terms of student engagement and academic progress. As one second-grade teacher stated, They definitely improved in their fluency rate. They definitely started reading faster and there was more of a flow. One teacher of middle school mathematics in her eighth year stated, They are more engaged in the lessons now. They have more confidence in the lessons because basically the lessons are tailor-made for them now. Recommendations for Improving Courses Based on the feedback from learners summarized above, the following are practical ways to improve courses: 1. Maintain a regular schedule for review and revision of course content. Several learners, particularly those in technology courses, noted that some content and resources appeared to be dated. Courses should be reviewed regularly to confirm that all URLs are PBS TeacherLine Evaluation: Executive Summary iv

7 operational. Course content should be reviewed to ensure that it is current, which is particularly important for technology courses. 2. Summarize course assignments and due dates in one place. Some learners stated that there was not a clear week-by-week summary of assignments and due dates. Each course should provide a summary of what is due in a particular week. 3. Instructors should provide flexible guidelines for discussions. Some learners noted that keeping up with the requirements for participating in discussions became onerous due to the large volume of comments. Facilitators might be able to provide some flexibility to learners in terms of requirements for reading and responding to posts so that these requirements should not be more time consuming than is warranted. 4. Gather feedback on the amount of coursework. Several learners perceived that the amount of coursework greatly exceeded their expectations for the course, based on the number of credits offered. It would be feasible to collect input from participants about the amount of time they spent on assignments and whether that amount of time was reasonable. It would be possible to adjust the amount of work in courses in response to this feedback. PBS TeacherLine Evaluation: Executive Summary v

8 PBS TeacherLine Presurvey and Postsurvey Evaluation Report PBS TeacherLine provides online professional development to educators across the country with funding through its Ready to Teach grant administered by the U. S. Department of Education. In 2000, PBS TeacherLine began offering cohorts of K 12 teachers a variety of noncredit online courses in connection with local PBS stations. PBS TeacherLine aims to improve teacher practices and help educators acquire the skills they need to become and remain highly qualified. PBS TeacherLine offers online graduate-level professional development courses for K 12 teachers in the following curriculum areas: reading/language arts, mathematics, instructional technology, and instruction. Most courses are 30-hour courses that are conducted for six weeks. According to the PBS TeacherLine website, all courses are designed to improve participant attitudes and practices by incorporating weekly assignments, discussion boards, and opportunities for reflection. The pedagogy of each course involves inquiry-based and projectbased learning culminating in a project that is a tangible activity, case study, or strategic plan for classroom implementation. Most courses are intended to provide teachers with specific resources or tools they can use in the classroom. All course facilitators undergo PBS TeacherLine s rigorous Online Facilitation Training program and are highly qualified teachers with master s degrees. The PBS TeacherLine program is flexible and customizable. There are two common modes of participation. First, districts and schools can enroll cohorts of teachers in customized courses to meet a district or school s strategic needs and objectives. Second, teachers can enroll independently to pursue their own particular interests or to align with their professional development requirements and goals. Most PBS TeacherLine courses allow educators to receive graduate credits from local partner colleges and universities, as four universities across the country grant credit for all PBS TeacherLine courses. Educators also may apply credits toward district professional development points or continuing education units (CEUs). The overall purpose of PBS TeacherLine courses, as mentioned earlier, is to improve teacher practices and help educators acquire the skills they need to become and remain highly qualified. Specifically, participation in PBS TeacherLine courses increases knowledge and understanding of high-quality teaching through reading, discussion, and assignments. Participation improves teaching skills through guided practice, reflection, and feedback from the instructor. Improved knowledge and teaching skills will improve the effectiveness of teaching (i.e., student learning and achievement). Currently, PBS TeacherLine offers more than 120 courses nationally to PK 12 educators. Although PBS TeacherLine initially targeted teachers in Title I schools, the clientele who currently participate in PBS TeacherLine are educators in schools with a wide range of demographic characteristics. Individual participants include classroom teachers as well as technology specialists and media/library specialists. Evaluation Purpose and Questions The primary goal of the evaluation is to evaluate the effectiveness of PBS TeacherLine courses on improvements to knowledge, skills, and practice. In a broader sense, the purpose is to PBS TeacherLine Evaluation 1

9 determine the value of courses. A second goal is to determine whether courses vary in their effectiveness and to try to understand the reasons for this variation. The broader purpose is to provide feedback that is relevant to the improvement of the quality and effectiveness of courses. A third goal of the evaluation is to describe the characteristics of repeat learners (i.e., teachers who participate in multiple PBS TeacherLine courses). The following evaluation questions address these goals: 1. To what extent has participation in Courses enhanced knowledge of subject content, pedagogy, and assessment? 2. To what extent has participation in Courses influenced learner practices related to curriculum, instruction, and assessment? 3. Which Courses, and which types of Courses, have the greatest impact on learner knowledge and practice, and for what reason(s)? 4. What is the level of satisfaction with course quality, and how can Courses be improved? 5. What are the characteristics of repeat learners in terms of professional development background, Courses taken, satisfaction with Courses over time, and changes in teaching practice? Data Sources The following sources of data addressed the evaluation questions: Review of PBS Presurvey and Postsurvey PBS TeacherLine has administered precourse and postcourse surveys to learners starting in fall 2004 and continuing through the last semester covered by this evaluation (fall 2009). These surveys were developed by PBS TeacherLine and have remained essentially the same throughout the past six years. A content review of the surveys conducted by indicates that the items address several different constructs. The 19 items on the precourse survey aligned with the following constructs: Reasons or goals for participating (five items) Level of comfort (six items)* Level of knowledge of course content (two items) District/school involvement in course selection (two items) Experience with other, non-teacherline professional development courses (four items) The 32 items on the postcourse survey aligned with the following constructs: Overall satisfaction with course (two items) Level of knowledge of course content (two items) Impact on teaching practice (three items) PBS TeacherLine Evaluation 2

10 Logistics of course participation (two items) Satisfaction with course facilitation (nine items)* Satisfaction with course quality (five items)* Satisfaction with customer service quality (five items)* Liked most, least, and how to improve (three items) Suggestions for additional course offerings (one item) Based on this review, we determined that there was one construct included on both the precourse and postcourse versions of the survey: level of knowledge of course content. Even though the items on this construct were not exactly the same across the precourse and postcourse versions, we nevertheless calculated the change in ratings of course knowledge from precourse to postcourse. Furthermore, we evaluated the psychometric properties of the four constructs marked with an asterisk to determine whether each set of items could be combined into a single scale score. 1 Only the items related to satisfaction with course quality could be combined into a scale score on the basis of this analysis. Survey Administration and Response Rate The analyses were based on 24,562 completed precourse surveys and 17,933 completed postcourse surveys. We were able to successfully match presurveys and postsurveys for 13,544 respondents, for a match rate of 75 percent. The number of survey responses per year is summarized in Table 1. Table 1. Number of Completed Presurveys and Postsurveys by Year of Completion Survey Type Total Precourse 0 6,094 6,985 5,655 5,828 24,562 Postcourse 360 3,991 4,794 4,903 3,880 17,928 The total number of registrations and unique participants are summarized for each year in Table 2. The response rate could not be calculated either by year or across all years because the registration database was incomplete, as evidenced by the fact that there were greater numbers of presurveys completed than learners enrolled. Table 2. Number of Registrations and Participants by Year Total Registrations 3,634 3,768 4,401 4,980 5,001 21,784 Participants 2,756 2,706 3,424 3,638 3,792 16,316 1 The psychometric analysis involved Rasch scaling (Wright & Masters, 1982). The four items were as follows: this course met my expectations, this course provided me with content knowledge that I can apply to my instruction, this course provided me with instructional strategies that I can apply to my instruction, and, my students academic performance will benefit because I have participated in this course. PBS TeacherLine Evaluation 3

11 Survey Analysis Procedures We conducted a descriptive analysis of all forced choice and multiple choice survey items, calculating the number and percentage of respondents who selected each response option. These frequency tables are displayed in Appendix A for both the precourse and postcourse surveys. Each item was disaggregated by the school year in which the course took place and the curriculum area of the course. We selected the criterion of eight raw percentage points as the magnitude of a difference among years and curriculum areas that would bear mentioning in the report. No differences of this magnitude were observed, however. For the three open-ended responses on the survey (liked most, liked least, and recommendations for improvement), we analyzed a sample of 100 valid responses for each item. The 100 responses were chosen randomly within the parameter of 20 responses from each curriculum area. Responses not pertaining to the question (e.g., nothing or this was an excellent class in response to the question about what they liked least about the course) were removed from the sample and replaced with valid responses. Responses to each question were coded inductively to identify common themes. Interviews To provide an in-depth perspective on the evaluation questions, we conducted interviews with 21 learners who had taken a PBS TeacherLine course. To be eligible to participate in the interviews, the learner must have completed a course occurring in summer 2009 or later. Priority for sampling was given to learners in the two courses per curriculum area with the highest levels of registration. We intended to sample at least 10 repeat learners and at least five nonrepeat learners. A repeat learner is someone who has taken more than one course. However, several of the nonrepeat learners already had registered and begun a second course by the time of the interview; therefore, the final sample included 18 repeat learners. The final sample of learners was distributed across curriculum area as summarized in Table 3. Table 3. Number of Interviewees by Curriculum Area Curriculum Area Number of Interviewees Instructional strategies 3 Mathematics 5 Reading/language arts 5 Science 4 Technology 6 All interviews were conducted by telephone using a semistructured interview protocol (see Appendix B for a copy of this instrument). Interviews were taped, with the permission of the interviewee, and transcribed to ensure accuracy. The evaluation team used a systematic approach for conducting the interview analysis using NVivo software for qualitative research to support systematic procedures for coding and categorizing the data. We identified common themes for PBS TeacherLine Evaluation 4

12 each question and examined whether the prevalence of certain types of responses varied by curriculum area. Interviewee Characteristics Most of the 21 interviewees were experienced teachers, with more than half having taught for 11 or more years and three quarters having taught for six or more years. The interviewees were fairly distributed across grade levels, as described in Table 4. Only one interviewee taught in the upper elementary level exclusively, although four taught in multiple grade levels (one was a counselor). Ten interviewees taught in a specific core curriculum area (i.e., mathematics, reading/language arts, science, or social studies), with the other half teaching either multiple subjects (i.e., as an elementary teacher) or a specific noncore subject (art, physical education) (see Table 5). Table 4. Frequency of Interviewee Grade Level Taught Grade Level Number of Interviewees PK Multiple grades/counselor 5 Table 5. Frequency of Interviewee Curriculum Area Taught Curriculum Area Number of Interviewees Mathematics 4 Noncore area (art, physical education) Reading/language arts 2 Science 3 Social studies 1 Technology-computer 1 All subjects (e.g., elementary or special education) 2 8 PBS TeacherLine Evaluation 5

13 Impact of Course on Knowledge and Practice This section summarizes findings pertaining to the first three evaluation questions: 1. To what extent has participation in Courses enhanced knowledge of subject content, pedagogy, and assessment? 2. To what extent has participation in Courses influenced learner practices related to curriculum, instruction, and assessment? 3. Which Courses, and which types of Courses, have the greatest impact on learner knowledge and practice, and for what reason(s)? Effect of PBS TeacherLine Courses on Knowledge of Content and Instructional Strategies In comparing the effect of PBS TeacherLine courses on teachers knowledge, two presurvey and postsurvey items while not identical are close enough to be compared. Both questions ask respondents to rate their level of knowledge (of the content and of the pedagogy addressed in the course, respectively). These findings suggest that learner ratings of their level of knowledge shift strongly upward after completing a PBS TeacherLine course. None of the findings in this section differed as a function of curriculum area or year. Regarding content knowledge, only about 30 percent of learners rated themselves as very knowledgeable (3 percent) or knowledgeable (27 percent) prior to their course, but more than 90 percent rated themselves as very knowledgeable or knowledgeable (46 percent for both) after completing their course. Moreover, most individuals increased their ratings of their content knowledge from presurvey to postsurvey. Excluding the 3 percent of participants who already rated themselves as very knowledgeable at presurvey, 84 percent of participants increased their ratings of knowledge level from presurvey to postsurvey. During interviews, learners were asked to describe the ways in which the course expanded content knowledge. The majority of participants (18 of 21) indicated that they expanded their content knowledge because they gained new knowledge and perspectives from their PBS TeacherLine course. As one participant said, The Web design was totally new for me so it developed and expanded all of my content knowledge there. With the math one, it just expanded my content knowledge as to newer online resources. Three respondents indicated that they gained knowledge of strategies for teaching students. Two of these respondents took a course focused on teaching mathematics to students in special education. Their course experience is reflected in the following quote: I have a much better understanding of special education labels and terms and of the way special education looks not only in a classroom but as a department. And because these special education teachers that were in this class along with the general education teachers like me, I could hear what is going on all over the nation and that was a huge, interesting point for me. PBS TeacherLine Evaluation 6

14 Regarding knowledge of instructional strategies, only 27 percent of learners rated themselves as very knowledgeable (2 percent) or knowledgeable (25 percent) prior to their course, but more than 90 percent rated themselves as very knowledgeable or knowledgeable (43 and 48 percent, respectively) after completing their course. Moreover, most respondents increased their ratings of knowledge of instructional strategies; excluding the 2 percent that rated themselves as very knowledgeable at presurvey, 84 percent of participants increased their ratings from presurvey to postsurvey. The same teacher quoted previously described the impact on knowledge of strategies as follows: I learned a lot about special education and those terms, and I learned how again to identify specific areas of disability in all students, and I learned how to deal with those types [of] students [and learned] good ideas for differentiation to deliver good lessons for those students. There is one important caveat about the survey findings, which is that the wording of these two items changed from the precourse to the postcourse survey. Although items on both surveys asked learners to rate their current level of knowledge, the two items on the postsurvey were prefaced with the phrase Now that you have completed the PBS TeacherLine course. The possibility that the difference in phrasing may have biased these results cannot be discounted. However, it is quite easy to test this possibility by modifying subsequent versions of the surveys so that the items are exactly the same on the presurvey and postsurvey. Impact of PBS TeacherLine Courses on Teaching Practice On the postsurvey, teachers indicated whether or not they had incorporated any of the content or strategies from their courses into their teaching practice. Responses to these items are summarized in Table 6. Table 6. Proportion of Participants Incorporating Knowledge Into Practice 37. To date, have you been able to incorporate any content area knowledge from this PBS TeacherLine course into your teaching practice? 39. To date, have you been able to incorporate any instructional strategies from this PBS TeacherLine course into your teaching practice? N Yes No Does not apply Not yet but I plan to 17, % 9.2% 15.4% 17,817 70% 9% 15.1% 5.9% Note: The response option of Not yet but I plan to was available only for item 39. Overall, nearly three quarters of those surveyed reported incorporating content knowledge from a PBS TeacherLine course into their teaching practice. Seventy percent of respondents reported incorporating instructional strategies into their teaching practice based on the course they had taken. Fewer than 10 percent responded that they had not incorporated content or strategies, with PBS TeacherLine Evaluation 7

15 the remaining portion (about 15 percent) indicating that they were not currently in a teaching position. During interviews, learners were asked to describe what ways, if any, the course made an impact on the way they taught. All but one respondent indicated at least one way that their PBS TeacherLine course influenced their teaching practice. Eleven of 21 respondents indicated that their course influenced the way they teach because they learned and applied new teaching strategies in the classroom. As one teacher stated: [Y]ou would do different strategies and different things to see if they were gaining words per minute, if they were getting faster, if their fluency rate was going up, all those things. So that class was a really good class. Several teachers also stated that the course affected student learning because they learned or used technology integration (8 of 21) or used and applied resources and materials from the course (7 of 21). One teacher stated the following about technological resources gained through the course: There are some websites where I can direct my students and we can do some activities together. There are some things they can use for further study at home. The interview questions asked participants to elaborate on the impact of the course on several specific aspects of their teaching practice: Additional teaching resources. Respondents indicated that PBS TeacherLine courses provided them with numerous materials that they could use for both lesson planning and within the classroom. One teacher summarized the number of resources as follows: I have more resources for my students, and access to more resources and activities and electronic activities. Looking at the types of tools and resources, 16 respondents indicated that their course provided them with links to Web-based resources. This response was particularly common among participants who completed a technology course (all five respondents). As one teacher said, I take my students to the computer lab more. It gave me confidence on [the] computer and Internet. Five teachers indicated that their PBS TeacherLine course provided them with templates, models, or graphic organizers that they could use in the classroom. Confidence in teaching. The majority of teachers (17 of 21) said that their PBS course made them more confident in their teaching ability. One teacher strongly agreed with this, stating, Significantly, I am not a trained reading teacher, but because of the PBS courses I have taken, I am confident to discuss aspects of reading instruction. I was not able to do so a year ago. Four respondents indicated that their course did not improve their confidence, but this may be reflected in the statement of one teacher: I do not really rely on a course to give me confidence in my teaching ability. Communities of practice. Only three respondents indicated that their course expanded their community of practice. Most respondents (14 of 21) reported that the course expanded their community of practice but only for the duration of the course. PBS TeacherLine Evaluation 8

16 Impact on Student Learning During interviews, learners were asked to describe how these changes in learning made an impact on student learning. The majority of teachers indicated that PBS TeacherLine courses have either directly or indirectly influenced student learning through the strategies they have applied or content knowledge they acquired. Nine of 21 teachers responded that courses appear to impact student performance or progress. As one teacher said: Oh, [it] definitely [impacts learning]. Well, I see it in performance. I am able to recognize it better when students either master a skill or need additional help in developing a skill. The resources I have been exposed to have a direct impact on student performance. The resources, research, collegial discussions, and facilitator [all made an impact]. Six teachers reported that students are more engaged in the classroom from the skills and knowledge they gained in the course. One teacher described the change in her students after incorporating the differentiated instruction she learned from her course: They are more engaged in the lessons now. They have more confidence in the lessons because, basically, the lessons are tailor-made for them now. So, they are given more chances to succeed now, and it [is] showing in their confidence. Five teachers reported that the course has had an impact because they know more about the content area and can pass this information on to students. Of these teachers, three had completed a course in science. Differences Among Courses in Level of Impact There were no noticeable differences in learner ratings of impact as a function of curriculum area, either in respect to impact on knowledge or impact on instructional practice. PBS TeacherLine Evaluation 9

17 Course Quality This section addresses the following evaluation question: 4. What is the level of satisfaction with course quality, and how can Courses be improved? This section summarizes learner opinions on course quality in respect to course facilitation, course content, and learning activities. Quality of Course Facilitation Course facilitation refers to the set of skills and best practices that the course instructor engages in. These include managing discussions, providing feedback on assignments, answering questions from learners, and reminding students about and clarifying assignments. On the postsurvey, learners answered nine questions addressing their course instructor s frequency of engaging in these best practices. In general, most respondents reported that their facilitator engaged in these practices all the time or most of the time (see Table 7). Table 7. Ratings of Facilitation Quality 43. In the discussion forums, your facilitator provided at least one summary post per week? 44. In the discussion forums, your facilitator asked probing questions to advance and deepen the discussion? 45. Your facilitator demonstrated knowledge of the content area? 46. Your facilitator established a weekly schedule with s and announcements? 47. Your facilitator responded to requests for assistance (via or discussion board) within 24 hours? 48. Your facilitator gave you weekly feedback via e- mail or the online journal or the grades tool? 49. Your facilitator began each discussion forum with a well-written question or prompt? 50. Your facilitator provided administrative information such as graduate credit information? N All the time Most of the time Sometimes Never 17, % 14.9% 8.5% 4.5% 17, % 22.4% 14.8% 5.4% 17, % 15% 6.9% 2.3% 17, % 8.9% 4% 1.6% 17, % 11.8% 5% 1.6% 17, % 14.9% 8.3% 2.6% 17,715 83% 10.8% 4.3% 2% 17, % 9.5% 4.7% 4.5% The only item in which fewer than 80 percent selected either of these two responses referred to the facilitator asking probing questions to advance and deepen the discussion. Perhaps, then, this is a more challenging aspect of facilitation. Overall satisfaction with facilitator. Overall, the majority of respondents were satisfied with their instructor; when asked if they would recommend that their colleagues take a course taught by their facilitator, 92 percent responded in the affirmative. PBS TeacherLine Evaluation 10

18 Facilitator support. In interviews, PBS TeacherLine learners were asked to describe the ways that their instructor supported them in completing coursework. Interview respondents indicated that course instructors showed support in the following ways: Feedback. Most frequently, participants (11 of 21) said that their instructor supported course completion by providing useful feedback. One participant remarked, [S]he presented sincere comments so you felt she read your paper, and she synthesized [her comments] and then provided suggestions. She personalized her feedback to you. Responsiveness. Nine of 21 respondents said that a timely response to questions was another way that instructors supported them in completing coursework. A respondent said, You can always them and they ll you back. Usually within 24 hours you get a reply. And they were all pretty good with replying. Clarifying grades and assignments. Six respondents described how instructors provided frequent information regarding grades, weekly schedule, due dates, etc. She certainly made it clear when the assignments were due and what was expected in each week. Areas for Improvement in Course Facilitation When asked to describe how the instructor could have been more supportive, most learners (15 of 21) said that support by their instructor was not lacking in any way. However, five of 21 suggested that the instructor should have provided more feedback or provided more clarity in the form of examples and explanations: [S]ome of his answers to help weren t very helpful they [the instructor] just kind of pointed you back to the syllabus, which, obviously, if we re asking for help, we read the syllabus and didn t understand. Three respondents also mentioned that the instructor was inflexible or rigid in one way or another. The following comment reflects this sentiment: [T]he major thing that makes people go away from online classes or gives them a bad name is that it s supposed to be very much more flexible, and, if it isn t, then there is no use taking the class because you could do that face-to-face the rigid types of demands that you had to do it [assignments or postings] at certain times. Another respondent felt that the lack of flexibility was an affront of sorts to her professionalism: I remember feeling like I m a professional give me the respect that a professional deserves, and don t tell me that I need to do ABCDEFG; just give me a question, and let me expand on it with my colleagues. PBS TeacherLine Evaluation 11

19 Quality of Course Content Course content refers to the topics covered, materials, and assignments in the course, along with their organization. On the survey, learners rated their agreement with four statements about the quality of course content. Their ratings of these items are summarized in Table 8. As with the ratings of facilitation quality, learners expressed strongly favorable opinions of course content. For every item, more than 90 percent strongly agreed or agreed that the course met expectations, provided content or strategies that were applicable to instruction, and prepared them to improve their students academic performance. In line with the overall positivity of these survey responses, six interviewees offered general words of praise about course quality, such as the following: The PBS class was hands down the best content I had ever come across in terms of an online class. Respondents elaborated on their opinions about the course in interviews and open-ended survey responses, as will be described. Table 8. Postsurvey Ratings of Course Quality N Strongly Agree Agree Neutral Disagree Strongly Disagree 53. This course met my expectations. 17, % 32.4% 6% 3% 1.2% 54. This course provided me with content knowledge that I can apply to my instruction. 55. This course provided me with instructional strategies that I can apply to my instruction. 56. My students academic performance will benefit because I have participated in this course. Relevance of Courses 17, % 30.8% 4.4% 1% 0.5% 17, % 30.8% 4.6% 1% 0.5% 17, % 33.5% 7.5% 1% 0.5% Learners reported that the courses focused on the topics that they purported to address. In the survey, 90 percent of respondents either strongly agreed (57 percent) or agreed (32 percent) that the course met their expectations. Six of 21 interviewees mentioned this for a reason why they liked the course content. A typical comment was, It was excellent quality because I knew what I was signing up for when I took it. I had to do exactly what I saw in the syllabus. Consistent with this finding, another interviewee praised the relevance of the course: I m teaching middle school right now and it addresses middle school learners. I thought that all of the topics were very relevant for what I do. Usefulness of Courses Learners typically reported that the courses were highly useful and applicable. In the survey, 94 percent of respondents either strongly agreed (63 percent) or agreed (31 percent) that the course addressed content knowledge and instructional strategies that were highly applicable to their instruction (two separate items). Moreover, in response to the open-ended survey item, 35 PBS TeacherLine Evaluation 12

20 percent of respondents identified the relevance of course content as what they liked best. Similarly, seven of 21 interviewees mentioned this as the reason for why they liked the course content. These respondents emphasized the applicability of what was learned. Some representative responses (from open-ended responses) are as follows: I really liked the content, the discussions, and that I was asked to apply some of the learning in a practical way. I liked that I could immediately take what I learned back to the classroom. The walkthroughs were very easy to follow. Usefulness of resources. Other respondents commented on the usefulness of the resources to which they were exposed. For example, one survey respondent stated: I loved the wide variety of resources offered. I was introduced to new websites such as Reading Rockets. I had no idea PBS offers so much for teachers, students, and parents on their website. In response to an interview question about the resources teachers gained, all respondents indicated that PBS TeacherLine courses provided them with materials that they could use for lesson planning or within the classroom. One teacher summarized the number of resources in the following comment: I have more resources for my students, and access to more resources and activities and electronic activities. The following were the most prevalent types of tools and resources: Links to Web-based resources (16 of 21 respondents). This response was particularly common among participants who completed a technology course (all five respondents). Templates, models, or graphic organizers that they could use in the classroom (five responses). Videos (four responses). Lesson plans (four responses) in their course. Quality of Materials Although there were no fixed-choice survey items addressing the course materials, learners expressed their opinions in open-ended survey items and in the interviews. The feedback was generally positive, with some specific criticisms as well. On the positive side, three interviewees praised the quality of materials, along with seven percent of the respondents to the open-ended item. 2 These comments typically mentioned the readings, videos of lessons, and websites, as expressed as follows by one interviewee: I really enjoyed the material that was presented, the little video clips and the readings that we had to do. Another interviewee remarked, The materials were great. They were very clear. It was all online. 2 Many more respondents praised course content in general, which may also have referred to the quality of materials. PBS TeacherLine Evaluation 13

21 On the other hand, several respondents offered criticism about the materials and suggested what could be improved. About seven percent of responses to the open-ended question liked least and how to improve focused on materials. These respondents fell disproportionately within the technology course area; five respondents from this area suggested that the materials needed to be updated to reflect changes in technology. One participant wrote: [Provide] updated reading materials the copyright on some of them was nearly 10 years old! One about search engines didn t even mention Google. A tech class should have more up-to-date information. Along similar lines, one interviewee pointed out that some course links were out of date Some [of] the materials for our course were no longer online, some of the assignments had to be changed at the last minute, and I like to work ahead when I have the time. Variation in Course Quality Course areas differed somewhat in their postsurvey ratings of course quality. While this difference is statistically significant, the magnitude of the difference does not appear to be large or important. This is illustrated in Figure 1. Given that the scale scores range from about 4 to 9, the range from the highest mean (6.8 in reading/language arts) to the lowest mean (6.0 in technology) is not large. 3 Figure 1. Mean Course Quality Scale Score as a Function of Curriculum Area 3 A single course quality measure was derived by combining the four course quality survey items into a single scale score. The course quality score was related to reported level of content knowledge; ratings of course quality were correlated with ratings of knowledge on the postsurvey. On a scale of 0 to 1, the correlation of course quality and content knowledge is 0.45, and the correlation of course quality and knowledge of teaching strategies is The test statistic is Spearman s Rho, with df = 17,815 and 17,805, respectively. PBS TeacherLine Evaluation 14

22 Discussions and Assignments In all courses, learners were expected to post comments to the discussion board and respond to the comments of other learners. When asked which learning activities were the most useful, interviewees (10 of 21) most frequently named the discussion forums and interaction with other participants (e.g., peer review). Similarly, 27 percent of respondents to the survey item about what they liked most mentioned interactions with peers or the facilitator. One interviewee remarked as follows: [W]ith the discussion board, you just learn so much. It s good to connect with other people all over the country. With PBS, you could be anywhere in the country and connecting and getting ideas, so I think that s just wonderful. Another respondent explained, [T]he discussion board is so helpful because you get ideas and hints from other teachers. Finally, one respondent indicated that participants also used the discussion boards to share useful resources: I was very happy with people bringing their different areas of expertise to the discussion and sharing resources and sharing good websites and ideas about how to present things or some people shared their PowerPoints that they had made. However, the main criticism of the discussion activity was that there was too much interaction due to too many requirements for posting and responding to one another. This was the second most frequent open-ended response to what learners liked least about the course, mentioned by 20 percent of respondents to that item. Many of these respondents said it took too much time to read all of the postings and respond. In addition, several respondents disliked the requirements regarding the discussions, which they considered onerous: The online discussions were not as beneficial as they could have been. Too much time was spent reading information and viewpoints that will not help me improve my teaching ability. I sometimes felt overwhelmed by the number of discussion posts. One participant explained how the point requirements for posting and responding to comments were burdensome: Being so worried about points, people stopped doing what the whole point was, and that was to converse and share thoughts by the end of it, [it] was absolutely just answer the question, get your posts in, and it was not meaningful. PBS TeacherLine Evaluation 15