PBS TeacherLine Pre-Survey and Post-Survey Evaluation Report

Transcription

1 PBS TeacherLine Pre-Survey and Post-Survey Evaluation Report June 2010 Prepared by: Jonathan Margolin, Ph.D. Debbie Davidson-Gibbs Natalie Tucker Data analyst: Jiuping Chen 20 N. Wacker Drive, Suite 1231 Chicago, IL Copyright All rights reserved. 4698_06/10

2 Contents Page Executive Summary... i PBS TeacherLine Presurvey and Postsurvey: Evaluation Report... 1 Evaluation Purpose and Questions... 1 Data Sources... 2 Interviews... 4 Impact of Course on Knowledge and Practice... 6 Effect of PBS TeacherLine Courses on Knowledge of Content and Instructional Strategies... 6 Impact of PBS TeacherLine Courses on Teaching Practice... 7 Differences Among Courses in Level of Impact... 9 Course Quality Quality of Course Facilitation Quality of Course Content Discussions and Assignments Customer Satisfaction Repeat Learners Conclusions and Recommendations Reference Appendix A. Survey Tables PBS TeacherLine Presurvey Tables Postsurvey Response Tables Appendix B. Interview Protocol Appendix C. Recommendations for PBS TeacherLine Presurvey and Postsurvey... 29

3 Executive Summary The evaluation has three key goals. First, the goal of the evaluation is to evaluate the effectiveness of PBS TeacherLine courses on improving teachers knowledge, skills, and practice. In a broader sense, the purpose is to determine the value of courses. A second goal of the evaluation is to determine whether courses vary in their effectiveness and to try to understand the reasons for this variation. The broader purpose is to provide feedback that is relevant to the improvement of the quality and effectiveness of courses. A third goal of the evaluation is to describe the characteristics of repeat learners (i.e., teachers who participate in multiple PBS TeacherLine courses). The following evaluation questions address these goals and purposes: 1. To what extent has participation in Courses enhanced knowledge of subject content, pedagogy, and assessment? 2. To what extent has participation in Courses influenced learner practices related to curriculum, instruction, and assessment? 3. Which Courses, and which types of Courses, have the greatest impact on learner knowledge and practice, and for what reason(s)? 4. What is the level of satisfaction with course quality, and how can Courses be improved? 5. What are the characteristics of repeat learners in terms of professional development background, Courses taken, satisfaction with Courses over time, and changes in teaching practice? Question 1. To what extent has participation in Courses enhanced knowledge of subject content, pedagogy, and assessment? Learner ratings of their level of knowledge of content and instructional strategies shift strongly upward after completing a PBS TeacherLine course. Eighty-four percent of participants increased their ratings of their knowledge of both content and instructional strategies from presurvey to postsurvey. This finding excludes the two to three percent of participants who already rated themselves at the highest level at the time of the pretest. During interviews, learners were asked to describe the ways in which the course expanded content knowledge. The majority of participants (18 of 21) indicated that they expanded their content knowledge because they learned a lot and better understood alternative perspectives from their PBS TeacherLine course. As one participant said, The Web design was totally new for me so it developed and expanded all of my content knowledge there. With the math one, it just expanded my content knowledge as to newer online resources. Question 2. To what extent has participation in Courses influenced learner practices related to curriculum, instruction, and assessment? Nearly three quarters of those surveyed after taking a course reported incorporating content knowledge from a PBS TeacherLine course into their teaching practice. Seventy percent of respondents reported incorporating instructional strategies into their teaching practice based on PBS TeacherLine Evaluation: Executive Summary i

4 the course they had taken. Fewer than 10 percent responded that they had not incorporated content or strategies, with the remaining portion (about 15 percent) indicating that they were not currently in a teaching position. During interviews, all but one respondent indicated at least one way that their PBS TeacherLine course influenced their teaching practice. Major themes identified by these interviewees were as follows: Teachers learned and applied new teaching strategies in the classroom. Teachers learned how to integrate technology. Teachers used and applied resources and materials from the course, most prevalently, links to Web-based resources, as well as templates, models, or graphic organizers that they could use in the classroom. Teachers gained confidence in their teaching ability. Question 3. Which Courses, and which types of Courses, have the greatest impact on learner knowledge and practice, and for what reason(s)? There were no noticeable differences in learner ratings of impact as a function of curriculum area, either in respect to impact on knowledge or impact on instructional practice. Question 4. What is the level of satisfaction with course quality, and how can Courses be improved? Course satisfaction was measured in terms of satisfaction with the instructor, the course content, and participants interactions with other learners. Overall, the majority of respondents were satisfied with their instructor; when asked if they would recommend that their colleagues take a course taught by their facilitator, 92 percent responded in the affirmative. Most survey respondents agreed that their facilitator frequently engaged in best practices such as providing feedback on assignments, answering questions from learners, and reminding students about and clarifying assignments. These three practices also were mentioned frequently by interviewees as the ways in which instructors supported their learning. A minority of learners suggested that the facilitator could have been more supportive, such as by providing more feedback on assignments or more clarity in the form of examples and explanations. Some would have preferred greater flexibility about assignments and due dates. Learners expressed strongly favorable opinions of course content. For every item, more than 90 percent strongly agreed or agreed that the course met expectations, provided content or strategies that were applicable to instruction, and prepared them to improve their students academic performance. Moreover, all interviewees indicated that their course provided them with useful resources. Several learners also praised the quality of course materials, such as the readings, videos of lessons, and websites. However, a number of learners, especially in technology courses, suggested that the materials needed to be more up-to-date. PBS TeacherLine Evaluation: Executive Summary ii

5 Learners frequently mentioned the discussions with other participants as the most beneficial aspect of the course. Learners stated that they benefitted from the opportunity to share ideas and resources with learners from around the country. However, several learners disliked the requirements regarding the discussions (e.g., the number of posts required), which they considered onerous. In general, the most prevalent criticism (expressed by 40 percent of learners) was that the amount of work was not commensurate with the number credits awarded. A minority of learners stated that the assignments were either not explained clearly or were too time consuming given the number of credits offered for the course. Question 5. What are the characteristics of repeat learners in terms of professional development background, Courses taken, satisfaction with Courses over time, and changes in teaching practice? High repeaters (who took four or more courses), in comparison with nonrepeaters (who took only one PBS course) and low repeaters (who took two courses), more frequently stated that they were taking the course to earn graduate credit. Repeat learners were asked in interviews why they took more than one course. Learners most frequently mentioned the convenience of the online format as the reason for taking more than one PBS TeacherLine course. Similarly, the convenience of the online format was the most frequent response to the survey item asking what learners liked best about their course. A smaller number of interviewees mentioned only their positive experiences in previous PBS courses. Finally, a review of registration records indicates that repeat learners are typically completing most of their courses in the same curriculum area. The pattern that emerges from these three findings is that learners take multiple PBS TeacherLine courses to collect graduate credit in a specific curriculum area, and they value the flexibility of the online format. Although the online format was a strong selling point for the program, a minority of learners voiced criticism about the way certain courses were implemented. About one fifth of learners reported experiencing some technical difficulties. A minority stated that the layout of the course lacked clarity, especially in regard to the listing of assignments and due dates. Summary of Program Strengths Based on these findings, the following is a summary of program strengths: 1. Learners typically increased understanding of content and instructional strategies after completing their PBS TeacherLine course. One English as a second language (ESL) teacher in her 34th year of teaching stated, I have a much better understanding of special education labels and terms and of the way special education looks not only in a classroom but as a department. 2. PBS TeacherLine courses provide useful teaching strategies. Most teachers were able to apply what they learned directly to the classroom setting. One survey respondent stated, I liked that I could immediately take what I learned back to the classroom. The walkthroughs were very easy to follow. PBS TeacherLine Evaluation: Executive Summary iii

6 3. The courses typically provide useful resources and materials that are applicable to instruction, such as Web-based curriculum resources, templates, models, and graphic organizers. One survey respondent stated, I loved the wide variety of resources offered. I was introduced to new websites such as Reading Rockets. I had no idea PBS offers so much for teachers, students, and parents on their website. 4. PBS TeacherLine improves confidence in teaching practice. For example, a high school science teacher in her 14th year stated, It made me more confident in having students do things on the computer because I got better at figuring out what was going wrong. 5. Facilitators of PBS TeacherLine courses typically exhibit best practices such as providing feedback on assignments, promptly answering questions, and clarifying assignments. A second-grade teacher in her third year stated, The instructor was great. Very hands on. If I asked her a question, she responded immediately. She gave me feedback on my work all the time. However, a minority of respondents stated that their instructors should have provided more feedback on work or explanations of assignments. A minority also described their instructors as lacking appropriate flexibility. 6. Courses present content through a variety of modalities, such as readings, videos of lessons, and websites. A high school mathematics teacher in her 10th year remarked, The materials were great. They were very clear. It was all online. 7. Courses provide frequent opportunities for learners to share ideas and resources. Many learners considered these interactions to be a strength of the program, including an art teacher who stated, With the discussion board, you just learn so much. It s good to connect with other people all over the country. With PBS, you could be anywhere in the country and connecting and getting ideas, so I think that s just wonderful. 8. The online format of PBS TeacherLine courses makes the program highly convenient, allowing teachers to incorporate their coursework into their daily schedules. One survey respondent noted, It allowed me to teach during the day, spend family time, then do PBS work. However, a minority of learners noted that the online layout of their course could be improved. 9. Most teachers agreed that the course made an impact on their students learning, in terms of student engagement and academic progress. As one second-grade teacher stated, They definitely improved in their fluency rate. They definitely started reading faster and there was more of a flow. One teacher of middle school mathematics in her eighth year stated, They are more engaged in the lessons now. They have more confidence in the lessons because basically the lessons are tailor-made for them now. Recommendations for Improving Courses Based on the feedback from learners summarized above, the following are practical ways to improve courses: 1. Maintain a regular schedule for review and revision of course content. Several learners, particularly those in technology courses, noted that some content and resources appeared to be dated. Courses should be reviewed regularly to confirm that all URLs are PBS TeacherLine Evaluation: Executive Summary iv

7 operational. Course content should be reviewed to ensure that it is current, which is particularly important for technology courses. 2. Summarize course assignments and due dates in one place. Some learners stated that there was not a clear week-by-week summary of assignments and due dates. Each course should provide a summary of what is due in a particular week. 3. Instructors should provide flexible guidelines for discussions. Some learners noted that keeping up with the requirements for participating in discussions became onerous due to the large volume of comments. Facilitators might be able to provide some flexibility to learners in terms of requirements for reading and responding to posts so that these requirements should not be more time consuming than is warranted. 4. Gather feedback on the amount of coursework. Several learners perceived that the amount of coursework greatly exceeded their expectations for the course, based on the number of credits offered. It would be feasible to collect input from participants about the amount of time they spent on assignments and whether that amount of time was reasonable. It would be possible to adjust the amount of work in courses in response to this feedback. PBS TeacherLine Evaluation: Executive Summary v

8 PBS TeacherLine Presurvey and Postsurvey Evaluation Report PBS TeacherLine provides online professional development to educators across the country with funding through its Ready to Teach grant administered by the U. S. Department of Education. In 2000, PBS TeacherLine began offering cohorts of K 12 teachers a variety of noncredit online courses in connection with local PBS stations. PBS TeacherLine aims to improve teacher practices and help educators acquire the skills they need to become and remain highly qualified. PBS TeacherLine offers online graduate-level professional development courses for K 12 teachers in the following curriculum areas: reading/language arts, mathematics, instructional technology, and instruction. Most courses are 30-hour courses that are conducted for six weeks. According to the PBS TeacherLine website, all courses are designed to improve participant attitudes and practices by incorporating weekly assignments, discussion boards, and opportunities for reflection. The pedagogy of each course involves inquiry-based and projectbased learning culminating in a project that is a tangible activity, case study, or strategic plan for classroom implementation. Most courses are intended to provide teachers with specific resources or tools they can use in the classroom. All course facilitators undergo PBS TeacherLine s rigorous Online Facilitation Training program and are highly qualified teachers with master s degrees. The PBS TeacherLine program is flexible and customizable. There are two common modes of participation. First, districts and schools can enroll cohorts of teachers in customized courses to meet a district or school s strategic needs and objectives. Second, teachers can enroll independently to pursue their own particular interests or to align with their professional development requirements and goals. Most PBS TeacherLine courses allow educators to receive graduate credits from local partner colleges and universities, as four universities across the country grant credit for all PBS TeacherLine courses. Educators also may apply credits toward district professional development points or continuing education units (CEUs). The overall purpose of PBS TeacherLine courses, as mentioned earlier, is to improve teacher practices and help educators acquire the skills they need to become and remain highly qualified. Specifically, participation in PBS TeacherLine courses increases knowledge and understanding of high-quality teaching through reading, discussion, and assignments. Participation improves teaching skills through guided practice, reflection, and feedback from the instructor. Improved knowledge and teaching skills will improve the effectiveness of teaching (i.e., student learning and achievement). Currently, PBS TeacherLine offers more than 120 courses nationally to PK 12 educators. Although PBS TeacherLine initially targeted teachers in Title I schools, the clientele who currently participate in PBS TeacherLine are educators in schools with a wide range of demographic characteristics. Individual participants include classroom teachers as well as technology specialists and media/library specialists. Evaluation Purpose and Questions The primary goal of the evaluation is to evaluate the effectiveness of PBS TeacherLine courses on improvements to knowledge, skills, and practice. In a broader sense, the purpose is to PBS TeacherLine Evaluation 1

9 determine the value of courses. A second goal is to determine whether courses vary in their effectiveness and to try to understand the reasons for this variation. The broader purpose is to provide feedback that is relevant to the improvement of the quality and effectiveness of courses. A third goal of the evaluation is to describe the characteristics of repeat learners (i.e., teachers who participate in multiple PBS TeacherLine courses). The following evaluation questions address these goals: 1. To what extent has participation in Courses enhanced knowledge of subject content, pedagogy, and assessment? 2. To what extent has participation in Courses influenced learner practices related to curriculum, instruction, and assessment? 3. Which Courses, and which types of Courses, have the greatest impact on learner knowledge and practice, and for what reason(s)? 4. What is the level of satisfaction with course quality, and how can Courses be improved? 5. What are the characteristics of repeat learners in terms of professional development background, Courses taken, satisfaction with Courses over time, and changes in teaching practice? Data Sources The following sources of data addressed the evaluation questions: Review of PBS Presurvey and Postsurvey PBS TeacherLine has administered precourse and postcourse surveys to learners starting in fall 2004 and continuing through the last semester covered by this evaluation (fall 2009). These surveys were developed by PBS TeacherLine and have remained essentially the same throughout the past six years. A content review of the surveys conducted by indicates that the items address several different constructs. The 19 items on the precourse survey aligned with the following constructs: Reasons or goals for participating (five items) Level of comfort (six items)* Level of knowledge of course content (two items) District/school involvement in course selection (two items) Experience with other, non-teacherline professional development courses (four items) The 32 items on the postcourse survey aligned with the following constructs: Overall satisfaction with course (two items) Level of knowledge of course content (two items) Impact on teaching practice (three items) PBS TeacherLine Evaluation 2

10 Logistics of course participation (two items) Satisfaction with course facilitation (nine items)* Satisfaction with course quality (five items)* Satisfaction with customer service quality (five items)* Liked most, least, and how to improve (three items) Suggestions for additional course offerings (one item) Based on this review, we determined that there was one construct included on both the precourse and postcourse versions of the survey: level of knowledge of course content. Even though the items on this construct were not exactly the same across the precourse and postcourse versions, we nevertheless calculated the change in ratings of course knowledge from precourse to postcourse. Furthermore, we evaluated the psychometric properties of the four constructs marked with an asterisk to determine whether each set of items could be combined into a single scale score. 1 Only the items related to satisfaction with course quality could be combined into a scale score on the basis of this analysis. Survey Administration and Response Rate The analyses were based on 24,562 completed precourse surveys and 17,933 completed postcourse surveys. We were able to successfully match presurveys and postsurveys for 13,544 respondents, for a match rate of 75 percent. The number of survey responses per year is summarized in Table 1. Table 1. Number of Completed Presurveys and Postsurveys by Year of Completion Survey Type Total Precourse 0 6,094 6,985 5,655 5,828 24,562 Postcourse 360 3,991 4,794 4,903 3,880 17,928 The total number of registrations and unique participants are summarized for each year in Table 2. The response rate could not be calculated either by year or across all years because the registration database was incomplete, as evidenced by the fact that there were greater numbers of presurveys completed than learners enrolled. Table 2. Number of Registrations and Participants by Year Total Registrations 3,634 3,768 4,401 4,980 5,001 21,784 Participants 2,756 2,706 3,424 3,638 3,792 16,316 1 The psychometric analysis involved Rasch scaling (Wright & Masters, 1982). The four items were as follows: this course met my expectations, this course provided me with content knowledge that I can apply to my instruction, this course provided me with instructional strategies that I can apply to my instruction, and, my students academic performance will benefit because I have participated in this course. PBS TeacherLine Evaluation 3

11 Survey Analysis Procedures We conducted a descriptive analysis of all forced choice and multiple choice survey items, calculating the number and percentage of respondents who selected each response option. These frequency tables are displayed in Appendix A for both the precourse and postcourse surveys. Each item was disaggregated by the school year in which the course took place and the curriculum area of the course. We selected the criterion of eight raw percentage points as the magnitude of a difference among years and curriculum areas that would bear mentioning in the report. No differences of this magnitude were observed, however. For the three open-ended responses on the survey (liked most, liked least, and recommendations for improvement), we analyzed a sample of 100 valid responses for each item. The 100 responses were chosen randomly within the parameter of 20 responses from each curriculum area. Responses not pertaining to the question (e.g., nothing or this was an excellent class in response to the question about what they liked least about the course) were removed from the sample and replaced with valid responses. Responses to each question were coded inductively to identify common themes. Interviews To provide an in-depth perspective on the evaluation questions, we conducted interviews with 21 learners who had taken a PBS TeacherLine course. To be eligible to participate in the interviews, the learner must have completed a course occurring in summer 2009 or later. Priority for sampling was given to learners in the two courses per curriculum area with the highest levels of registration. We intended to sample at least 10 repeat learners and at least five nonrepeat learners. A repeat learner is someone who has taken more than one course. However, several of the nonrepeat learners already had registered and begun a second course by the time of the interview; therefore, the final sample included 18 repeat learners. The final sample of learners was distributed across curriculum area as summarized in Table 3. Table 3. Number of Interviewees by Curriculum Area Curriculum Area Number of Interviewees Instructional strategies 3 Mathematics 5 Reading/language arts 5 Science 4 Technology 6 All interviews were conducted by telephone using a semistructured interview protocol (see Appendix B for a copy of this instrument). Interviews were taped, with the permission of the interviewee, and transcribed to ensure accuracy. The evaluation team used a systematic approach for conducting the interview analysis using NVivo software for qualitative research to support systematic procedures for coding and categorizing the data. We identified common themes for PBS TeacherLine Evaluation 4

12 each question and examined whether the prevalence of certain types of responses varied by curriculum area. Interviewee Characteristics Most of the 21 interviewees were experienced teachers, with more than half having taught for 11 or more years and three quarters having taught for six or more years. The interviewees were fairly distributed across grade levels, as described in Table 4. Only one interviewee taught in the upper elementary level exclusively, although four taught in multiple grade levels (one was a counselor). Ten interviewees taught in a specific core curriculum area (i.e., mathematics, reading/language arts, science, or social studies), with the other half teaching either multiple subjects (i.e., as an elementary teacher) or a specific noncore subject (art, physical education) (see Table 5). Table 4. Frequency of Interviewee Grade Level Taught Grade Level Number of Interviewees PK Multiple grades/counselor 5 Table 5. Frequency of Interviewee Curriculum Area Taught Curriculum Area Number of Interviewees Mathematics 4 Noncore area (art, physical education) Reading/language arts 2 Science 3 Social studies 1 Technology-computer 1 All subjects (e.g., elementary or special education) 2 8 PBS TeacherLine Evaluation 5

13 Impact of Course on Knowledge and Practice This section summarizes findings pertaining to the first three evaluation questions: 1. To what extent has participation in Courses enhanced knowledge of subject content, pedagogy, and assessment? 2. To what extent has participation in Courses influenced learner practices related to curriculum, instruction, and assessment? 3. Which Courses, and which types of Courses, have the greatest impact on learner knowledge and practice, and for what reason(s)? Effect of PBS TeacherLine Courses on Knowledge of Content and Instructional Strategies In comparing the effect of PBS TeacherLine courses on teachers knowledge, two presurvey and postsurvey items while not identical are close enough to be compared. Both questions ask respondents to rate their level of knowledge (of the content and of the pedagogy addressed in the course, respectively). These findings suggest that learner ratings of their level of knowledge shift strongly upward after completing a PBS TeacherLine course. None of the findings in this section differed as a function of curriculum area or year. Regarding content knowledge, only about 30 percent of learners rated themselves as very knowledgeable (3 percent) or knowledgeable (27 percent) prior to their course, but more than 90 percent rated themselves as very knowledgeable or knowledgeable (46 percent for both) after completing their course. Moreover, most individuals increased their ratings of their content knowledge from presurvey to postsurvey. Excluding the 3 percent of participants who already rated themselves as very knowledgeable at presurvey, 84 percent of participants increased their ratings of knowledge level from presurvey to postsurvey. During interviews, learners were asked to describe the ways in which the course expanded content knowledge. The majority of participants (18 of 21) indicated that they expanded their content knowledge because they gained new knowledge and perspectives from their PBS TeacherLine course. As one participant said, The Web design was totally new for me so it developed and expanded all of my content knowledge there. With the math one, it just expanded my content knowledge as to newer online resources. Three respondents indicated that they gained knowledge of strategies for teaching students. Two of these respondents took a course focused on teaching mathematics to students in special education. Their course experience is reflected in the following quote: I have a much better understanding of special education labels and terms and of the way special education looks not only in a classroom but as a department. And because these special education teachers that were in this class along with the general education teachers like me, I could hear what is going on all over the nation and that was a huge, interesting point for me. PBS TeacherLine Evaluation 6

14 Regarding knowledge of instructional strategies, only 27 percent of learners rated themselves as very knowledgeable (2 percent) or knowledgeable (25 percent) prior to their course, but more than 90 percent rated themselves as very knowledgeable or knowledgeable (43 and 48 percent, respectively) after completing their course. Moreover, most respondents increased their ratings of knowledge of instructional strategies; excluding the 2 percent that rated themselves as very knowledgeable at presurvey, 84 percent of participants increased their ratings from presurvey to postsurvey. The same teacher quoted previously described the impact on knowledge of strategies as follows: I learned a lot about special education and those terms, and I learned how again to identify specific areas of disability in all students, and I learned how to deal with those types [of] students [and learned] good ideas for differentiation to deliver good lessons for those students. There is one important caveat about the survey findings, which is that the wording of these two items changed from the precourse to the postcourse survey. Although items on both surveys asked learners to rate their current level of knowledge, the two items on the postsurvey were prefaced with the phrase Now that you have completed the PBS TeacherLine course. The possibility that the difference in phrasing may have biased these results cannot be discounted. However, it is quite easy to test this possibility by modifying subsequent versions of the surveys so that the items are exactly the same on the presurvey and postsurvey. Impact of PBS TeacherLine Courses on Teaching Practice On the postsurvey, teachers indicated whether or not they had incorporated any of the content or strategies from their courses into their teaching practice. Responses to these items are summarized in Table 6. Table 6. Proportion of Participants Incorporating Knowledge Into Practice 37. To date, have you been able to incorporate any content area knowledge from this PBS TeacherLine course into your teaching practice? 39. To date, have you been able to incorporate any instructional strategies from this PBS TeacherLine course into your teaching practice? N Yes No Does not apply Not yet but I plan to 17, % 9.2% 15.4% 17,817 70% 9% 15.1% 5.9% Note: The response option of Not yet but I plan to was available only for item 39. Overall, nearly three quarters of those surveyed reported incorporating content knowledge from a PBS TeacherLine course into their teaching practice. Seventy percent of respondents reported incorporating instructional strategies into their teaching practice based on the course they had taken. Fewer than 10 percent responded that they had not incorporated content or strategies, with PBS TeacherLine Evaluation 7

15 the remaining portion (about 15 percent) indicating that they were not currently in a teaching position. During interviews, learners were asked to describe what ways, if any, the course made an impact on the way they taught. All but one respondent indicated at least one way that their PBS TeacherLine course influenced their teaching practice. Eleven of 21 respondents indicated that their course influenced the way they teach because they learned and applied new teaching strategies in the classroom. As one teacher stated: [Y]ou would do different strategies and different things to see if they were gaining words per minute, if they were getting faster, if their fluency rate was going up, all those things. So that class was a really good class. Several teachers also stated that the course affected student learning because they learned or used technology integration (8 of 21) or used and applied resources and materials from the course (7 of 21). One teacher stated the following about technological resources gained through the course: There are some websites where I can direct my students and we can do some activities together. There are some things they can use for further study at home. The interview questions asked participants to elaborate on the impact of the course on several specific aspects of their teaching practice: Additional teaching resources. Respondents indicated that PBS TeacherLine courses provided them with numerous materials that they could use for both lesson planning and within the classroom. One teacher summarized the number of resources as follows: I have more resources for my students, and access to more resources and activities and electronic activities. Looking at the types of tools and resources, 16 respondents indicated that their course provided them with links to Web-based resources. This response was particularly common among participants who completed a technology course (all five respondents). As one teacher said, I take my students to the computer lab more. It gave me confidence on [the] computer and Internet. Five teachers indicated that their PBS TeacherLine course provided them with templates, models, or graphic organizers that they could use in the classroom. Confidence in teaching. The majority of teachers (17 of 21) said that their PBS course made them more confident in their teaching ability. One teacher strongly agreed with this, stating, Significantly, I am not a trained reading teacher, but because of the PBS courses I have taken, I am confident to discuss aspects of reading instruction. I was not able to do so a year ago. Four respondents indicated that their course did not improve their confidence, but this may be reflected in the statement of one teacher: I do not really rely on a course to give me confidence in my teaching ability. Communities of practice. Only three respondents indicated that their course expanded their community of practice. Most respondents (14 of 21) reported that the course expanded their community of practice but only for the duration of the course. PBS TeacherLine Evaluation 8

16 Impact on Student Learning During interviews, learners were asked to describe how these changes in learning made an impact on student learning. The majority of teachers indicated that PBS TeacherLine courses have either directly or indirectly influenced student learning through the strategies they have applied or content knowledge they acquired. Nine of 21 teachers responded that courses appear to impact student performance or progress. As one teacher said: Oh, [it] definitely [impacts learning]. Well, I see it in performance. I am able to recognize it better when students either master a skill or need additional help in developing a skill. The resources I have been exposed to have a direct impact on student performance. The resources, research, collegial discussions, and facilitator [all made an impact]. Six teachers reported that students are more engaged in the classroom from the skills and knowledge they gained in the course. One teacher described the change in her students after incorporating the differentiated instruction she learned from her course: They are more engaged in the lessons now. They have more confidence in the lessons because, basically, the lessons are tailor-made for them now. So, they are given more chances to succeed now, and it [is] showing in their confidence. Five teachers reported that the course has had an impact because they know more about the content area and can pass this information on to students. Of these teachers, three had completed a course in science. Differences Among Courses in Level of Impact There were no noticeable differences in learner ratings of impact as a function of curriculum area, either in respect to impact on knowledge or impact on instructional practice. PBS TeacherLine Evaluation 9

17 Course Quality This section addresses the following evaluation question: 4. What is the level of satisfaction with course quality, and how can Courses be improved? This section summarizes learner opinions on course quality in respect to course facilitation, course content, and learning activities. Quality of Course Facilitation Course facilitation refers to the set of skills and best practices that the course instructor engages in. These include managing discussions, providing feedback on assignments, answering questions from learners, and reminding students about and clarifying assignments. On the postsurvey, learners answered nine questions addressing their course instructor s frequency of engaging in these best practices. In general, most respondents reported that their facilitator engaged in these practices all the time or most of the time (see Table 7). Table 7. Ratings of Facilitation Quality 43. In the discussion forums, your facilitator provided at least one summary post per week? 44. In the discussion forums, your facilitator asked probing questions to advance and deepen the discussion? 45. Your facilitator demonstrated knowledge of the content area? 46. Your facilitator established a weekly schedule with s and announcements? 47. Your facilitator responded to requests for assistance (via or discussion board) within 24 hours? 48. Your facilitator gave you weekly feedback via e- mail or the online journal or the grades tool? 49. Your facilitator began each discussion forum with a well-written question or prompt? 50. Your facilitator provided administrative information such as graduate credit information? N All the time Most of the time Sometimes Never 17, % 14.9% 8.5% 4.5% 17, % 22.4% 14.8% 5.4% 17, % 15% 6.9% 2.3% 17, % 8.9% 4% 1.6% 17, % 11.8% 5% 1.6% 17, % 14.9% 8.3% 2.6% 17,715 83% 10.8% 4.3% 2% 17, % 9.5% 4.7% 4.5% The only item in which fewer than 80 percent selected either of these two responses referred to the facilitator asking probing questions to advance and deepen the discussion. Perhaps, then, this is a more challenging aspect of facilitation. Overall satisfaction with facilitator. Overall, the majority of respondents were satisfied with their instructor; when asked if they would recommend that their colleagues take a course taught by their facilitator, 92 percent responded in the affirmative. PBS TeacherLine Evaluation 10

18 Facilitator support. In interviews, PBS TeacherLine learners were asked to describe the ways that their instructor supported them in completing coursework. Interview respondents indicated that course instructors showed support in the following ways: Feedback. Most frequently, participants (11 of 21) said that their instructor supported course completion by providing useful feedback. One participant remarked, [S]he presented sincere comments so you felt she read your paper, and she synthesized [her comments] and then provided suggestions. She personalized her feedback to you. Responsiveness. Nine of 21 respondents said that a timely response to questions was another way that instructors supported them in completing coursework. A respondent said, You can always them and they ll you back. Usually within 24 hours you get a reply. And they were all pretty good with replying. Clarifying grades and assignments. Six respondents described how instructors provided frequent information regarding grades, weekly schedule, due dates, etc. She certainly made it clear when the assignments were due and what was expected in each week. Areas for Improvement in Course Facilitation When asked to describe how the instructor could have been more supportive, most learners (15 of 21) said that support by their instructor was not lacking in any way. However, five of 21 suggested that the instructor should have provided more feedback or provided more clarity in the form of examples and explanations: [S]ome of his answers to help weren t very helpful they [the instructor] just kind of pointed you back to the syllabus, which, obviously, if we re asking for help, we read the syllabus and didn t understand. Three respondents also mentioned that the instructor was inflexible or rigid in one way or another. The following comment reflects this sentiment: [T]he major thing that makes people go away from online classes or gives them a bad name is that it s supposed to be very much more flexible, and, if it isn t, then there is no use taking the class because you could do that face-to-face the rigid types of demands that you had to do it [assignments or postings] at certain times. Another respondent felt that the lack of flexibility was an affront of sorts to her professionalism: I remember feeling like I m a professional give me the respect that a professional deserves, and don t tell me that I need to do ABCDEFG; just give me a question, and let me expand on it with my colleagues. PBS TeacherLine Evaluation 11

19 Quality of Course Content Course content refers to the topics covered, materials, and assignments in the course, along with their organization. On the survey, learners rated their agreement with four statements about the quality of course content. Their ratings of these items are summarized in Table 8. As with the ratings of facilitation quality, learners expressed strongly favorable opinions of course content. For every item, more than 90 percent strongly agreed or agreed that the course met expectations, provided content or strategies that were applicable to instruction, and prepared them to improve their students academic performance. In line with the overall positivity of these survey responses, six interviewees offered general words of praise about course quality, such as the following: The PBS class was hands down the best content I had ever come across in terms of an online class. Respondents elaborated on their opinions about the course in interviews and open-ended survey responses, as will be described. Table 8. Postsurvey Ratings of Course Quality N Strongly Agree Agree Neutral Disagree Strongly Disagree 53. This course met my expectations. 17, % 32.4% 6% 3% 1.2% 54. This course provided me with content knowledge that I can apply to my instruction. 55. This course provided me with instructional strategies that I can apply to my instruction. 56. My students academic performance will benefit because I have participated in this course. Relevance of Courses 17, % 30.8% 4.4% 1% 0.5% 17, % 30.8% 4.6% 1% 0.5% 17, % 33.5% 7.5% 1% 0.5% Learners reported that the courses focused on the topics that they purported to address. In the survey, 90 percent of respondents either strongly agreed (57 percent) or agreed (32 percent) that the course met their expectations. Six of 21 interviewees mentioned this for a reason why they liked the course content. A typical comment was, It was excellent quality because I knew what I was signing up for when I took it. I had to do exactly what I saw in the syllabus. Consistent with this finding, another interviewee praised the relevance of the course: I m teaching middle school right now and it addresses middle school learners. I thought that all of the topics were very relevant for what I do. Usefulness of Courses Learners typically reported that the courses were highly useful and applicable. In the survey, 94 percent of respondents either strongly agreed (63 percent) or agreed (31 percent) that the course addressed content knowledge and instructional strategies that were highly applicable to their instruction (two separate items). Moreover, in response to the open-ended survey item, 35 PBS TeacherLine Evaluation 12

20 percent of respondents identified the relevance of course content as what they liked best. Similarly, seven of 21 interviewees mentioned this as the reason for why they liked the course content. These respondents emphasized the applicability of what was learned. Some representative responses (from open-ended responses) are as follows: I really liked the content, the discussions, and that I was asked to apply some of the learning in a practical way. I liked that I could immediately take what I learned back to the classroom. The walkthroughs were very easy to follow. Usefulness of resources. Other respondents commented on the usefulness of the resources to which they were exposed. For example, one survey respondent stated: I loved the wide variety of resources offered. I was introduced to new websites such as Reading Rockets. I had no idea PBS offers so much for teachers, students, and parents on their website. In response to an interview question about the resources teachers gained, all respondents indicated that PBS TeacherLine courses provided them with materials that they could use for lesson planning or within the classroom. One teacher summarized the number of resources in the following comment: I have more resources for my students, and access to more resources and activities and electronic activities. The following were the most prevalent types of tools and resources: Links to Web-based resources (16 of 21 respondents). This response was particularly common among participants who completed a technology course (all five respondents). Templates, models, or graphic organizers that they could use in the classroom (five responses). Videos (four responses). Lesson plans (four responses) in their course. Quality of Materials Although there were no fixed-choice survey items addressing the course materials, learners expressed their opinions in open-ended survey items and in the interviews. The feedback was generally positive, with some specific criticisms as well. On the positive side, three interviewees praised the quality of materials, along with seven percent of the respondents to the open-ended item. 2 These comments typically mentioned the readings, videos of lessons, and websites, as expressed as follows by one interviewee: I really enjoyed the material that was presented, the little video clips and the readings that we had to do. Another interviewee remarked, The materials were great. They were very clear. It was all online. 2 Many more respondents praised course content in general, which may also have referred to the quality of materials. PBS TeacherLine Evaluation 13

21 On the other hand, several respondents offered criticism about the materials and suggested what could be improved. About seven percent of responses to the open-ended question liked least and how to improve focused on materials. These respondents fell disproportionately within the technology course area; five respondents from this area suggested that the materials needed to be updated to reflect changes in technology. One participant wrote: [Provide] updated reading materials the copyright on some of them was nearly 10 years old! One about search engines didn t even mention Google. A tech class should have more up-to-date information. Along similar lines, one interviewee pointed out that some course links were out of date Some [of] the materials for our course were no longer online, some of the assignments had to be changed at the last minute, and I like to work ahead when I have the time. Variation in Course Quality Course areas differed somewhat in their postsurvey ratings of course quality. While this difference is statistically significant, the magnitude of the difference does not appear to be large or important. This is illustrated in Figure 1. Given that the scale scores range from about 4 to 9, the range from the highest mean (6.8 in reading/language arts) to the lowest mean (6.0 in technology) is not large. 3 Figure 1. Mean Course Quality Scale Score as a Function of Curriculum Area 3 A single course quality measure was derived by combining the four course quality survey items into a single scale score. The course quality score was related to reported level of content knowledge; ratings of course quality were correlated with ratings of knowledge on the postsurvey. On a scale of 0 to 1, the correlation of course quality and content knowledge is 0.45, and the correlation of course quality and knowledge of teaching strategies is The test statistic is Spearman s Rho, with df = 17,815 and 17,805, respectively. PBS TeacherLine Evaluation 14

22 Discussions and Assignments In all courses, learners were expected to post comments to the discussion board and respond to the comments of other learners. When asked which learning activities were the most useful, interviewees (10 of 21) most frequently named the discussion forums and interaction with other participants (e.g., peer review). Similarly, 27 percent of respondents to the survey item about what they liked most mentioned interactions with peers or the facilitator. One interviewee remarked as follows: [W]ith the discussion board, you just learn so much. It s good to connect with other people all over the country. With PBS, you could be anywhere in the country and connecting and getting ideas, so I think that s just wonderful. Another respondent explained, [T]he discussion board is so helpful because you get ideas and hints from other teachers. Finally, one respondent indicated that participants also used the discussion boards to share useful resources: I was very happy with people bringing their different areas of expertise to the discussion and sharing resources and sharing good websites and ideas about how to present things or some people shared their PowerPoints that they had made. However, the main criticism of the discussion activity was that there was too much interaction due to too many requirements for posting and responding to one another. This was the second most frequent open-ended response to what learners liked least about the course, mentioned by 20 percent of respondents to that item. Many of these respondents said it took too much time to read all of the postings and respond. In addition, several respondents disliked the requirements regarding the discussions, which they considered onerous: The online discussions were not as beneficial as they could have been. Too much time was spent reading information and viewpoints that will not help me improve my teaching ability. I sometimes felt overwhelmed by the number of discussion posts. One participant explained how the point requirements for posting and responding to comments were burdensome: Being so worried about points, people stopped doing what the whole point was, and that was to converse and share thoughts by the end of it, [it] was absolutely just answer the question, get your posts in, and it was not meaningful. PBS TeacherLine Evaluation 15

23 Course Assignments: Strengths and Areas for Improvement Courses had several readings and assignments that learners had to complete on a weekly basis as well as a larger project (or projects) that they worked on for several weeks. When asked which learning activities were the most useful, nine of 21 interviewees mentioned the project assignments. Said one learner, I think the most useful was the final project that involved putting together a unit under whatever subject I chose. Reading assignments was an activity mentioned as useful by seven of 21 interviewees, and four participants mentioned that viewing videos or accessing Web resources was particularly useful. However, one participant said that the readings were too lengthy and another characterized the information presented in the course as a little overwhelming: There was so much reading, there were so many places to look there was so much information presented that it was a little overwhelming. The most prevalent criticism about course assignments of all types was their excessive length. This criticism, expressed in 40 percent of the open-ended responses to what participants liked least about the course, somewhat echo the concerns about the discussion requirements that some found burdensome. Many respondents indicated that they felt the amount of work was not commensurate with the number credits awarded. As one learner stated, This was not a 10-hour class. The final project was extremely time consuming. A portion of these comments mentioned the inconvenience of deadlines falling in the middle of the week or during school holidays. In the words of one participant, [The] weekly deadlines [were] on a Tuesday night [the] middle of the week was tough. In open-ended responses, 15 percent of learners criticized the clarity or difficulty of assignments. Regarding the lack of clarity of assignments, one learner noted, I wasn t always completely sure that I was completing the assignments correctly when I was working on them. Elaborating on this difficulty, one learner stated, Simply providing students with a rubric is not enough. We should have an example to follow for at least the final project. Another participant pointed out that there was a lack of clarity in assignment due dates. Some learners described overly stringent or unrealistic expectations for the grading. One respondent simply stated, The grading was overly hard. Another elaborated on one assignment with unrealistic expectations: The partnership plan that we had as an assignment was not difficult to write, but it seemed in order to write it to fit the rubric, it had to be written for more than just my future classes. This seemed idealistic but not realistic. PBS TeacherLine Evaluation 16

24 Customer Satisfaction The final section of this report summarizes several topics related to customer satisfaction, including opinions about the program as a whole, opinions and experiences of repeat learners (who demonstrate their satisfaction by taking multiple courses), and satisfaction with customer service. This section relates to the final evaluation question: 5. What are the characteristics of repeat learners in terms of professional development background, Courses taken, satisfaction with Courses over time, and changes in teaching practice? Repeat Learners The surveys and interviews provide insight into the motivations and course-taking behavior of repeat learners. Repeat learners were asked in interviews why they took more than one course. Two thirds of the repeat learners (12 of 18) mentioned the convenience of the online format, namely, being able to work on courses at a time and place of their choosing. One respondent remarked, I like the way the chat is set up on the PBS courses asynchronously, actually, because people log on at different times and you can respond. This finding was reflected in the responses to the liked best question; 58 percent of all responses mentioned the online format. Two other reasons for taking multiple courses were positive experiences with a previous PBS TeacherLine course and cost or reimbursement (each mentioned by three participants). In regard to this latter reason, some learners explained that courses were paid for by the district either through a grant or because a certain grade was earned. One interviewee indicated that the the price is good for graduate credits. Data from the survey also address this question. In response to the presurvey question What are your primary objectives in taking this course? high repeaters more typically stated that they were taking the course to earn graduate credit (see Table 9). None of the other reasons for taking the course (e.g., to increase knowledge and skills, to gain CEUs) differed by explainer category. Table 9. Primary Objectives in Taking Course by Repeater Category (2009 Data) Repeater Category Nonrepeater (one course) N To fulfill highly qualified teacher requirements To earn graduate credit To increase my knowledge and skills To earn CEUs toward professional development Other 1, % 41.1% 76.7% 39.8% 7.6% Low (two courses) % 48.0% 68.2% 33.0% 7.3% Medium (three courses) High (four and more courses) % 56.9% 69.3% 33.5% 7.3% % 63.1% 72.4% 30.2% 7.1% Total 3,039 31% 46% 74% 37% 7% Note: Respondents could select all that applied. PBS TeacherLine Evaluation 17

25 One question is whether learners who take several courses tend to branch out across course areas or whether they delve into a single area. We examined registration records going back to the school year to answer this question. On average, high repeaters took more than two thirds of their courses in the same area, medium repeaters took three quarters in the same area, and low repeaters took 78 percent in the same area. Thus, it appears that repeaters tend to take most of their courses in the same area. The pattern that emerges from these three findings is that learners take multiple PBS TeacherLine courses because of the flexibility of the online format and to collect graduate credit in a specific curriculum area. Areas for Improvement in Online Format Although the online format was a strong selling point for the program, a minority of learners voiced criticism about the way certain courses were implemented. Although nearly three quarters of survey respondents agreed (30 percent) or strongly agreed (43 percent) with the statement, I did not have any technical difficulties while taking this course, nearly a fifth disagreed (16 percent) or strongly disagreed (3 percent). In response to the open-ended question about what they liked least, seven percent mentioned some technical problem related to the online environment. These respondents reported that the navigation through the site and interface for completing quizzes were cumbersome. Finally, 4 percent of respondents to the survey noted that the layout of the course lacked clarity regarding assignments and their due dates. The following response was the most specific about this problem: It was very hard to decipher assignments and due dates. Information regarding assignments was present in several places, but it didn't seem that there was one, all-inclusive area to check what was due next. PBS TeacherLine Evaluation 18

26 Conclusions and Recommendations Looking across the findings related to the five evaluation questions, there is strong evidence that learners typically have strong positive opinions about the quality of PBS TeacherLine courses. In particular, most learners praised the relevance of course content, the value of discussions and assignments, the usefulness of the resources about which they learned, and the effectiveness of course facilitation. Most learners believed the courses influenced their teaching practice and believed they gained new content knowledge and instructional skills. These findings are based on self-reported data collected in a large number of surveys and through 21 targeted interviews, mainly with repeat learners. This approach is well suited for drawing conclusions on learner satisfaction with courses. However, for making judgments about the impact of the courses on knowledge and practice, the design of the presurvey and postsurvey created limitations. Only two survey items were consistent from presurvey to postsurvey, and even these two items were not exactly the same. Therefore, it is not possible to draw a strong conclusion about the impact of courses on learner ratings of knowledge and skills that is, their change from the beginning of the course to the end. Moreover, the surveys omit several measures that could be relevant to course impact, such as attitudes toward teaching or toward the curriculum area addressed by the course. Appendix C presents several recommendations for enhancing the usefulness of the surveys. As mentioned, the majority were satisfied with the quality of PBS TeacherLine courses. Several themes emerged from the minority of learners who voiced criticism of their courses in particular or the PBS TeacherLine format in general. Based on the strength and consistency of these themes, the following recommendations are warranted: 1. Maintain a regular schedule for review and revision of course content. Regular course review would confirm that all Web URLs linking to articles and resources are still available online. Course content also should be reviewed to ensure that it is current; this is particularly important for technology courses, which cover topics that change with great frequency. 2. Summarize course assignments and due dates in one place. This summary would ensure that learners could easily find a schedule of what is due in a particular week. 3. Provide flexible guidelines for discussions. A common frustration of participants was the sheer volume of comments they believed they were obligated to read and respond to. Without having the opportunity to review the actual discussion forum, it is difficult to confidently suggest how to address this concern. It may be possible, however, that facilitators could provide some flexibility to learners in terms of requirements for reading and responding to posts so that these requirements should not be more time consuming than is warranted. 4. Gather feedback on the amount of coursework. Another common concern of participants was that the time required to complete assignments was greater than what they exceeded based on the stated course workload. It would be feasible to collect input from participants about the amount of time they spent on assignments and whether that amount of time was reasonable. It would be possible to adjust the amount of work in courses in response to this feedback. PBS TeacherLine Evaluation 19

27 Reference Wright, B. D., & Masters, G. N. (1982). Rating scale analysis: Rasch measurement. Chicago: MESA Press. PBS TeacherLine Evaluation 20

28 PBS TeacherLine Presurvey Tables Appendix A. Survey Tables The items in the following tables are forced-choice except where noted. Table A-1. Previous Online Course Experience 67. What is your experience in taking online courses? N Percentage This is my first facilitated online course. 10, % I have taken a facilitated course once before. 7, % I have taken several facilitated online courses. 6, % Total 24, % Table A-2. How Participants Heard About PBS TeacherLine 68. How did you first hear about PBS TeacherLine? N Percentage Conference % 4, % Colleague, friend, or family 6, % Web search 1, % District or administrator referral 6, % PBS TeacherLine fact sheet % PBS TeacherLine catalog 2, % Other 1,958 8% Total 24, % Table A-3. Teacher Learning Expectations 70. What statement best describes your learning expectation for this course? N Percentage To build content knowledge 16, % To learn different approaches to instruction 19, % To enable my students to become more familiar with technology 8, % To understand how online learning takes place 6, % To become more comfortable with technology 6,880 28% Other 975 4% PBS TeacherLine Evaluation 21

29 Table A-4. Items Addressing Level of Comfort in an Online Course 71. I know what to expect as a learner in a facilitated online course. 72. I am comfortable with the idea of learning in an online environment. 73. I am comfortable with participating in professional development activities outside of a traditional classroom setting. 74. I am comfortable expressing myself on an online discussion board. N Agree Neutral Disagree 24,426 61% 29.8% 9.3% 24, % 19.1% 3.6% 24, % 10.5% 1.4% 24,407 79% 18.4% 2.6% 75. I have a good sense of my learning style. 24, % 8.1% 0.3% 76. I feel comfortable working with technology (such as loading software or troubleshooting problems). 24, % 26.5% 9.4% Table A-5. Rating of Current Knowledge of Content and Pedagogy 77. How would you rate your current knowledge of the content this course will address? 78. How would you rate your current knowledge of the instructional strategies that this course addresses? N Very Knowledgeable Knowledgeable Moderately knowledgeable Slightly knowledgeable Not knowledgeable 24, % 27% 43.9% 21.2% 4.6% 24, % 24.7% 46.4% 21.6% 5% Table A-6. School Policy Toward PBS TeacherLine 79. Does your school district include PBS TeacherLine courses among its list/catalog of teacher professional development options? N Yes No Don t know 24, % 12.8% 31.1% Table A-7. Awareness That Participant Is Taking the Course 80. Of the following, who are aware that you are taking this course? (check all that apply) N Percentage Principal 8, % School district professional development office 9,574 39% None 6, % Other 7, % PBS TeacherLine Evaluation 22

30 Table A-8. Likelihood of Recommending the Course 83. How likely would you be to recommend these other professional development courses to others? N Very Likely Somewhat Likely Not Likely or Unlikely Unlikely Very Unlikely 23, % 32.8% 6.3% 0.7% 0.8% Table A-9. Intention to Seek Graduate Credit N Yes No 86. Will you seek graduate credit for this course 7, % 41.8% Table A-10. Primary Objectives for the Course 87. What are your primary objectives in taking this course? (check all that apply) To fulfill my school district/state s highly qualified teacher requirements N Percentage 2,344 20% To earn graduate credit 3,276 29% To increase my knowledge and skills in this area 5,314 46% To earn CEUs toward professional development 2,687 23% Other 502 4% Note: These data are based on responses to the new version of the survey, for which respondents could select all that apply. PBS TeacherLine Evaluation 23

31 Postsurvey Response Tables Table A-11. Likelihood of Recommending the Course or Taking Another 35. How likely are you to recommend this course to a colleague? 36. Based on your experience in this course, what is the likelihood that you will take another PBS TeacherLine course? N Very Likely Somewhat Likely Neither Likely nor Unlikely Somewhat Unlikely Very Unlikely 17, % 22.3% 3.9% 2.9% 1.9% 17, % 20.9% 4.8% 2.6% 1.6% Table A-12. Incorporating Knowledge and Skills Into Practice 37. To date, have you been able to incorporate any content area knowledge from this PBS TeacherLine course into your teaching practice? 39. To date, have you been able to incorporate any instructional strategies from this PBS TeacherLine course into your teaching practice? N Yes No Does not apply 17, % 9.2% 15.4% 17,817 70% 9% 15.1% Table A-13. Rating of Current Level of Knowledge of Content and Pedagogy Now that you have completed the PBS TeacherLine course 38. How would you rate your knowledge of the content this PBS TeacherLine course addresses? 40. How would you rate your knowledge of the instructional strategies that this PBS TeacherLine course addresses? N Very Knowledgeable Knowledgeable Moderately knowledgeable Slightly knowledge -able Not knowledge -able 17, % 45.7% 7.1% 1.2% 0.2% 17, % 47.7% 7.4% 1.3% 0.3% Table A-14. Where Participant Worked on Course 41. Where did you spend most of your time working on your PBS TeacherLine Course? N At home with phone line access At home with broadband access At school Other 17, % 69.6% 10.5% 1.8% PBS TeacherLine Evaluation 24

32 Table A-15. Average Time Spent on Course per Week 42. On average, how much time did you devote each week to this PBS TeacherLine course? N Less than 2 hours 2 4 hours 4 6 hours 6 8 hours 8 10 hours hours hours Greater than 14 hours 17, % 19.3% 32.3% 21.5% 11.1% 5.5% 3.7% 5.1% Table A-16. Rating of Facilitation Quality 43. In the discussion forums, your facilitator provided at least one summary post per week? 44. In the discussion forums, your facilitator asked probing questions to advance and deepen the discussion? 45. Your facilitator demonstrated knowledge of the content area? 46. Your facilitator established a weekly schedule with s and announcements? 47. Your facilitator responded to requests for assistance (via or discussion board) within 24 hours? 48. Your facilitator gave you weekly feedback via or the online journal or the grades tool? 49. Your facilitator began each discussion forum with a well-written question or prompt? 50. Your facilitator provided administrative information such as graduate credit information? N All the time Most of the time Sometimes Never 17, % 14.9% 8.5% 4.5% 17, % 22.4% 14.8% 5.4% 17, % 15% 6.9% 2.3% 17, % 8.9% 4% 1.6% 17, % 11.8% 5% 1.6% 17, % 14.9% 8.3% 2.6% 17,715 83% 10.8% 4.3% 2% 17, % 9.5% 4.7% 4.5% Table A-17. Recommendation of Facilitator 51. I would recommend that my colleagues take a course taught by this facilitator. N Yes No 17, % 8.5% PBS TeacherLine Evaluation 25

33 Table A-18. Rating of Course Quality N Strongly Agree Agree Neutral Disagree Strongly Disagree 53. This course met my expectations. 17, % 32.4% 6% 3% 1.2% 54. This course provided me with content knowledge that I can apply to my instruction. 55. This course provided me with instructional strategies that I can apply to my instruction. 56. My students academic performance will benefit because I have participated in this course. 17, % 30.8% 4.4% 1% 0.5% 17, % 30.8% 4.6% 1% 0.5% 17, % 33.5% 7.5% 1% 0.5% Table A-19. Rating of Technical Quality 57. I did not have any technical issues while taking this course. 58. I was able to enroll in this course easily. N Strongly Agree Agree Neutral Disagree Strongly Disagree 17, % 29.5% 7.8% 16.1% 3.3% 17,797 71% 24.4% 2.4% 1.6% 0.5% Table A-20. Rating of Customer Service 59. A customer support representative responded in a timely manner. 60. A customer support representative provided accurate information. 61. A customer support representative acted in a professional manner. N Strongly Agree Agree Disagree Strongly Disagree Neutral 16, % 11.6% 1.2% 0.6% 66.1% 16,356 20% 11.3% 1.4% 0.5% 66.8% 16, % 10.3% 0.3% 0.3% 67.3% PBS TeacherLine Evaluation 26

34 Appendix B. Interview Protocol I m from. We are conducting an evaluation of the PBS TeacherLine program to provide feedback about the opinions of participants such as yourself. Thanks for agreeing to speak with me today. This interview covers several topics related to your experiences in your PBS course. Your participation is entirely voluntary. Your responses to my questions will be completely confidential. We do not identify respondents in the information we provide to PBS. I would like to audio record our interview in order to accurately capture everything you tell me. Do I have your permission to record this interview? INTERVIEWER: [IF YES, TURN ON RECORDER AND PROCEED.] For our records, could you once again confirm that I have your permission to record this interview? For our records, this is interview number [user ID] for course area [RDLA/INST/TECH/MATH/ SCIE]. Teacher Background 1. How many years have you been teaching? 2. What grade do you teach? 3. What subject or subjects do you teach? Reasons for Participating 4. How many PBS TeacherLine courses have you completed to date? 5. Were they in the same area or different areas (e.g., reading/language arts, mathematics, science, technology, instruction)? 6. What was your main reason for taking your most recent TeacherLine course? (Probe: CEUs, master s program, professional development, etc.) 7. If they have taken more than one: What was the reason why you have taken more than one course from PBS TeacherLine? Course Experiences For the following questions, focus on the PBS TeacherLine course you completed most recently. Course Quality 8. What was your overall opinion of the quality of this course? Why? (Probe: organization, pacing, materials, assignments.) PBS TeacherLine Evaluation 27

35 9. What was your opinion about the course content, that is, the topics and assignments it covered? Why? (Probe: organization, pacing, materials, assignments.) 10. What types of learning activities were particularly useful? (Examples include reading assignments, discussions, group work.) Facilitation 11. How did your instructor support you in completing the course work? (Probe: feedback, answering questions.) 12. In what ways, if any, could your instructor have been more supportive? Student Interactions 13. In what ways did you collaborate with other participants? 14. Did you think there was enough interaction among participants? If not, what else would you have liked? 15. What, if anything, would you recommend changing about the course? Impact on Teaching and Learning For the remaining questions, consider all of the courses you have taken. 16. Overall, in what ways did the course(s) expand your content knowledge? 17. In what ways, if any, has [have] your course(s) made an impact on the way you teach? (Probe for specific examples: content, teaching strategies, technology integration.) 18. If they have taken more than one: Did the courses build on each other, or were they mostly independent of each other? 19. Did the course(s) provide you with tools or resources that you could apply in your teaching practice? If so, please describe. (Probe: lesson plans, templates, graphic organizers, links to resources.) 20. Has your experience with this course or these courses made an impact in your level of confidence in your teaching ability? If so, please explain why. 21. Do you think these courses have influenced student learning? In what ways have you seen this? 22. Has this course expanded your community of practice? (Probe: community of practice is the network of educators who share questions, resources, etc.) Closing Comments 23. Is there anything else that I should know regarding your experiences with your PBS TeacherLine course that we didn t discuss? PBS TeacherLine Evaluation 28

36 Appendix C. Recommendations for PBS TeacherLine Presurvey and Postsurvey 1. Simplify and clarify wording of questions. The wording of several questions could be simpler and clearer. Some examples and suggestions for improvement are as follows: In the postcourse survey, question 36 asks Based on your experience in this course, what is the likelihood that you will take another PBS TeacherLine course? The question could be reworded to be more direct: How likely are you to take another PBS TeacherLine course? Question 65 in the postcourse survey reads If you have any suggestions for improving PBS TeacherLine or this course, please share them here. This question asks for input from two different perspectives, namely, the course itself and the PBS TeacherLine program in general. Breaking this into two separate questions could elicit more feedback directed toward each aspect, and it would be easier to interpret the findings. Question 76 in the precourse survey is I feel comfortable working with technology (such as loading software or troubleshooting problems). The parenthetical text is not clear enough. A better approach would be to provide more specific descriptors before the word technology (e.g., interactive websites). 2. Provide response options. Question 83 in the precourse survey requires an open-ended response for the question How likely would you be to recommend these other professional development courses to others? A series of response options should be provided (see recommendation 4 in this list). The survey should then provide a message directing those who selected None to skip to item 86 (because items 84 and 85 would not apply to them). 3. Reduce number of response categories. There are eight response options for Question 42 on the postcourse survey ( On average, how much time did you devote each week to this PBS TeacherLine course? ). This large number of categories is unwieldy for analysis. A new scale is proposed in the column on the right in Table C-1. Table C-1. Response scale for Question 42: Hours per Week Spent on Course Current Scale Less than 2 hours Proposed Scale Less than one hour 2 4 hours 1 3 hours 4 6 hours 3 6 hours 6 8 hours 6 9 hours 8 10 hours 9 12 hours hours 12 or more hours hours Greater than 14 hours 4. Simplify the likelihood scale. This response scale (used for postcourse items 35 and 36) should measure only how likely something is, not how unlikely it is. This would eliminate the ambiguous midpoint of the scale. PBS TeacherLine Evaluation 29

37 Table C-2. Current and Proposed Likelihood Response Scale Current Scale Very likely Somewhat likely Neither likely or unlikely Unlikely Very unlikely Proposed Scale Very likely Moderately likely Somewhat likely Slightly likely Not at all likely 5. Use a single agree/disagree scale. There are three different versions of the agree/disagree scale being used between the two different surveys. It is proposed to use the version currently used for postsurvey items because it excludes the neutral category that can be interpreted in different ways by different respondents. However, it does not appear that it is necessary to include the N/A only for items 59 61; this option should be eliminated for the other items. Table C-3. Current and Proposed Agree/Disagree Response Scale Presurvey Questions Postsurvey Questions Strongly agree Agree Agree Agree Postsurvey Questions Strongly agree Neutral Neutral Disagree Disagree Disagree Strongly disagree Strongly disagree 6. Use a moderator for each response option in the knowledgeable scale. The scale measuring the extent to which respondents are knowledgeable (on both the presurvey and postsurvey) does not have a modifier for one of the rating categories the second highest rating is simply knowledgeable. The proposed scale is described in Table C-4; it uses the same set of five modifiers as the likelihood scale described in Table C-2. N/A Table C-4. Current and Proposed Knowledgeable Scale Current Scale Very knowledgeable Knowledgeable Moderately knowledgeable Slightly knowledgeable Not knowledgeable Proposed Scale Very knowledgeable Moderately knowledgeable Somewhat knowledgeable Slightly knowledgeable Not at all knowledgeable 7. The knowledge items should be consistent across versions. The items about level of knowledge differ slightly on the precourse and postcourse versions. These should be exactly the same on both versions; the phrase Now that you have completed your PBS PBS TeacherLine Evaluation 30

38 TeacherLine course should, therefore, be removed from the postcourse version. There are two additional suggestions regarding these items: There are two versions of the knowledge question, addressing content and instructional practice. The similarity in response rates of the two versions suggests that they can be combined into a single version. This item could be phrased as follows: What is your level of knowledge of the content that this PBS TeacherLine course addresses? On the postcourse survey, there should be an additional question that asks learners to rate their level of knowledge before taking the course (e.g., Prior to this course, what was your level of knowledge of the content that it addressed? ) The reason is that many surveys are not able to be linked to the precourse version, and such an item would permit a comparison of the change from precourse to postcourse. Second, after taking a course, many learners realize how much they did not understand beforehand therefore, it is important to have a postcourse perspective on the amount learned. 8. Provide additional response options for impact on practice. Items 35 and 37 on the postcourse survey have nearly identical scales. The current scales are as follows: 35. To date, have you been able to incorporate any content area knowledge from this PBS TeacherLine course into your teaching practice? Yes. No. Does not apply. I am not a classroom teacher/i am not currently teaching. 37. To date, have you been able to incorporate any instructional strategies from this PBS TeacherLine course into your teaching practice? Yes. No. Does not apply. I am not a classroom teacher/i am not currently teaching. Not yet, but I plan to. One problem with the current set of options is that the yes/no format does not offer much detail about the extent of impact or influence on teaching practice. It is possible to provide additional response options. The precourse and postcourse surveys both should ask learners to rate their extent of incorporation: PBS TeacherLine Evaluation 31

39 To what extent have you incorporated what you learned from this PBS TeacherLine course into your teaching practice? Very much so Moderately Somewhat Slightly Not at all N/A not currently teaching 9. Add items about other aspects of course quality. The four items addressing course content (items 53 56) could be combined to form a valid scale score. However, it seems that a few additional items should be included, as identified by the evaluation report. These items include ratings of course materials, discussions, assignments, and resources. 10. Modify the facilitator quality items. The nine items about the facilitator (43 51) did not form a valid scale score, which limits the usefulness of these findings. The following revisions to these items would make it more likely that they could be used to create a single scale score. First, the phrasing of the items should be similar, with all items starting with the same phrase (e.g., Your facilitator, rather than the phrase In discussion forums that introduces items 43 and 44). Second, the current rating scale (addressing frequency) does not align well with all of the items. For example, it is not easy to rate the frequency with which an instructor demonstrated knowledge of the content area (item 47). Therefore, these items should employ an agree/disagree scale, similar to the items addressing course content. PBS TeacherLine Evaluation 32