17 Stories Program Evaluation

Transcription

1 17 Stories Program Evaluation East Africa: Uganda February 2013 OneHope 600 SW 3rd Street Pompano Beach, FL T

2 CONTENTS Key Findings 3 Introduction 4 Evaluation Methods 5 Results 6 Program Implementation 6 Meta-Narrative Assessment 6 Story Retelling Assessment 10 Classroom Observation 14 Teacher Interviews 16 2

3 KEY FINDINGS Meta Narrative: The meta narrative scores increased significantly from pretest to posttest. Older children did better at posttest than younger children did. Card Identification: Students did much better at identifying the Old Testament (Jacob and Esau) story at posttest compared to at pretest. Most already knew the New Testament (Jesus feeds 5,000) at pretest. Story Retelling: The students also did significantly better at telling both of the stories at posttest compared to pretest. Classroom Observations: Each class was observed once. The reviewer specifically commented on the introduction, storytelling, discussion question, activity, and reviewing portion of the lessons. Overall, the observer noted that the lessons went as planned and that the teachers were well prepared. Teacher Interviews: A teacher from each class answered questions about their experience teaching the 17 stories program. Teachers were generally positive about the content of the program as well as the implementation. The number of teachers interviewed and classrooms observed was greater than the final number of sites included in the analysis. Only three of the seven sites had complete data that could be used in the statistical analysis. 3

4 INTRODUCTION Description of 17 Stories 17 Stories is a scripture engagement and discipleship course for children and youth ages As an oral, story-based program, 17 Stories is designed to be even effective with children and youth who are unable to read. The course presents 16 Bible stories which teach relevant themes to address the needs of children and youth in the community where it is taught. Each story is depicted on a card which is given to the child when the story is taught. When the 16 story cards are put together in order and turned over, there is a large picture on the back of all the cards which shows the meta-narrative (overarching story) of the Gospel from the Old and New Testaments. Program Outcomes 17 Stories is designed to achieve several outcomes (or goals) in the lives of the children and youth who attend. 1. Learn: Participants develop an understanding of God s overall plan of the Gospel through the Old and New Testaments. 2. Share: Participants can orally share the Bible stories. Purpose of the Program Evaluation The program evaluation sought to better understand what was working well in the 17 Stories program in East Africa and how the program could be improved. The evaluation answered two research questions: 1. Is the 17 Stories program being implemented as intended? 2. Are the program outcomes of 17 Stories being achieved? The results enable OneHope to highlight successes of 17 Stories and identify where changes need to be made to the course content (the teacher s manual and/or the story cards) or the teachers trainings. In addition, each participating country is encouraged to review and summarize the data from their evaluation, and to use the findings to guide future 17 Stories programs. 4

5 EVALUATION METHODS Indicators and Measurement Tools Indicators and measurement tools were developed to answer each of the research questions, as shown in the following table. Research Question Indicator Measurement and Recording Tools Are the program outcomes of 17 Stories being achieved? Outcome 1: Learn: Participants develop an understanding of God s overall plan of the Gospel through the Old and New Testaments. Outcome 2: Share: Participants can orally share the Bible stories. Is 17 Stories being implemented as intended? After completing the 17 Stories course, students improve in their ability to arrange the 16 story cards in the proper order. After completing the 17 Stories course, students improve in their ability to retell the Bible stories. Teachers follow the lesson format in the Teacher s Manual Each child receives a story card for each lesson they attend The child-to-teacher ratio is 30:1 or lower Meta-Narrative Assessment Story Retelling Assessment Classroom Observation Teacher Interview Site Selection & Personnel 17 Stories is being used in many different regions of Africa, as well as other parts of the world. For this project, it was important to evaluate 17 Stories throughout the various contexts where it is being used. To that end, evaluations are occurring in Anglophone West Africa, East Africa, Francophone Africa, and Southern Africa. The evaluation in East Africa was conducted in 3 countries that are believed to be representative of the region: Kenya, Tanzania, and Uganda. In Uganda, a coordinator selected seven- 17 Stories program sites to participate in the evaluation. The selection of the sites to participate in the project attempted to include the full range of 17 Stories programs in the country, in terms of format (school-based, church Sunday school, church VBS, etc.), regions of the country, and urban or rural location. This report includes results from Uganda. In-country evaluation personnel included a national coordinator, research manager, site observers, and teachers. 5

6 Data Processing & Analysis Data were entered in SPSS. Statistical analyses were conducted at a 90% confidence level (p = 0.10). Therefore, there is a 90% likelihood that statistically significant differences were not due to random variation. RESULTS Program Implementation The evaluation began in Uganda in March 2012 and was completed in September The in-country national coordinator and research manager managed the site selection and implementation. Meta-Narrative Assessment Purpose To see if students who complete the 17 Stories course improve in their ability to arrange the 16 Bible story cards in the order that the stories take place in the Bible. This ability shows their understanding of God s overall plan of salvation through the Old and New Testaments. Process At each 17 Stories program site the meta-narrative assessment was given to a set of randomly-selected students as a pretest (before the first 17 Stories lesson) and as a posttest (after the final lesson). The same individual students participated in the pretest and posttest. Each student was tested in a private location. The testing was carried out by the teacher with the help of a visiting observer. For the test, each student was given the 16 Bible story cards in random order and was asked to put the story cards in the order that the stories take place in the Bible. On the back of the teacher s manual is a picture showing the cards in their correct order. Points were assigned to each story card using the scoring system below. A maximum of 96 points could be earned for all 16 story cards. 6 points = card is in the right location 3 points = card is 1 space away from the right location 1 point = card is 2 spaces away from the right location 0 points = card is 3 or more spaces away from the right location 6

7 Results Pretest and posttests were completed by a total of 40 students at 3 different sites. Original data collection involved closer to 60 students at 7 sites but four sites returned incomplete data in which pretest and posttest data could not be matched for each child. Thus, the following report includes pretest and posttest data collected by a total of 40 students at 3 different sites. The sample was made up of more female children (65.0%) than male children (35.0%) and the sample was balanced by age group (see Table 1). Table 1. Demographic Characteristics of the Children (N=40) Frequency % Age Gender 6 to 12 Years to 14 Years Male Female As shown in Table 2, across all 3 sites, the mean pretest score was 29.4 (out of a possible 96). The pretest results indicate that children began the program with minimal knowledge of the Bible stories included in the Meta Narrative Assessment as almost half of the children received pretest scores of less than 20 (45.0%). There were no significant differences in the mean pretest scores between younger and older children or between male and female children. Also shown in Table 2, at posttest, the overall mean score was This was a significant increase from the pretest score of 29.4 (p <.001). Overall, the results show that the children did very well at posttest as over half of the children received perfect scores (57.5%) and most of the children answered at least half of the questions correctly (90%). There were significant differences in mean posttest scores between younger and older children (p =.002), older children scored significantly better on the posttest compared to younger children. There were no significant differences between males and females at posttest. The figures below show overall pretest and posttest scores as well as the score comparisons for age and gender. 7

8 Table 2. Comparison of Pretest and Posttest Meta Narrative Scores Overall and by Age and Gender N Mean SD Min Max p Meta Narrative <.001 Pretest Posttest Meta Narrative Pretest to 12 Years to 14 Years Meta Narrative Posttest to 12 Years to 14 Years Meta Narrative Pretest.744 Male Female Meta Narrative Posttest.591 Male Female Note. There was a significant increase in scores from pretest to posttest 8

9 Note. At pretest, there was no significant difference between younger and older children. At posttest, the older children did significantly better than the younger children. Note. There were no significant differences between males and females at pretest or posttest. 9

10 Story Retelling Assessment Purpose The purpose of this research was to see if students who completed the 17 Stories course improved in their ability to identify Bible stories from story cards and to retell Bible stories. Process The Story Retelling assessment was given as a pretest before the first 17 Stories lesson and as a posttest after the final lesson to the same students who were randomly selected for the Meta-Narrative assessment. Students were tested on the Meta-Narrative and Story Retelling during the same testing session. Each student was tested using one Old Testament story (Card #4: Jacob and Esau) and one New Testament story (Card #13: Jesus Feeds 5,000). Stories that students may be less familiar with before taking the course were intentionally chosen, to make it easier to assess how much the students learned after the course. First, each student was shown the Old Testament story card and was asked to name the story. Then, the student was asked to tell the story. The teacher or observer recorded the student s responses. Students were given credit for mentioning any part of a story point. The process was repeated with the New Testament story. Students were not expected to know all of the story points. The goal was to assess how many additional story points the students learned as a result of taking the course. Results Pretests and posttests were given to the same 40 students who participated in the Meta-Narrative assessment (see the Demographics on p. 7). Story Identification. Students improved greatly from pretest to posttest in their ability to identify the story of Jacob and Esau. Students did not differ much in their ability to identify the story of Jesus feeds the 5000 but that is due to most children naming that story correctly during the pretest (91.7%). This could be because all of the children were familiar with the story (see Table 3). Table 3. Percent of Students who Correctly Identified the Story Card Pretest Posttest Card 4: Jacob and Esau 3% 100% Card 13: Jesus Feeds 5,000 92% 100% 10

11 Story Retelling. For the story retelling, students were asked to tell the stories. The number of phrases that they mentioned in the story was tallied and recorded for each student for the Old Testament Story of Jacob and Esau, and for the New Testament story of Jesus feeds 5,000. The number of possible phrases was 21 for the story about Jacob, and there were 12 possible phrases for the story about Jesus feeding the 5,000. Table 4 shows that for both stories, students did significantly better at posttest than at pretest (see Table 4 and the Figure). Table 4. Pretest and Posttest Differences for the Old Testament (Jacob and Esau) and the New Testament (Jesus feeds 5,000) Stories N Mean SD Min Max p Old Testament Story <.001 Pretest Posttest New Testament Story <.001 Pretest Posttest Note. There were significant differences between pretest and posttest for both the Old Testament and New Testament Stories. 11

12 The Old Testament Story about Jacob and Esau was compared by age and gender both at pretest and posttest (see Table 5). There were no differences by age or gender between either pretest or posttest scores. Table 5. Pretest and Posttest Differences for the Old Testament Story (Jacob and Esau) by Age and Gender N Mean SD Min Max p Card 4 Pretest to 12 Years to 14 Years Card 4 Posttest to 12 Years to 14 Years Card 4 Pretest.176 Male Female Card 4 Posttest.370 Male Female

13 The New Testament Story about Jesus feeding the 5,000 was compared by age and gender both at pretest and posttest (see Table 6). There were no differences in the pretest or scores for age or gender. Table 6. Pretest and Posttest Differences for the New Testament Story (Jesus feeds 5,000) by Age and Gender N Mean SD Min Max p Card 13 Pretest to 12 Years to 14 Years Card 13 Posttest to 12 Years to 14 Years Card 13 Pretest.220 Male Female Card 13 Posttest.951 Male Female

14 Classroom Observation Purpose To see if 17 Stories is being taught the way it was designed. Specifically, observers sought to answer the following questions: Do teachers cover the key parts of the lesson, as described in the Teacher s Manual? Does each student receive a story card for the current lesson? Are there enough teachers and assistants? (The target ratio is 30 students to 1 teacher, but teachers are encouraged to have an assistant if possible.) Process An outside observer (who was unaffiliated with the church or Kids Club where the 17 Stories program was being conducted) visited each 17 Stories classroom once during the course of the program. The observer completed the Classroom Observation form while observing the lesson. Prior to the lesson, the observer explained to the teacher that the purpose of the visit was to evaluate the overall 17 Stories program, not to evaluate individual teachers. Results Do teachers cover the key parts of the lesson, as described in the Teacher s Manual? Introduction: The observer stated that the introduction was completed in all seven locations and also that the teachers introduced the lesson very well. The observer mentioned that in one location, the Bible passages were not discussed and in another location, one of the children was allowed to read the Bible story. Additionally, in six of the seven of the classrooms, teachers reviewed one or more of the 14

15 previous lessons during the introduction and the students retold the stories from the previous lessons. Each teacher had a manual. Storytelling: The teacher told the story for the current lesson in all of the classrooms. Cards were handed out before the lesson (with the exception of one classroom in which cards were handed out after the story) and there were enough cards for each child to have one at six of the locations, one location did not provide data about card amount. The observer noted that the children were engaged and attentive during the lesson and specifically mentioned one teacher who was lively and involved the class in the lesson, which helped the children understand more. Discussion Questions: The teacher discussed the story with the class in all seven classrooms. The observer also indicated that the children were actively engaged in answering the discussion questions. Activity: In four of the classrooms, the teacher explained the meaning of the activity and the observer noted that the activity related to the story well, was interesting, and that it enhanced the meaning of the story. In three classrooms, the activity was not conducted due to time constraints. Reviewing Card Set: At the end of the lesson, children in all of the locations reviewed the cards and in three locations, the children had an opportunity to retell the story from the current lesson and previous lessons. At one location, children retold stories from previous lessons at the beginning of the class and not again at the end. In six locations, children were given the opportunity to put the story cards in order (meta-narrative). In the seventh location, the cards were reviewed with the children but they did not have a chance to arrange the cards themselves. Overall: The observer stated that the teachers were prepared and knowledgeable overall, and made generally positive remarks about each stage of the lesson. Does each student receive a story card for the current lesson? Cards were handed out in all classrooms and there were enough cards for each child to have one. One location did not provide information about the amount of cards available. Are there enough teachers and assistants? During the observed lesson, classroom size reportedly ranged from 15 to 70 students. Across classrooms, the ratio of students to adults ranged from as low as 1:4 up to 1:35. Six of the seven classrooms were below the target ratio of 1:30 and one was above. 15

16 Teacher Interviews After each of the classroom observation sessions, the observers interviewed the teachers of the classrooms to get the teachers impressions of the strengths, weaknesses, and ways to improve the 17 Stories program. The following eight questions were asked. 1. How did you feel about today s lesson? What went well? Is there anything you would like to change? All teachers expressed satisfaction with the lesson. One teacher specifically mentioned that teaching the lessons had become easier and they all agreed that everything went well. 2. How well did the activity work in today s lesson (or previous lessons)? How did the children respond to the activity? How easy was it for you to carry out? Six of the seven teachers mentioned that the children were interested and excited with the activity and that it related well to the lesson. One teacher said the activity didn t go well but did not expound. 3. Did the children understand how the activity was related to the lesson? (Probing question: Did you find the outcome lesson useful for relating the activity to the theme of the lesson? The teacher who said the activity did not go well in the previous question said that the children did not understand how the activity was related to the lesson. The remaining teachers mentioned that the activity was useful to the lesson but one teacher remarked that they didn t understand at the beginning how it was related and later appreciated how the activity and lesson were related. The teacher who moved the activity to the first part of the class remarked that he/she kept relating the activity and the story throughout the lesson and that the two complimented each other. 4. Are there any different activities or games you would like to use as an alternative to those in the manual? If so, please describe them. Three teachers appeared to be satisfied with the activities at that point and mentioned that after the conclusion of the course they may have some suggestions for activities. The other four teachers simply stated they did not know of any alternative activities that they would like to use. 5. Think about all the 17 Stories lessons that you have taught so far. Is there a particular theme or story that has especially impacted the students? Three of the teachers mentioned the story of Joseph and the conduct of his brothers and how the children learned the importance of forgiveness. One teacher mentioned Abraham and Isaac and how it 16

17 was impactful in teaching about obedience to both parents and God, another mentioned David and Goliath and yet another mentioned Jesus feeding the In general, do you like the way each lesson flows between sections? All of the teachers remarked that they like the lesson flow and one of the teachers mentioned that he/she would like the activity to occur first (this was not the teacher who had already moved the activity to occur first). 7. Are there any other Bible stories you would like to include in the 17 Stories course? All teachers mentioned stories from the Old Testament, two specifically mentioned the Tower of Babel and several mentioned Esther. Also mentioned were: Daniel and the Lions Den, Cain & Abel, Sodom & Gomorrah, and Shadrach, Mesach, and Abednego. 8. If you could change anything about the course, what would you change? One teacher suggested changing the picture of Adam and Eve because it doesn t show the specific things they were teaching (e.g., the fruit, the snake). Two teachers thought there should be more pictures and that the pictures should be larger. 17