Student Feedback and Interview Questions From Film A Classroom Tale

Transcription

1 Consulting Pupils on The Assessment of their Learning2 Report Students feedback on the impact and effectiveness of the short film A Classroom Tale as a learning resource for improving their experiences od assessment and leaning Ruth Leitch Stephanie Mitchell with Erik Cownie

2 1. Introduction The Student Feedback sheets (see template in Appendix) encompassed 11 questions which provided data for both quantitative and qualitative analysis. In the first and fourth questions, students were asked how much they liked the film and how relevant it was to their experience of school by completing Likert type response scales. In the eighth question, students were asked to pick three pupil principle statements (most important to their learning) from a list of ten; and in the ninth question, they were asked if the film could help learners and teachers talk about some of these principles by selecting Yes, No, or Maybe. Full tabular presentations of the findings from these questions can be found in the Appendix. The remaining questions in the Student Feedback sheets were interpretive and sought qualitative explanations by asking students, for example, what they liked about the film and why they thought it was or was not relevant to their experience of school. In Section 2, the methodology adopted is detailed; and following this, a summation of the quantitative findings is presented which outlines the study wide findings and details comparisons both across the participating schools and in terms of gender. In the fourth section, the qualitative findings are presented in thematic categorisations which emerged during the coding and analysis of the qualitative data. In the fifth section, conclusions from the findings and suggestions for the future application of the film as a learning resource are presented. 2. Methodology A Classroom Tale is a short animation which seeks to encourage students and teachers to explore important aspects of previously reported research which consulted pupils on the assessment of their learning (CPAL). The aim of this element of the study is to assess the effectiveness of A Classroom Tale as a learning stimulus resource. To meet this aim, researchers from Queen s University Belfast adopted the survey questionnaire to solicit the views of the (N=) 497 young people who participated in the study. The value of the survey questionnaire in examining the social world is well established (Sarantakos (2005; Bryman et al 2008; Teddie and Tasshakkori 2009). Moreover, Scott (2000) and Creswell (2012) have outlined specific benefits of the survey questionnaire method when researching young people in school environments around respondent privacy and required commitment levels. However, purely quantitative surveys are limited in terms of giving respondents the opportunity to provide explanations for their answers. Thus, the heuristic benefits of combining quantitative and qualitative methods are increasingly recognised in the literature (Kumar 2011; Gomm 2008). The interpretive element of the Student Feedback sheets was designed to support the quantitative data in a search for patterns by examining the meaning that respondents attached to their answers (Bernard 1995). In other words, the quantitative questions here seek to identify patterns and the qualitative questions seek to explain the existence of those patterns. 2.1 Integrative analysis of the data The quantitative responses from the Likert type scales and multiple choice questions were collated into tables and matrices to facilitate measurement and cross comparison. Concomitantly, the qualitative data from the interpretive questions was assessed, coded and categorised into emergent themes in processes commonly referred to in the literature as content or thematic analysis (Krippendorff 2004; Guest 2012). The two data form were then analysed together using a basic convergence coding matrix to search for discrepancies and corroborations.

3 2.1 Presentation of analysis Summations and abridged tables of the quantitative findings are provided in Section 3; and full tables detailing inter school and gender comparisons are presented in the Appendix. The findings from the interpretive questions are presented in inter related themes in Section 4 and include indicative comments from the Student Feedback sheets. To protect the identity of respondents, no name, age, or school is assigned to these comments which, thus, should be read simply as a reflection of the sample s range of opinions. 3. Quantitative findings from the Student Feedback sheets 3.1 Assessing students liking of the film In the first question, How much did you like this film? students were asked to complete a Likerttype response scale where they could rate from 1 (a little) through to 5 ( a lot). Table 1: How much did you like this film? 1 (a little) (a lot) average rating Total selections Total percentages 3.0% 9.5% 27.2% 43.7% 16.7 Across all the participating classes and encompassing a total of (n=497) students, only 15 (3.0%) gave a rating of 1 (a little); 47 (9.5%) gave a rating of 2 ; 135 (27.2%) gave a rating of 3 ; 217 (43.7%) gave a rating of 4 ; and 83 (16.7%) gave a rating of 5 (a lot). This means that the average rating across all respondents was There was a relatively significant difference in terms of gender. The average rating for the (n=244) boys was 3.46 and the average rating for the (n=253) girls was 3.76 (see Appendix for full tables). Ballysally (4.41), Newcastle (4.24) and Banbridge (4.12) had the highest average ratings; however, the very small sample sizes of these schools (29, 21 and 16 respectively) need to be taken into consideration. The other schools samples ranged from 64 (St Pats Keady year 8) to 174 (Our Lady) with average ratings 3.25 (Our Lady) to 3.85 (Shimna). 3.2 Assessing the film s relevance The fourth question in the Student Feedback sheet was also based on a Likert type response scale and asked students How relevant is the film to your experience of school? Again, the students were asked give a rating from 1 (a little) through to 5 (a lot). Across all participating schools, 55 out of (n = 497) students (11.1%) gave a rating of 1 (a little); 104 students (20.9%) gave a rating of 2 ; 123 (24.7%) gave a rating of 3 ; 160 (32.2% gave a rating of 4 ; and 55 (11.1% gave a rating of 5. This means that the average rating of all respondents to the question of the film s relevance to their school experience was 3.11 (gender and cross school comparisons can be found in the Appendix). Table 2: How relevant is the film to your experience of school? Likert scale 1(a little) (a lot) average rating Total selections Total percentages 11.1% 20.9% % 11.1% There was only a marginal difference in terms of gender; the average rating given by the male respondents was 3.10 and the average for female respondents was On this occasion, two of the small samples (Banbridge and Ballysally) returned the highest and lowest ratings (3.93 and 2.41)

4 respectively). The rating responses of the other schools ranged from 3.82 (Shimna) to 2.62 (Newcastle). 3.3 Assessing the most important pupil principles The eighth question on the Student Feedback sheet invited the pupils to select three statements / pupil principles from a list of ten by asking them: Which of these are the most important to YOUR learning? The top three choices for both male and female respondents were: firstly, You know what you re supposed to do in any task, test or homework (selected by (n=362) students); secondly, You are told individually how to improve and can work out what you need to do (selected by (n=196) students); and thirdly, You are not made to feel embarrassed about your work (selected by (n=180) students). Table 3: Which of these are the most important to YOUR learning? Statement / Pupil Principle selections made male/female/total You know what you re supposed to do in any task, test or homework (176) = 362 Your teacher talks with you about how your work will be assessed 82 + (62) = 144 You get positive feedback on your work BEFORE you are told how to improve 79 + (76) = 155 Feedback from your teacher is clear and happens soon 46 + (40) = 86 You are told individually how to improve & can work out what you need to do 93 + (103) = 196 You get detailed comments & a mark from your teacher 64 + (89) = 153 You are not made to feel embarrassed about your work 81 + (99) = 180 You get classmates comments AND teacher feedback 33 + (28) = 61 You can look at a classmate s work and suggest improvements 25 + (28) = 53 You know how to assess your own work 51+ (38) = 89 Although there were slight variations in the rating orderings between the fourth and eighth most popular statement selections, both male and female respondents also agreed that the least two important statements in terms of their learning were: You can look at a classmate s work and suggest improvements which attracted only (n=53) selections; and You get classmates comments AND teacher feedback, which attracted (n=61). This consistency was also reflected across the schools and there was very little difference between the six participating groups (see full table in Appendix). 3.4 Assessing the value of the film in relation to the pupil principles In the ninth question, students were asked Could the film help learners and teachers to talk about some of these things (the listed pupil principles)? The students were then given three answer options: Yes, No and Maybe. Across all the schools, only 26 students (5.2) answered No ; 256 students (51.5%) answered Yes ; and 215 students (43.2%) answered Maybe. Table 4: Could the film help learners and teachers to talk about some of these things? Yes No Maybe 125 males females = males + 8 females = (114) = 215 In terms of gender, there were only marginal differences in the responses. 51.2% of the boys answered Yes ; 7.3% No ; and 41.4% Maybe. The corresponding figures for the girls was 51.8% Yes ; 3.2% No ; and 45.0 % Maybe (full table in Appendix).

5 The highest proportion of Yes responses was given by Shimna (75%) and the lowest by Our Lady (35.1%). Both St Pats Keady (year 8) and Banbridge recorded zero No responses, while Our Lady gave the highest proportion of No responses (8.6%). 3.5 Key findings from the quantitative data With an overall average rating of 3.62 out of 5 and the fact that over 60% of all respondents rated the film 4 or 5, it is clear that the young people from participating schools liked the film A Classroom Tale. However, it is also clear that there is something of a mismatch between liking the film and its relevance to their experience of school where a significantly lower rating of 3.11 was recorded and only 43% gave a rating of 4 or 5. The explanations provided by the students for this variation are outlined in the qualitative section. Although the female respondents rated the film and its relevance slightly higher than the male respondents, there were only marginal differences in the quantitative responses in terms of gender. This convergence was also apparent when both groups selected the precise same ordering of the top three and bottom two pupil principles. In terms of variations between schools, and in cognisance of the small sample sizes of Ballysally, Newcastle and Banbridge, only marginal differences were recorded and, in general, the findings reported were consistent across all participating schools. 4. Qualitative findings from the Student Feedback sheets In addition to the four quantitative questions as detailed above, students were also asked a series of qualitative / interpretive questions: Question 2: What did you like about the film? Question 3: Is there anything you don t like about the film? Question 5: Why was the film relevant / not relevant to your experience of school? Question 6: How would you describe the film to someone who hasn t seen it? Question 7: Will you encourage your friends and family to watch it? Question 10: How might the film help learners and teachers talk about the (listed) pupil principles? Question 11: The questionnaire ended by inviting students to provide any concluding remarks In this section, the qualitative responses from these questions have been categorised into the following themes which emerged during analysis of the data. Following a summation of the general views of the film, the most positive aspects of the film and related concerns are outlined. The section then examines the assumed relevance of the film and its future application as a learning resource. 4.1 General views of the film as a learning resource The qualitative responses concur with the quantitative data and clearly indicate that a significant majority of the (n=497) students enjoyed the film and found it valuable as a learning resource. The consensus being that the film was a truthful, honest and helpful account which pretty much sums up what school life is like. It was also frequently suggested that it showed you how to deal with worry. A further common response was that the film gave students an insight into problem solving and the confidence to express your feelings.

6 A nice film about being able to talk to your teachers... the problems you face in school and how to solve them. It s about honesty, how you can get lost in school and how the teachers can help. It s a funny short film with lots of meaning about a boy who worries a lot. 4.2 Most positive aspects of the film In terms of the technical aspects of the film, the students reported that the animations, music, characters, and plot of the story were really very good and that the film was put together brilliantly. In terms of substance, the consensus was that A Classroom Tale was easy to understand, realistic and showed real life situations and how to resolve them. It is also clear that the film resonated with many students experience of being reluctant to ask questions and maybe (be) made fun of. More broadly, the most frequently mentioned positive aspects were: firstly, the way the teacher helped the child ; secondly, that it shows what school is like from a child s point of view ; and thirdly, the way it portrayed issues like embarrassment and teaches you to be more confident. That s the way I feel sometimes... this is true this really happens! I liked that the teacher didn t shout at the pupil when she didn t understand. It shows how teachers make you feel scared to ask questions... being put in the spotlight is embarrassing. Shows how scary school can be but makes you feel you are not alone. It shows you how to be brave and ask for help. It was what school should be like and how teachers and children should get along. 4.3 Issues raised in connection with the film There were very few issues raised in relation to the film, of those which were the most frequently cited was the film s lack of relevance to their own school experience which is examined in the following section. The remaining issues, primarily, related to when Alex was having a bad dream ; when he/she was didn t understand the homework and was scared to tell the teacher. Some respondents also claimed they could not read all the writing, particularly when the letters jumped about ; that the credits went too fast ; and that the film should have been in colour. It s not like that... It s not that easy to say you don t understand. I didn t like it when the boy was scared. Worst bit of the film was when the wee girl was having a nightmare. I could not read some of the writing... it wasn t on long enough to read it all.

7 4.4 Assumed relevance of the film as a learning resource Perhaps the most significant finding from the Student Feedback sheets was the fact that many students made a clear distinction between liking the film and finding it relevant to their own experience of school. While the majority of students could easily relate to Alex and the challenges he/she sought to overcome, many did not think the teacher s supportive intervention was believable. A number of students confessed to often lying in bed worrying about school ; being very anxious with homework ; being really stressed when they don t understand; and feeling too embarrassed to ask for help. Similarly, several respondents admitted that the day dream part was real because it s sometimes hard to concentrate. However, in terms of teachers reactions, there was a fairly broad consensus that it does not happen like this in real life. A significant proportion of respondents claimed that: the teacher would not take it this way ; that some of our teachers are not that easy to talk to ; that if you didn t do your homework the teachers would shout at you, give you detention or make you lose a merit ; and that asking for help can lead to people making fun of you. Conversely, many other students claimed that we would never be scared to ask our teacher ; that teachers here do help and don t embarrass you if you re stuck ; and that some people say they didn t understand homework when they really couldn t be bothered. More broadly, there was a consensus amongst the older students that although the film fairly reflected teacher pupil relations in primary education, it is not like this at all in secondary school. Sometimes I feel just like the girl in the film... I worry in bed too about school and I m nervous to hand homework in. If your work is not right, you worry about the teacher s reaction. This crisis happened to me... I always feel everyone else understands the homework and I don t. Teachers are not that nice; they don t just smile and explain you lose a merit or get detention. It is sometimes difficult to pay attention when you have been sitting for a long while. Teachers here will help if you ask but nobody wants to ask and look stupid. 4.5 Future application of the film as a learning resource This brief section is based on the responses to questions around: whether and why the students would recommend the film to others; how the film could be used to facilitate teacher pupil conversations around the ten pupil principles as detailed in Section 2; and any concluding remarks made by the students in respect of the film as a learning resource. A Significant majority of the (n=497) students claimed that they would encourage friends and family to watch A Classroom Tale. Indeed, many claimed that the film should be shown in every school and that the film would greatly benefit primary school and early post primary pupils, particularly, those who struggle with homework or lack the confidence to ask for help.

8 I think the film would comfort them if they were worried and it shows you not to be scared. It shows a normal day in school; my wee sister should watch it because she worries all the time about school. It encourages people to ask if they need help. In terms of encouraging dialogue between teachers and students around the 10 pupil principles, the consensus was that the film is a great way of showing the best way to learn. It was also suggested that big class discussions should follow viewing of the film and that in these discussions, teachers would: be encouraged to be more understanding and not to shout ; get to know how we worry sometimes ; know what embarrassed feels like ; and learn to be more like the teacher in the film. The potential sensitivities of such conversations were addressed by one respondent who proposed that these class discussions should be with teachers from different classes to prevent shyness. Have a discussion on it afterwards and ask if anyone feels like that sometimes. More communication between teachers and pupils would make teachers more approachable. Teachers shouting hurts your self esteem. Talking about the film will show teachers not to embarrass children. It could help with children s stress if teachers watch it they will see that we do listen we just don t understand. In the final question, which asked students to make any concluding comments, the majority of respondents offered suggestions as to how the film as a learning resource could be improved; and, more generally, how their learning experience at school could be enhanced. There was a definite hope amongst the students that sequels of A Classroom Tale would be made and that any future films would encompass a wider range of school based issues. There was also a widely held view that teachers have as much to learn from the film as the students themselves. Make the film longer and cover more school issues like bullying and shyness. The film should also show someone asking friends for help too because I would ask friends before teachers. The next film should let teachers know that some people just don t like talking out loud in class. Teachers should watch this and see what it feels like to be worried all the time. The best way to learn is to make the lesson fun; a mix of practical and classes would make you day dream less. Well done to the people who made this film it was brilliant and I am going to be less afraid to ask for help in future.

9 5. Conclusions The quantitative and qualitative data from the Student Feedback sheets concur that the participating students enjoyed watching the film A Classroom Tale ; found it easy to relate to the challenges encountered by the central character (Alex); and found the animation a useful resource in terms of identifying the issues which can either help or inhibit their learning and assessment. The data also indicated a similar convergence in terms of inter school and cross gender comparisons where only minor variations were evidenced. Perhaps the clearest example of this consistency was the precise same ordering by male and female respondents of the top three and bottom two pupil principles. The quantitative data on the students liking of the film produced an overall rating of 3.62 out of 5 and the qualitative data revealed that the film s most positive aspects were: the quality of the animations; the fact that it was made by children for children; and the way it encouraged pupils to seek help in situations where learning is difficult or not understood. The scene in the film where Alex is lying in bed worrying about school / having a bad dream about homework clearly resonated with the students. It was frequently recounted that these anxieties were familiar to many of the young respondents. A similarly widely held view was that the film accurately depicted how uncomfortable many of the students feel when (a) they don t understand the task or homework they have been set; (b) have to ask for help (and risk looking stupid ); or (c) are singled out by the teacher to answer a question, because being put in the spotlight is embarrassing. This is clearly an area where the film, its resources and post viewing class discussions could help students mediate these confidence barriers, advance past the comfort of passive approaches towards more co constructive and collaborative forms of learning and assessment. However, the film s relevance (most apparent in terms of Alex s challenges and associated worrying) did not appear to extend to the teacher s reaction to Alex s plight. This mismatch between liking the film and its relevance was reflected in the significantly lower Likert scale average rate of While some students rejected the premise because they claimed that teachers were more supportive than was shown in the film, many more thought the teacher s reaction was not believable because they are not that helpful. These critical observations aside, the film has clearly encouraged students to reflect on their learning and many respondents commented that following this reflection on the relationship between Alex s school experience and their own, they feel less isolated and more confident. In terms of the film s overall impact on the students, perhaps the most telling suggestions forwarded were that sequels of the film should be made to cover other school related issues (such as bullying, friendships and shyness ). This would appear to confirm that the way the film was made and disseminated: firstly, was a highly effective method of illustrating the fears and apprehensions of young people in school; secondly, provided an accessible and informative account of Alex successfully overcoming his/her challenges; and thirdly, affords a range of opportunities for teachers and students to examine individual and collective barriers to an improved learning and assessment environment.

10 Bibliography Bernard, H. R. (1995) Research Methods in Anthropology: Qualitative and Quantitative Approaches (2 nd ed.), Thousand Oaks, California: Sage Publications. Bryman, A., Becker, S., & Semptik, J. (2008) Quality criteria for quantitative, qualitative and mixed methods research: A view from social policy, International Journal of Social Research Methodology, 11, Creswell, J. W. (2012) Educational research: Planning, conducting, and evaluating quantitative and qualitative research (4th edition). Upper Saddle River, New Jersey: Merrill Prentice Hall. Gomm, R. (2008) Social research methodology: a critical introduction (2nd edition). Palgrave Macmillan, Basingstoke. Guest, G. (2012) Applied thematic analysis, Thousand Oaks, California: Sage Publications. Krippendorff, K. (2004) Content Analysis: An Introduction to Its Methodology (2nd edition). Thousand Oaks, CA: Sage Publications. Kumar, R. (2011) Research methodology: a step by step guide for beginners (3rd edition). London: Sage Publications. Sarantakos, S. (2005) Social research (3rd edition). Basingstoke: Palgrave Macmillan. Scott, J. (2000) Children as Respondents: The Challenge for Qualitative Researchers, in P. Christensen and A. James (eds) Research with Children: Perspectives and Practices, pp London: Falmer Press. Teddlie, C. & Tashakkori, A. (2009) Foundations of mixed methods research: Integrating quantitative and qualitative approaches in the social and behavioural sciences. Thousand Oaks, CA: Sage.