The influence of teaching strategies on student achievement in higher order skills

Transcription

1 The influence of teaching strategies on student achievement in higher order skills Patrick Griffin University of Melbourne, Vic. Professor Patrick Griffin holds the Chair of Education (Assessment) at the University of Melbourne and is the Director of the Assessment Research Centre. He is the Associate Dean in the Melbourne Graduate School of Education. Professor Griffin has published widely on assessment and evaluation topics including competency development, language proficiency, literacy and numeracy, professional standards, and online assessment and calibration. Professor Griffin is currently the Executive Director of the Assessment and Teaching of 21 st Century Skills project, a multi-year, multi-national public-private-academy partnership project sponsored by Cisco, Intel and Microsoft. Co-authors: Esther Care, Michael Francis, Danielle Hutchinson and Masa Pavlevic University of Melbourne, Vic. Abstract Over the past seven years the Assessment Research Centre at the Melbourne Graduate School of Education has worked with the Catholic Education Office in Melbourne. The work has emphasised the development of reading comprehension performances by students, which were promoted and assisted by teachers targeting instruction to the level of development or the Vygotsky zone of proximal development (1974). The hypothesis was that if the teachers targeted instruction where students were most ready to learn, improvements in performance would be pronounced. By and large this has remained the case for the Catholic schools in Melbourne. Four years ago the project was expanded to include DEECD schools in Victoria and expanded to focus on mathematics as well as reading comprehension. Similar results were obtained but gains were less pronounced. The hypothesis was still that if targeted instruction could be aimed at the level of development or the zone of proximal development, increased improvement would be achieved. It became clear that most

2 of the improvement occurred at lower levels of proficiency. Smaller gains were made at the higher order skills level of reading comprehension or mathematics. This was examined further using a series of workshops with teachers. The participating teachers were able to freely offer options and strategies for student development in mathematics and reading at lower order skill levels. However, they were unable to provide strategies to develop higher order skills in either mathematics or reading comprehension. This led to some intriguing issues associated with the rhetoric of closing the gap and may have serious implications for both in-service and preservice teacher education. Introduction This project examined the way teachers used data to teach literacy and numeracy. It examined the implications of a shift from a deficit model to a developmental approach. In recent PISA results Australia s position had slipped while other countries had improved. McGaw (2008) argued that improving nations encourage high-performing students as well as low performers to improve, whereas Australia focuses on remedial action for low-performing students. Our objective is to enable teachers to use data within a developmental framework to improve performance of all students. The teachers work in a culture where evidence is challenged and discussed rather than one in which there is only mutual endorsement of shared teaching strategies. They become increasingly skilled in the theory and application of assessment and the developmental construct they are teaching and better able to link evidence of student learning readiness to targeted intervention. The study had its origins in a project with the Catholic Education Office (Melbourne) (CEOM). In 2004 the CEOM began trials of a range of reading tests in 20 schools, seeking advice on how the test data could be used to improve students reading comprehension. The pilot study was known as the LAP (Learning Assessment Project) (Murray & Rintoul, 2008; Griffin, Murray, Care, Thomas, & Perri, 2008). Professional Learning Teams of teachers (PLTs) were led by the schools literacy coordinators. The PLT members engaged in collaborative discussions based on challenging peer evidence of learning and links between intervention and learning gains. Gains in reading comprehension were compelling (Griffin et al., 2008). Several hypotheses were formulated and this study examined and systematically tested those hypotheses in order to generalise and scale up the procedures across systems, year levels and subjects. The premise was that teachers who used a specific style of evidence-based teaching, and operated within a developmental learning paradigm had an increased effect on student learning outcomes. The study examined the role of collaborative teaching teams (PLTs) in the use of data to enhance decisionmaking regarding teaching and learning strategies. The pilot work suggested that with a data-driven, evidence-based approach to teaching and learning, teachers could manipulate the learning environment

3 and scaffold learning for every student, regardless of the student s development or intellectual capacity (Griffin, 2007). In the LAP project, teachers were shown how to differentiate between deficit and developmental teaching and learning approaches. The pilot study was exploratory and explanations for the improved outcomes were suggested, but have not been tested. The relationship between teacher behaviour, knowledge and values with student learning is the key issue addressed. The criterion was measured using standardised tests of reading and mathematics. The effectiveness of the intervention was assumed to depend on teacher knowledge and understanding of how best to use assessment data to improve learning outcomes. In examining this relationship teachers were assisted in interpreting data and in linking their interpretation to targeted intervention in a differentiated instruction framework model (Perkins, 2006). There is a convergence of research that this is an effective practice in improving teaching and learning (Snow, Burns & Griffin, 1998; Taylor, Pearson, Peterson & Rodriguez, 2005). Merely having and using tests is, on its own, an insufficient condition to inform teaching and improve learning (Halverson, Grigg, Prichett, & Thomas, 2005). Ways to access and interpret test data in an evidence-based approach to teaching and learning was central. Using standardised assessments formatively requires that tests can provide sufficient information to profile students learning and to identify the zone of intervention for individual students. It also requires teachers to link their interpretation of data at both group and individual levels to teaching interventions to examine and explain any improvement in student learning. This has been enhanced by a process of critical and collaborative analysis and discussion of data (Griffin et al., 2008). The common theme among previous studies has been that it is essential to have a process by which teachers can be engaged in interpreting the data, linking the information to their own teaching, and testing the links using the discourse of evidence and accountability among peers. Teachers often do not link their teaching to student achievement, but attribute outcomes to factors beyond their control, such as home background. This is despite evidence that teacher/classroom effects can account for up to 60 per cent of the variance in student achievement (Alton-Lee, 2004). Teachers need to understand their own practice and how it affects student achievement. They need an understanding of the developmental nature of the construct areas in which they teach, and this must precede or underpin their understanding of the developmental assessment. Critical and collaborative discussions, where teachers test their ideas about these links, are an important vehicle for doing this. Teambased models are an effective form of professional development in comparison to traditional workshop models. Change in teaching practice can occur when teachers are engaged in examining their own theories of practice (Deppeler, 2007). The LAP project emphasised this approach which in this project will be implemented in more than 100 schools, over six year levels, and in literacy and numeracy. Teachers collaborative reflections have been linked to improved student achievement (Phillips et al., 2004) and changed teacher perceptions (Timperley & Alton-Lee, 2008). Collaborations in professional learning teams enable teachers to have access to a greater number and divergence of theories to

4 test their own against, particularly if the community draws on differing expertise, but it can be a slow and painful process of cultural change (Ladson-Billings & Gomez, 2001). In the LAP study, it was hypothesised that this approach instilled a peer approach to accountability within the team and enabled teachers constructively to draw on and challenge the expertise and experience of their colleagues (Griffin et al., 2010). Teams of teachers, school leaders, policymakers and researchers appeared to accelerate learning when they were involved in rigorous examinations of teaching and learning, rather than comfortably sharing ideas. The shift from sharing to challenge was important and facilitated when the discourse of challenge was based on observable evidence what students do, say, write or make; not on the interpretation or inferences that are deduced from that evidence (Griffin, 2007). This changed the discourse from a teacher-centred mode to student-based evidence. Deficit approaches to diagnosis of student learning focus on the things that students cannot do and are insufficient to improve learning. In particular they focus on a rescue package for low achievers. Developmental models scaffold existing knowledge bases of all students. They focus on readiness to learn and follow a generic thesis of developing the student. For this approach the expertise of the teacher both in content and in developmental learning and assessment is critical (Wilson & Draney, 1999). The normal practice in teacher professional development programs and in pre-service training is to focus first on teaching strategy. The LAP study made it explicit that there was a prior student condition that had to be measured and generalised to a level of development. Only after these steps was it appropriate to design intervention linked to an overall level of development (Griffin et al., 2008; Murray & Rintoul, 2008). Resource allocation and decision making about instruction then follow the generalisation. In this study it was proposed to measure learning team activity and cohesion and to relate the discussion to clarification of decisions and their links to learning outcomes. In LAP, how teacher teams developed the capacity to use data to improve student learning was also linked to the way in which teacher teams developed data-driven instructional systems to improve classroom practice and monitor student learning. Griffin et al. (2006, 2007) and Alton-Lee (2008) have shown how team leaders and teachers developed formative feedback systems. Timperley and Alton-Lee (2008) have also shown that teachers in teams need to develop as members of their teams. Cohorts of teachers learned how to challenge each other and use evidence to discuss specific issues in a professional experience-based learning approach. Follow-up and support was needed in the school. Professional development was shown to match the learning needs of students so that the new skills can transfer into the classroom. The LAP project incorporated learning opportunities for teachers consistent with principles that underpinned the CLaSS project (Hill & Crévola, 1997) as a school improvement strategy (Hill, Crévola & Hopkins, 2000). It was also consistent with the recommendations of Fullen, Hill and Crévola (2006), who highlighted the importance of professional learning. They identified three core elements that enhanced sustained change in schools: personalization, precision and professional learning. They

5 argued that assessment for learning, although frequently spoken about, was not broadly or effectively practised in schools. In this study the emphasis is placed on assessment for teaching. Method The LAP project provided the opportunity to integrate Fullen et al. s (2006) emphasis on professional learning with Johnson s (2000) recommendations on the effectiveness of teams working at different levels. Level 1 teams consisted of teachers operating in the classroom, level 2 teams consisted of the team leaders, and level 3 teams consisted of the research and system level project management personnel. Within each level team members were accountable to each other rather than to an external system or top-down accountability structures. Their work suggested that the combination of internal, work-based and external input of theory and practice may have the best chance of improving teacher effectiveness if it is linked to PLT activities. The outcomes of the LAP project and other studies discussed above led to a range of research propositions: 1 Student achievement is a function of teacher pedagogy, values, beliefs, knowledge and peer accountability. 2 Teachers classroom pedagogy and use of resources is a function of their theoretical and practical knowledge, beliefs and peer accountability. 3 Teachers beliefs, values and attitudes about evidence-based and developmental learning are a function of peer accountability, their theoretical knowledge of the construct they are teaching, and the theory and practice of assessment and data interpretation. 4 A supportive but challenging environment of a PLT is a function of teachers theoretical and practical knowledge and understanding of the constructs they are teaching and of data use and assessment. 5 Peer accountability and increased emphasis on an evidence based culture and challenge within PLTs is a function of networking PLTs across schools. A relational function linking student achievement (Yi) with teacher characteristics (T x ) summarises the literature and the propositions set out in the foregoing discussion. Yi = f{ta.tb,tu,tk/}/ X, Z and H, (1) where... Tb is the domain of teacher beliefs, values and attitudes about influences on student learning, teacher roles and class and school actions. Measures of attitudes, beliefs and values associated with developmental models, accountability and peer collaboration will be developed and used to monitor these factors and their relationship to student learning outcomes; T u is the domain of teacher use of strategies, resources and data. Measures of pedagogical activities and evidence-based use of data in the classroom and of the accountability mechanisms within the PLT will be developed and monitored; Tk is the domain of teacher knowledge and expertise relevant to classroom management and teaching and the learning

6 outcomes associated with the professional development sessions including understanding of theory and practice of the constructs they are teaching as well as understanding of assessment and reporting and data interpretation; T a is the domain of teacher peer accountability. Measures of peer accountability and the use of challenge in the team will be developed. Evidence-based decision making and PLT culture will be developed to examine the relationships between student achievements and intervention strategies. X, Z and H represent the given characteristics of the student, the school and home background respectively. Measures of each of the teacher domains will be developed in order to test the propositions. Measures of variation can be obtained through an examination of the effects of training and monitoring. This can be achieved via a survey of teacher variables such as beliefs, attitudes and the learning outcomes of PD sessions. Student achievement in the key learning outcome will be measured using standardised tests of reading comprehension and mathematics. The overall conceptual model is illustrated in Figure 1. This paper examines the link between the teaching strategy for reading comprehension and mathematics and the level of skill being developed among the students in those disciplines. Figure 1. Conceptual framework for explaining variance in student learning outcomes Results A series of tests were constructed in reading comprehension in mathematics, covering the year levels from Grade 3 through to Grade 10. The tests were all delivered online and student results were fed back to the teachers in terms of skill levels rather than scores. In order to produce the skill levels, individual test items were analysed for the cognitive skill involved. This skills audit provides the skills descriptions that enables the item response modeling variable maps to be interpreted in terms of levels of proficiency on an underpinning learning progression or construct. Levels on the construct were interpreted in terms of reaching skill competencies for the reading test and numeracy skills were interpreted for the mathematics underlying constructed. The reading progression shown in Figure 2 yielded reports which

7 were given to teachers. The reports indicated which level on the progression each student had reached and these were interpreted as levels of Vygotskian (1970) readiness to learn. Teachers were encouraged to intervene and devise reading or mathematics strategies as appropriate to move each student from one level to the next. This targeted or differentiated instruction approach was the core of the project.

8 Level A B C D E F G H I J Reading skill description Insufficient data to assign a level Matches words and pictures involving concrete concepts and everyday objects; follows short simple written instructions; locates familiar words in a short one-line text; matches words to pictures and follows short familiar instructions Matches words and pictures involving prepositions and abstract concepts; uses cuing systems (by sounding out, using simple sentence structure and familiar words) to interpret phrases by reading on; reads familiar words and identifies some new words; uses simple and familiar prepositions and verbs to interpret new words; matches and recognises words and simple phrases Interprets meaning (by matching words and phrases, completing a sentence, or matching adjacent words) in a short and simple text by reading on or reading back; uses context and simple sentence structure to match words and short phrases; uses phrases within sentences as units of meaning; locates adjacent words and information in a sentence Reads on or reads back in order to link and interpret information located in various parts of the text; interprets sentence and paragraph level texts; matches phrases across sentences; reads forwards and backwards in order to locate information in longer texts Reads on and reads back in order to combine and interpret information from various parts of the text in association with external information (based on recalled factual knowledge) that completes and contextualises meaning; locates, interprets, and reads forward to join multiple pieces of adjacent information; uses multiple pieces of information to interpret general purpose of a document; paraphrases and interprets non-adjacent pieces of information Reads on and reads back through longer texts (narrative, document or expository) in order to combine information from various parts of the text so as to infer the writer s purpose; interprets, and draws inferences from different types of texts by reading backwards and forwards to confirm links between widely separated information pieces; extracts information from a non-traditional (left to right) document; makes judgements about an author's intentions or purpose beyond the text content Locates information in longer texts (narrative, document or expository) by reading on and reading back in order to combine information from various parts of the text so as to infer the writer s personal beliefs (value systems, prejudices, and/or biases); combines several pieces of information from a range of locations in complex and lexically dense text or documents; analyses detailed text or extended documents for an underlying message; identifies meaning from different styles of writing Locates information in longer and dense texts (narrative, document or expository) by reading on and reading back in order to combine information from various parts of the text so as to infer and evaluate what the writer has assumed about both the topic and the characteristics of the reader; can develop and defend alternative points of view to those of the author Combines and evaluates the relevance of multiple pieces of information from a range of locations in complex and lexically dense text or documents in order to determine how the message is constructed; analyses and compares parts of the various texts for cohesion and contribution to an underlying message; explains the deeper significance of sub messages in the text, and differentiate between analogy, allegory; identifies innuendo and undertone in the text Figure 2. Developmental progression for Reading comprehension This approach has been shown to be successful and to increase in effectiveness the longer the school remains in the project and the longer the strategies were involved. It indicates that this is a slow but effective approach to improving student achievement over a long period of time. It is not an instant success strategy because it involves a change in teaching practices and a change in the school culture associated with the use of data. These take time. Teachers needed to learn how to reorganise their class to enable ability grouping for instructional purposes and how to use data to make instructional decisions. Hattie s (2009) data indicates that an average annual growth could be expected for an effect size of 0.4. Table 1 below illustrates that these results are not only achieved but are double or even triple those reported by Hattie (2009) and increase with time involved in the project..

9 Grade 3 Grade 4 Grade 5 Grade 6 Year Year Year Table 1: Effect sizes for growth over time by grade level Figure 3. Distribution of Reading levels over test retest measures for Grade 3 It is not a uniform growth. Major gains were achieved at the lower levels of proficiency. A bar chart indicates (Figure 3) shows the growth taking place across all levels with diminishing numbers of students remaining at the lower levels of proficiency and increasing numbers at higher levels of reading proficiency. But this may be deceptive. A cumulative frequency chart indicates that growth is predominant at the lower levels, but diminishes at higher levels of reading comprehension. An even more pronounced effect is noticed in numeracy development. This is illustrated in Figure 4. This effect is replicated over grade levels in that growth is greatest in Grade 3 but diminishes as the proficiency level increases. Growth in one calendar year for reading 3-year pattern of growth

10 Growth in one calendar year for Mathematics 3-year pattern of growth Figure 4. Growth patterns for Grades 3 to 4 in one year and three years The lack of growth in higher order skills is not as pronounced in schools where the differentiated opportunities to learn are implemented. Growth does occur at the higher levels and can be seen to emerge over a three-year period. This is shown in Figure 4. In an effort to understand how this occurred, workshops were organised with leading teachers from schools were greatest growth had been identified. An interesting and disturbing outcome was identified. Teachers were proficient in recommending strategies for developing lower order skills among their students. However, they were unable to identify strategies at the top levels of the reading or mathematics continua. The results of these workshops are presented in Table 2a and 2b. The level of proficiency is shown in the left column of the table levels are A through L, with L being the most sophisticated or the highest order of skill. Level A was used to indicate that insufficient data was obtained in order to make a decision largely because students have not completed enough work to illustrate their competence in mathematics. Across the top of the table the labels indicate the number of suggestions made in the workshop (113 approaches to differentiated instruction), which the teachers then examined for suitable strategies. They classified the strategy according to its potential use: they could use the suggested teaching strategy without modification; they could use it if it was modified; and they could use it but it would have to be applied to a different level to that suggested. The results are presented in Table 2a and 2b for numeracy and literacy respectively. Of the 147 strategies that could be identified for mathematics across all levels 73 per cent were associated with number skills, only 1 per cent was associated with space geometry. Only 3 per cent were associated with the higher order skill levels of the learning progression. Decision re strategy Level Suggestions use modify re level total L K J I H

11 G F E D C B A misc total Table 2a: Frequency of suggested Math strategies by level A similar pattern emerged for strategies associated with teaching reading comprehension. More than 400 strategies were identified, of which the teachers decided that they could use 72 without modification, 79 if they were modified and a further seven if they could be applied to a different proficiency level to that recommended. The trend in both tables indicates that there are numerous strategies for teachers to help develop lower order skills. However, the workshop was bereft of suggestions for strategies for the development of higher order skills at the top of the developmental progression. Decision re strategy Level Suggestions Use Modify Re-level Total L K J I H G F E D C B A Misc Total Table 2b: Frequency of suggested Reading strategies by level The teachers in the workshop came from schools where improvement in the test scores was most marked over a six-month period with test and retest measures. These were teachers whose students demonstrated

12 the largest improvement and therefore it was assumed that these teachers had available the widest of potentially successful strategies. This was not an artifact of selecting schools with high ability students. The teachers were from schools where the improvement covered all levels of this proficiency scale. However, even with a selection of schools and teachers based on student results, larger improvement was focused at the bottom end of the scale or the development of low order skills and not at levels of higher order skills. The possible explanations for a lack of strategies at higher order skill levels included the following: 1 The format or language of the proficiency levels inhibited teachers interpretation. This in turn diminished their capacity to offer suggestions of intervention strategies and resources. 2 Strategies for higher order skills development are not documented in that they are identified and implemented intuitively. 3 Intervention strategies are reliant on commercially prepared resources. This means that teachers implement the strategies without necessarily understanding how they link to a developmental framework. 4 Teachers lack confidence in being able to articulate their own strategies despite the evidence that their students improve. 5 Teachers have no systematic record on which to draw on in terms of articulating teaching and intervention strategies for students developing at levels of higher order capabilities. 6 Teachers do not know how to intervene with students at a higher order level. The last explanation may be unpalatable. But the replication of the plateauing effect with students at higher order levels may be due to the final rationale offered. It may be that the emphasis on closing the gap means that teachers are encouraged to emphasise intervention at the bottom end of the proficiency scale. The logic says that students at the top end of the scale are higher ability students. As such they should be able to improve at a faster rate than those at the lower levels. This study is showing the opposite. Students at the bottom levels of the proficiency scale are improving rapidly. Students at the top end of the scale are hardly improving at all. The link to teacher strategies and teacher resources is a disturbing link. Because of the way in which the developmental progressions are formulated it is possible to argue that each level in the progression should provide an opportunity for developing skills amongst the students already placed at that level. Because the scales are developed using item response modeling with the response probability of 0.5, students at each level of the scale have approximately 50 per cent chance of being able to demonstrate skills at that level. Lower ability students are identified as being at the lower order skill levels associated with their Vygotskian zone proximal development. This applies to students based at a higher order skill levels as much as it applies to students based at the lower order skill levels. The ability of the students is matched to the difficulty of the skills embedded in the levels on the

13 developmental progressions. Hence it can be expected that the higher ability students have the same chance of success at the higher order skills as do the lower ability students have of success in the lower order skill levels. Teachers were less able to offer intervention strategies at the top end of the proficiency scale, but they were able to offer numerous intervention strategies at the bottom end of the scale. Emphasising improvement at the bottom end of the skill level continuum perhaps indicates that the rhetoric of closing the gap may be denying students at the top end of the scale an opportunity for accelerated progress. It also suggests that at a national or state level overall improvement is constrained by the emphasis on intervention at the bottom end while allowing the top end students to develop unaided. This perhaps means that if this is a systemic problem replicated in the PISA and NAPLAN data there may be a national and systemic problem of a lack of teaching strategies or resources to encourage higher ability students to improve or progress at a rate commensurate with their ability. Perhaps there is a need for a shift in rhetoric. References Alton-Lee, A. (2004). A Collaborative Knowledge Building Strategy to Improve Educational Policy and Practice: Work in progress in the Ministry of Education s Best Evidence Synthesis Programme. Paper presented for the Symposium at the New Zealand Association for Research in Education National Conference - Turning the Kaleidoscope. Deppeler, J. (2007). Collaborative inquiry for professional learning. In A. Berry, A. Clemens, & A. Kostogriz, (Eds.), Dimensions of Professional Learning. Rotterdam: Sense Publishers. Griffin, P. (2007). The comfort of competence and the uncertainty of assessment. Studies in Educational Evaluation, 33, Griffin, P., Murray, L., Care, E., Thomas, A., & Perri P. (2010). Developmental assessment: Lifting literacy through Professional Learning Teams. Assessment in Education: Principles, Policy & Practice, 17(4), Halverson, R., Grigg, J., Prichett, R., & Thomas, C. (2005). The New Instructional Leadership: Creating Data-Driven Instructional Systems in Schools (WCER Working Paper No ). Madison: University of Wisconsin Center for Education. Hattie, J.C. (2009), Visible Learning: A Synthesis of Over 800 Meta-Analyses Relating to Achievement. London & New York: Routledge, Taylor& Francis Hill, P.W., & Crévola, C. (1997). The literacy challenge in Australian primary schools. IARTV Seminar Series, No. 69, November, Hill, P., Crévola, C., & Hopkins, D. (2000). Teaching and Learning as the Heartland of School Improvement. IARTV Seminar Series. December. No Melbourne. Johnson, R. (2000). Research: Teachers as Investigators. In D.P. Chambers (Ed.). CLICK: Computers and Learning in Classrooms K-6. Department of Science and Mathematics Education, University of Melbourne.

14 Ladson-Billings, G., & Gomez, M.L. (2001). Just showing up: Supporting early literacy through teachers professional communities. Phi Delta Kappan, 82(9), Fullen, M., Hill, M., & Crévola, C. ( 2006). Breakthrough. Sydney: Corwin Press. McGaw, B. (2008). The research culture and strategy for improvement. Presentation at the University of Melbourne Planning Seminar, Mt Elisa, Melbourne Business School, October, Murray, L., & Rintoul, K. (2008). The Literacy Assessment Project. An initiative of the Catholic Education Office for continuous school improvement in literacy. Paper presented at the Australian Council for Educational Leaders, Melbourne, September, Perkins, D. (2006). Teaching for understanding. Retrieved 18 October 2011, from Snow, C. E., Burns, M. S., & Griffin, P. (1998). Preventing reading difficulties in young children. Washington: National Academy Press. Taylor, B. M., Pearson, P. D., Peterson, D., & Rodriguez, M.C. (2005). The CIERA School Change Framework: An evidence-based approach to professional development and school reading improvement. Reading Research Quarterly, 40(1), Timperley, H., & Alton-Lee, A. (2008). Reframing teacher professional learning: An alternative policy approach to strengthening valued outcomes for diverse learners. Review of Research in Education, (32), Vygotsky, L. S. (1974). Thought and language. Boston: MIT Press. Wilson, M., & Draney, W. (1999). Developing maps for student progress in the BEAR Assessment System. Berkeley: University of California Press.