ACER Progressive Achievement Tests (PAT) 2013 National Update. Australian Council for Educational Research

Transcription

1 ACER Progressive Achievement Tests (PAT) 2013 National Update Australian Council for Educational Research

2 First published 2014 by the Australian Council for Educational Research Ltd 19 Prospect Hill Road, Camberwell Victoria, 3124, Australia Text copyright Australian Council for Educational Research Ltd 2014 This report is copyright. All rights reserved. Except under the conditions described in the Copyright Act 1968 of Australia and subsequent amendments, and any exceptions permitted under the current statutory licence scheme administered by Copyright Agency Limited ( no part of this publication may be reproduced, stored in a retrieval system, transmitted, broadcast or communicated in any form or by any means, optical, digital, electronic, mechanical, photocopying, recording or otherwise, without the written permission of the publisher.

3 Contents Message from the CEO 2 Message from Director, Professional Resources 3 Progressive Achievement Testing in Australia 4 The PAT journey 4 Future developments 5 Transition from paper to online testing 5 PAT at a glance 6 PAT content 7 Key PAT features 8 Case studies 10 Individualised learning 10 Monitoring an intervention 11 Online reporting 12 The PAT team 14 Meet the expert 14 Get to know your PAT support team 15 Education consultants 15 Online support 15 Test scoring and analysis 15 Customer service and contacts 15 Test scoring and analysis service 16 PAT professional learning 16 1

4 Message from the CEO Over recent decades, higher expectations have been placed on the assessment of student learning. Better information is being sought to assist in: clarifying starting points for action; investigating details of student learning and performance; monitoring improvements and evaluating educational interventions; motivating effort and encouraging self-monitoring; and providing feedback to guide teaching and learning. In my essay Towards a Growth Mindset in Assessment (Masters, 2013) I describe how the approaches we take to assessing and reporting learning send powerful signals to students, parents and the community. I quote Carol Dweck on the importance of promoting a growth mindset through our assessment processes: When [teachers and students] change to a growth mindset, they change from a judge-and-be-judged framework to a learn-and-help-learn framework. Their commitment is to growth, and growth takes plenty of time, effort and mutual support. (Dweck, 2006, 244) An assessment approach that focuses on assessing and monitoring student growth over time is underpinned by an understanding that students of the same age and in the same year of school can be at very different points in their learning and development, and that all students are capable of progress given the right learning opportunities. This approach stands in contrast to the lockstep assumption that all students in the same year of school are at the same point in their learning and draws on research showing that, typically, the most advanced students in any school year are five to six years ahead of the least advanced. Under a growth mindset, high expectations are held for every student s learning progress, regardless of their starting point. This approach to assessment requires teachers to investigate and diagnose the details of each student s current situation, including by identifying difficulties the student is experiencing and misunderstandings that are preventing further progress. The Progressive Achievement Tests (PAT tests) have been developed over many years by ACER s expert team of test developers and psychometricians to meet a need for high quality tools that support the investigation and diagnosis of student learning. The PAT tests are focused on supporting teachers to gather evidence to establish where students are in their long-term learning, identify the best next steps for action, decide on appropriate evidence-based interventions, monitor what progress their students are making over time, and evaluate the effectiveness of their own teaching decisions and approaches. In adopting and using the PAT tests to investigate and diagnose student learning and to monitor progress over time, teachers are working within a growth mindset. The benefits are likely to be better targeted teaching and thus increased levels of student engagement and improved learning outcomes. Professor Geoff Masters AO 2

5 Message from Director, Professional Resources Thousands of schools across Australia have adopted ACER s Progressive Achievement Tests (PAT tests) as the standard for lowstakes diagnostic assessment and monitoring of student learning. Over many years, ACER s team of expert test developers, psychometricians and assessment consultants have worked closely with teachers, principals and systems staff across Australia to develop and deploy the suite of ACER PAT assessments. According to data from 2013, more than 4000 schools across Australia use PAT as part of their regular assessment practice, with around two million PAT assessments administered each year. With a new generation of PAT tools now available online, teachers and schools have more access than ever to timely, targeted and reliable assessments which help identify where students are in their learning and monitor learner progress and the success of teaching and learning strategies and interventions over time. Having helped cement the critical role of assessment in educational decision-making, ACER has increasingly turned its attention to supporting teachers and school leaders to build their capacity to understand and use evidence of student learning gathered from PAT assessments to drive effective and targeted teaching and learning practices. Increasingly sophisticated reporting (including longitudinal monitoring of student learning outcomes), data interpretation services and a rich suite of professional resources, workshops and courses all aim to help teachers, schools and systems make the most of the evidence they are gathering to achieve their ultimate goal of improved learning outcomes for students. In 2013, ACER expanded its professional learning support by offering PAT professional learning workshops as online webinars will see a continuation of that program as well as the release of a new suite of integrated, online PAT teaching resources. ACER remains committed to working with teachers, school leaders and systems staff to continuously improve the PAT tools and professional resources so that they meet the needs of teachers, schools and systems. I congratulate the ACER team and the thousands of teachers and leaders across Australia who have made PAT such a valuable tool in our shared efforts to deliver improved learning for all students. Ralph Saubern 3

6 Progressive Achievement Testing in Australia The PAT journey The ACER Progressive Achievement Tests (PAT tests) have been available for use in Australian schools since the 1970s. Following the model originally developed by the New Zealand Council for Educational Research, PAT Reading Comprehension and Vocabulary was first published by ACER in 1973 and PAT Maths in Both test programs are now in their fourth editions, with reading published in 2008 and mathematics in In 2009 PAT Science and in 2011 PAT Written Spelling, Punctuation and Grammar were published. A hallmark of new editions of the PAT tests has been continuous improvement in content and reporting based on feedback from schools using the tests. In the early days a small number of schools purchased the PAT tests, many to identify students current level of performance for placement purposes. However, with increased reliance on data to inform teaching and learning, and the expectation of governments and parents for schools to collect objective data to supplement school-based assessments, the number of schools now using the PAT tests to measure student progress from year to year has significantly increased. The PAT tests have been embraced by schools to provide immediate, timely, nationally norm-referenced data on student achievement. Schools use the PAT tests to supplement information received from NAPLAN testing, school-based assessment and teacher judgements. Some of the advantages reported by schools include: the ability to implement a test program that fits in with the school schedule; improvement in teachers ability to effectively interpret test results; the opportunity to interrogate the data to identify patterns of responses for individual students and groups of students; analysis of data to inform teaching and learning programs; and use of results to report growth in skills over time. PAT Reading PAT Reading, Revised PAT Reading, 3rd ed PAT Maths PAT Maths, Revised PAT M

7 Future developments Test content development Advances in technology and the increasing capability of the ACER online assessment and reporting platform bring the opportunity to develop new and engaging item types for online delivery, for example, interactive drag and drop, ordering and hot spot items. During 2014, new item development for PAT Maths and PAT Reading Comprehension will complement the current tests, providing schools with more flexibility in tailoring tests to students current level of ability. Item development is also planned to provide teachers with increasingly in-depth information about students strengths and weaknesses that can be used to inform differentiated, individualised teaching and learning strategies. Analysis of data The extensive collection of data through online assessment and reporting enables the analysis of results to check item performance and adjust norm reference data. This in turn leads to more accurate information for schools and an increase in the quality of the test items and improved descriptions of student achievement. Reporting Advances in customised and interactive reporting are focused on addressing the identified needs of teachers, schools and systems to use data collected from PAT tests for a variety of diagnostic, benchmarking and monitoring purposes. Two new longitudinal reports to measure improvement over time are now available. New customised reporting templates are also available for schools to use through ACER s Test Scoring and Analysis service Transition from paper to online testing In 2010, ACER began the process of transitioning the PAT tests to online delivery. By 2014, all the PAT tests were available for use by schools through ACER s innovative online assessment and reporting system. The system was built using the Agile software development approach, a highly collaborative development process that focuses on understanding and responding to the needs of users through iterative and incremental development. ACER s team of software developers have worked closely with test developers, psychometricians, classroom teachers, systems staff, students and the ACER customer support team to build a continuously improving system attuned to the needs of users. The result? In the last two years, over 2,500 schools have signed up to use the system and between them administered millions of PAT tests. That success hasn t meant the completion of the project. ACER continues to invest in system development to address the increasing use of PAT tests and to support the effective use of PAT data to support improved learning outcomes for students. PAT Reading, 4th ed PAT Maths Plus Online PAT SPG All PAT tests online aths, 3rd ed PAT Science PAT Maths, 4th ed 5

8 7 PAT ASSESSMENTS 2,500,000 PAT assessments administered 1000 schools using ACER Test Scoring and Analysis PAT at a glance Test program Publication dates Brief summary of content (latest editions) Range In development PAT Reading Comprehension, Vocabulary and Spelling 1973, 1986, 2001, 2008 (spelling introduced in 2008) Assesses the prereading, reading comprehension, vocabulary knowledge and spelling skills of students Year 1 to Year 10 New items for online delivery at all levels. Test structure to provide more flexibility in tailoring the test to students level of ability PAT Maths 1984, 1997, 2005, 2013 Assesses six strands of mathematics: number, algebra, geometry, measurement, statistics and probability Year 1 to Year 10 New items for online delivery at all levels. Linking of all items to Australian Curriculum strands PAT Science 2009 Assesses four broad areas: life and living, energy and change, earth and space; materials Year 3 to Year 10 Expansion of test content to mirror Australian Curriculum; new interactive items for online delivery PAT Spelling, Punctuation and Grammar 2011 Assesses grammar, punctuation and written spelling in two separate test programs Year 2 to Year 10 Implementation of interactive functionality for online tests 6

9 Over 4000 schools using PAT in 2013 Over 2500 schools using PAT online in teachers attending PAT workshops and seminars 40% 60% of primary schools using PAT of combined and secondary schools using PAT PAT content Mathematics PAT Maths assesses number, algebra, measurement, geometry, statistics and probability in multiple-choice format. The assessments also address the mathematical processes of understanding, fluency, problem solving and reasoning. The skills assessed by each question are mapped against the Australian National Curriculum for Mathematics. Reading PAT Reading Comprehension assesses retrieving directly stated information, interpreting explicit information, interpreting implied information and reflecting on texts in multiple-choice format. These skills reflect comprehension skills described in the Australian National Curriculum for English. Vocabulary PAT Vocabulary assesses word knowledge through synonyms. The questions are all multiple choice. Grammar and punctuation PAT Punctuation and Grammar assesses a range of language convention skills including the usage and purposes of grammar and punctuation at the word and sentence level. The skills reflect those outlined in the Australian National Curriculum for English. Spelling There are two PAT spelling tests: dictated spelling and written spelling: Dictated spelling requires the teacher to administer and score. The teacher reads a sentence aloud containing a word that students then spell. Written spelling requires students to correct a misspelt word in a written sentence. Half the questions in each test have the misspelt word identified and half do not identify the misspelt word. Detailed diagnostic information is provided for the spelling of each word. Science PAT Science tests student knowledge of the concepts, skills and processes of science. Questions relate to four broad areas life and living, energy and change, earth and space and materials. Tests are multiple-choice format. Short answer questions for each year level provide additional information on students scientific literacy. 7

10 Key PAT features Professionally developed test content The development of each edition of the PAT tests follows a rigorous procedure to ensure that the final tests are of the highest quality and address a range of content strands and features at an appropriate level of difficulty. Careful consideration is given to the item content to ensure that relevant curriculum areas are covered and question format and language are appropriate. Expert test developers work together with psychometricians to develop and select item content with reference to the test construct, mapped against relevant curriculum frameworks. A described proficiency scale to measure and describe progress At the centre of each PAT test is a described proficiency scale constructed using the Rasch Measurement model. The scales provide both quantitative data on student performance, which can be compared fairly and reliably between tests and over time, and qualitative data, which describe typical student performance at different levels of achievement. These make the PAT tests ideal for understanding students current strengths and weaknesses, informing teaching and learning strategies and monitoring progress. Norm referencing to compare and benchmark achievement A sample of schools across Australia, representing all systems and states, participate in each PAT norming study, providing schools with an opportunity to compare and benchmark their students achievement with a national student sample. 8

11 Reporting to get the most out of PAT data A range of reporting formats provides teachers and schools with the tools they need to get the most out of student data collected. Individual student reports provide rich information about student strengths and weaknesses. Group reports allow teachers and school leaders to group students according to their learning needs and identify common areas of difficulty for focused attention and resourcing. Longitudinal reporting allows schools and systems to track student achievement over time to monitor progress and the effectiveness of programs and interventions. Customised reporting is provided to schools by ACER Test Scoring and Analysis to meet specific school needs. The flexibility of paper or online The PAT tests are available in paper and online formats providing maximum flexibility to schools. ACER Test Scoring and Analysis services provides scoring and marking services for large-scale paper testing, allowing schools and teachers to concentrate on understanding and using the data. A professional learning approach ACER is committed to supporting the ongoing professional development of teachers to get the most out of PAT testing. ACER education consultants, customer support services and professional development trainers work with teachers and schools to build professional capacity and maintain a focus on improving learning outcomes for students. 9

12 Case studies Individualised learning Jane is a grade 4 teacher in a primary school in Queensland that uses PAT Maths to monitor the progress of all students across the school. We test all our students early in term 1 with PAT Maths, she explains. I get the results back for my own class straight away and I use them to help prepare an individualised learning plan for each student. A key strategy that Jane employs is grouping students at similar levels so that she can organise activities that are relevant to their different learning needs. Often I ll have three groups of students working on different things or different aspects of the same topic, she says. With the first group, I ll do some intensive work on concepts and trying to work around entrenched misunderstandings. A second group might be doing some practice on the concept to cement and master understandings, while a third group might be working on some applications of the concept. The PAT data is also used to identify resourcing and planning needs across the whole school. In looking across all our PAT data, we are able to identify common areas of weakness at particular year levels or across year levels, explains Jane. That often helps inform our plans for our major learning projects for that term or year. It also helps us to understand where we as teachers need to put our focus and build our expertise. What s next? Jane is looking forward to using the new longitudinal reporting function in the online reporting system. When we moved to using PAT online, we noticed straight away how much easier it was to interpret and understand the reports. The longitudinal reporting is going to be another really useful tool which will help us identify those students who aren t progressing at the rate that we d like to see and also help us reflect on how effective our practices are. 10

13 Monitoring an intervention A key funding requirement of a major educational improvement initiative was the collection of evidence of the impact of the program. In 2012 and 2013, ACER worked closely with a number of systems to collect and analyse pre- and post-intervention data using PAT Maths and PAT Reading for over 200 schools. We realised early that the easiest way to collect the data in a consistent and timely way was to set up all the schools in the project to use online testing, explains Mette Høyberg, Project Director for ACER s Online Assessment and Reporting System. We put together a package of support for the schools which included some webinars and online question and answer sessions. Mette and her team helped the systems to manage school registration, uploading of student data, creation of consistent tags and assigning of tests. Some of the schools were already using PAT for local monitoring, says Mette. So, we just made sure they did their usual testing during the identified time frames. We spent more time with the schools that were new to PAT making sure they understood the administration protocols and could get the most out of the data they were collecting on individual student achievement. The ACER team also worked with the central office staff from each sector to implement some new features in the online system that made it possible to group schools together for reporting purposes to reflect the structure and needs of the initiatives. This enabled people involved in the project to view the data being collected in a way most relevant to their needs. For example, teachers were able to view data on students in their own class, senior staff and principals could view whole school data, and regional and central office staff could view aggregated group data. Because all the PAT tests use measurement scales at their heart, we are able to provide really robust and reliable information about student progress, explains Mette. The final reports we provided to the system and the schools clearly showed which groups of students had made progress during the program and how the cohort as a whole had fared. We were also able to provide a comparison of that progress against expected progress for students at each level. The client was really pleased with the reports and most of the schools have signed up to keep using PAT online, says Mette. So it s a real win-win. In the debrief sessions following completion of the intervention some of the key ingredients identified as being crucial to the success of the program included collaborative discussion of results, support from the Principal, time and support for teachers to use results to develop individual learning plans and in-school professional development. 11

14 Online reporting A key feature of the ACER Progressive Achievement Tests has always been the carefully designed reports that help teachers and school leaders understand and unpack the evidence gathered from the tests. With the move to online assessment, PAT reports are better than ever, with interactive features that enable teachers to interrogate the data in a variety of ways, drill down from class to student to individual item views, and generally make the monitoring of student progress easier and clearer. Online group reports These reports show the responses of all the students in a group or class to the same test. The relative difficulty of each of the questions is shown as a scale score to support teachers in identifying questions students might be expected to get right and those that are likely to be too challenging. Each student response is identified as correct, missed or incorrect with the incorrect multiple choice option shown to support teacher diagnosis. 12

15 A range of tools allow teachers to filter and sort data, display data in a variety of formats, including graphical representations, and export data as excel files or printer friendly PDFs. A valuable feature is the ability to call up individual test items by clicking on a question number. Viewing the question will give teachers access to statistics about how their student cohort answered this question. This screen also shows the level of difficulty, the question classification, the Australian Curriculum links and the skill descriptor based on the PAT scale. The navigational tools at the top right corner of this window can be used to navigate through all the questions in the test. Measuring progress (longitudinal) reports From the beginning of 2014 a new advanced reporting feature is available. Longitudinal reports enable schools to measure student growth over time or to compare different cohorts over time. The Box Plot Report compares selected cohorts and shows the cohort median scale score and distribution against the PAT scale. The Scatter Plot Report shows the distribution of students in the selected cohort against the PAT scale, and allows tracking of individual students between the two sittings. Over time, more sittings can be added, providing the opportunity to track students progress over a number of years. 13

16 The PAT team ACER has many decades of experience developing high quality assessments for use by systems, schools and teachers in Australia and internationally. An expert team of ACER staff work together to ensure that our tests are helping support teachers in their commitment to improve learning outcomes for all students. Expert test developers and psychometricians Consultation with teachers, students, schools and systems Linking to most upto-date curriculum documents and research Research and Development Expert publishing team for online and paper delivery Publishing and Delivery Research-based design Regular updating of item statistics and normreference data Expert, timely customer service for online and paper test delivery Test scoring and analysis service Customer Support Professional Learning Workshops and webinars Expert consultants to assist with effective test use Customised school-based professional development Meet the expert Prue Anderson is a Senior Research Fellow in the Assessment and Reporting Humanities and Social Sciences research program. Prue joined the developmental assessment team in 1999, where she was responsible for developing a wide range of assessment materials for system level testing programs, mapping assessments against curriculum outcome statements and frameworks. Prue was a major contributor to PAT Reading Comprehension, 4 th edition and is currently the team leader for development of teaching and learning resources for the PAT Resources Centre and the coordinator of the new PAT online test development. Prue has conducted PAT professional learning workshops across Australia, including the successful 2013 Purposeful Assessment series. 14

17 Get to know your PAT support team Education consultants Barbara Smith Schools Program Manager Terri Cornish National Sales Manager Julia Inglis Education Consultant (VIC, TAS) Maxine O Shannassy Education Consultant (WA) Taylor Hayes Education Sales (Qld) Noel McBurnie Education Sales (Qld, NT) Jo Porter Education Consultant (SA) Online support Mette Høyberg Project Director, Online Assessment and Reporting (OARS) Scott Haines Project Officer (OARS) Tarli Dowling Project Officer (OARS) Daniel O Loughlin Project Officer (OARS) Test scoring and analysis Robyn Hall, Senior Project Director, Test Scoring and Analysis (TSA) Pei Lin Soo Senior Project Officer (TSA) Monty Choi Project Officer (TSA) Customer service and contacts Customer service and orders: Further information about PAT tests: Further information about online assessment: Further information about the PAT Resources Centre: Login to the PAT Resources Centre: Login to Online Assessment and Reporting System: 15

18 Test scoring and analysis service ACER test scoring and analysis service is available to schools conducting PAT paper tests. ACER scans answer sheets and prepares group and individual reports which are sent to schools within 10 working days. ACER also provides collated results across groups of schools for systems and sectors. Services include detailed analysis of school results, for example, monitoring of students growth over time, interpretation of patterns of responses and comparisons of cohort results from year to year. PAT Score Year Level Pat norming 90th percentile Pat norming 75th percentile Pat norming 50th percentile Pat norming 25th percentile Pat norming 10th percentile School Cohort Comparison of mean scores for matched cohort from 2010 to 2011 PAT Score Yr Yr Yr Yr Yr Yr Yr Yr PAT professional learning ACER is committed to supporting teachers to build their professional capacity to use assessment effectively to inform teaching and learning strategies and practices and improve learning outcomes for students. PAT professional learning is available in a variety of formats: face-to-face workshops, webinars, and school-based seminars. ACER education consultants are also available to consult with schools on implementing and getting the most out of school-wide assessment programs. Workshops In 2013 a series of Purposeful Assessment workshops were conducted in Sydney, Adelaide, Perth, Melbourne and Hobart. Over 500 teachers participated and the feedback was very positive. Fantastic presentation I appreciated the clarity of explanations and the summary at the end of Session 1 Great explanation of the scoring of the tests and what parts are relevant/most useful and how to group them. Well-paced and very targeted at what I came for. Excellent course notes and good balance of information with time for clarification. 16

19 Webinars In 2013, ACER introduced interactive webinars for online clients. Initially the webinars focused on helping schools use the online assessment and reporting system efficiently and demonstrating new features as they became available. The use of webinars will be expanded in 2014 to focus more broadly on interpreting and understanding PAT data and linking to appropriate teaching and learning strategies. Webinars will be run at convenient times for teachers and duration will be 50 minutes. Recordings are available of each webinar so participants can look at each session as many times as required to reinforce content and share with colleagues. School-based professional development ACER s six expert consultants conduct introductory presentations and professional development at individual schools and network meetings throughout Australia. Advice is provided on selecting appropriate test programs, online and paper administration, and interpreting and applying PAT results. Nearly 400 sessions were conducted in schools and regional offices across Australia in Registrations for PAT Professional Learning Seminars: To book a school visit with an ACER education consultant call Introducing the PAT Resources Centre As schools collect more and better quality data on student learning, ACER has turned its focus to supporting teachers to tailor teaching and learning informed by evidence gathered from PAT tests. The PAT Resources Centre provides teachers with a direct link between measured levels of student achievement on PAT tests and targeted teaching and learning strategies and activities. Find out more and view samples at Teachers log in to the PAT Resources Centre to access: Video tutorials supporting data literacy Interpretations of PAT data Teaching notes Lesson plans Resources to support reading comprehension and maths will be released on the PAT Resources Centre throughout

20 Australian Council for Educational Research