Progress in Reading Assessment (PiRA) Second edition

Transcription

1 Progress in Reading Assessment (PiRA) Second edition Interim Manual for Autumn tests Years & 2 (updated) Colin McCarty & Kate Ruttle

2 Contents Introduction 3 This interim manual for PiRA Autumn What is PiRA? Why use PiRA? Measuring and following progress using PiRA Progression in reading demand in PiRA 2 Administering the PiRA tests 6 When to test Group size Timing Preparation Administering the tests 3 Answers and mark schemes 8 Marking the answers Finding the total raw score Obtaining other scores Answers and mark scheme: PiRA Autumn Answers and mark scheme: PiRA 2 Autumn 4 Obtaining and interpreting test scores 3 Summative measures Diagnostic and formative interpretation 5 Standardised score tables 7 Age-standardised scores Standardised scores Copyright 205 Hodder and Stoughton Ltd. Version Two 23//205 2

3 Introduction This interim manual for PiRA Autumn This free teacher guidance provides everything you need to administer and mark the Autumn tests. More extensive teacher guidance, including information relating to the PiRA Spring and Summer tests will be provided in the full PiRA Manuals Stages and 2, which will be published in January 206, together with the PiRA tests for Spring and Summer. The PiRA Manuals will also include the following information, to assist you when using PiRA across the whole school year: Diagnostic and formative information Pupil profile sheets for each term, to enable you to review patterns of strengths and weaknesses across the year Further information about interpreting and analysing results Technical information about the standardisation and equating study Answers and mark schemes for the PiRA Spring and Summer tests. In the meantime, should you have any queries about using the PiRA Autumn tests, please What is PiRA? This is the second edition of Progress in Reading Assessment (PiRA). It provides a termly standardised assessment of a pupil s reading attainment, plus a profile of reading skills, which helps you identify those pupils who may need further teaching and practice, as well as enabling you to celebrate success. Separate tests are available for the spring and summer terms in Reception and for each of the autumn, spring and summer terms in Years 6. PiRA is designed for whole-class use, with pupils of all abilities. This new edition retains all the texts of the original edition, but about 20 per cent of the questions have been replaced so that it fully matches the 204 National Curriculum and the content domain-assessable elements of the reading test framework. The new questions have been written to reflect the styles of questions exemplified in the test framework. The tests provide thorough coverage of the new National Curriculum Programme of Study for each year. This has been assured by systematically sampling Key Stage and 2 performance descriptors for English using the 205 reading test framework and being informed by the sample material for the Key Stage and Key Stage 2 national tests. A sophisticated equating exercise was carried out for each termly test between January and July 205 (comprising over 5,000 pupils), to ensure that the marks pupils obtain on each new PiRA test are equated to the marks they gained in the original tests. This enables continuity of data for schools using the new edition of PiRA. Further details of this process will be provided in the technical chapter of the full published manual. Copyright 205 Hodder and Stoughton Ltd. Version Two 23//205 3

4 Why use PiRA? PiRA provides reliable summative information. For example: It provides three carefully designed tests for each year, enabling you to follow the progress of your pupils from term to term, as well as year to year throughout primary school. Marks have been calibrated onto the Hodder Scale, which is an independent measure of progress throughout primary. This enables you to follow progress term by term and compare progress to national norms. It allows you to predict what score pupils should obtain in subsequent terms and so set meaningful targets. PiRA also has a diagnostic capability, enabling you to investigate the strengths and weaknesses of your pupils reading skills. PiRA tests give you five distinct types of information to inform you of each child s, class s and cohort s progress: age-standardised scores, which take into account a pupil s chronological age so that you can see how a pupil s performance compares with other pupils of the same age; standardised scores, which show the standard score for a year cohort; reading age for a quick at-a-glance-reference; performance indicators to help you make a judgement against the year-related expectations of the 204 National Curriculum; the Hodder Scale, which is an independent measure of progress throughout primary. To enable you to use the information in a diagnostic/formative way, total scores can be broken down into the following distinct aspects of reading. Reading analysis Comprehension Inference Language, structure and presentation Content domain reference b d, e a, c Details of the content domain are given in the mark schemes, along with the reading analysis, which defines the relevant area of the curriculum. PiRA systematically assesses pupils reading of different text types or genres fiction, various forms of non-fiction, poetry and playscripts in line with national guidelines across the primary phase. You can also examine the performance of pupils on each question. Using the percentage of pupils that answered each question correctly in the national standardisation (technically, the facility value), you can easily compare the performance of your own pupils with those in the national sample. You will find facility values alongside every question in the mark scheme for the test. Measuring and following progress using PiRA The PiRA test results have been statistically linked from term to term and year to year to enable you to track or predict progress through the whole primary phase, using the Hodder Scale. This enables you to monitor and compare in detail individual patterns of performance against the norms and patterns for the term or year. The Hodder Scale acts as a common spine on which all of the PiRA tests across the whole primary phase are plotted. It provides the statistical basis for predicting pupil progress and future attainment, based on the termly performance data of over 5,000 pupils nationally. The Hodder Scale is presented in Table 4.2. Copyright 205 Hodder and Stoughton Ltd. Version Two 23//205 4

5 Profiling performance by category The reading category that the question focuses on is indicated by a code letter beneath each mark box. (We have used the abbreviations comp for comprehension literal understanding and retrieval from text; inf for making inference including prediction from text; and for language, structure and presentation understanding structure and purpose of text.) If you wish to profile the pupil s performance, add up the number of correct answers the pupil has obtained in each coded category and record these in the boxes at the bottom of the grid on the front cover. You can then compare the pupil s performance to the national averages from the equating study, which are provided in a table beneath the mark scheme for each test. Progression in reading demand in PiRA Each test has been designed to ensure there is a steady progression in the demand of both the reading texts and the questions. We have evaluated the demand of each PiRA question, taking the DfE performance descriptors into consideration, as well as the facility for each question (see below) and our own professional judgement. More detail will be provided about this in the full published manuals. Performance indicators We have provided an algorithmically derived set of thresholds that indicate where the child is with respect to the National Curriculum performance descriptors, by applying the summer standard for the whole year to a test being taken in the autumn or spring. In the mark schemes, we have provided the facility for each question, which is the percentage success of children who took the test in the standardisation. (Note: a high facility indicates an easy question and a low facility a difficult question.) It is possible to set thresholds drawing on the performance of children in the equating standardisation trials, by using the facilities of the questions and applying a reasonable set of expectations (presented in the table below). Please note that these thresholds apply to the curriculum age-expectation for complete years. Therefore, in autumn they will be seen as very challenging and hard to achieve, but any child achieving the expected standard in autumn will be doing very well. This information is given at the end of the mark scheme for each year. It links the performance (facilities) to the language most commonly used to describe children s performance (for example, working towards, emerging ), based on our test-development experience. Table.: Thresholds for performance indicators Working towards Emerging Expected Exceeding 90 00% success (i.e. facility) 60 89% success 20 59% success 0 9% success Our thinking is: to be at the expected standard of performance, for example, children need to get 90 per cent of the emerging questions of that term s test correct, over 60 per cent of the expected questions correct, all of the working towards but none of the exceeding questions. (The thresholds for emerging and exceeding will be explained in the full published manual.) There is no official definition or golden rule that gives us this. However, many years of working in test development and analysing related data have given us the considerable experience to create algorithms to produce thresholds. The critical factor is that the rule we use has to have a sense of reasonableness, that is to say it is based on our experience, which has been informed by discussions with a number of headteachers and literacy co-ordinators who concur with our thinking and methodology for creating a data-driven way to evolve thresholds. Copyright 205 Hodder and Stoughton Ltd. Version Two 23//205 5

6 2 Administering the PiRA tests When to test PiRA should ideally be used just before or shortly after the relevant half-term, as this exactly mirrors when they were standardised and will therefore give the most dependable data. Group size You can administer the tests to whole classes or large groups, if you feel comfortable doing so. With younger children, however, it may be better with small groups supported by a teaching assistant. In Year, it is permissible and often advisable to administer the tests in three sessions with breaks (so, a pair of pages at a time), but this should not normally be necessary at Year 2. Timing For the Year tests, the time taken is controlled by the pace of the teacher s delivery. A maximum time limit of 40 minutes is set for the Year 2 tests. In practice, in all years, the time is likely to take less than 30 minutes for most pupils, unless they are particularly slow readers or hesitant pupils. Preparation Each pupil will need the appropriate test booklet plus a pencil or pen and an eraser. The tests Year and Year 2 use questions which are embedded after short sections of reading material, wherever possible on the same page spread, so that pupils are not required to hunt through the test for answers. Some questions are read aloud by the teacher to the pupils. Where relevant, a teacher's script is provided on the back cover of the test booklet. At Key Stage, the PiRA tests vary in their phonics content, and the text and questions to be read aloud to the pupils (and hence whether the children will be working at their own pace), so when explaining a test to the class you should always be guided by the particular test you are giving. Key points are: - Tell pupils that they will be reading a number of short stories, poems and non-fiction texts, and answering questions about them. - Tell them that some questions will be read to them, but others they will read for themselves. (The details of this are given on the front cover of the test booklet, with any specific script on the back.) - They should do their best to answer all the questions. Say that there will be some questions they will be able to do easily, but that the test tends to get harder towards the end. They should not worry if they find some questions difficult, but just try their best and move on to see if they can answer some of the following questions. Administering the test Give each pupil a test booklet. Ask them to write their names on the front cover at Key Stage, the other information is best supplied by the teacher or teaching assistant when marking the test. The general guideline is that all questions and instructions for you to read aloud are printed in colour, in a different, smaller font. Text and questions for the children to read themselves is printed in black. Copyright 205 Hodder and Stoughton Ltd. Version Two 23//205 6

7 In the Year Autumn test, almost all of the questions are mediated orally, including all the phonics at the beginning of the test. Most of the reading-for-meaning questions are also introduced orally, to ensure pupils are not left struggling. A script is provided on the back page of the test for the phonics. In the Year 2 Autumn test, only the first one or two pages of the test (phonics) are mediated orally (see the details on the front of the test). Following this, the pupils are required to read the questions for themselves. Teachers should follow the relevant teacher s scripts strictly, to ensure that results are both valid and reliable. If any pupils are not clear about what to do, you may give additional explanation to help them understand the requirements of the test, but do not read any of the actual questions, unless it is indicated they are to be mediated by a teacher or teaching assistant. Do not help with individual words. For the results to be reliable, it is important that pupils work alone, without copying or discussing their answers. Copyright 205 Hodder and Stoughton Ltd. Version Two 23//205 7

8 3 Answers and mark schemes Marking the answers Use the score box in the right-hand margin alongside each question in the test booklets to record marks. Follow the mark scheme carefully and use your professional judgement some questions have more than one part, or attract more than one mark. For scores to be valid, you should not award half-marks. Please use your professional judgement when marking, recognising that children often write more words than the brief, crisp answers given in the mark scheme. Capital letters are not required unless specifically stated in the mark scheme. Do not penalise spelling: as long as the meaning is clear, always award the mark. Where a question asks for, say, three answers ticked and a pupil ticks four, deduct one mark. Finding the total raw score You can record total marks for the page at the bottom of each page in the test booklets. Then add together the page scores to find each pupil s total raw score and record this at the bottom of the front cover. Obtaining other scores Refer to the appropriate tables in this manual to obtain the standardised score, age-standardised score, reading age, Hodder Scale score and performance indicator for each pupil. Copyright 205 Hodder and Stoughton Ltd. Version Two 23//205 8

9 Answers and mark scheme: PiRA Autumn No. Answer Mark Content domain reference Reading analysis Facility % Early reading bag and cat joined to correct pictures a 8 2 The cat is eating. a 74 3 banana and orange joined to correct pictures mark for each correct answer. 4 boat and barn joined to correct pictures mark for each correct answer. a a frogs b comp 6 6 fish in a child s fishing net 72 Do not accept two lines to net and fish above the net or a line down from the fish to the net. a 7 A bee can quack. buzz. a a 7 63 An owl can A duck can hoot. 2 marks for three correct; mark for one or two correct. total 0 Hen in the Shed 8 hen b comp 83 9 I can not get into the shed joined to the c 68 fox. I am going to sleep joined to the hen. 0 fox a 8 No No No 2 marks for three correct; mark for one or two correct. b b comp comp chick b comp 66 3 The chick will get out of the egg. 3 c 27 The chick is in the egg. The chick will crack the egg. 2 4 Eggs and Chicks b comp 59 total 8 Bob s Hat 5 red a 60 Accept red written in sentence. 6 To keep his hair dry. d inf 24 7 No No No 2 marks for three correct; mark for one or two correct. b b comp comp a a Copyright 205 Hodder and Stoughton Ltd. Version Two 23//205 9

10 8 paint, hat, coat, boots joined to or pointing b comp 63 at correct parts of picture. mark for three or four correct answers. Do not accept lines from paint to clothes, hat and boots for the paint label. 9 2 Bob got some paint. c 53 3 Bob painted his boots black. Both required for the mark. 20 black d inf 42 total 7 Overall 25 Analysis of performance by category Text type Questions Marks National average mark Word reading Story Story Total marks 25 Reading analysis Comprehension Making inference Language, structure and presentation Raw score Performance indicator 0 3 Working towards 4 6 Emerging 7 22 Expected Exceeding Copyright 205 Hodder and Stoughton Ltd. Version Two 23//205 0

11 Answers and mark scheme: PiRA 2 Autumn No. Answer Mark Content domain reference Reading analysis Facility % My Week 2 b comp 52 2 sleep c 63 3 pig d inf 45 4 It is where I eat. a 59 5 It has rhyme. c 9 It has rhythm. Both required for the mark. 6 hen b comp 82 Accept chicken, bird, snake, duck or spider 7 horse d inf 48 8 stable b comp 5 9 course c 45 total 9 Sam 0 Sam b comp 75 at school d inf 8 2 Arrows correctly labelled: head (do not accept hair) ear or ears throat (do not accept neck) 3 Any one from: looked up shook his head put his head down (on his hands) cry or cried Do not accept felt all wobbly or wants his mum. a a a b comp 43 4 Mrs Jacobs boy b comp 62 Sam office lady Mrs Patel teacher All required for the mark. 5 Mrs Jacobs was going to let him go home. b comp 58 6 Sam is ill. d inf 68 total 9 Motorbikes 7 On the Road b comp or 3 c comp 52 9 Motorbikes (accept Motorcycles) Capital b comp 58 not required or 5 b comp 48 2 Either of: Why do motorbike riders wear helmets? How do you steer a motorbike? Award no mark if more than one ticked unless both correct answers are ticked. d inf 29 Copyright 205 Hodder and Stoughton Ltd. Version Two 23//205

12 22 2 marks for three or four words appropriately linked; mark for one or two words appropriately linked. Accept for handlebars the lump above the headlight. total 7 Overall 25 a a Analysis of performance by category Text type Questions Marks National average mark Poem Story Non-fiction Total marks 25 Reading analysis Comprehension 5.5 Making inference Language, structure and presentation Raw score Performance indicator 0 5 Working towards 6 6 Emerging 7 22 Expected Exceeding Copyright 205 Hodder and Stoughton Ltd. Version Two 23//205 2

13 4 Obtaining and interpreting test scores Summative measures The results obtained from PiRA will also enable you to report pupil performance in terms of: age-standardised score (see Chapter 5) standardised score (linked to the Hodder Scale; see Chapter 5) reading age (Table 4.) the Hodder Scale (previously known as the PiRA Scale; Tables ) and new performance indicators (see Chapter ). Age-standardised scores There are a number of advantages of using age-standardised scores for comparing summative performance. These include: They are standardised to an average score of 00, immediately showing whether a pupil is above or below average, relative to PiRA s national standardisation sample. They allow comparisons to take into account the pupils ages: older pupils are likely to have higher raw scores than younger pupils, but could have a lower age-standardised score. This enables you to rank pupils in order of achievement after age has been accounted for. (Note: with younger pupils, exposure to teaching is likely to have a significant if not a greater impact on achievement than the chronological age of the child.) Standardised scores Standardised scores are also standardised to an average of 00. In many ways they are similar to age-standardised scores, except no allowance is made for the age of the child. They are useful to compare children from one cohort to another, but require that the children take the same test for this comparison to be made. We have taken standardised scores and linked them to the Hodder Scale to enable teachers to monitor progress term by term using the scale, and to predict progress too, which standardised scores are not designed to do. A simple way is to look up the marks and use the standardised scores (and Hodder Scale) in Chapter 5 and assume that all children within the range 90 0 of the mean (00) are at the appropriate level in that term and are on target to achieve the expected level at the end of the year. Please note that age-standardised and standardised scores are quite different measures and are calculated differently. (For example, if a class includes a significant number of older children, then an older child could have a high standardised score, but a lower age-standardised score.) Therefore it is not appropriate to relate a child s age-standardised score to their standardised score. Further explanation about using and interpreting age-standardised and standardised scores will be provide in the full published PiRA manuals. Reading ages Many teachers use reading ages as a quick reference: they show the average chronological age of the pupils who obtained each particular raw score (i.e. the chronological age at which this level of performance is typical). For more detailed comparative information, however, and especially for tracking progress over time, please refer to age-standardised scores. Copyright 205 Hodder and Stoughton Ltd. Version Two 23//205 3

14 Table 4.: Reading ages for PiRA & 2 Autumn tests PiRA Autumn raw score Reading age PiRA 2 Autumn raw score Reading age <5:5 4 <4:6 4 5:5 5 4:6 5 5:6 6 4:7 6 5:9 7 4:9 7 5: 8 5:0 8 6:2 9 5:2 9 6:4 0 5:4 0 6:7 5:7 6:9 2 5:9 2 6: 3 5: 3 7:2 4 6:2 4 7:4 5 6:4 5 7:7 6 6:7 6 7:9 7 6:9 7 7: 8 6: 8 8:0 9 7:0 9 >8:0 20 >7:0 20 Diagnostic and formative interpretation Summative measures are valuable, but only give an overall picture of the child s performance relative to his or her peers. Such data may, for example, confirm that the pupil is doing well for his/her age and indicate that no intervention strategy is required. However, a more detailed check may show, for example, that good literal reading accuracy is masking a weakness in comprehension and inference. Profiling pupil performance You can analyse pupils performance in the different reading categories, to see if there are patterns of strengths and weaknesses. You may also go one stage further and check a pupil s individual performance on a specific question and compare how they have performed relative to other pupils in the same year group. Refer to the mark scheme to see what proportion of pupils in that year group answered each question correctly. This is called the facility and is shown as a percentage: 60 per cent shows that 60 per cent of pupils in the national sample answered the question correctly. Reporting progress using the Hodder Scale In developing the PiRA tests, seven cohorts of pupils - over pupils - were tracked termly over a full academic year. Using this information, plus Optional and Key Stage test data, it was possible to link pupil performance from term to term and year to year, to identify patterns that provide a firm basis on which to project future performance and establish realistic expectations. The equating trials carried out in 205 ensure that the linking of Hodder Scale scores remains secure for this new edition. Copyright 205 Hodder and Stoughton Ltd. Version Two 23//205 4

15 Table 4.2: Relating PiRA raw scores to the Hodder Scale PiRA Autumn raw score Hodder Scale PiRA 2 Autumn raw score Hodder Scale Predicting future performance with the Hodder Scale The Hodder Scale enables teachers to monitor progress term by term at a much finer level than the scaling used by most data-tracking programs. The tests for each term have been designed to cover a range of demand appropriate to the year and term. In Table 4.3 you can see at a glance the Hodder Scale score of a pupil in the Autumn term and track to the next column to see the anticipated Hodder Scale score they will obtain if they make average progress the next term. As the tests have been designed to challenge the pupils around the level at which they are expected to be working, you may find that pupils get similar raw scores from term to term across each year, but their level of performance, as shown in the Hodder Scale score, will continue to increase. You may wish to set targets for the future and monitor progress over a term or year. In Key Stage, expected progress is usually at least one-third of a Hodder Scale score each term. Some children do better than this, others less well. Table 4.3 shows how, on average, pupils gaining different Hodder Scale scores move on in subsequent terms up the Hodder Scale. Monitoring the difference between the actual Hodder Scale score and the predicted average Hodder Scale score for an individual pupil or for a whole class enables you to see if there is increasing divergence or convergence to normal progress. Copyright 205 Hodder and Stoughton Ltd. Version Two 23//205 5

16 Table 4.3: Monitoring and predicting progress from Autumn to Spring terms Average Hodder Scale score Average Hodder Scale score PiRA Autumn PiRA Spring PiRA 2 Autumn PiRA 2 Spring Copyright 205 Hodder and Stoughton Ltd. Version Two 23//205 6

17 5 Standardised score tables Age-standardised scores for PiRA Autumn Raw Score Age in years and completed months 5:0 5: 5:2 5:3 5:4 5:5 5:6 5:7 5:8 5:9 5:0 5: 6:0 6: 6:2 6:3 6:4 6:5 6:6 6:7 6:8 6:9 6: Award >30 for all scores in this area :0 5: 5:2 5:3 5:4 5:5 5:6 5:7 5:8 5:9 5:0 5: 6:0 6: 6:2 6:3 6:4 6:5 6:6 6:7 6:8 6:9 6:0 Raw Score

18 Age-standardised scores for PiRA 2 Autumn Raw Score Age in years and completed months 6:0 6: 6:2 6:3 6:4 6:5 6:6 6:7 6:8 6:9 6:0 6: 7:0 7: 7:2 7:3 7:4 7:5 7:6 7:7 7:8 7:9 7: Award <70 for all scores in this area Award >30 for all scores in this area :0 6: 6:2 6:3 6:4 6:5 6:6 6:7 6:8 6:9 6:0 6: 7:0 7: 7:2 7:3 7:4 7:5 7:6 7:7 7:8 7:9 7:0 Raw Score Copyright 205 Hodder and Stoughton Ltd. Version Two 23//205 8

19 Standardised scores PiRA Autumn PiRA 2 Autumn Hodder Scale Raw Score Standardised Score Hodder Scale Raw Score Standardised Score