Manual for Judges Undertaking Paired Comparisons of Student Work Samples

Transcription

1 ACACA Comparability Project 2015 Phase 5: Paired Comparisons Manual for Judges Undertaking Paired Comparisons of Student Work Samples Participating assessment and certification agencies have welldeveloped, robust and cost-effective procedures to demonstrate the extent and nature of year-to-year and cross-jurisdictional comparability of standards. Project Objective This manual provides necessary information for judges undertaking the paired comparison judging process. Judgements should be based solely on information in this manual supported by the professional experience of judges experienced in the assessment processes of their respective agencies. Information on obtaining additional and more specific advice including technical help and frequently asked questions is included. 1

2 CONTENTS ACACA Comparability Project 2015 Phase 5: Paired Comparisons... 1 Quick Guide... 3 ACACA Comparability Project Overview... 4 The Role of Judge... 4 Planned Outcomes from Paired Comparison Process... 5 Organisational Arrangements for Paired Comparisons Using the Paired Comparison Engine... 5 Technical Requirements... 6 Specific Details about the Judging Process... 6 Being a Judge Not a Marker... 7 Using the Paired Comparison Engine... 7 System Requirements Handy Hints Confidentiality, privacy and security Commonly asked questions and answers Help Desk JUDGES OF STUDENT WORK: CONFIDENTIALITY AGREEMENT

3 Quick Guide 1. Judges are required to look at items of work done by pairs of students in year 12 from participating agencies and then rank them on the basis of their relative merits. 2. In 2015 this process will be applied to two English subjects (including EAL/D); two Mathematics subjects; three Science subjects (Biology, Chemistry, Physics); one History subject, Geography and, where relevant, Psychology. The process will also include samples from England in a Mathematics subject, English (language and literature), Psychology and History. 3. Paired samples of student work are presented and preferences are recorded through an online computer software program (or engine ). 4. Judges will be given a login and password allowing online access to scripts for the period 19 February to 15 March 2015 when judging takes place. 5. The paired comparison engine is accessed by going directly to the website at 6. Judges are expected to be familiar with the section in the Guide providing Specific Details about the Judging Process page Privacy considerations require that the work, and where it is from, must be kept confidential. Please complete the CONFIDENTIALITY AGREEMENT (page 13) and directly to the person in your respective agency who appointed you. 8. The quality in terms of written presentation of scripts will vary - make your judgement about the entire collection of work on the basis of everything that you are able to read. 9. Frequently asked questions and answers are provided at page Remember you are being a judge not a marker - the essential task is to make a judgement about the relative quality of each sample of the students work. Do not be distracted by any marking annotations and focus on the work itself. 3

4 ACACA Comparability Project Overview This project builds on the earlier work of ACACA reflected in its Costs and Feasibility of Reporting on Comparability of Standards Project in A second phase, the ACACA Comparability Project Phase 2 was undertaken during A third phase continued in 2013 and An overview report of the project so far is attached. The ACACA Project has the principal objective of ensuring that participating agencies have welldeveloped, robust and cost-effective procedures to demonstrate the extent and nature of year-to-year comparability of standards and any cross-jurisdictional comparability in the senior secondary sector that may be required in the future. The project currently has four planned outcomes: 1. There are tried, tested, sound and costed procedures for comparing standards of student work within senior secondary courses between-years and across jurisdictions 2. Continuous improvement through publicly available reviews of agencies comparability procedures is widely accepted and understood 3. There are tried, tested, sound and costed procedures for developing and publishing datadriven course cohort comparisons 4. Stakeholders (at jurisdictional and national level) understand the nature, costs and success (validity and reliability) of agencies comparability processes. In 2012 a paired comparisons process was applied across five subject areas one English course, three Mathematics courses and one Physics course. This process presented pairs of student work samples to judges from different jurisdictions, using a web-based delivery process developed in Tasmania. Judges were asked to enter a simple preference for the first or second sample. In 2013 this process was applied to one English subject, two levels of Mathematics, Physics, History and Japanese. In 2014, this process was applied to two English subjects, two levels of Mathematics, Biology, Chemistry, Physics and History. The detailed technical reports provide agencies with information on the sorts of results that this process can provide. The Role of Judge Judges are required to look at work done by pairs of students in Year 12 and then to make a judgement about the ranking of samples. Work samples are provided by different participating agencies and then arranged into pairs to allow nominated judges to make a judgement about which is preferred or is better. Judges are chosen by each state authority based on their expertise in assessing in one of the subject areas selected for this project this year. Participating agencies in Australia, New Zealand and England appreciate the involvement of subject and assessment specialists as judges in the paired comparison process. 4

5 Planned Outcomes from Paired Comparison Process Through the paired comparison process agencies will gain insight into: 1. Developing low-cost and timely processes within each agency for the collection and scanning of work samples. 2. Developing good estimates of the cost (money and time) of paired comparisons involving samples of work from multiple agencies. 3. Exploring the methods, precision and cost of paired comparisons of senior secondary work samples that involve on-line comparisons of substantial bodies of work. 4. Providing a perspective on the results of cohort comparisons. Organisational Arrangements for Paired Comparisons Paired-comparison samples will be collected by agencies by the end of 2014, scanned and forwarded to the Tasmanian Qualifications Authority where they will be incorporated into the paired comparison engine, the online tool with which judges will work. 2. Each participating agency will appoint a contact who is responsible for selection, appointment and liaising with judges. 3. Agency contacts will provide support for judges including co-ordinating the judging process across different subjects, making arrangements for accessing agency computers if necessary, guiding judges through using the on-line program detailed in the next section, and arrangement payment for the work completed. 4. It is not envisaged that judges will receive any face-to-face training in how to make judgements. Using the Paired Comparison Engine Paired samples of student work are presented through an online computer software program referred to as an engine. This means judges have to view all student work, and record judgements, electronically. Judges require access to appropriate computer technology either or both in their workplace or at home. Judges will be provided with a login and password allowing access to scripts via the web for the period of time over which judging takes place. Judging can be done at any time across a 24-hour period seven days a week. In this current phase of the project judging is planned to take place between 19 February to 15 March

6 Technical Requirements 1. Internet connection that is fast enough to download web-based scripts which can be quite large. 2. Computer with sufficient capacity to enable fast download of scripts around 1 GB of disc space. 3. For best results the most recent version of Microsoft Internet Explorer on a PC or Safari on a Mac is required. 4. For best results the most recent version of Adobe Acrobat Reader is required. 5. A computer screen large enough to allow scripts to be more easily read as well as allowing the software option to view both pair items simultaneously. Specific Details about the Judging Process 1. Recording the judgement as to which of the two pieces of work is better is done by clicking on the corresponding button on the screen. 2. Judges are not markers - marking criteria, guides and correct solutions are not needed. 3. Judgements are made simply on the basis of the quality of the work presented, drawing on your experience of what learning is valued in your subject and how this learning is exhibited in the actual work. 4. While it is important that all the work of each student is considered, this consideration need only to be sufficient to make a judgement with some confidence. In many cases, making the judgement will be fast and easy. You certainly aren t being asked to give a mark to the work, or to comment on it. 5. Avoid deciding that two sets of student work are of the same standard - a judgement of differentiation always has to be made. 6. Expect to make judgements on about 65 or so pairs of scripts this may vary depending on your subject and particular situation. Check with your agency contact before exceeding this number significantly. A counter on the engine will allow you to keep track of the number of pairs completed. 7. Expect to do about six judgements an hour the total time to complete the paired comparisons should therefore not exceed the equivalent of about two full working days. 8. Judging should occur within the time period organized by your state contact. 6

7 9. When you undertake the judging is up to you - judging in small blocks over a longer time period rather than in large blocks over a shorter time period is preferable Being a Judge Not a Marker 1. The essential task is to make a judgement about the quality of the students achievement as shown in the work as it appears in front of you. 2. Treat the work of each student as a collection of the student s own authentic responses to valid assessment instruments. You can take into account any actual assessment questions or tasks. 3. Do not take notice of any grades, comments or marks that appear on scripts - this project needs information about what happens when judgements are made by you alone. 4. Do not talk to the other judges about particular work you encounter, about the project work in general, or about the criterion or standards you are applying when making your judgement decisions. 5. Don t try to remember what decision you made previously for a different pair of students, or why. Finish judging the work of one pair of students, then move on to the next pair and view that work on its own merits. Consistency will come from your making judgements across the many pairs. Using the Paired Comparison Engine The paired comparison engine is accessed by going directly to the website at You will receive an or call from the TQA giving you the user name and password that you ll need to use to log in. If you have any problems logging on or call the Help Desk direct at justin.bryen@tqa.tas.gov.au or call Calling from NZ or the UK Once you ve logged in, using the software is quite intuitive as the screen shots from the website show. 7

8 Figure 1 Log on screen At first login, you will be presented with an empty screen with toolbars at the top and the bottom. To begin the judgement process, click on the Show PDF button at the lower left hand side of the screen. Figure 2 Show PDF Screen The two documents will then load into the system. Pairs are determined in the background and will be presented based on internal business rules. 8

9 Figure 3 Presentation of Paired Documents In a typical browser setup, the two documents will display a loading progress bar, the time taken to load will depend on your connection speed. The documents may be viewed one at a time or side by side, using the buttons in the upper left side of the screen, A+B will display both documents, A will display the left document (Document A) and B will display the right document (Document B) Figure 4 Presentation of Single Document 9

10 Figure 5 Scrolling and Zooming using Adobe Acrobat tools The documents may be scrolled and zoomed using the Adobe Acrobat tools. When ready to make a judgement, simply use the Select A and Select B buttons at the bottom left to choose your preferred document, remembering that document A is on the left side and document B is on the right. When you have selected, click on the Submit button to register your judgement and load the next pair. If for some reason you wish to skip a pair, you may click on the Show PDF button again to load a new pair without submitting a judgement This should however be avoided. Figure 6 Select buttons By clicking onto User Information you will be taken to a screen that shows the total number of completed judgements and the number completed in a particular session for individual judges. 10

11 Figure 7 Counter button System Requirements The system will perform best using Internet Explorer 8 or 9 for Windows 7, 8 or XP, or Safari in the Macintosh environment. Either system should have the latest version of Adobe Acrobat reader installed and correctly configured to display inline PDF files (this is the default). Earlier versions of browser and Acrobat have been tested and do work, although their behavior may not be consistent with the above instructions. In some cases depending on an individual computer s configuration a progress bar may not be displayed but in most cases the PDF will still load there will just not be any feedback as to the progress. It is therefore advisable to wait a reasonable amount of time to see if the PDF appears. Do not repeatedly click the Show PDF button as this may cause you to skip over documents. If the PDFs never load, or start downloading to your machine or behave in any manner other than displaying within the system you should seek assistance. In many cases re-installing Adobe Acrobat Reader can rectify these issues. Due to the potential diversity of computer systems used by judges, it is not possible to test on every possible configuration. The project s Help Desk staff will endeavor to work with you through any issues you may encounter. Handy Hints 1. Assessment questions are not always included in the samples. This should not detract from making a judgement about the overall quality of the student s work. 11

12 2. Pages might not always be in order, and single-page drafts or notes may inadvertently be included and occasionally key passages or questions will be given after, rather than before, the response (or even given elsewhere), especially when the response relates to more than one question. 3. It is worth getting into the habit of quickly checking that the piece of work you are looking at is the final version to be assessed, and not just a draft or notes that also had to be handed in. 4. There are a few things that can help you judge more efficiently: Work with both scripts at the same time. Scan a few pieces of work from each student rather than focus on just one. If it seems clear that one student is achieving better than another, look at other pieces of work only to the extent that you feel you need to confirm this decision. There s no rule that you have to read all the work! If the two students work seems to be of much the same standard, don t concentrate on making fine decisions about the work you ve seen. Instead, look at more of their work. Concentrate on finding tasks that are sufficiently demanding so that the better quality work will become more obvious. Confidentiality, privacy and security Privacy considerations demand that the student work, and where it is from, must be kept confidential. As part of this, you must keep secure your login and password and any associated data or materials. If you leave your computer when other people are around, log out or lock your screen. A Form for Judges detailing these requirements is attached. You must sign, date and return this form to your state contact before beginning the paired comparison task. This acknowledges that you are aware of, and will abide by, the confidentiality and security requirements. Commonly asked questions and answers There are a few issues that might arise, depending on what subject you are judging and what work you see. This questions and answers section might be reassuring if you happen to hit any of these issues. What do I do if I can t read the writing? If it s just the odd, individual script that you can t read, try to read it a few times and then decide what, if anything, is legible enough for you to admit it as evidence of achievement (or lack of it). Treat everything that you can t read as non-existent, rather than as diminishing the overall quality of the work. That is make your judgement about the entire collection of work just on the basis of everything that you are able to read. 12

13 Can I use the correct answers that have been provided for the multiple-choice questions or any other questions with a single answer, or do I have to work out the answers myself? Yes, you can use the correct answers given. Do I give more weight to some types of assessment than to others? The assessment instruments themselves can guide you on this. They will, for example, give more weight to writing an essay than to answering an individual multiple-choice question. Avoid the temptation, though, to base your decisions on only those particular types of item that you happen to prefer. What if one student has to answer questions that are much more difficult than the other student s? The difficulty of the questions matters only to the extent that, when more difficult questions are asked, students who have success on them have the opportunity to demonstrate greater achievement. The main thing to remember, though, is that you are not making judgements about the difficulty of the assessment instruments themselves but of the demonstrated achievement in the students responses to them. What do I do if I am having trouble deciding which work is better? Occasionally, the two sets of work forming a pair are very similar in standard and hence making the decision is a bit more difficult. The important thing is not to agonize but to make a considered decision and then move on to the next pair of students. Where do I go to get answers to questions not already covered in this manual? All administrative or financial issues connected with your contribution to the project are handled by your contact and through them by your state authority. If you are unclear about anything to do with your involvement, please contact them. The contact is also the person to ask if you are having issues with the use of the technology. You might have questions about making judgements on the quality of the work itself. Nevertheless, we ask you not to go looking for assistance but just make the best assessment decision you can. One of the aims of this project is to find out what happens when people have to make independent judgements with minimal training and assistance. Help Desk If you have any difficulties associated with using the paired comparison engine contact the Help Desk at the TQA at: justinbryen@tqa.tas.gov.au or call mobile Calling from NZ or UK requires or

14 JUDGES OF STUDENT WORK: CONFIDENTIALITY AGREEMENT Other than as instructed or required as part of carrying out my duties for this project: I,... WILL NOT: disclose any of the information obtained through undertaking these duties to anyone outside the project (this includes not disclosing anything that identifies students or schools) use any information obtained from the duties I undertake for any purposes outside the project AND WILL: keep confidential the login and/or password that I have been given for this project ensure that no other person views the on-screen materials keep any physical materials associated with the project private and secure and return or destroy them at the end of my involvement in the project inform the officer responsible for my involvement in the project if any of these guidelines are accidentally breached or potentially breached. I hereby acknowledge that I have read and agreed to abide by the above guidelines in relation to my involvement in the project. Judge s Signature:... Date:... State contact s signature:... State authority name:... Date:... PLEASE RETURN TO THE ACACA COMPARABILITY PROJECT CONTACT PERSON IN YOUR STATE 14