Standardisation: Resources 1
Internal standardisation of judgments Why? What? How? Who? Where? When? Why? To maintain the standard To keep the team s eye in to level To address particular aspects of assessment which may be causing a problem:- methods of assessment; types of evidence; feedback to learners; variation in practice To get exemplars for consistency To celebrate and share good practice What? What s the risk? New Course or programme? New site? New assessor? Underperforming assessor? Problem with a method of assessment? Problem with type of evidence? Problem with feedback to learners? Problem with a unit or aspect of a unit? 2
Who? Who should be involved? Assessors Internal Verifiers (IV) Peers Learners Colleagues including support workers Others where delivery is off site with instructors or work experience providers How, Where and When? All the time? In small chunks through the year? At a distance? At big face to face events? At Learning Review weeks every half term? At Team Meetings? A combination of all or some of these? 3
Standardisation Activities Standardisation can take many forms. Below are some examples: Review procedures for the registration / certification of learners and agree improvements and changes to meet externally set standards. Identify agree and use common tracking / recording systems which capture external standards and meet internal requirements. Take a learner s piece of written evidence and discuss against the relevant criteria to agree the level between a group of assessors/ivs. Take a selection of photographs submitted as evidence and discussing against the relevant criteria to agree validity and authenticity between a group of assessors/ivs. Take a unit/learning outcome/assessment criteria and agree on the required evidence for a candidate to be successful to agree validity, sufficiency and inclusiveness between a group of assessors/ivs. Take a video recording of a learner and the judge the performance against the relevant criteria to agree standard between a group of assessors/ivs. Take an audio recording of a learner being asked questions and judge against the relevant criteria to agree standard between a group of assessors/ivs. Run a workshop on witness testimonies to establish a common agreement on key features and standard expected. Run a workshop on how to write effective assessment plans and agree key features. Run a workshop on how to write effective feedback and agree key features. 4
Run a workshop observing a range of learners against the relevant criteria. Compare requirements at different levels. Use of alternative assessment methods when learners are finding it difficult to meet the criteria. Peer observation and feedback of the assessment process. Evidencing it It s always a good idea to write down how you plan to operate sampling and standardisation. A brief report after a meeting or workshop, using a template which prompts answers and which is action focused is usually effective. Cascade this amongst the team and follow up! 5
NOCN and OCN Eastern Region Context of Standardisation and Expectations NOCN and OCNER NOCN Focuses on National Qualifications and how far standards are consistent nationally in that qualification. OCN Eastern Region One standardisation per year for each group of NOCN qualifications. NOCN An annual cycle covering a range of qualifications. OCN Eastern Region An annual cycle reflecting the profile of qualifications and units offered. CENTRE (Internal Standardisation) Broader in remit, can include qualifications, units and can focus on a variety of aspects of these depending on the level of risk they present but still looks at consistency of standard See Standardisation Activities More flexible-frequency determined by need but must be across subjects, levels, assessors and verifiers. QR/EV Scrutinises Centre records of standardisation. Carries out review of every unit of every qualification across all sites, all assessors, all verifiers over a period of time. NOCN Attended by Quality Reviewers (QR) / Regional representatives using samples of work and includes updates on Includes tutor/assessors, IVs and others involved in assessment. 6
qualifications. OCN Eastern Region Attended by representatives from Centres and a QR Includes standardisation of QR/External Verifier (EV) judgments NOCN audit OCN Eastern Region to ensure compliance Outcomes are reported to NOCN and back to the who in turn report back to the providers at National Level. The outcomes of OCN Eastern Region standardisations go to NOCN. Leads to national exemplars. Supports consistency. Opportunities to share good practice in assessment, including writing assignment briefs, benchmarking, evidencing, providing feedback to learners. Outcomes inform self assessment and Quality Improvement Plans; help to improve assessment and IV; are recorded for internal use and external evidence for awarding organisations (EV /QR) and Ofsted. Leads to centre exemplars. Supports consistency. Opportunities to share good practice in assessment, including writing assignment briefs, benchmarking, evidencing, providing feedback to learners. 7
Documentation you need to run a Formal Standardisation of Judgments session with a brief reason for each. Checklist for the sample The Unit and Unit plus information The assessment materials /briefing sheet/ guidance The student s work Assessor feedback IV records Results summary for the group or class list to show which samples have been selected showing the levels achieved by all in the class Representative samples of To ensure all components have been included and any omissions noted and explained To understand the context and judge whether the LOs and ACs have been targeted correctly and the indicative assessment method and evidence have been adhered to. To check VARCSI To check in relation to the Unit and the student s work To check helpfulness of the briefing sheet / guidance To check VARCSI To check assessor feedback To check accuracy of assessment To see whether the assessor has matched feedback to the LOs and ACs accurately To see whether it is constructive and developmental To see if the assignment has been verified prior to use To examine the tracking and recording system and feedback to the assessor To note the context and validity of the sample 8
learners work which has been assessed and where some samples have been IV d Centre policies and procedures for assessment IV and standardisation NOCN Centre Handbook Standardisation pro formas Evaluation Form Agenda For reference These may need improvement/amendment after standardisation For reference To record outcomes To evaluate the session To show the direction of the session 9
Unit title Sample SAMPLE CENTRE STANDARDISATION EVENT PARTICIPANT FEEDBACK SHEET PARTICIPANT FEEDBACK PART A: ASSESSMENT and ASSESSMENT EVIDENCE VALIDITY Will the assessment enable the learner to meet the assessment criteria with integrity? i.e. if the criteria requires that an artefact is produced the assessment does not expect something else e.g an essay. Is it set at the right level, so that the learner has the opportunity to meet the criteria? Yes No Comments/Evidence Does the evidence generated match the level of the unit? Is it the evidence suitable for the assessment criteria? i.e. if the criteria requires that an artefact is produced the evidence is not writing about the artefact. 10
AUTHENTICITY Has the assessment been designed so that the learner can demonstrate their learning? Can you be confident that this is the learner s own work? CONSISTENCY AND RELIABILITY If the assessment was used again in a different setting or at another time are you confident that the result would be the same? SUFFICIENCY Would the assessment generate enough evidence to cover all the relevant criteria? Is the evidence sufficient? Enough to convince you the learner met the assessment criteria? INCLUSIVENESS Are there opportunities to present evidence in a range of suitable formats? Can the assessment cater for a range of learners and their needs? Is the language suited to the level targeted? Could reasonable adjustments be made? 11
PART B: ASSSESSMENT FEEDBACK AND DECISION Are the assessor s comments clear, focussed and helpful for the learner? e.g. are they specific, particularly if more work is needed to meet the criteria Do you agree with the assessor s decision? Please explain your judgement PART C: TRACKING AND RECORDING ASSESSMENT Comment on the effectiveness of the tracking and recording of assessment PART C: GOOD PRACTICE- practice that should be shared and celebrated Good Practice in :- assessment ; in types/range of evidence used/in tracking and recording/in feedback to learners 12