Portfolios as a method of student assessment

Transcription

1 Porfolios as a mehod of suden assessmen AMEE Medical Educaion Guide No 24 This AMEE Educaion Guide was firs published in Medical Teacher: Friedman Ben David M, Davis M H, Harden R M, Howie P W, Ker J and Pippard M J (2001) AMEE Medical Educaion Guide No 24: Porfolios as a mehod of suden assessmen. Medical Teacher 23(6). The Auhors Miriam Friedman Ben David is Professor of Medical Educaion, formerly Co-Direcor of he Clinical Skills Cerificaion Programme a he Educaional Commission for Foreign Medical Graduaes (ECFMG) and Associae Dean a he Medical College of Pennsylvania, Philadelphia. Professor Friedman Ben David is currenly a he Cenre for Medical Educaion, Dundee, Scoland. Her main areas of ineres are physician and suden assessmen, programme evaluaion and developmen of leadership in he medical profession. Margery Davis is Senior Lecurer in Medical Educaion a he Universiy of Dundee s Cenre for Medical Educaion. She has provided educaional suppor for he Dundee Medical School since Ronald M Harden is Direcor of he Cenre for Medical Educaion, Universiy of Dundee and Direcor of he Scoish Council for Posgraduae Medical and Denal Educaion s Educaion Developmen Uni. Peer Howie is Professor of Obserics and Gynaecology a he Universiy of Dundee and Chairman of he Scoish Council for Posgraduae Medical and Denal Educaion. He has had a long-sanding ineres in medical educaion and he developmen of porfolio assessmen in paricular. Jean Ker is a Clinical Lecurer in Medical Educaion in he School of Medicine, Universiy of Dundee. She is a pracising General Praciioner and provides suppor o he undergraduae curriculum in clinical skills and assessmen. Marin J Pippard has been Professor of Haemaology in Ninewells Hospial and Medical School, Universiy of Dundee, since He rained in Oxford UK and Seale USA and was formerly Consulan Haemaologis a he MRC Clinical Research Cenre and Norhwick Park Hospial, Harrow UK. As convenor for Phase 2 (years 4 and 5) of he Dundee undergraduae medical curriculum, he has coordinaed he implemenaion of his phase of a new curriculum over he las 3 years. Guide Series Edior: Pa Lilley Producion and Deskop Publishing: Lynn Bell AMEE 2001 ISBN: Copies of his guide are available from: AMEE, Cenre for Medical Educaion, Tay Park House, 484 Perh Road, Dundee DD2 1LR, UK Tel: +44 (0) ; Fax: +44 (0) ; amee@dundee.ac.uk -1 -

2 Conens Purpose of his guide Page Summary A: Porfolios as a mehod of suden assessmen Background o he use of porfolios for assessmen purposes Wha are porfolios? Types of porfolios Wha porfolios migh conain Why use porfolios? The assessmen of professionalism B: The Dundee case sudy C: Psychomeric issues relaing o use of porfolios for assessmen Summary of he porfolio s main assessmen feaures Raers consisency Decision consisency Sampling The consensus approach Forms of validiy Exernal vs. inernal examiners D: Implemenaion of porfolio assessmen Defining he purpose Deermining compeences o be assessed Selecion of porfolio maerial Developing a marking sysem Selecion and raining of examiners Planning he examinaion process Suden orienaion Developing guidelines for decisions Esablishing reliabiliy and validiy evidence Designing evaluaion procedures E: Conclusions References

3 Purpose of his guide This guide informs medical eachers abou he use of porfolios for he purpose of suden assessmen. I looks a: Wha are porfolios? Why use porfolios and he conribuions ha porfolios can make o suden assessmen and he psychomeric issues relaing o use of porfolios in assessmen. Guidelines for he implemenaion of porfolio assessmen. The educaional/heoreical consideraions presened in his guide along wih principles of pracical applicaions should enable he reader o design and implemen a porfolio assessmen. -3 -

4 Summary This guide is inended o inform medical eachers abou he use of porfolios for suden assessmen. I provides a background o he opic, reviews he range of assessmen purposes for which porfolios have been used, idenifies possible porfolio conens and oulines he advanages of porfolio assessmen wih paricular focus on assessing professionalism. The experience of one medical school, he Universiy of Dundee, is presened as a case sudy. The curren sae of undersanding of he echnical, psychomeric issues relaing o porfolio assessmen is clarified. The final par of he paper provides a pracical guide for hose wishing o design and implemen porfolio assessmen in heir own insiuions. Five seps in he porfolio assessmen process are idenified: documenaion, reflecion, evaluaion, defence and decision. I is concluded ha porfolio assessmen is an imporan addiion o he assessor s oolki. Reasons for using porfolios for assessmen purposes include he impac ha hey have in driving suden learning and heir abiliy o measure oucomes such as professionalism ha are difficul o assess using radiional mehods. A: Porfolios as a mehod of suden assessmen 1 Background o he use of porfolios for assessmen purposes Recen changes in medical educaion and raining emphasise he developmen of eaching and learning programmes which mee he needs of he medical profession and sociey as a whole. Professional organisaions and he public a large demand demonsraion of professional aribues which ensure docors finess for pracice while adhering o high sandards of care (GMC 1997). The medical schools and posgraduae raining programmes have responded by inroducing new eaching and learning sraegies which will enhance docors accounabiliy. One imporan innovaion in curriculum reform is he developmen of an oucome-based educaional framework (Harden e al 1999). Learning oucomes of undergraduae and posgraduae raining are defined broadly o allow sudens o recognise heir progress along he scope and deph of heir professional compeences. The use of such frameworks faciliaes he developmen of radiional learning oucomes, such as clinical skills, as well as learning oucomes defining he docor as a professional which, in he pas, have been ignored. The growing ineres and concern wih docors professional aribues is shared worldwide as indicaed by he vas number of medical educaion papers published on he opic of professionalism during he year 2000 alone (Ginsburg 2000). Furhermore, he concep of professionalism has filered ino eaching and learning sraegies where sudens are encouraged o ake responsibiliy for heir own learning, and personalise heir learning experiences. Suden-cenred programmes increasingly incorporae he learners needs, as well as lisen o he learners voice. Consequenly, medical schools curricula are seeking o broaden suden experiences inside and ouside he hospials o allow beer appreciaion of he muli-conex sysem wihin which docors are now pracising. The new direcion of medical schools curricula resuls in somewha less srucure and more auhenic experiences for he rainee which, in urn, increases self-direced learning. Concern wih he lack of coninuiy beween undergraduae and posgraduae educaion is anoher imporan facor. Transiion from he undergraduae o he posgraduae phase of medical educaion should be consisen wih progression from echnical discree abiliies o full inegraion of professional compeencies, which is he ulimae oucome of medical raining. Concurren o hese educaional reforms, new assessmen sraegies are being developed o mee he needs of recen innovaions in he healh professions. The search for new assessmen ools is a reacion agains exising mehods of assessmen, which ofen have had adverse effecs on he learner, he eacher and he curriculum as a whole. I is recognised ha assessmen ools should enhance and suppor learning as well as measure performance. Much curren ineres is in auhenic, performancebased assessmen (Korez e al, 1998) which also encourages learners o ake responsibiliy for heir own learning and guides he learners o accumulae evidence of learning, while incorporaing a crierionreferenced inerpreaion of heir performance. -4 -

5 The porfolio is an aemp o counerac he limiaions of a reducionis approach o assessmen. I faciliaes assessmen of inegraed and complex abiliies and akes accoun of he level and conex of learning. I provides an assessmen soluion for a curriculum ha designs learning along broad educaional and professional oucomes. I personalises he assessmen process while incorporaing imporan educaional values. I suppors he imporan principle of Learning Through Assessmen (Friedman 2000). This guide describes he use of porfolios as a ool for assessmen and explores he conribuions hey can make. I examines how hey can be implemened in pracice and looks a heir psychomeric properies. A previous AMEE guide, Porfolio-based Learning and Assessmen (Snadden e al 1999), inroduced he concep of porfolios in undergraduae, posgraduae and coninuing medical educaion and focused on heir conen and consrucion. 2 Wha are porfolios? Definiions The idea of a porfolio is drawn from he sudy of ar, where he erm signifies a purposeful collecion of work. Secher 1998 A porfolio is a collecion of papers and oher forms of evidence ha learning has aken place. Davis e al 2001 I is a collecion of suden work ha exhibis he suden s effors, progress and achievemens in one or more areas This collecion represens a personal invesmen on he par of he suden an invesmen ha is eviden hrough he suden s paricipaion in he selecion of he conens, he crieria for selecion, he crieria for judging he meri of he collecion and he suden s self reflecion. Gisselle O, Marin-Kniep 2000 These definiions imply ha he purpose of suden assessmen will guide he conen of he porfolio, i.e. bes work, ypical work, mos diverse work, work on a heme, or work on one projec as i evolves over ime (Secher 1998). Definiion of purpose, from an assessmen perspecive, will deermine he porfolio conen, he process of creaing i and he inerpreaion of he evidence. Therefore, porfolios will vary from conex o conex depending on heir specialised purpose. Mos porfolios will share a common characerisic of being cumulaive, i.e. conain work compleed over a period of ime relevan o sudens ongoing work or sudies. 3 Types of porfolios Porfolios are used in all sages of educaion: elemenary, higher educaion and in professional and vocaional programmes. Porfolios are used in elemenary school classrooms as par of ongoing assignmens (Herman & Winer 1994). Sudens prepare draf wrien assignmens over ime and, a a pre-deermined dae, choose he bes work for a show case. Such porfolios could be scored a a school level or a sae level (a a sae scoring sie) - for high sakes examinaions or for qualiy conrol procedures. In he USA, he Sae of Vermon inroduced in 1992 a large-scale porfolio assessmen for grades 4 o 8 in wo subjecs, mahemaics and wriing. Sudens selec from heir on-going assignmens in mahemaics, five o seven bes pieces of work which are scored on seven dimensions of performance, hree peraining o communicaion and four o problem solving. The sudens submi a porfolio wih one bes piece of work and oher specified pieces. The bes piece was scored separaely and he remaining pieces were scored as a se. The wriing pieces were scored on five dimensions: purpose, organisaion, deails, voice/one, usage/mechanism/ grammar. Porfolio maerials were scored by eachers o provide individual scores and by he Sae o provide school profiles in hose wo subjecs (Korez 1998). The Sae of Kenucky developed a similar largescale porfolio assessmen in wriing for 4 h grade sudens. Over a period of one year, he sudens compiled a porfolio comprising six pieces: a personal narraive, a poem, a play, a piece of ficion, one informaion or persuasive piece, one piece from any subjec area oher han English and language ars, a bes piece and a leer o he reviewer abou he bes piece and suden s growh as a wrier (Korez 1998). In he professions, porfolios are used, for example, in recording nurses career and professional developmen hrough formal and experienial learning for periodic regisraion wih he UK Cenral Council (Jasper 1995). In medicine, porfolios are used o sudy criical hinking and self-direced learning in he daily pracice of general praciioners (Al Shehri 1995). The Royal College of Physicians and Surgeons of Canada inroduced a learning porfolio, The Mainenance of Compeence Program (MOCOMP), using compuer sofware for physicians o creae a personal learning porfolio. The collecion of -5 -

6 informaion shows how physicians learn, which facors affec changes in pracice and wha sources of learning physicians use. I also creaes daabases for esablishing sandards of physicians performance and for coninuing medical educaion programmes (Bashook e al 1998). The Naional Council for Vocaional Qualificaions esablished assessmen procedures for qualificaion in England, Wales and Norhern Ireland in which porfolios play a cenral role. I is aimed for young people in upper secondary schooling and young people and aduls in work or in governmen raining. Wolf (1998) repors ha he porfolio approach o assessmen has affeced he desired learning syles, bu major problems have arisen relaing o he manageabiliy of he approach and reliabiliy of assessors judgemens. The General Medical Council in UK has pu forward a revalidaion proposal for pracising physicians (GMC 2000). The 3-sage proposal suggess a porfolio (folder) which will include informaion and evidence regarding docors cumulaive performance o be assessed by groups of docors. 4 Wha porfolios migh conain Porfolios conain sudens ongoing work over ime which may provide evidence for learning and progress owards educaional and professional oucomes or learning objecives. These may include mainenance of compeences, keeping up o dae wih professional pracice, finess for pracice, adherence o professional sandards and more. Porfolios may also conain suden s or candidae s reflecion on he submied evidence. By definiion any maerial which provides evidence for he above menioned educaional and work relaed crieria can be included in porfolios. This may include: bes essays wrien repors or research projecs samples of evaluaion of performance, e.g. uor repors from clinical aachmens video apes of ineracions wih paiens or wih peers records of pracical procedures underaken (log books) annoaed paien records leers of recommendaion CVs wrien reflecion on he evidence and on professional growh. Evidence included in porfolios is limied only by he degree of he designer s creaiviy. 5 Why use porfolios? The use of porfolios for sudens assessmen enables sudens and eachers o engage in a process of learning hrough assessmen. This implies ha he assessmen procedure no only measures and reinforces he desired learning oucomes bu raher enhances he developmen of sraegies, aiudes, skills and cogniive processes essenial for life long learning. Consequenly, he use of porfolios no only broadens he scope of assessmen bu also inroduces a number of educaional benefis. Porfolios conribuion o assessmen The assessmen of learning oucomes: This includes oucomes no easily assessed by oher mehods, e.g. personal growh, self-direced learning, reflecive abiliy, self-assessmen of personal growh, professionalism and more. An exended discussion of he use of porfolios for assessmen of professionalism appears a he end of his secion. The provision of evidence of performance: This evidence may be colleced from a range of sources, e.g. suden s on-going work over ime, over seings and over subjec maers. The represenaion of evidence colleced over a period of ime: Porfolios provide evidence of suden developmen over ime and no jus a snapsho a one ime or a series of snapshos, as in radiional assessmen mehods. Suden progression owards he learning oucomes: Porfolios allow assessmen of progress owards he learning oucomes by using chronological work samples colleced a differen poins of ime. Summaive and formaive assessmen: The evaluaion of he porfolio conen generaes summaive saemens regarding suden performance for promoion or pass/fail decisions. I also provides an in-deph profile of suden abiliies. Summaive evaluaion saemens may also conain informaion regarding sudens srenghs and weaknesses, hus inegraing summaive decisions wih formaive profiles. The curren challenge facing porfolio assessmen is o judge he qualiaive and quaniaive evidence and ye mainain reliabiliy and validiy -6 -

7 of he judgemen. Secion C in his guide expands he discussion of his issue. In addiion o he conribuion ha he porfolio makes o suden assessmen, i also reinforces oher imporan educaional aspecs cenral o effecive eaching and learning. Focus on personal aribues Provides personal and professional educaional evidence for suden learning (Glen and High, 1992). Makes provision for sudens o receive feedback abou heir own personal values, feelings, syles of ineracion and ways of handling significan experiences (Beaie 1991) Provides a personalised sensiive porrai of wha sudens know and are able o do (Herman & Winer, 1994). Conexualises learning and links experience wih personal inerpreaion. Enhances ineracions beween sudens and eachers Allows dialogue beween sudens and educaors and ensures individual proeced ime. Reminds sudens ha learning is a wo-way process beween learner and educaor. Mirrors sudens and eachers work and simulaes eachers o re-assess eaching sraegies (Aschlacher 1993). Raises eachers expecaions in relaion o hinking abiliy and problem solving (Finlay e al 1998). Simulaes he use of reflecive sraegies Faciliaes he use of pas experiences o define learning and o recognise progress. Simulaes he use of reflecive skills in order o analyse and synhesise experiences. Sudens can also describe and analyse learning sraegies hus engaging in meacogniive processes. Allows educaors o separae he qualiy of evidence from he suden abiliy o reflec on he evidence. Expands undersanding of professional compeence The variaions in suden percepion and inerpreaion of heir experiences increases undersanding of sudens professional growh. Those newly acquired insighs inform medical educaion and pracice (Herman & Winer 1994). Encourages a holisic and inegraive approach o medical pracice. 6 The assessmen of professionalism Why assess professionalism? The acquisiion of he professional behaviour required o pracise medicine is slowly emerging as a cenral focus in he new undergraduae medical curriculum (Cohen 2001) and wih i increasing consensus on professional behaviour (Swick 2000). The inroducion of new mehods of assessmen o expand professional horizons (Friedman 2000) presens he profession wih he opporuniy o realign curricula o incorporae he developmen of professionalism as a cenral oucome and find ways o assess professionalism and professional growh. Wha is professionalism? One of he unique problems for medicine in defining professional ideniy is ha i develops from pracice which in recen imes has seen such enormous changes boh in biomedical advances and in shifing role boundaries in he delivery of care (Shapiro 1999). There has been lile appreciaion ha professionalism is a dynamic concep which needs o be defined in he conex of changing expecaions of sociey and changing imes. Recenly, however, clearer definiions of professionalism have sared o emerge. For example, reflecive abiliy is increasingly being idenified as a key componen of medical professionalism (Grundy 1987, Lundberg 1991, Parboosingh 1996, Wear 1997, Haffery 2001). This builds on Schon s work in relaion o he developmen of he reflecive praciioner for professional pracice (1983). Reflecion requires boh cogniive and humaniarian aribues. I is a process ha enables praciioners o deermine heir own acions, criically review hese acions and ac on he oucome in he bes ineres of he clien or paien. Projec Professionalism, which was inroduced by he American Board of Inernal Medicine (ABIM) (1994), defined medical professionalism as an ideology consising of he following elemens: a commimen o: The highes sandards of excellence in he pracice of medicine and in he generaion and disseminaion of knowledge (cogniive aribues) Susain he ineress and welfare of paiens Be responsive o he healh needs of sociey -7 -

8 These elemens were furher defined as he humaniarian qualiies of alruism, accounabiliy, excellence, duy, honour, inegriy and respec for ohers. These are required of all candidaes seeking cerificaion and recerificaion from he ABIM. The Associaion of American Medical Colleges (AAMC) (1998), in heir Medical Schools Objecives projec, have also focused on professional developmen in relaion o boh humaniarian and cogniive qualiies. They have idenified four core aribues ha all medical sudens should be able o demonsrae on graduaion. The four aribues include alruism, knowledge (cogniive) skill and duifulness. They furher idenify learning objecives for each of hese aribues. In relaion o alruism AAMC include characerisics such as compassion, honesy, inegriy, eam relaions, commimen o paiens ineress, undersanding hreas o medical professionalism and an awareness of one s own limiaions. In he UK, he Scoish Deans Medical Curriculum Group (2000), which consiss of five medical schools has idenified a Scoish medical graduae as a compeen and reflecive praciioner. In heir new oucome based model, hey have idenified an apiude for personal and professional developmen as one of he welve curriculum oucomes of a Scoish docor a graduaion. The key feaures of his oucome include self awareness, (developing reflecive pracice o be aware of one s own compeence), self learning (idenifying one s own srenghs and weaknesses) self care (being aware of inernal and exernal facors on one s own professional developmen, career choice and commimen). Why are porfolios a useful means of assessing professionalism? The developmen of muliple sampling in he assessmen of clinical pracice is receiving increasing aenion paricularly in relaion o he assessmen of professional behaviour. Turnbull (2000) has used a sysem of clinical work sampling o assess sudens during a roaion and found i o be a reliable, valid and feasible form of assessmen. Porfolios, which sample evidence over ime, also demonsrae a poenial and reliable ool for he assessmen of acquisiion of knowledge, skills, aiudes, undersanding and achievemens and...reflec he curren sages of developmen and aciviy of he individual (Brown 1995). The use of porfolio assessmen has recenly been inroduced a he Universiy of Dundee School of Medicine (Davis 2001) as a key componen in he final undergraduae degree examinaion, in order o assess aspecs of medical pracice such as professional developmen. Porfolios presen an ideal mehod of assessing professional developmen as hey enable he assessmen of he docor s progress mached agains professional crieria over ime. Porfolios also enable professional developmen o be assessed a differen sages of a praciioner s career. The sampling of work-relaed aciviy reflecs he realiy or auheniciy of an individual s professional clinical pracice. Having a variey of auhenic evens recorded in he porfolio enables he examiner o idenify any cogniive or humaniarian deficiencies ha candidaes may have in relaion o heir professional developmen. The porfolio assessmen process allocaes ime for he examiner o focus on he sudens developmen, during he oral componen of he assessmen, by asking he examinees o reflec on heir professional role and relaed experiences. The porfolio provides an ideal conex o assess reflecive abiliy as a key componen of professional developmen. Time is provided during he porfolio assessmen for examiners o assess he abiliy of sudens o reflec in relaion o heir porfolio evidence. In addiion, reflecive wriing for a porfolio assessmen demonsraes he examinee s experience separaed from his/her personal inerpreaion of he experience. Organising sudens experiences hrough reflecive wriing modifies heir percepion of he experience and enables i o be inegraed ino prior learning or used o iniiae new learning (Eisner 1991). The porfolio assessmen process also provides a safe opporuniy o explore errors in professional medical pracice and sraegies o avoid fuure errors. These sensiive maers can be discussed wihin he oucome of an apiude for personal and professional developmen. -8 -

9 B: The Dundee case sudy Dundee medical school moved o oucome-based educaion in he academic year wih 12 learning oucomes idenified. A hree-circle classificaion model was developed for he presenaion of he learning oucomes (Harden e al 1999). The inner circle describes wha he docor is able o do and his includes seven compeencies: 1) clinical skills, 2) pracical procedures, 3) invesigaing a paien, 4) paien managemen, 5) healh promoion and disease prevenion, 6) communicaion skills and 7) informaion handling and rerieval. The middle circle describes how he docor approaches he ask and i includes 3 compeencies: 8) undersanding of basic, clinical and social sciences, 9) appropriae aiudes, ehical undersanding and legal responsibiliies, 10) appropriae decision making, clinical reasoning and judgemen. The ouer circle describes he docor as a professional and i includes: 11) he role of he docor wihin he healh service and 12) apiude for personal developmen. In Dundee Universiy School of Medicine, he learning oucomes provide he framework for suden learning, eaching and assessmen. Individual sudens progress owards he 12 oucomes, is he focus of he porfolio assessmen. In heir porfolios he sudens provide he evidence ha hey have achieved he oucomes a he appropriae level expeced of a graduaing suden. The oucome definiions are he crieria for he porfolio assessmen. The final examinaions were redesigned o mee he needs of he Dundee new curriculum (Davis e al 2001). The curriculum incorporaed an inegraed, ask-based approach ha is inerwoven wih he oucome model. The final examinaions are adminisered in wo pars: Par 1 occurs in he end of year 4 when sudens have compleed heir core clinical aachmens. I includes a knowledge and problem solving componen measured by exended maching iems (EMI), and consruced response quesions (CRQ); and an OSCE o assess clinical skills. Par 2 of he final examinaions akes place owards he end of year 5 by employing a porfolio assessmen. Sudens and faculy were given he opporuniy o evaluae, in a holisic approach, suden progress owards all he oucomes including hose dealing wih he docor as a professional and personal developmen which are no easily assessed by radiional assessmen mehods. The conen of he porfolio includes sudens summaries of progress owards each of he oucomes and a variey of maerials prepared during years 4 and 5 (Table 1). The noes o examiners and sudens conain deailed descripions of suden work which are lised in Table 1. The porfolio examinaion in Dundee is par of a sequenial assessmen process in ha sudens need o pass he year 4 final examinaions before hey are permied o si for he porfolio. End of year 5 sudens, who pass he porfolio examinaion, require no furher assessmen and hus fulfil requiremens for graduaion. Sudens who presen wih minor or major deficiencies are referred eiher for furher remedial work or for a year 5 final examinaion comprising an OSCE and review of he porfolio afer appropriae correcion or addiions. Figure 1 oulines he process of porfolio examinaion in Dundee from he examiners perspecive. Two examiners independenly read each suden porfolio a few days prior o he dae of he porfolio oral review wih he suden. The wo examiners independenly grade he porfolio conen employing crieria and guidelines se ou by he medical school. They use a marking shee, he porfolio assessmen scoring shee (PAS) o inser heir oucome grades based on heir reading of sudens porfolio work. The examiners gradings of he 12 oucomes are independen bu ake accoun of he already pre-marked suden work. They focus on he overall paerns of achievemen and progress over he wo years raher han he grading of specific pieces of evidence. They make judgemens on how he differen ses of porfolio evidence relae o he suden progress owards he 12 oucomes. In a prereview session on he day of he oral review, he wo examiners mee and discuss he sudens porfolios. They agree on he sudens srenghs and weaknesses and specific issues ha should be explored during he oral review of he porfolio wih he suden. During he oral review of he porfolio wih he suden, which lass 40 minues, he examiners pursue he issues of concern and, a he end, hey independenly assign grades for each oucome based on sudens performance during he oral review. The examiners hen reach consensus on a hird se of oucome grades which consiues he final grading based on which he pass/referred decisions are made. They also make recommendaions for disincion. A he end of each examinaion day, all examiners for his day mee as an Examiners Commiee o consider he referred and disincion sudens, o challenge he decisions -9 -

10 Suden work Year Number Pre-marked* Suden s personal summary of progress owards each oucome No Paien presenaions: shor summaries of paiens seen by he suden, seleced from he 100 core clinical problems on which eaching and learning is based on his phase of he curriculum Someimes marked Case discussions: repors of approximaely 1500 words, each analysing a paien s hisory and findings in erms of one of he curriculum hemes. 5 7 Yes Year 4 assignmen: a projec repor wih a grade awarded by he projec supervisor ogeher wih feedback for he sudens. 4 1 Yes Record of achievemen: a record of procedures ha he suden was expeced o have compleed or observed during he phase. 4&5 1 Signed by Faculy GP special sudy module assessmen form: a repor on he suden by heir general pracice supervisor wih a grade awarded. 5 1 Yes Clinical special sudy module assessmen form: heme special sudy module assessmen form: a repor on suden performance during he module wih a grade awarded for relevan oucomes. 5 2 Yes PRHO appreniceship in medicine assessmen form: a learning conrac beween he suden and his/her educaional supervisor wih grades awarded for each learning oucome. 5 1 Yes PRHO appreniceship in surgery assessmen form: a learning conrac beween he suden and his/her educaional supervisor wih grades awarded for each learning oucome. 5 1 Yes Elecive repor: a repor compleed by he suden afer he elecive period. This was read by one of he wo members of saff responsible for elecive sudies and wrien feedback provided. 5 1 Yes * The pre-marked suden work is scored using he Universiy grading sysem (A-G) and conains commens provided by Faculy. Table 1: Descripion of porfolio conens -10 -

11 Porfolios as a mehod of suden assessmen Porfolio maerial Porfolio reading process Broad sampling of sudens work (graded & no graded) Examiner I Independen grading of oucome Examiner II Independen grading of oucome Pre-porfolio oral review Agreemen on srenghs and weaknesses, o be explored during he porfolio oral review wih he suden Oral review of porfolio by examiners wih he sudens Pos-porfolio oral review Examiner I Independen oucome grading Examiner II Independen oucome grading Consensus on 12 oucome grades by Examiners I and II Consensus by group of examiners Pass Disincion Referred Figure 1: The porfolio examinaion process: he examiners perspecive and reach consensus for he final lis of he passed, referred and disincion sudens. Grades awarded for each oucome correspond o he following crieria: Grade A excellen Grade B very good Grade C saisfacory Grade D borderline pass Grade E marginal fail Grade F definie fail Grade G bad fail Following he Examiners Commiee meeing, sudens are allocaed o one of he following caegories: Exemp from furher pass/fail oucome-based assessmen, (all grades are D or above). Exemp from furher pass/fail oucome-based assessmen, bu o be considered for disincion. These sudens will si for a disincion examinaion which will deermine who is awarded a disincion

12 Poenial exempion from furher examinaion (only one oucome is graded E wih all ohers graded D or above). Furher work will be assigned o address he limied deficiency idenified in he porfolio examinaion. Proceeds o furher examinaion (abou 6 weeks laer) if wo or more oucomes a Grade E or below or one or more oucomes a Grade F or G. Those sudens will si an OSCE and a furher porfolio assessmen. In summary, he porfolio examinaion in Dundee is one sequenial componen of he final examinaion process which sars a he end of year 4. Sudens need o pass he year 4 final examinaion componens before hey can si he porfolio assessmen. Sudens who pass he porfolio assessmen, mee he final examinaion requiremens. Sudens who fail he porfolio assessmen proceed o eiher remediaion or furher esing via an OSCE and furher review of he porfolio. Figure 2 demonsraes he porfolio process from he suden perspecive. For each phase of he process here is an associaed educaional value which indicaes how he porfolio process suppors suden learning. In he beginning of year 4 sudens learn in advance wha is expeced of hem. As hey selec he evidence during years 4 and 5, hey decide wha informaion hey need in order o mee he 12 oucome crieria. When hey compile he evidence in he porfolio, hey demonsrae heir organisaional skills, ime managemen and prioriisaion abiliy. The 12 oucome summary shee allow sudens o reflec on heir progress and prepares hem o hink on heir fee during he oral review. Sudens are asked o complee an evaluaion form a he end of he porfolio assessmen which allows hem o furher reflec on he whole process. Sudens who pass he porfolio complee he cycle of demonsraing performance achievemens. Sudens who were seleced for disincion have furher opporuniies o demonsrae problem solving abiliy, informaion handling and rerieval and knowledge. The ones who were referred o furher work or esing have o ac on heir own deficiencies. Educaional value Task Time Advance organiser Mee learning needs wihin he oucome framework Organisaional skills, ime managemen and prioriisaion Reflecive abiliy Read orienaion book + aend inducion Sar of Year 4 Selecion of he evidence Sar of Years 4&5 Compiling evidence in he porfolio Year 5 Wrie 12 oucome summary shees Year 5 Think on one s fee. Demonsrae professional behaviour Reflecion on he whole process. Have curriculum inpu. Oral review. Defending he evidence Year 5 Complee evaluaion forms Year 5 Pass Mee requiremen Disincion Demonsrae problem solving abiliy, informaion handling and rerieval and knowledge Referred Ac on own deficiencies Figure 2: The porfolio process: suden perspecive -12 -

13 C: Psychomeric issues relaing o he use of porfolios for assessmen 1 Summary of he porfolio s main assessmen feaures Table 2 provides a summary of he porfolio s main assessmen feaures. 1 Formaive and summaive 2 Qualiaive and quaniaive 3 Personalised 4 Sandardised 5 Auhenic Table 2: The porfolio s main assessmen feaures Formaive and summaive The main assessmen characerisics of porfolios focus on he FORMATIVE and SUMMATIVE aspecs in which sudens collec evidence of heir own performance for pass/fail decisions, feedback, diagnosic purposes and inervenion. The formaive value of he porfolio, when linked o summaive decisions, presen a very powerful assessmen ool. In his secion, a forward sraegy is discussed for he use of porfolios for summaive decisions such as promoion, graduaion, cerificaion and oher high sakes applicaions. Qualiaive and quaniaive An imporan feaure of porfolios is he combined QUALITATIVE and QUANTITATIVE approach, unlike many oher performance ess ha rely mainly on he quaniaive aspec of performance. The porfolio conains descripive recorded maerial as well as graded evidence. This approach o assessmen allows he inegraion of qualiaive judgemen wih quanified informaion o generae a more comprehensive inerpreaion of suden achievemen. Personalised The STUDENT CENTERED aspec of porfolios ensures ha he suden assumes an acive role in direcing he evidence while eacher involvemen is essenial in monioring he process. By personalised, we imply he suden individualised selecion of evidence, he individualised experience conained in he evidence, he review process by which he examiner learns he unique suden s characerisics and he opporuniy given o sudens o reflec and defend heir work. This process creaes a highly PERSONALISED assessmen mehod. When he porfolio approach is conrased wih he common forms of an OSCE clinical saion, he OSCE will focus mainly on STANDARDISATION of he encouner while providing an individual assessmen profile. The porfolio seeks o balance he personalised approach o assessmen and he sandardisaion of he process. Sandardised I is difficul o achieve maximum conrol of sandardisaion for he porfolio conen and process. I is possible o deermine he ype of work o be submied (case sudies, projecs, essay, ec). However, human performance wih regard o srucure, deph, volume, maerial exhibiion is highly variable (Pis e al 2001). In heir argumens Pis e a (2001) fear ha a oo srucured approach o developing guidance for using a porfolio as an assessmen ool migh diminish is srengh. The dilemma is how o mainain he personalised aspec of porfolio bu a he same ime sandardize he approach o enhance he porfolio s reliabiliy. Some suggesions for sandardisaion are provided in Table 3. 1 Same porfolio s unis of evidence are assigned o all sudens. 2 Tasks and crieria for assessmen are defined and made clear. 3 Insrucions o sudens provide clear guidelines. 4 The porfolio reading process and raing of maerial follow sandardised guidelines mainly by wrien insrucions and raining workshops for examiners. 5 The probing in an oral review of he porfolio wih he suden follows sandardised guidelines. 6 Pass/fail decisions follow a pre-deermined policy. Table 3: Suggesions for sandardisaion of porfolio assessmen -13 -

14 In cases where porfolio assessmen incorporaes an oral porfolio review componen (noe his is no a radiional oral examinaion) one has o consider sraegies o avoid he many pifalls of oral examinaion (Missin 1985). Facors which accoun for problems in oral examinaions include: axonomic level of quesions (Des Marchais 1995), he subjeciviy of examiners, he lack of sandardised crieria employed by examiners eiher for probing or for grading, lack of shared inerpreaion of examinees behaviour and varied levels of expecaions from examinees. All may conribue significanly o lack of sandardisaion and low reliabiliy. However, some of hese problems could be recified by raining and by exensive reading of individual suden s porfolios which helps prepare he examiners for a conexualised and personalised oral encouner while mainaining a holisic framework of suden performance. The sandardisaion process is cenral o he reliabiliy and he validiy of he porfolio. The validiy of grading relaes o he inerpreaion by he examiners of sudens work, he expecaions, shared undersanding of crieria for assigning he grade and he use of he resuls (Messick 1994). Auhenic Auheniciy is defined as he exen o which he oucomes measured represen appropriae, meaningful, significan and worhwhile forms of human accomplishmens. (Archibald and Newmann 1988). Characerisics of auhenic achievemens as judged in porfolio assessmen are provided in Table 4. 1 Producion of knowledge raher han reproducion or ciing ohers work 2 Disciplined enquiry, dependen on: A prior knowledge base (o be used o help produce knowledge). In-deph undersanding Inegraion he producion of knowledge requires he abiliy o organise, synhesise and inegrae informaion in new ways. 3 Value beyond assessmen aesheic, uiliarian or personal value. 4 Higher order hinking. Table 4: Characerisics of auhenic achievemens as judged in porfolio assessmen (Cumming e al 1999) The porfolio is auhenic in he sense ha i incorporaes pas academic and work-relaed experiences which form he foundaions for he above-menioned crieria. Messick (1994) saes ha a porfolio is an auhenic simulaion in ha i does no simulae a paricular ask, bu raher requires ha he examinees by concepualizing abou he case will generalize o oher occasions. The auhenic aspec of he porfolio is cenral o he validiy of he porfolio maerial and he scores inerpreaions. I is also an essenial aspec of developing professional aiudes while reflecing on one s own experiences (Schon 1987). For his reason Case (1994) saes her concern for he loss of porfolio auheniciy in large scale, more sandardised, less personalised examinaions. The personalised approach of porfolio assessmen is he unique componen which conribues o is auheniciy, i.e. is validiy. The lack of i may increase is reliabiliy bu may also decrease is validiy. 2 Raers consisency Discussions of reliabiliy are ofen cenered around erms of generalisabiliy ; ha is, he exen o which performance is consisen across various sources of error labelled faces. (Korez 1998). In he porfolio case, raers are considered as one face of error ha hreaens generalizabiliy of performance. The raers effec includes issues of raers sabiliy over ime (Tes - rees), sabiliy over raers (Inerraer) and reproducibiliy or decision consisency of pass/fail marks (Livingson 1995). Given he complexiy of porfolio assessmen, he amoun of ime involved in processing he es and he acual knowledge learned abou examinees from he firs adminisraion, presen difficulies for sudying raers sabiliy over a relaively shor ime. As for iner raer agreemen, Le Mahier e al (1993) repored.60 o.70 iner-raer correlaions and generalizabiliy esimaes beween wo raers for each uni of porfolio evidence in he range of.80. Herman e al (1995) repored average correlaions beween pairs of raers of.82. Percenages of absolue agreemens of all pairs averaged resuled in 98%. Korez (1998) saes ha much of he oal error in scores was aribued o facors oher han raers in he porfolio. High agreemen depends on clear crieria, adequae examiners raining, communicaing crieria o sudens, good suden orienaion maerials, examiners familiariy wih he conex and shared undersanding of expeced -14 -

15 suden performance and he assessmen purpose. In he Dundee case sudy, as in oher porfolio assessmens, he sabiliy over raers is mainly relaed o he consisency of heir inerpreaion of paerns of performance over ime and over asks, raher han agreemen on a single uni of performance or on a single grade. The shared undersanding of suden progress owards he curriculum learning oucomes in Dundee is he focus of examiners raining. Auhenic crieria, which incorporae progressional benchmarks owards he global oucomes, is he Dundee curren challenge for achieving high inerraer reliabiliy. We will furher discuss developmen of crieria in he secion regarding implemenaion. However, one needs o quesion he meaning of inerraer consisency in porfolio assessmen. For example, in a porfolio which conains one sample of bes work, for he purpose of assessing an abiliy such as wriing, high absolue iner-raer agreemen will be essenial if decisions are based on he wriing score aained. However, for porfolios which conain broad, complex, muliple asks, one needs o quesion he likelihood of wo independen examiners arriving a idenical grading of individual ses of evidence. In his case, i migh be imporan ha examiners agree on he consequences of he grading raher han on he acual grade obained. 3 Decision consisency Misclassificaion of sudens ino pass/fail or referred caegories due o raers source of error, is a main concern regarding disagreemen among raers. The lack of reproducibiliy as a funcion of low inerraer agreemen, is a major hrea o he reliabiliy of he porfolio examinaion in high sakes examinaion or any porfolio examinaion wih summaive consequences. Iner-raer agreemens for unis of porfolio may resul in high correlaions. However, wha is imporan is he reproducibiliy of pass/fail decisions among raers. For example, do he pass/fail sandards/ crieria or oher policy decisions assigned o he marks produce he same classificaion consequences across raers? If correlaions are high among raers, while one raer consisenly raes lower, sudens who obained he low raings may fail based on he examinaion sandards, while sudens who achieved higher raings may pass even hough he wo raers grading correlaed very highly. Consequenly, agreemens expressed in correlaion saisics are no sufficien if pass/fail sandards are employed. Correlaion saisics and percen agreemens should accompany he following quesions: 1) Wha is he exen of agreemen for individual unis among raers in reaching he same pass/fail resuls? (Livingson 1995). 2) Irrespecive of he grades assigned, do raers reach he same pass/fail decision? Daa available from he firs year of he Dundee porfolio adminisraion, shows 98% pass/referred agreemen beween wo pairs of examiners, who independenly examined he sudens. Generalizabiliy coefficiens will define he amoun of error aribued o raers. I will also assis in esimaing he number of raers needed o achieve high reproducibiliy (Brennan 1983). 4 Sampling Sampling of asks is anoher imporan face of measuremen error in porfolio assessmen. In fac, given he complexiy of behaviours measured by he porfolio assessmen, and in order o achieve consisency of performance across a number of asks, sampling needs o be exensive. Linn (1994) raises he issue of limied sampling in performance assessmen and urges he use of sampling of suden work over ime and over asks o reduce sampling error due o ask, by employing porfolios and/or muliple sources of performance assessmen. Appropriae sampling is a common problem in performance assessmen due o limied examinaion ime. The Dundee approach allows such sampling hrough he preparaion of evidence over ime, over occasions, employing muliple asks while addressing he 12 oucomes. This approach is differen from a porfolio in which one uni of bes work is presened as evidence. In he bes work case, when a limied number of compeences are assessed, he suppor provided by more evidence of he same abiliy as in he Vermon example, will increase he porfolio validiy due o broader sampling. 5 The consensus approach In he Dundee porfolio examinaion, judgemens on suden performance are generaed by incorporaing a consensus approach. Can one employ a consensus approach for pass/fail decisions? How scienific and accurae is he consensus process beween wo raers or among commiee members? There is evidence ha scienific heories are deermined by a consensus approach raher han by a criical hinking process, or by consisency among independen research findings (Hand 1999). I is common pracice in assessmen cenres o incorporae a consensus approach for judging rainees performance (Gagler e al 1987). In pracice, he selecion of rainees and oher higher level -15 -

16 personnel is conduced by selecion commiees which employ a consensus process raher han inerraer agreemens. Juries also operae on a consensus principle. In boh cases evidence is presened, he panel argues he case and decision is graned following a consensus process. Relevan o personalizing assessmen in a performance domain, is he issue of how one guards agains subjeciviy of raers. The porfolio approach, by is complex naure, calls for judgemen of muliple performance aspecs, cogniive, aiudinal, professional and echnical skills ha are bes expressed a he personalised level. The hreas o reliabiliy among raers, in such a complex and personalised underaking, raises he quesion: should consensus procedures be recognised as a legiimae and reliable assessmen sraegy and can consensus procedures replace independen raings? Linn (1994) menions he work of Moss (1994) on validiy versus reliabiliy and brings examples of search commiees examining he credenials of candidaes and arriving a an inegraed decision. Delandshere and Perosky (1994) alk abou confirmaion of prior judgemen raher han replicaion, analogous o a second opinion from a physician. Linn (1994) saes ha he confirmaion approach may be a useful way of looking a suden porfolios and exhibiions? However, in such cases examinee performance is judged by inernal examiners who are familiar wih he educaional conex and he oucome framework as i is pracised, for example, in Dundee. Due o he highly qualiaive naure of porfolios, i seems reasonable o mainain independen raing and srive for high iner-raer agreemen. A he same ime, we may ask: how can we equip examiners wih he righ ools and he appropriae definiions o judge examinees independenly and furher submi hese judgemens o a rigorous consensus process? The use of examples and evidence o suppor raers judgemens agains pre-deermined crieria is one way o achieve a meaningful raings and consensus process. Jusificaion of grading wihin defined frameworks and shared crieria may creae a meaningful and reliable consensus process. In he Dundee example, wo sages of independen raings provide he basis for he final consensus process, which leads o an overall judgemen and agreemen beween he wo raers. I seems ha he hree forms of reliabiliy: agreemen among independen raings, reproducibiliy of pass/fail decisions and rigorous consensus procedures may all provide an inegraed soluion for achieving high reliabiliy and for he use of porfolios for summaive purposes. When raers provide heir independen grading hey own his decision. In he consensus process hey mus negoiae he grades by supplying evidence of suden performance and explain heir inerpreaion of he performances. Iner-raer agreemens could be sudied as well as reproducibiliy of pass/fail decisions o esablish he overall reliabiliy of he porfolio examinaion. A sronger design may incorporae wo pairs of examiners, each providing individual raings as well as consensus judgemen. This approach was implemened in Dundee in he firs year of he porfolio examinaion. Each pair of examiners conduced a 20-minue oral review of he porfolio wih he suden. However, i was fel ha some sudens needed more ime wih each pair and, on many occasions, sudens faced he same quesions by he wo pairs albei concerning differen componens of he porfolio. There was also an issue of manpower. Currenly, Dundee employs only one pair of examiners for he 40 minue oral review of he porfolio wih he suden, bu his policy is under review. One proposal is o have wo pairs of examiners wih a suden for 30 minues each. Work needs o be done o furher sudy he validiy and reliabiliy of he consensus approach as complemenary o oher forms of reliabiliy. 6 Forms of validiy Discussions of validiy include he appropriaeness of score inerpreaion and he specific inferences made from he es (Messick 1994). The validiy of he inferences depends also on he reliabiliy of he es. If es scores suffer from low accuracy due o low iner-raer agreemen or poor sampling, inferences canno be made. The auheniciy aspec of he porfolio srenghens he predicive validiy of he porfolio assessmen. If porfolio examinaions are adequae o measure high level, complex abiliies applicable in real life siuaions, hen is predicive validiy should demonsrae srong relaionship wih laer professional or career performance. Such long-erm sudies were no found. Finlay e al (1998) conduced a randomized conrolled sudy and has shown beer resuls for he porfolio (experimenal) group on Oncology facual knowledge. Those submiing porfolios had overall higher marks han hose who have seleced no o submi. This sudy may also indicae differences in moivaion. As for consruc validiy, Korez e al (1993) found moderae correlaions ( ) beween wriing -16 -

17 porfolio scores and oher direc wriing assignmen. Where pass/fail decisions are considered, he dilemma of low correlaions beween differen measures becomes more problemaic. One has o ensure ha sudens will achieve masery on boh he porfolio and oher forms of wriing abiliy o esablish comparabiliy of measures. Also i was found ha differen scoring approaches o he porfolio may produce differen classificaion resuls. (Korez 1993). Currenly, a cenral validiy aspec of he porfolio assessmen is is face validiy. The process, he evidence and he personalised approach are aracive o boh sudens and educaors. However, work is needed o furher esablish is predicive validiy and is relaionship o oher measures of suden performance. 7 Exernal vs inernal examiners I is argued ha porfolio assessmen is more suiable for inernal examiners (Linn 1994) where familiariy wih he educaional programme and wih suden progression ensures consisency of marking. The personalised approach of porfolio assessmen requires inernal knowledge of he educaional milieu. Exernal examiners a he sae or disric level may be recognised more as examiners for qualiy conrol of he porfolio process. This may have implicaions for judging porfolio work a a cenral sie wihou seeing he examinee or lack of familiariy wih he educaional conex. The purpose of such cenral marking of porfolio assessmen should be clarified and he crieria employed for judgemen should incorporae sandardisaion procedures as well as generic crieria suiable o all examinees independen of educaional programme. Exernal examiners who si in a personalised porfolio may differ in heir inerpreaion of examinee performance due o a differen educaional philosophy in heir own schools. Exernal examiners, however, are crucial in he assessmen sysems of UK medical schools. When exernal examiners paricipae in porfolio examinaions, seps should be aken o ensure ha hey do no bias he resuls. Specific raining programmes should be designed for exernal examiners o familiarise hem wih he curriculum, suden assessmen and educaional philosophy of he school. The conexual effec on porfolio performance as well as he personalised approach o sudens are forces which aemp o localize he porfolio raher han use i for exernal esing. The large number of sudies on porfolio implemenaion indicae ha inernal as well as exernal uses are repored. The difference in he applicaion lies in he purpose of he porfolio and he crieria employed for judgemen. Exernal uses may employ general crieria for sudens work which is independen of educaional or work-relaed conex. The exernal marking of suden work may generae group daa on school profiles, sudens level of achievemen and he qualiy conrol of educaional programmes. D: Implemenaion of porfolio assessmen Implemenaion of porfolio assessmens commonly incorporaes a sequence of seps. These seps are idenified in Table 5. For each sep we will bring examples o illusrae some pracical consideraions. 1 Defining he purpose 2 Deermining compeences o be assessed 3 Selecion of porfolio maerial 4 Developing a marking sysem 5 Selecion and raining of examiners 6 Planning he examinaion process 7 Suden orienaion 8 Developing guidelines for decisions 9 Esablishing reliabiliy and validiy evidence 10 Designing evaluaion procedures Table 5: Seps in developing porfolios 1 Defining he purpose Porfolios are used for summaive and formaive decisions, for differen rainees levels and in a variey of seings. The variaion in is conen, process and applicaion requires ha he purpose will be clearly defined. The non-radiional assessmen approaches employed by porfolios may inroduce confusion and uncerainy regarding is purpose. The clearer he purpose, he higher he qualiy of he porfolio assessmen. An example of a purpose definiion The purpose of he porfolio assessmen is o deermine, in line wih he GMC guidelines, ha he graduaing suden is ready o proceed o he Preregisraion House Office (PRHO) year and ha he -17 -

18 suden has achieved he required leaning oucomes. The porfolio is only one componen of he final examinaion. Successful performance on he porfolio is a requiremen for graduaion. The purpose saemen direcs he es designers o consider he compeencies essenial for enering he PRHO year, o deermine which compeencies are already measured by he oher final examinaion componens, and how he porfolio should supplemen he oher assessmen mehods. In his saemen, he porfolio is one hurdle which necessiaes a summaive decision in order o fulfil graduaion requiremens. The purpose defines he assessmen sysem and he relaionship beween he porfolio and he oher assessmen componens. 2 Deermining compeences o be assessed Faculy, medical educaors, assessmen commiee and oher medical school saff will mee o discuss he purpose of porfolio assessmen. They will review he exising assessmen procedure and will idenify gaps in assessmen which should be supplemened by he porfolio. In Dundee, where he porfolio is defined as a requiremen owards graduaion, he firs 11 oucomes are assessed a he knows and knows how level (Miller 1991) by EMIs and CRQs, and a he shows how level by he OSCE. A he level of does, however, oucome 12 is only assessed by he porfolio. Table 6 shows he grid of final examinaion and he associaed mehods for each oucome. The porfolio, hrough sudens cumulaive work, idenifies srenghs and weaknesses in all oucomes wih emphasis on oucomes The porfolio designers may also emphasise he inerrelaions among he oucomes and sudens overall progress owards all oucomes. The process of idenificaion of compeencies o be assessed signifies a sysemic and sysemaic approach o assessmen (Friedman 1999) which ensures a comprehensive sraegy o deermine qualificaion for graduaion. The principle ha only one assessmen mehod is inadequae o deermine finess for pracice guides his design. 3 Selecion of porfolio maerial The idenificaion of compeencies o be measured is a saring poin for selecion of porfolio maerial. I is imporan o idenify key performance behaviours relevan o he compeency measured. The idenified behaviours will guide he porfolio conen selecion. For example, in assessing rainees paien managemen skills, one of he seleced performance behaviours could be paien educaion. The porfolio designers will search for a work sample of paien educaion. Evidence migh include wrien ouline of a paien educaion programme in he communiy, or a video of individual paien educaion session wih paien discussing smoking cessaion. I is imporan o remember ha evidence is seleced from ongoing rainee s work and is no creaed in an arificial way jus for he porfolio examinaion. I is also essenial ha he compilaion of evidence will serve as learning and educaional experience for he rainee. For example, in selecing a video for paien educaion, rainees will employ crieria for selecing he bes paien educaion encouner. In defending he selecion, he rainee demonsraes undersanding of crieria for opimal performance. For oucome 9, appropriae aiudes, ehical and legal undersanding and responsibiliy, he porfolio maerial in Dundee conains he elecive repor which provides evidence if he suden shows ehical undersanding of issues inheren in he elecive. In a case discussion on ehics, he sudens will provide evidence of ehical judgemen and moral reasoning and could be quesioned abou he case. In he Dundee case sudy, where he purpose of assessmen is progress owards he 12 oucomes, he evidence should indicae progress over ime. A selecion of samples of suden work will faciliae discussion of paerns of performance and overall srenghs and weaknesses. 4 Developing a marking sysem In he Vermon applicaion of porfolios, generic crieria were developed specifically for grade 4 in mahemaics. Sudens work is judged by crieria which will specify he level of heir academic achievemens and will deermine heir progress owards sae sandards. This approach is possible when one uni of porfolio conen is evaluaed. However, in he Dundee case sudy, he 12 broadly defined oucomes do no allow he applicaion of specific crieria, bu raher general saemens regarding suden performance relaive o he oucome specificaion. For example, under he oucome Informaion Handling he following dimensions are specified: he docor is compeen in recording, rerieving and analysing informaion using a range of mehods including compuers. Crieria for marking may include evaluaion of -18 -

19 Oucomes EMI 1 CRQ 2 OSCE Porfolio 1 Clinical skills xxx xxx xxx xx 2 Pracical procedures xx xx xxx xx 3 Invesigaing a paien xx xx xxx xx 4 Paien managemen xxx xxx xxx xx 5 Healh promoion and disease prevenion xx xx xxx xx 6 Communicaion skills x x xxx xxx 7 Informaion handling xx xx x xxx 8 Undersanding of basic, clinical and social sciences xxx xxx x xxx 9 Appropriae aiudes ehical and legal responsibiliy xx xx xx xxx 10 Decision making, clinical reasoning xxx xxx xxx xxx 11 Role of he docor x x xx xxx 12 Personal developmen x x xx xxx 1 EMI (Exended Maching Iems), 2 CRQ (Consruced Response Quesions) x considered for use, xx appropriae and is currenly used, xxx mos appropriae assessmen mehod Table 6: Final examinaion grid in Dundee oucome-based curriculum paien records (including specific marks for qualiy of records). Crieria for search sraegies (including marks for formulaing research quesions and generaing key words for muliple daabases) and crieria for analysing informaion (including marks for criique of a research paper). The porfolio maerial should direc he examiner o consider suden progress according o he oucome specificaion and should enable he examiner o idenify srenghs and weaknesses. As menioned earlier, highly specific crieria could be employed if one compeence is assessed. For muliple compeences, general sandards should be developed. I is also possible as in he Dundee case sudy o assign grades o suden global work. However, prior o grades assignmen, examiners mus share undersanding of he specific or general crieria and consisenly negoiae wha consiues an A level grade (excellen) versus an F level (fail) and furher define and refine he poins on he coninuum. Schools may be bound by universiy grading sysems. The bes approach will be o follow he above procedures. Definiion of crieria for performance and idenificaion of level of performance which will correspond o he universiy grading sysem. 5 Selecion and raining of examiners If one considers only an inernal examiners sysem for porfolio assessmen, selecion of examiners will follow he inernal srucure and philosophy of he educaional program. Each porfolio according o is purpose will dicae he appropriae examiners. These will include a wide range of saff; eachers in he basic sciences and laboraory-based disciplines, clinicians, faculy who indicae special ineres in educaion and in suden developmen, faculy who are moivaed o spend he exra ime needed and faculy who are good oral examiners (mainly suden cenred). Anoher selecion issue relaes o he senioriy of examiners. Pairing of new examiners wih more senior colleagues wih experience of porfolio assessmen has been found o be helpful in Dundee. Whaever he selecion process employed, he raining of faculy examiners and mainaining hem in he examiners pool is a key poin for he success of he programme. Naurally he amoun of ime and effor invesed in he examiners pool hrough raining and orienaion and heir accumulaed experience wih porfolio examinaion, should be preserved and reinforced. Consequenly, hey should be reaed accordingly (dinners, lunches) and given help o form a group wih recognised academic respec and appreciaion. The fac ha a faculy member says I can join you for omorrow s meeing I am a porfolio examiner, should bring pride o he speaker and respec from he lisener. Faculy in general appreciae heir paricipaion in porfolio examinaions (Davis 2001). I provides an opporuniy o know he sudens and o share opinions wih heir colleagues. They obain beer perspecives on suden accumulaed work and progress. Alhough he amoun of reading migh be -19 -

20 excessive (a problem if one decided o mainain he broad sampling principle) realisic approaches o reading plans can keep his burden a manageable levels. 6 Planning he examinaion process Forunaely, he use of OSCEs has inroduced he concep of individualised suden assessmen. This concep, ogeher wih he ime needed, faculy manpower, logisics, adminisraion issues and availabiliy of suppor saff are all issues shared wih porfolio examinaions. In Dundee, wih a class of 150, each pair of examiners sees five sudens for a oal examinaion ime of 200 minues (40 minues for each oral review of he porfolio wih he suden). If we include lunch, coffee breaks, discussion ime beween candidaes, filling up forms and he Examiners Commiee meeing a he end of he day five sudens per day is he maximum number. If 10 pairs of examiners are recruied for each day all 150 sudens could si he oral review examinaion in hree days wih differen ses of examiners. The number of examiners needed for such an assessmen operaion is 60 wih each examiner being commied for jus one day. There is, of course, he addiional ime examiners spend reading he porfolios before he meeing wih he sudens, approximaely one hour per porfolio in Dundee. Faculy needs o buy ino he concep of porfolio o be willing o paricipae in such an imporan underaking. As wih all aspecs of reform, srong leadership is needed wihin he medical school o faciliae he launch of he firs adminisraion and keep he momenum in subsequen years in spie of any reservaions among faculy, sudens and adminisraors. when he guidelines are oo broad. However, in he Dundee case sudy, sudens use heir on-going work for selecion of maerial and usually, if hey demonsrae good progress in heir achievemens, hey have confidence in heir abiliy o pass he porfolio. The more informaion given o sudens he more posiive hey become owards he porfolio. Following he porfolio examinaion, sudens are pleased ha senior examiners have given deailed aenion o heir work over heir las wo years of medical school. The sudens, in general, like his personalised, suden cenered, conexualised assessmen approach. 8 Developing guidelines for decisions Once a grading sysem is developed, he porfolio designers mus develop policy and guidelines for sandards of performance. If he porfolio is used for summaive pass/fail decisions, sandards should specify wha consiues a fail or a pass. If porfolios are used for formaive decisions, sandards for srenghs and weaknesses should be defined along wih due process for inervenion and remediaion. The process should also incorporae he procedures for evaluaing he inervenion and he decision mechanisms o furher promoe or o hold back/expel he suden. The decision making process may be oulined by he following flow diagram. Porfolio examinaion Pass Fail Referred Si for anoher exam Undergo remediaion 7 Suden orienaion Sudens mus be informed a he beginning of he undergraduae course abou he porfolio examinaion and kep up o dae wih any changes in subsequen years. Pass Graduae Fail Commiee review Fail Pass Graduae A special orienaion bookle for sudens is issued o inform hem of he purpose and conen of porfolio building, he porfolio assessmen process, he examinaion day, he marking sysem and he use of resuls. The more complex he ask of porfolio building, he more anxious he suden becomes. Problems may occur if he guidelines and crieria for judging performance are no enirely clear as may happen Figure 3: Flow diagram for he decision-making process 9 Esablishing reliabiliy and validiy evidence I is imporan o deermine, prior o he implemenaion of he porfolio, wha will consiue good reliable evidence and plan he examinaion -20 -