Training Experts for Quality Assurance: Lessons Learned

Transcription

1 Training Experts for Quality Assurance: Lessons Learned MarkFrederiks,StephanieZwiessler,MariaE.Weber

2

3 Training Experts for Quality Assurance: Lessons Learned The results of the E-TRAIN Pilot Trainings Copyright* *2012*by*the*European)Consortium)for)Accreditation)in)higher)education* Author(s):*Mark*Frederiks,*Stephanie*Zwiessler,*Maria*E.*Weber* ISBN/EAN:)978H94H90815H07H3) All*rights*reserved.*This*information*may*be*used*freely*and*copied*for*nonHcommercial* purposes,*provided*that*the*source*is*duly*acknowledged** Additional*copies*of*this*publication*are*available*via*

4

5 Table of content 1. Introduction,...,9 2. Training,Practices,of,ECA,agencies,..., Startingpoint:whenissomeoneanexpertandagoodpanelmember Whatcanandshouldbeaccomplishedinstead? The,Pilot,Trainings,..., ThepilottrainingsforEuropeanexperts ThefirstpilottrainingofEuropeanexperts ThesecondpilottrainingofEuropeanexperts ThepilottrainingsforQAstaff ThefirstpilottrainingofQAstaff ThesecondpilottrainingofQAstaff Conclusions,..., ThetrainingofEuropeanexperts ThetrainingofQAstaff(trainthetrainers) Annexes,...,37 Annex1:DetailedoverviewoftrainingcharacteristicsofQAagencies Annex2:Preparatorymaterial Annex3:Programmeofthetraining Annex4:ProgrammeofthefirstpilottrainingofQAstaff Annex5:Recommendations(finaldraft) Annex6:Programmeforthetrainingdays

6

7 European cooperation in higher education calls for internationally experienced and trained experts. * *

8

9 1. Introduction A fundamental part of external quality assurance procedures is peer review. The connection between the quality of the procedure and the quality of the team members is apparent: no matter how competently the quality standards, frameworks and procedures are composed if the experts are not conversant with the applicationofthesestandards,thequalityoftheproceduremightsuffer.thisisonereason whytheeuropean,standards,and,guidelines,for,quality,assurance,(esg)insistthat ( )the experts undertaking the external quality assurance activity have appropriate skills. Additionally the ESG point out that Quality Assurance Agencies should provide appropriate briefingandtrainingforexperts. Within the higher education sector, collaboration across regional and national borders is more and No matter how competently the quality standards, frameworks and more the rule instead of the exception [ especially procedures are composed if the withintheeuropeanhighereducationarea (EHEA), experts are not conversant with the and correspondingly the field of work for experts is application of these standards, the nolongerrestrictedtotheirownnationalsystem(s). quality of the procedure might This fact is also stressed by the ESG s emphasis on suffer. theuseofinternationalexperts andwouldleadto theconclusionthatthetrainingofexpertsshouldnot be restricted to the national level, either. Drawing on international expertise can help to ensure the independence of the panel activity from the influences of national conflicts of interests and foster the orientation towards European and international developments for standardsandprocedures.

10 SupportedbyEU,fundstheEuropeanConsortiumforAccreditationinhighereducation(ECA) launchedinoctober2010the EuropeanTrainingofQAExperts(E[TRAIN) project.eca se[ TRAINprojectintendstostrengthentheEuropeandimensioninqualityassurancebymakingit possible for Quality Assurance Agencies to identify trained and experienced experts from countrieswithintheehea. The overall, aim of E[TRAIN was to facilitate the sharing, of, trained,, knowledgeable,, and, internationally, experienced, experts who will be better equipped to participate in quality assurance (QA) procedures, to the benefit of higher education institution institutions under review.twoobjectiveswerespecifiedinordertoensurethattheoverallaimofe[trainwould bemet: 1. todevelopaeuropeantrainingprogrammeforexpertsinqaproceduresand 2. todevelopadatabasetoshareexpertsamongqualityassuranceagenciesineurope. Thefirstobjectiveisthemostimportantoneforthishandbook:theprovisionoftrainingfor staffmemberofqualityassuranceagenciesandfortrainingofexperts(peers/auditors)who willbepartofpanelsinqaprocedures.accordingtothefirstobjectivetwosortsoftrainings havebeenconductedsincee[trainwaslaunched: 1. trainingforqualityassuranceagencystaff thetrain[thetrainer[session;and 2. pilottrainingsessionsforexperts Thefocuswithinthepilottrainingsessionswasnotuponspecificapproachesforthetraining of panel chairs and panel secretaries. Both types of trainings delivered were open for participation for staff of ECA member agencies but as well for experts nominated by the agencies.participatingagenciesandexpertsinvolvedinthepilotsreceivedpracticalhands[on approachesforthetrainingofexperts. 10

11 The first pilot session for staff members of quality assurance agencies was held on 3rd November2011inViennaandthesecondwasheldon10thMay2012inMadrid.Thefirstpilot sessionforexpertswasheldon6thoctober2011inthehagueandthesecondoneon23and 24April2012inVienna.Inadvancetothesesessionstheagencieswereaskedtonominate expertstoparticipateinthesepilotsessions. Thispublicationincludesasummaryoftheexperiencesandachievementsacquiredthrough thevariouspilotsessionsduringthee[trainproject. Chapter 2 provides a general overview on the actual status quo of training practices within agencies.chapter3summarizesthetrainingsactuallydeliveredwithinthee[trainprojects. Since the handbook first and foremost deals with the aspect of how to train experts undertakingexternalqualityprocedures,asecondpublication[thehandbookfortrainingof Experts [ provides guidance for preparing trainings for expert panel members. With this Handbook the author Gillian King, who was also a trainer in the pilot sessions for quality assurance agency staff members, provides a valuable guidance for preparing trainings for expertpanelmembers. 11

12 2. Training Practices of ECA agencies T hefollowinginformationisderivedfromasurveyamongstqaagencieswhichwas conducted in order to explore which training practices exist among the quality assurance agencies and to identify what the agencies consider to be good training practices. It served as an important input and inspiration for the design of the European trainingcourse.thedatawerecomplementedduringthefirstpilotstafftraininginnovember 2011wherestafffromtendifferentEuropeanagencieselaboratedontheircurrentpractices. Thesurveyhasrevealedcommonelementsoftheagencies trainingpractices. Almostalltheagenciescarryoutsomekindoftrainingexperts.Thefocusareasofthetrainings arecomparableandinvolvethequalityassuranceprocess,nationalhighereducationsystem andpracticalskills.theformatvaries,though. Some agencies use online trainings, some arrange seminars, some have trainings which lead to The national context is relevant for certification, and some combine the training with a meetingrightbeforethesitevisit. the needs and means of quality assurance and influence the outlook of trainings delivered; Whenlookingatwhatisconsideredtobegoodpractice, quiteafewagenciesfinditappropriatetospecialisethe training according to the background or role of the experts, e.g. distinguishing between newcomers and nevertheless the main focus of the trainings is laid on QA standards & criteria, the higher education system, practical skills. experienced experts, or training secretaries and chairs separately, or having separate student trainings. Furthermore, the practical elements (interactiveelementssuchascasestudies,role[playsetc.)areoftenhighlightedandthisisvery 12

13 muchappreciatedbytheparticipatingexperts.thisisonereasonwhytheeuropeanexperts trainingattachessomeimportancetointegratingpracticalelementsintothetraining. Considering these findings of the survey amongst agencies and taking into account the Europeandimension,theEuropeantrainingprogrammeofexpertshasbeendeveloped. ThetableinANNEX1givesamoredetailedoverviewoftrainingcharacteristicsofQAagencies Starting point: when is someone an expert and a good panel member Beforestartingtocreateatrainingprogrammeforinternationalexpertstwoverybasic questionshadtobeanswered: 1. Whatconstitutesa good expert?andcantheskills,qualitiesandcompetenceswhich mightbeneededbysuchanexpertbeacquiredthroughtrainingatall? 2. Whatisagoodpanelmember?Ofcoursethereisnoclearoreasyanswertothis question:the quality ofanexpertdependsonavarietyofaspects. Inthefollowingitisattemptedtodepictashortsetofcharacteristicsofexternalexperts.Due totheexperienceinthepilotsessionswithinthee[trainproject,thefollowingoutlinesare solelyfocussingonsocalled academicexperts.neverthelessvariousagencies proceduresdo notonlyinclude academicexperts butalsoexpertsfromotherrelevantstakeholdergroups (e.g. students, experts with relevant professional background). Several procedures have to takeintoaccountthedifferenttargetgroupsandbasedonthattheneedsofinformationand skillsofpanelmembers. Hence, if the expert is an academic, a high level of expert knowledge in their academic disciplineorfieldshouldbeagiven she/heshouldbeup[to[dateondevelopmentswithinher orhissubjectarea,she/heshouldhaveabroadscopeofexperienceinresearchandteaching, she/he should have experience with the development and implementation of study programmes.wheretheexpertisaseniorheadministratoritisexpectedthatheorshehas been responsible at a senior level in the institution for educational policy and processes. 13

14 She/heshouldhavebeeninvolvedataseniorlevelintheirinstitutionwithqualityassurance processesandassessmentprocesses. Student experts will be expected to be familiar with educational and/or quality assurance processes,andprobablytohave(orhavehad)somerepresentativeroleintheirinstitution. Qualityassuranceagenciescancheckthesebasicrequirementsforworkinginanexpertpanel rather easily by scrutinising documents such as the curriculum vitae, list of publications, personalstatements,oftheexpertcandidateinquestion. There are practical skills which competent experts possess. These comprise for example communication techniques, interview techniques, knowledge about how to write reports evidence based, knowledge about relevant national frameworks, awareness for the specific roleinthegroup,awarenessaboutculturaldifferences,theabilitytobegoodteamworkersor the ability to accurately analyse strengths and weaknesses of the study programme/ institutionunderreview. Whatexpertsshouldpossess Awarenessofculturaldifferences Abilitytoworkinteamsandgroups Subjectspecificknowledge Analyticalskillsandcompetences Communicationalskills&interviewtechniques KnowledgeofQAstandardsandprocedures&nationalframeworks 14

15 Thatiswhytrainingsessionsnormallyincludeapracticalcomponent: 1. Testingandenhancingtheseskillse.g.byusing(partsof)thehighereducation institutions self[evaluationreportstobeanalysedincasestudies. 2. Role[playsinwhichinterview[situationswithrepresentativesoftheinstitutionare simulatedandthuspossible(problematic)situationscanbeanticipated.role[playsare alsoagoodtrainingmethodtofocustheteamaspects(variousroles,diversity aspects). Obviouslyoneofthemainaimsof(national)trainingcoursesforexpertsistoguaranteethat theexpertsarewellacquaintedwiththestandardsandguidelinestobeappliedinthereview process.theyshouldknowthelegalframeworkoftheprocedure,theirroleinthereviewand thelimitsoftheirdecision[makingpowers.thisiswhyatrainingprogrammeshouldensure thatexpertsarefamiliarwiththeframeworkconditionsofthereviewprocess. Developingapan[Europeantrainingcourse,afirstimportantrealisationregardingthetarget group and the aims of the pilot training session was that it could not possibly cover all European external quality assurance systems with their varying standards, criteria and procedures. 1. Firstly,thiswouldbeunfeasibleregardingthesheerextent, 2. Secondlynoteverypotentialexpertisinterestedineachnationalsystem, 3. Thirdlyitisthecoretaskofeachregional/nationalQAagencytomaketheirexperts acquaintedwiththerespectivestandards. AEuropeantrainingprogrammecannotandshouldnottrytoaccomplishthis. 15

16 2.2. What can and should be accomplished instead? Asnotedabove,academicsubjectknowledgeisanimportantprerequisiteforexpertsbutis notteachableinthecontextofatrainingprogramme.alwaysbearinginmindthatthetarget groupareexpertswhoalreadyhavesomeexperienceinexternalqualityassuranceprocedures at the national and international level, this would leave practical and communication skills. Sincethetrainingcourseis embeddedintoaeuropeandimension,thecommunicationalskills mayhavetobecomplementedbyinputoninterculturalawarenessandcommunication.the practicalskillsmayincludereflectionontheroleoftheexpertwithintheprocedure,andwork onacasestudy. WhataEuropeanTrainingProgrammeshouldcomprise Practical/CommunicationalSkills Analyticalskillsandcompetences KnowledgeaboutexternalQAapproachesinEurope InterculturalAwareness Theknowledgedimension,aspreviouslydiscussed,cannotcompriseextensiveinformationon all European systems but could focus instead on a brief but representative overview of external QA approaches, of different philosophies across Europe (for more details see chapter3). 16

17

18 3. The Pilot Trainings T he focus of this chapter is to summarise the experiences from the trainings deliveredforeuropeanexpertsandforstaffofqaagencies.first,theexperiences withthetwopilottrainingsforeuropeanexpertsaredescribed,followedbythe two pilot trainings for QA staff. For each of the pilot trainings a brief description of the targetgroup(participants),objectives,learningoutcomes,contentanddidacticsisgiven. Thisisfollowedbythemainresultsoftheevaluationofeachofthepilottrainings The pilot trainings for European experts WhoaretheEuropeanexpertsthatshouldbetrainedbyE[TRAIN?TheE[TRAINproposal indicates that experts who are part of assessment panels in national external QA procedurescouldtakepartinthepilottrainings. In its meeting on 5th May 2011 the E[TRAIN Stakeholder Group (representing the QA agencies) specified that as institutional reviews were emerging in several countries it would be most useful if the trainings focused on experts who could participate in institutionalreviews. The experts trained by E[TRAIN should bring internationalexperiencetonationalpanelsandalsohelp to legitimize decisions made when institutional reviews are carried out. They should also be able to indicate ways in which the institutions can improve their practices. The experts trained by E[TRAIN should bring international experience to national panels andalsohelptolegitimizedecisionsmadewheninstitutionalreviewsarecarriedout.

19 They should also be able to indicate ways in which the institutions can improve their practices.itwasarguedthatinorderfortheexpertstoreachthishighlevel,theyneedto havesolidexperiencefromqaalreadybeforeenteringthee[trainpilottraining.hence, thefocusofthetrainingshouldbeonahigherlevelthanbasicelementsofqa.however, thee[trainfocusgroup(representingexperiencedexperts)hadwarnedagainsta new class of elitist QA professionals. It would threaten the nature of peer review and ultimatelythelegitimacyofthereviewprocessitselfifasmallnumberoftrainedexperts travelledaroundeuropetotaketheirseatsinreviewpanelsasifitwereaprofession.to avoidthisthreat,itwassuggestedthatthetrainingsshouldalsobeopentoexpertswith little or no experience in QA. A mediating solution was found in the agreement by the StakeholderGroupthatlessexperiencedexpertsmightjointhetrainingsafterundergoing onlinee[traintrainingmodulesleadingtotheentrylevel.moreover,itwasdecidedthat theexpertsforthepilottrainingsshouldbeproposedbythenationalagencies(i.e.theeca members).nominatedexpertsshouldhavesomeexperiencewiththenationalprocedures in the country concerned. It was agreed that it would be possible to nominate student experts.thetrainershouldreceiveinformationfromtheparticipantsontheirbackground andexpectationsinordertocustomizethetrainingaccordingtoneeds.itwasagreedthat ProfessorNickHarris(independentconsultantandformerQAADirectorofDevelopment andenhancementgroup)wouldactastrainerforthefirstpilot.withregardtothecontent ofthetrainingthestakeholdergroupendorsedthedistinctionmadebythefocusgroup (and included in the Guidelines for Training of Experts ) between the awareness dimension, the cognitive dimension and the interactive dimension. With a view to the training of experts for institutional reviews the Stakeholder Group emphasized the following: 1. Theawarenessdimension;thereshouldbeadditionalfocusonintercultural attitudes(thiscanincluderesearchpapersoninternationalcommunication). 2. Thecognitivedimension;hereadditionalfocusshouldbeontheEuropean dimensionandonvariationsininstitutionalqualitymanagement/autonomyin highereducationinstitutionsindifferentcountries.aftercompletingthetraining theexpertsshouldbeuptodateoninternationalstandards. 19

20 3. Theinteractivedimension;theexpertsshoulde.g.learnaboutintercultural communication,effectiveinterviewingskills,andother softskills. Itwasagainstthisbackgroundthatthepreparationsforthefirstpilottrainingofexperts started The first pilot training of European experts Forbothbudgetaryreasonsandbusyagendasofexpertsitwasdecidedtolimitthefirst pilottrainingto1daywithadinneronthepreviousevening.thetrainerandtheproject partners responsible for the training (ÖAR 1 and NVAO) met once to prepare the pilot training.theenvisagedcontentforthetrainingwasderivedfromtheguidelinesforthe TrainingofExperts: 1. Whatshouldexpertsbeawareof? - Differentorganisationalcultures,differentqualitycultures - Beingbiased(priorperceptions/prejudices) - Interculturalawareness/Interculturalcommunication 2. Whatshouldexpertsknow? - KnowledgeoftheEuropeandimensionofQA - KnowledgeofthenationaldimensionofQA - Knowledgeofculturaldifferences 3. Whatshouldexpertsbeabletodo? - Audittechniques/ tools&tricks,e.g.howtoaskgoodquestions - Teamwork:howtoeffectivelyworktogetherasareviewteam - Howtowritea(joint)reportthatincludestheopinionsofall - Do sanddont s atthesitevisit - Howtoactininterculturalsituations 1 WiththelegalvalidityoftheHS[QSGanewandsingleagency(AgencyforQualityAssuranceandAccreditationAustria)was establishedwhichisresponsibleforexternalqualityassuranceinthehesector(publicuniversities,universitiesof AppliedSciences,PrivateUniversities).The Qualitätssicherungsrahmengesetz (QSRG)enteredintoforceon1stMarch 2012.TheÖARpersisteduntil31stAugust

21 Itwasquicklyagreedthatthisagendawastoomuchforaonedaytraining.Instead,the trainer had made a proposal which included several dimensions of QA training. A distinctionwasmadebetweenknowledgeandunderstandingwhichcouldbeprovidedby onlinetraining,andskills(includingrolesandpractice)whichwouldbethemainfocusof thetrainingday.eachofthesedimensionsconsistedofabreakdowninto4categorieson an introductory level and 4 subsequent levels of achievement. This resulted in a matrix whichwouldhavetobefurtherdeveloped.learningoutcomesshouldbeincludedforeach ofthedimensions,categoriesandlevels.itturnedoutthatthiswasatoocomplicatedtask to achieve in time for the start of the pilot training. Instead the trainer would ask participantsatthestartofthetrainingwhattheyexpectedaslearningoutcomesofthe training. The first question in the evaluation form was then whether these learning outcomeshadbeenachievedintheviewoftheparticipants.theotherquestionsinthe evaluation form dealt with opinions on the strong and weak points of the training. In addition, the participants were specifically asked to rate the following aspects from 1 (excellent)to5(poor): - Clarityofsessiongoals - Trainer spresentationskills - Relevanceofinformation - Clarityofinformation - Scope(coverage)oftraining - Relevanceofmaterialspresentedduringtraining - Organisationoftraining - Overallratingoftraining. Theknowledgebasewithonlinematerialsforexpertswasnotavailablebythetimeofthe first training (this was meant to be an outcome by the end of the E[TRAIN project). Therefore,itwasdecidedtostartthetrainingdaywithadiscussionontwoassignments whichtheexpertswereaskedtocomplete.oneassignmentonstakeholdersperspectives wastobeaccomplishedbytheexpertspriortothemeeting,andanotherassignmenton placingtheexpert sownnationalqasysteminamatrixofsystemswouldbeincludedasa 21

22 plenaryexerciseduringthetrainingday.theattentionintheprojectpartners preparatory meetingthenturnedtotheskillsdimensionandtheroleplayingexerciseswhichwouldbe themainpartofthetraining.firstthetrainingprogrammewiththedifferentsessionswas designedandthen,afterthepreparatorymeeting,thetrainingmaterialswerefinalisedby thetrainerandsenttotheparticipants.theroleplaywouldbeintroducedtoparticipants during the dinner on the night before the training. This would give some time for participantstopreparetheroleswhichwerewrittendownforthem.unfortunatelynotall participantswerepresentatthedinnerandsothepreparationdidnottakeplacethen,but hadtobedelayeduntilthenextdayofthetraining. Eight experts, including one student expert, were proposed by the agencies and participatedinthetraining.allexpertshadexperiencewithnationalqaproceduresbut notallofthemwithinstitutionalreviews.itturnedoutthatamongecamembersthere was less experience with institutional reviews in practice than envisaged by the Stakeholder Group. In addition to these eight experts to be trained, four experienced expertsfromthefocusgroup(includingonestudentmember)participatedinthetraining. Theseexpertsparticipatedfullyinthetrainingbutalsohadaroleasobservers.TheFocus Groupmetimmediatelyafterthetrainingtodiscusstheirobservations.TheStakeholder Grouphaddecidedthateachagencycouldsendanobservertothepilottraining.Thisled toanadditionalgroupof7observersfromagencies.thegroupoforganisersconsistedof thetrainerand4membersofstaffoföarandnvaowhoalsohadaroleinthetraining (particularlyintheroleplay). The training was held on 6th October 2011 at the NVAO offices in The Hague (with a precedingdinneronoctober5th).abackgroundpaperintroducingtheprogrammeforthe trainingdayandthepreparatorymaterialwassenttoparticipantsinadvance:annex2. The pilot training was evaluated by the Focus Group, the Stakeholder Group and by an observationreportoftheresponsibleprojectpartneröar.themainconclusionsfromthis evaluationwere: 1. Theoverallratingofthefirstpilottrainingwas3onascalefrom1(excellent)to5 (poor).theratingsofthedifferentaspectsandcommentsmadebyparticipants madeitclearthatsignificantimprovementswereneededforthesecondpilot training. 22

23 2. Thetrainingfocusedmainlyontheinteractivedimension(roleplay),lesson knowledge,andverylittleonawareness.itwasconcludedthatthesecondtraining shouldhaveastrongerfocusonthelatterdimensions,particularlyonintercultural awareness. 3. Morestructureforthetrainingwasneeded:explicitlearningoutcomesforeachof the3dimensionsshouldbedeveloped. 4. Thereshouldbemoreclarityandcoherencewithregardtodidacticsandmaterials used. 5. Theroleplayneededarealcase[study. 6. Aclearerprofileofthetrainedexpertsisneeded(e.g.programmeorinstitutional reviews). 7. Thereweretoomanyobservers;therewereasmanyagencystaffpresentasthere wereexperts,andwithinthewholegroupthe8expertstobetrainedformedonly athirdofthegroup. 8. Onetrainingdayistooshort. Thefirstpilottrainingofexpertsfocusedmainlyonthe interactivedimension(roleplay),lessonknowledge,and verylittleonawareness.itwasconcludedthatthe secondtrainingshouldhaveastrongerfocusonthe latterdimensions,particularlyonintercultural awareness. * KeepinginmindthecriticalresultsofthisevaluationtheE[TRAINSteeringGroup,ÖARand NVAOinparticular,tookresponsibilityfordesigninganewstructureforthesecondpilot training. 23

24 The second pilot training of European experts It was decided that the training should be prolonged to 1 ½ days to enable a better coverageofthe3dimensions.thetargetgroupofexpertstobetrainedinthesecondpilot training were described as experts who can participate in institutional external quality assurance procedures (audits/accreditations/evaluations) in different national systems. The training was structured along the lines of the three previously agreed dimensions whichwerespecifiedasthefollowingaimsforthetraining: To increase experts knowledge of the European dimension of external quality assurance and of the characteristics of different European external quality assurancesystems; To raise intercultural awareness when working as an expert in distinct (sub)nationalexternalqualityassurancesystems; Toenhancetheauditskillsneededtofunctioneffectivelyasanexpertinvarious (sub)nationalexternalqualityassurancesystems. The training was divided into three modules: European external QA approaches; Intercultural awareness; and, Audit skills for diverse national settings. Each of these modulescorrespondedwithoneormoreoftheoverallaimsofthetraining.foreachofthe modulesobjectivesandintendedlearningoutcomesweredrawnup.theseobjectivesand learningoutcomeswerebasedontheoverallaims.basedonthelearningoutcomesthe didacticalapproachandthedesiredprofileofthetrainerweredescribed.therefore,the description of each of the modules followed the same format: Objectives Learning Outcomes Didactical Approach Trainer s profile. We will limit ourselves here to an overviewofthelearningoutcomes: 24

25 Itwasdecidedthatthetrainingshouldbeprolongedto1 ½daystoenableabettercoverageofthe3dimensions. Thetargetgroupofexpertstobetrainedinthesecond pilottrainingweredescribedas expertswhocan participateininstitutionalexternalqualityassurance procedures(audits/accreditations/evaluations)in differentnationalsystems. A., European,external,QA,approaches, 1. discussthepositionofexternalqualityassuranceintheeuropeanhighereducation Area; 2. describehowhighereducationsystemsandtheirspecificpolicychoicesimpactthe externalqualityassurancemethodologies; 3. classify(sub)nationalmodelsofexternalqualityassurance; B., Intercultural,awareness, 4. position one s own national culture in a generic frame of reference for cultural differencesandcharacteristics; 5. frame questions related to the quality of higher education in the perspective of culturaldifferencesandcharacteristics; 6. perceiveculturedependentframesofreferenceinordertoavoidbeingmisleadby them; C., Audit,skills,for,diverse,national,settings, 7. communicate effectively and appropriately with all actors in diverse external qualityassurancesettings; 8. employ advanced communication techniques for successful and relevant informationgatheringinexternalqualityassuranceprocedures; 25

26 9. assume suitable behaviour for particular expert roles in diverse intercultural settingsofexternalqualityassuranceprocedures; 10. cooperate effectively and appropriately in decision[makingprocessesofanaudit team. Thetrainershadtomeetthetrainer sprofileandwerecarefullyselected.apreparatory meetingofthe3trainersandtheresponsiblee[trainprojectpartners(öarandnvao) wasorganised.atthismeetingthecontentoftheprogrammewasagreed,thedidactical approachwasdiscussedandtheassignmentsandmaterialswereagreed.thepre[training assignmentsforparticipantsconsistedof:readingaselectedcase[studywhichwasusedfor theroleplay;atextoneuropeanqa;shortprofilesof3nationalagencies;and,amatrixon which experts were asked to position their own national QA system. This matrix would thenbediscussedduringthetraining. The programme followed the sequence of the modules but not exactly; to increase the span of attention during the day it was decided to break the intercultural awareness module in two sessions. Intercultural awareness part 1 was scheduled before lunch and Part2beforetheendoftheday.FortheprogrammeofthetrainingseeANNEX3. Thetrainingwasheldon23and24April2012inVienna.Tenexpertsparticipatedinthe training.asinthefirstpilottrainingtheywerenominatedbytheecamemberagencies.in addition, three experienced experts from the Focus Group participated in the training. Theyalsoevaluatedthetrainingimmediatelyaftertheendofthetraining.Theorganising teamconsistedofthethreetrainers,twostaffmembersoftheaustrianagency(ofwhich oneactedasobserver) 2,andtwostaffmembersofNVAO.Asaconsequencetherewasa mixedgroupofpeoplefromaustria,belgium,germany,thenetherlands,norway,poland, andspain. Aswiththefirstpilottrainingthesecondonewasthoroughlyevaluated.Duringthefinal session participants were asked to fill in an evaluation form and these were collected before going to lunch. As a consequence there was a very high response rate. The achievementofthelearningoutcomeswereratedandanumberofotherquestionswere 2 AQAustria 26

27 asked.afterlunchthefocusgroupsharedtheirobservations,andnotburgadammfrom AQAustria,wroteanobservationreport.Themainresultsoftheseevaluationswere: Thetrainingwasclearlyappreciatedbytheparticipants;theoverallratingwas7.8 onascalefrom1(poor)to10(excellent). Mostparticipantsfeltthat1½daysoftrainingwassufficientandfeasible. Agroupsizeof10expertsworkedwell,especiallyinthegroupwork(2groupsof5 experts).whensplittingin2groupsforaroleplayordiscussion(e.g.whenmaking judgements) there are always 2 trainers needed to give maximum feedback to participants. The structured approach taken worked well; the content structure (e.g. the sequenceofthesessions)couldevenbefurtherrefinedandimproved. The case[study and role play generated a lot of enthusiasm and there was agreementthatthesewerethemostbeneficialpartsofthetraining. Interculturalawarenessshouldbeintegratedintheroleplay/case[study. More attention is needed for: the classification of national QA systems; the role andattitudeoftheauditor;panelchairskills. The structured approach taken in the second pilot trainingof expertsworked well.the case[studyandrole play generated a lot of enthusiasm and there was agreementthatthesewerethemostbeneficialpartsof thetraining. 27

28 3.2. The pilot trainings for QA staff OriginallyintheE[TRAINprojectproposalthestafftrainingsweremeanttotrainQAstaff memberstobecometrainersof European experts after which they could go on to train European experts themselves. However, it turned out that there was no clear profile of theseeuropeanexpertswhichwassharedbytheagenciesatthestartoftheproject.asa consequence, a lot of time and energy went into discussions among the agencies (Stakeholder Group and Steering Group) and experts (Focus Group) on reaching an agreementaboutwhoshouldbetrainedandwhatthiswouldmeanforsettingupthese trainings.moreover,itgraduallybecameapparentthatagenciespreferredsocalled train thetrainers sessionswhichwouldhelpstaffmemberstobecomebetternationaltrainers. Thesequenceoftrainingswasthereforereversed:firstthepilottrainingofexpertstook place,followedbythetrainingofstaff The first pilot training of QA staff TheStakeholderGroupagreedthatthefirstpilottrainingofQAstafftrainingshouldtake placeafterthefirstpilottrainingofexperts.thiswouldallowfortheinvolvedpartiesto reflectupontheexpertstrainingbeforecarryingoutthestafftraining.thestafftraining wouldfunctionasa trainthetrainers meeting.theoriginalideawasthatobserversfrom theagencieswhowerepresentatthefirstpilottrainingofexpertscouldparticipateinthe staff trainings so that they could act as future trainers of European experts. However, whenthesestaffmemberswereaskedwhattheirpreferenceforthefirstpilottrainingwas itturnedoutthatthevastmajoritywantedatrainingwhichwouldhelpthemimproveas national trainers. Therefore, it was decided that the first pilot training should have the formatofaworkshopoftrainerswheretheycouldsharehowtheytrainexpertsintheir agencies. Mutual learning and sharing good practices were important goals for this workshop.moreover,thereshouldbeoverarchingoutcomesforthegroupofagenciesin the form of an overview of training practices in agencies, drafting recommendations concerningnationaltrainings,andinitiatingthedevelopmentofaskillsprofileoftrainers. Theworkshoptookplaceon3rdNovember2011inVienna.Fortheprogrammeofthefirst pilottrainingofqastaffseeannex4. 28

29 The first pilot training of staff members from QA agencies should have the format of a workshop of trainers where they could share how they train experts in their agencies. Mutual learning and sharing good practiceswereimportantgoalsforthisworkshop. There were in total 15 participants in this workshop representing (prospective) trainers fromaustria,denmark,germany,thenetherlandsandflanders,spain,switzerland,and the UK. The survey of good practices in experts trainings had shown that the Quality AssuranceAgencyUKhadalonglastingexperiencewithanextensivetrainingprogramme, particularlyinthefieldthatwasmostlynewbutofincreasingrelevanceforecamembers, namely the training of institutional reviewers. The first part of the programme was therefore reserved for a presentation by Dr Gillian King (consultant and former Deputy Director of the Reviews Group at QAA) of the QAA training programme. Then the participantspresentedthetrainingpracticesintheirownagenciesinwhichtheyfocused primarilyonthefollowingaspects: Descriptionofexpertsprofile(whichexpertsaretrained) Contentandprogramme Preparationandmaterials Conductionofthetraining Learningoutcomes Didacticapproachandmethods Skillsoftrainers 29

30 ForanoverviewofthetrainingprogrammecharacteristicsoftheseagenciesseeANNEX1. Althoughtherearesomesignificantdifferencesinhowthesetrainingsareconductedthere arealsocommonalities.itbecameapparentthatmostagencieshave: Thesametrainingorbriefingforallexperts; Afocusonthereviewframework; Legalaspectsaspreparatorymaterials; Onedayoftrainingorless; Atendencytofocusdidacticsonpresentationsanddiscussions,althoughtheuseof casestudiesisincreasing; Trainersfromstafforthemanagementoftheagency. Theparticipantsalsodiscussedtendraftrecommendationstoagenciesforthetrainingof experts.afinaldraftoftheserecommendationsisincludedasannex5. The workshop was evaluated which resulted in an overall rating of 2 on a scale from 1 (excellent) to 5 (poor). During the concluding session the participants expressed their preferencesforthesecondpilottrainingtoincludeguidanceonthestructuralaspectsof setting up a training and for opportunities to advance their communication skills as a trainer The second pilot training of QA staff Allstaffmemberswhoareresponsibleforthetrainingofthepanelmemberswithintheir agency have an extensive knowledge of the subject, i.e. external quality assurance, and they are experienced with the practicalities of external quality assurance procedures as well.atthesametimethemajorityhavenotundergoneanypreparationforthespecific taskofconvertingthatknowledgeintoatrainingcourse.theyhadnostructuredtrainingin how to train panel members but follow more or less a Do[it[yourself instead of a didactic[methodicalapproach.thisiswhyitwasdecidedthatthesecondpilottrainingfor QAstaffshouldbedesignedasa Train[the[trainers workshop,conductedbyprofessional trainers.thestaffmemberswouldbegiventoolswhichtheycanusewhendesigningtheir 30

31 trainingcoursebackhome.on10and11may2012thetrain[the[trainersworkshopwas heldinmadrid.theworkshopwasdividedintwoparts;onthefirstdaygilliankingwasthe trainerforthestructuralpartofthetrainingandontheseconddaythepolishconsultants Przemyslaw Kotecki and Anna Giniewicz were responsible for the communication skills training.thelearningoutcomesandthecontentofthetwotrainingdaysweredeveloped bythetrainers. TheprogrammeforthetwotrainingdaysisincludedasANNEX6.Thecontentofthefirst trainingdayismainlyincludedin the publication Handbookforthetrainingofexperts. Fifteenlearningoutcomesweredevelopedforthetwotrainingdays. Table 1. Learning Outcomes for the two days training of staff members Day,1, 1. beawareoftheirownskillsasatrainerandhaveidentifiedareasfordevelopment highereducationarea 2. beawareoftheneedtoknowtheirparticipantsandthematerialtobe communicated 3. beabletowritesessionaimsandoutcomes 4. beabletochoosedeliverymethodswhichsuittheparticipantsandthematerial 5. beabletodeveloptrainingmaterials(notes,visuals)whichdelivertheoutcomes 6. considerassessmenttools,ifrequired 7. understandtheimportanceofevaluatingtheprogramme. Day,2, 8. beabletoidentifythebarriersincommunication 9. beabletobreakdownthecommunicationbarriers 10. beawareofdifferentaspectsofcommunication 11. beabletoworkwiththegroupdynamics 12. beabletoworkwith difficult participants 13. understandtheroleofauto[presentationindeliveringthetraining 14. beabletouseauto[presentationtechniquesfordeliveringofthetraining 15. beawareofmethodsandstrategiesforstressmanagement Therewere10participantsfromthesamecountriesasinthefirstpilottraining.Aneeds questionnaire was sent to participants asking them for their experience as a trainer, whethertheyhadbeentrainedasatrainerbefore,theirstrengthsasatrainer,andareas fordevelopment. 31

32 The training was evaluated by participants filling in an evaluation and by one of the participantswhohadactedasobserver.theevaluationledtothefollowingresults: Theoverallratingonascalefrom1(poor)to6(excellent)was:5.2forday1and 4.8forday2. Participantsconsideredday1tobeparticularlyusefulandwell[structured. Theagendasonbothdayswereabitoverloaded;thetimepressurewas noticeable. Apossiblesolutioncouldbethatthefirstdaywouldbestretchedinto2daysand someoftheobjectivesoftheseconddayintegratedinthefirstday. The second pilot training of staff members from QA agencieswasatrainthetrainersworkshop.itwasvalued highlybytheparticipants. 32

33 4. Conclusions T he experiences with the pilot trainings and the discussions during the E[TRAIN disseminationconferenceheldinmadridon14and15june2012gaveveryuseful feedbackonapossiblefollow[uptotheproject.thefollowingobservationsmay guidetheproposalsforfuturetrainingsofexpertsandstafforganizedbyeca: 4.1. The training of European experts Experttrainingsshouldbe tailor[made dependingonthetargetgroup,e.g.: o Expertsforinstitutionalreviews o Expertsforassessmentsofjointprogrammes(singleaccreditation procedures) o ExpertsfortheassessmentofHEIsorprogrammesrequestingtheECA certificateforinternationalisation(cequintproject). Acommunityoftrainedexpertsshouldbeestablishedbycarefullyconsidering: o Recruitment:topreventanelitecorpsof(retired)expertstrainingsshould notsolelyrelyonnominationsofexperiencedexpertsbutbeopento newcomerswithlessexperienceinexternalqa.recruitsshouldsharethe agencies values,ethos,andideasofprofessionalism. o InclusionintheEuropeandatabase,preferredparticipationinpanels,and maintenanceofcommunication(e.g.newsletter,providingopportunities forupdatingskills,expressionofappreciation) o Performanceofexpertsshouldbereviewed o Studentexpertsshouldbeincludedintrainingsandinthecommunityof expertsbuttheymayrequiresomespecialisedcontent. Structureandcontentofthetraining: o o Thestructuredapproachtakeninthesecondpilottrainingshouldbe maintainedandrefined Thecontentofthetrainingshouldbeadaptedaccordingtotheprofileand experienceofexperts 33

34 o o o o KnowledgeofEuropeanexternalQAapproachesshouldbeenhancedby expandingtheecapediaandmakingusefulreferencestomaterialsinthe ECApediathroughoutthetrainingandinadesignatedassignment specifically Interculturalawarenessshouldbeintegratedinthediscussionon EuropeanQAsystemsandinthetrainingofauditskills(includingacase[ studyandroleplay) Improvethelearningeffectoftheroleplaybyprovidingcountry[specific informationonlocalhighereducation Moreattentionneededfor:theroleandattitudeoftheauditor;panel chairskills;enhancement[ledvs.assessment[focusedreviews;creatingan atmosphereoftrustandopenness. Organisationandfinances: o o o o Trainingsshouldbeorganizedonaneasilyaccessiblelocation,ifpossiblein cooperationwithothernetworkstoincreaseeuropeansupportandthe fundingbase Financialsustainabilitycanbeenhancedbysharingcostsbetweeninvolved networks,agencies,andexperts Organizea1½daytrainingforapproximately10expertswhocanbesplit in2groupsguidedby2trainers Trainingsshouldbeorganisedatleasttwiceayearandmoreoftenwhen theneedoccurs. Expert trainings should be tailor[made depending on the target group and a community of trained experts shouldbeestablished. 34

35 4.2. The training of QA staff (train the trainers) Trainthetrainersworkshopsof2daysshouldbeorganisedannuallyfortrainersof nationalagencies Thetrainthetrainersworkshopsmayadapttheobjectivesandlearningoutcomes accordingtotheneedsoftheparticipants Futuretrainthetrainersworkshopmayfocusonthedevelopmentofacommunity ofexpertsandonincreasingtheeffectivenessofrecruitmentandperformanceof experts Acommunityoftrainersshouldbeestablishedbyaregularexchangeoftrainers amongnationalagenciesandmaintenanceofcommunication(e.g.newsletter, providingopportunitiesforupdatingskills,etc.) Thecostsofthetrainingshouldbesharedbytheagenciesinvolved. Train the trainers workshops of 2 days should be organisedannuallyfortrainersofnationalagencies. 35

36 36

37 5. Annexes 37

38 Annex 1: Detailed overview of training characteristics of QA agencies

39

40 Quality Assurance Agency ACE Denmark Training s target audience Same training for representatives of all stakeholder groups ANECA Same training for representatives of all stakeholder groups + international experts Selection criteria for choosing experts; gender and territorial balance assured AQU Catalunya Same training for representatives of all stakeholder groups; Additionally ad hoc courses for students Selection criteria for choosing experts Training content Preparatory material Assessment framework; accred. criteria; roles and tasks of experts Evaluation context +process; tools; accred. criteria and their interpretation; ANECAs role, practical issues No excessive preparation (legal framework, guides, example of a report) Standardized material (legal aspects, guidelines and standards, evaluation protocols, case study for small group works) Evaluation process, context, accred. criteria, IT tools Management of experts is part of internal QA system Special content for student courses: purpose AQU, students role in committees, etc Training process Learning outcomes Didactics Trainers skills 1 day training; lunch as an important social component 1 day training Practical orientation, with feedback loops 1 day training Ad hoc courses for students 0.5 day Know assessment framework and criteria/interpretation; minimal threshold level; know roles and tasks of experts; social competences Know legal aspects; know evaluation process; skills to analyse a degree; interpretation of criteria; soft skills; ethics and professionalism, team play; empathy Improvement oriented aspects; application of accred. criteria; team work and soft skills Presentation based training; discussions; little case work, experiences of experts Lecturing about evaluation process; Workshop I: standards and guidelines; interpretation of criteria Workshop II: small groups, case studies, simulation of evaluation Lecturing; discussion of examples; case studies (PBL) including feedback Two trainers, teaching experience, accreditation experiences Experts from ANECAs IQM team with experiences in evaluation; academics with QA experiences; experiences in management of degrees Strategic aspects taught by AQU management; soft skills, problem based, improvement skills by project managers

41 EVA All experts are trained (except existing experts) Selection is based on pedagogical know how and QA experiences FHR 1 Workshops for council members and academic members Selection criteria for expert panel members FIBAA academics, professionals students: regular trainings of new reviewers (2 per anno) and seminars for experienced experts (2 per anno) EVA s concept; accreditation as method, accred. criteria; roles and tasks of experts Purpose of assessment; assessment framework; national context, roles of experts and staff, good questions Newcomers: learn about assessment procedure, national regulations and European standards assessments framework. Experienced experts: new regulations, standards and trends, consistency of procedures Letter with tasks of experts; Legislative texts; LO material No formal training yet Background documents (regulations); FIBAA assessment guide 3 hours training Knowledge of accreditation as method; role of experts, know assessment criteria No formal training yet 1 day training; 4 trainings per year organized Needed LO according to surveys: Knowhow about assessment framework and criteria/interpretation; know roles and tasks of experts; know how to ask smart questions; know how to write a report Newcomers learn roles and tasks of experts; differences conditions/recommendations; assessment methods, national and European regulations. Experienced experts learn about changes of regulations, interpretation of regulations and about special items (i.e. joint programmes) Presentations and plenary discussions (case studies planned for future) No formal training yet Present case studies; discuss examples, working on texts, reporting of own experiences EVA management + external counsellor with thorough experiences in accreditation No formal training yet Experienced staff of agency; experiences in QA policies and procedures 1 "With"the"legal"validity"of"the"HS2QSG"a"new"and"single"agency"(Agency"for"Quality"Assurance"and"Accreditation"Austria)"was"established"which"is"responsible"for"external"quality"assurance"in"the"HE"sector"(Public" Universities,"Universities"of"Applied"Sciences,"Private"Universities)."The" Qualitätssicherungsrahmengesetz "(QSRG)"entered"into"force"on"1st"March"2012."The"FH"Council"persisted"until"31st"August"2012.""

42 NVAO Training of top level persons for institutional audits; separate trainings for students and secretaries Selection criteria for choosing experts OAQ Briefing for all members of expert panels; separate training for students Selection criteria for choosing experts QAA Same training for representatives of all reviewerstakeholder groups Strict selection process: nomination by institutions, application forms, selection criteria. QA in wider perspective; Audit framework; assessment criteria and interpretation; leadership; audit trails; role of experts; judgement; diversity Framework of assessment; roles and tasks of experts and staff of agency Assessment/ judgement process, roles and tasks, ethical element of peer reviews Audit framework; realistic case, examples of audits trails; Literature on leadership and on cultural differences Legal documents; Guides; Q- Standards; template for expert report Reading material Quiz Preparatory exercises 2 days training programme (small group learning); 3 times a year 3 hours training/briefing for all experts 1.5 day training programme for students Online preparatory work. 3 days residential training; no mass training trained once assigned to a specific review. Refresher training and CPD. Understand audit framework; able to lead audit team; awareness of cultural differences (Netherlands and Flanders) Not specified; know assessment framework and criteria/interpretation; minimal thresholds; know roles and tasks of experts Understand objectives of assessment process and process itself; know roles and tasks of experts, Ethics; social skills, consensus finding Lecturing; group discussions; case studies; Diner pensant, Ice- breakers; Folder with articles and study material Presentation, discussions among experts, sharing of experiences Student training interactive with role plays,etc. Inputs, small groups, simulation, plenary discussion, reflection spots, videos NVAO senior staff + senior external consultant; both experts on audits in HE Senior staff /management of agency Experienced staff of agency (QAA assistant directors) and experienced review team members

43 ZEvA Training for all members of expert panels excl.students (they have been trained elsewhere) National and regional regulations, assessment frame- work, examination regulations, criteria for nonstandard study programmes Standardized material on regulations (guidelines and standards, templates etc.). Presentation material. One-daytrainings for experts and students, about three times a year. Short trainings (2 hours) before each specific review. Knowledge of regulations and evaluation framework; know ledge in change management, module and programme design Exerts talks, open discussion group work (in future e- learning material, role plays, awareness of cultures) Table 1. Table 1. Overview of current practices in training experts - Summary of Quality Assurance Agencies delivered in Staff Training Workshop Speakers with evaluation experiences.

44

45 Annex 2: Preparatory material

46

47 Backgroundandpreparatorymaterial 1 st TrainingSessionforPanelExperts TheHague5 th and6 th October2011 Background E(TRAINwilldevelopmaterialsandamethodologyfortrainingbothmembersforExpertPanelsandQA agencystaffinarangeofaspectsconcernedprimarilywithevaluationatthelevelofheinstitutions.it willincludetwomaincomponents:onlinematerialsanda facetoface dayconcernedwithallaspects oftheevaluationofheinstitutions.thefocuswillbeonthe Europeancontext ;particularattentionwill be paid to the variety of institutional evaluation procedures used and also language, including terms that are frequently employed but may be interpreted in (slightly) different ways depending on individual s own backgrounds. The first sentence for example includes two words training and evaluation ( that may be interpreted in different ways; whilst training is a somewhat generic term it might,probablywould,beregardedbysomeasslightlyanachronisticwhenputwith experts.itwould forsomebemoreappropriatetospeakofinductionwithinaeuropeancontext(ratherthantraining)for thosewhohavealreadygainedsubstantialexperience/expertisewithinanational,regionalorsectoral role.thetraining/inductionwillfocuson evaluation asthecoreactivitysincethisisagenerictermfor the wide range of accreditation, review, audit, etc methods that are used within different countries, regionsorsectors. Purposeoftheday ThepurposeofthispartoftheE(TRAINprogrammeistoshareexperiencesandalsotoextend knowledge,understandingandpracticalexpertiseamongstestablishedexperts,relatedtothe wide range of contexts and approaches used across the EHEA for the evaluation of HE institutions. Asitisalsothefirstsuchsessionandthusa pilot,importantlyitshouldalsoidentifytheextent to which the contents and approaches could be improved for subsequent similar events. Session 5 is thus perhaps a little longer than might normally be expected, and the preceding Session4alittleshorter,inordertomaximiseparticipant sfed(backonthispilotday. Those who attend the first session will have already gained significant experience in some forms of evaluation but it is anticipated that they will depart with a much wider and deeper understanding(andpracticalknowledge)thatcaninfuturebeappliedwithbenefitstoawider rangeofcontexts. Page 1