Proof of concept trial of online peer review tool. Dr Sara Booth

Transcription

1 Proof of concept trial of online peer review tool Dr Sara Booth

2 01 Testing an online peer review tool SESSION AIMS 1. To provide overview of pilot with Education Services Australia 2. To discuss different models of peer review of assessment 3. To identify information for the testing 4. To discuss next steps with Colleges of Peers 2

3 Introduction A recent review of OLT projects focused on academic standards (Freeman & Ewan, 2014) Evidence of improved assessment practices with the development of threshold learning outcomes (TLOs); the establishment of networks; and the important role Deans Councils play in leading efforts on academic standards. Gaps: 1) the absence of non-self-accrediting and private providers in these academic quality projects; 2) the lack of an evidence base for quality assurance; and 3) the lack of external referencing.

4 Overview of Pilot with Education Services Australia (ESA) Principles for Proof-of-Concept Testing Context-sensitive; Streamlined, efficient, cost-effective, sustainable; Fit for purpose; Engaged (Krause, 2014). Options 1) an online tool that can be used by two or more HE providers to engage in external referencing and benchmarking of academic standards and assessment outcomes 2) an online repository to enable staff across the HE sector to share validated assessment items.

5 Methodology of the Project Phase 1: Development of a proof-of-concept online peer review tool and clearinghouse of good practice in assessment. The online peer review tool product development was undertaken from March-October, 2015 by technical support staff at ESA. Key deliverables include Windows environment set up; reviewer/reviewee survey types; survey rendering; user stories, test scripts; introductory texts; user manual; interface design; and continuous product development

6 Methodology of the Project Phase 2: Pilot project to test the proof-of-concept online peer review tool and clearinghouse of good practice in assessment with stakeholder groups and collect feedback on how to improve these resources; Phase 3: Engage with key stakeholders in the HE community about the proof-of-concept project;

7 Methodology of the Project Phase 4: UTAS to lead the coordination of support for the College of Peers process, not unlike Bristol University with the Economics Network by providing an expansion of PRAN website, s and communications, web resources and forums; Phase 5: UTAS and ESA to set up an organisational support structure, business plan and registration process for future subscribing institutions and Deans Councils to use the peer review tool;

8 Methodology of the Project Phase 6: Develop online resources in peer review [assessment and teaching], including international journal in peer review and a training package on how to give peer review [working with HERDSA, OLT Fellows, CADAD and ACODE]; and Phase 7: Set up forums and workshops for professional development in peer review. Liaise with national and international HE organisations such as the QAA, AQA, Ako Aotearoa, HEA, Higher Education Services, TEQSA, and Universities Australia to set up national and international linkages in peer review.

9 Stakeholder Groups Stakeholder Groups involved in testing Group 1: RMIT University, Queensland University of Technology, Curtin University, University of Wollongong Group 2: Edith Cowan University,University of New England, Central Queensland University, Western Sydney University, Swinburne University of Technology Group 3: Council of Deans in Nursing and Midwifery (CDNM): Southern Cross University, The University of Newcastle, Edith Cowan University, Griffith University, RMIT University, University of Tasmania, Federation University Australia, University of South Australia, The University of Notre Dame Australia and Monash University Group 4: Australasian Council of Deans of Information and Computing Technology (ADICT): under discussion Group 5: Council of Deans of Engineering: under discussion Group 6:AIPE and Top Education: under discussion Group 7: Eastern College of Australia, Tabor Adelaide, Harvest Bible College, Avondale College Group 8: Avondale College and Charles Sturt University Disciplines Fashion Design Engineering Range of disciplines Research unit in nursing Groups of sub-disciplines in ICT To be confirmed To be confirmed Theology, education Education

10 Verification (eg. QVS, IRU) Moderation (eg. LaTS) Primary intent QA (& QE) QA (& QE) QA & QE Calibration & double blind (eg. AMA) Scope Selected final UoS Selected final UoS Selected degree standards/tlos Disciplines Multiple (11) Multiple (12) Accounting Level Bachelor Bachelor Bachelor + Master Reviewers 1 academic 1 academic per partner 2 anonymous academics +/or professionals (3 rd potentially) Standards Implicit in reviewers Implicit in reviewers Explicit (agreed nationally 2010) Calibrated Not explicitly Not explicitly Yes by workshops Products viewed Tasks (inputs) & outputs in final unit of study Tasks (inputs) & outputs in final unit of study Tasks (inputs) & outputs in degree evidencing standards Data selection Stratified Stratified & de-identified Random & de-identified Sample outputs Reviews 12=3 per grade 4=1 per passing grade Minimum 5 per agreed standard Manual submission files & aggregation Manual submission files & aggregation Authority Institutional Institutional Disciplinary Online submission & auto aggregation

11 What this site can enable you to do This site can enable you to: Use external peers to confirm that you have soundly validated program level learning outcomes (using multiple reference points Use external peers to blind review and confirm the reliability of grading Identify good practice and examples for mapping, powerful assessment; accurate grading and calibration and aligned to engaging learning methods and resources Need a local A/Dean or HOP s network to identify, quality assure and load up good practice guidelines and exemplar

12 Homepage

13 Institutional Administrator

14 Institutional Administrator

15 Institutional Administrator

16 Institutional Administrator

17 Review Manager

18 Review Manager

19 Review Manager

20 Review Manager

21 Review Manager

22 Reviewee

23 Reviewer

24 Next Steps Establish College of Peers Engagement with Universities Australia Satellite Event Involve other forms of peer review [peer review of teaching; promotion etc.] Resources, including online resources and presentation on how to give effective peer review feedback