Reliable Pre-interview Selection of Domestic & International Post- Graduate Candidates: CASPer Kelly Dore, PhD, Brad Petrisor, Michelle Ghert, Moyez Ladhani, Harold Reiter, Sharyn Kreuger McMaster University
I do not have an affiliation (financial or otherwise) with a pharmaceutical, medical device or communications organization. Je n ai aucune affiliation (financière ou autre) avec une entreprise pharmaceutique, un fabricant d appareils médicaux ou un cabinet de communication. Author: Kelly Dore, PhD Date: Sept 26 th, 2013
Dilemma Large applicant pool (Domestic & International) How do bring in the right applicants? Once admitted, it is difficult to remove a trainee High costs of interviews to program and applicants
Discipline Knowledge C2LEO, OSCE, PHELO National Licensing Exam ugpa, MCAT MMI Pre-Interview Selection
Computer-based Assessment for Sampling PERsonal characteristics (CASPer)
Online Computer Test CASPer Format Computer System Pre-test 8 Video Vignettes (SJT format) Applicant s perspective 90 seconds 4 Self-Descriptive Questions Behavioural Descriptive 3 follow-up questions 5 min open-ended response
You are Dr. Jones
Evaluation 2 Independent Raters using online portal Faculty, Community Members and Trainees Access to resources 9-point Likert scale global rating Red Flags Comments Benchmarking
Our Experience in Undergraduate Assessment 2010 - n = 3510 2011 - n = 4209 2012 - n = 4558 Completed CASPer on 2 days (one unadvertised backup day) Reliability always above 0.7 for 90 minutes of testing
Pilot Application: Postgraduate 2012 CASPer piloted for CaRMS IMG and Domestic Orthopedics IMG Pediatrics Completed CASPer as part of preinterview selection 162/178 (Ortho) 213/281 (Peds) Cost absorbed by a research grant
CASPer Postgraduate Selection Ortho IMG (0.85) Overall Test Reliability Ortho Domestic (0.65) Peds IMG (0.77) Correlation with Interview (MMI) Correlation was 0.539 for domestic applicants IMG was underpowered Only 3 were interviewed
Take Home Messages CASPer has demonstrated high reliability at the UG applicants, pilot data supports continued review for PG esp with IMGs Preliminary correlation with MMI supports the use of CASPer pre-interview, predictive validity still needed Wider-spread implementation data to follow in the next couple of years
Thank You dore@mcmaster.ca http://fhs.mcmaster.ca/casper/
Help us improve. Your input matters. Download the ICRE App, Visit the evaluation area in the North Building, Upper Level, near registration, or Go to: http://www.royalcollege.ca/ icreevaluations to complete the session evaluation. Aidons-nous à nous améliorer. Votre opinion compte. Téléchargez l application de la CIFR Visitez la zone d évaluation, au bâtiment nord, niveau supérieur, près du kiosque d inscription, ou Visitez le http://www/collegeroyal.ca/evaluationscifr afin de remplir une évaluation de la séance. You could be entered to win 1 of 3 $100 gift cards. Vous courrez la chance de gagner l un des trois chèques-cadeaux d une valeur de 100 $.
CASPer Postgraduate Selection Mean scores Ortho Domestic (3.85) Ortho IMG (2.7) Peds IMG (3.0)
You are Dr. Jones
UG Correlation with the MMI 1 0.9 0.8 0.7 0.6 0.5 0.51 0.6 0.48 Study 1 2006 Study 2 2008 2010 Admissi on 0.4 0.36 2011 Admissi on 0.3 0.2 0.1 0 1
UG CASPer Inter-Rater Reliability 1 0.95 0.9 0.8 0.81 0.84 0.7 0.6 0.5 0.4 Study 1 2006 Study 2 2008 2010 Admissi on 2011 Admissi on 0.3 0.2 0.1 0 1
UG CASPer Overall Reliability 1 0.9 0.83 0.8 0.72 0.76 0.78 0.7 0.6 0.5 0.4 Study 1 2006 Study 2 2008 2010 Admissi on 2011 Admissi on 0.3 0.2 0.1 0 1
Wong et al, McMaster IMG s cite loss of professional identity, practice in context
Zulla et al, U of Toronto PDs cited concern about communication, collaboration
Hall et al, U of Ottawa Multi-source cite communication, practice in context, culture
Reuveni et al, Schabort et al greater use of screening predicts for better outcome
Implications: Cost Currently charging students $30 to cover cost 3 days of helpdesk + pretest helpdesk 21 new videos Programming Updates Rater Resources 1 rater uses about 180 raters for 4500 applicants
October 2011 CASPer Administration
Professionalism, 92% Knowledge, 8% $50,000,000,000
Please rate the candidate s performance on this station relative to the pool of all applicants. You may adjust your scores as necessary. Place marks inside the boxes. Score the applicant s overall performance on this station. Unacceptable Marginal Good Excellent Superior Comments: Red flags: Yes No If Yes please provide comment: