Reliable Pre-interview Selection of Domestic & International Post- Graduate Candidates: CASPer

Size: px

Start display at page:

Download "Reliable Pre-interview Selection of Domestic & International Post- Graduate Candidates: CASPer"

Theodore Flowers
10 years ago
Views:

1 Reliable Pre-interview Selection of Domestic & International Post- Graduate Candidates: CASPer Kelly Dore, PhD, Brad Petrisor, Michelle Ghert, Moyez Ladhani, Harold Reiter, Sharyn Kreuger McMaster University

2 I do not have an affiliation (financial or otherwise) with a pharmaceutical, medical device or communications organization. Je n ai aucune affiliation (financière ou autre) avec une entreprise pharmaceutique, un fabricant d appareils médicaux ou un cabinet de communication. Author: Kelly Dore, PhD Date: Sept 26 th, 2013

Je n ai aucune affiliation (financière ou autre) avec une entreprise

3 Dilemma Large applicant pool (Domestic & International) How do bring in the right applicants? Once admitted, it is difficult to remove a trainee High costs of interviews to program and applicants

4 Discipline Knowledge C2LEO, OSCE, PHELO National Licensing Exam ugpa, MCAT MMI Pre-Interview Selection

5 Computer-based Assessment for Sampling PERsonal characteristics (CASPer)

6 Online Computer Test CASPer Format Computer System Pre-test 8 Video Vignettes (SJT format) Applicant s perspective 90 seconds 4 Self-Descriptive Questions Behavioural Descriptive 3 follow-up questions 5 min open-ended response

perspective 90 seconds 4 Self-Descriptive Questions

7 You are Dr. Jones

8 Evaluation 2 Independent Raters using online portal Faculty, Community Members and Trainees Access to resources 9-point Likert scale global rating Red Flags Comments Benchmarking

9 Our Experience in Undergraduate Assessment n = n = n = 4558 Completed CASPer on 2 days (one unadvertised backup day) Reliability always above 0.7 for 90 minutes of testing

10 Pilot Application: Postgraduate 2012 CASPer piloted for CaRMS IMG and Domestic Orthopedics IMG Pediatrics Completed CASPer as part of preinterview selection 162/178 (Ortho) 213/281 (Peds) Cost absorbed by a research grant

Completed CASPer as part of preinterview selection

11 CASPer Postgraduate Selection Ortho IMG (0.85) Overall Test Reliability Ortho Domestic (0.65) Peds IMG (0.77) Correlation with Interview (MMI) Correlation was for domestic applicants IMG was underpowered Only 3 were interviewed

12 Take Home Messages CASPer has demonstrated high reliability at the UG applicants, pilot data supports continued review for PG esp with IMGs Preliminary correlation with MMI supports the use of CASPer pre-interview, predictive validity still needed Wider-spread implementation data to follow in the next couple of years

Preliminary correlation with MMI supports the use of CASPer pre-interview,

13 Thank You

14 Help us improve. Your input matters. Download the ICRE App, Visit the evaluation area in the North Building, Upper Level, near registration, or Go to: icreevaluations to complete the session evaluation. Aidons-nous à nous améliorer. Votre opinion compte. Téléchargez l application de la CIFR Visitez la zone d évaluation, au bâtiment nord, niveau supérieur, près du kiosque d inscription, ou Visitez le afin de remplir une évaluation de la séance. You could be entered to win 1 of 3 $100 gift cards. Vous courrez la chance de gagner l un des trois chèques-cadeaux d une valeur de 100 $.

Téléchargez l application de la CIFR Visitez la zone d évaluation, au bâtiment nord, niveau supérieur, près du kiosque d inscription, ou Visitez le

15 CASPer Postgraduate Selection Mean scores Ortho Domestic (3.85) Ortho IMG (2.7) Peds IMG (3.0)

16 You are Dr. Jones

17 UG Correlation with the MMI Study Study Admissi on Admissi on

18 UG CASPer Inter-Rater Reliability Study Study Admissi on 2011 Admissi on

19 UG CASPer Overall Reliability Study Study Admissi on 2011 Admissi on

21 Wong et al, McMaster IMG s cite loss of professional identity, practice in context

22 Zulla et al, U of Toronto PDs cited concern about communication, collaboration

23 Hall et al, U of Ottawa Multi-source cite communication, practice in context, culture

24 Reuveni et al, Schabort et al greater use of screening predicts for better outcome

25 Implications: Cost Currently charging students $30 to cover cost 3 days of helpdesk + pretest helpdesk 21 new videos Programming Updates Rater Resources 1 rater uses about 180 raters for 4500 applicants

26 October 2011 CASPer Administration

27 Professionalism, 92% Knowledge, 8% $50,000,000,000

28 Please rate the candidate s performance on this station relative to the pool of all applicants. You may adjust your scores as necessary. Place marks inside the boxes. Score the applicant s overall performance on this station. Unacceptable Marginal Good Excellent Superior Comments: Red flags: Yes No If Yes please provide comment:

Developing EPAs (Entrustable Professional Activities) for Family Medicine

Developing EPAs (Entrustable Professional Activities) for Family Medicine Author: Jane Griffiths, Karen Schultz Date: September 27, 2013 (Queen s, Laval, Western, Calgary, MUN) I do not have an affiliation