Approaching ( )

Transcription

1 School of Physician Assistant Studies Student Learning Outcomes Assessment Program Goal #1: Apply knowledge from their discipline in a context reflecting real, complex situations in their profession. Assessment level: Baseline Database to compare the following grades: 1. Anatomy, physiology, biochemistry, microbiology, and science GPA in pre-admission work 2. Human Science (biochemistry, anatomy, physiology) and Microbiology /Infectious Disease 3. Pharmacology 1,2 &3 and Clinical Medicine 1,2 & 3 Evidence of quality: This instrument has just been developed and so the strength of it has not been tested. It is intended to compare the academic quality of the entry level student, through the formative process of the program to a level of understanding in courses that directly impact on patient care; pharmacology and the clinical medicine courses. Logistics: Upon matriculation, the student s pre-admission grades, science GPA and number of science hours are entered into the database. As the student finishes the courses; eg Human Science, Microbiology/Infectious diseases, Pharmacology, and Clinical Medicine, the grades will be entered. Two things will be considered. One, whether there is any predictive value of pre-admission scores on success in the Human Science course, Pharmacology, and Clinical Medicine, reflecting the quality of the admissions process; and second, whether the grades in the Human Science and/or Microbiology/Infectious disease course predict success in the Pharmacology and Clinical Medicine courses, reflecting the preparatory value of the basic science courses. Criteria: Evaluating student performance will be based on grades. Uses: Information from this instrument will be helpful in evaluating not only the admissions process, but the quality of education in preparing the students for the clinical courses. Preadmission scores Course Cohort Below ( ) ( ) ( ) (>3.75) Anatomy 2008B (29/45) (32/52) (21/53) Physiology 2008B (29/45) (32/52) (18/53) Biochemistry 2008B (16/45) (19/52) (11/53) Microbiology 2008B (23/45) (25/52) (12/53) Immunology 2008B (5/45)

2 2009 (4/52) (5/53) Science GPA (average GPA over all 3 cohorts was 3.0) Cohort Below ( ) ( ) ( ) (>3.75) 2008B (0 data on ) Number of coursework hours in Science (average number over all 3 cohorts was 75.2) Cohort Below (59 or less) (60-69) (70-79) (>80) 2008B(0) data on ) Professional evaluations Assessment type: Indirect Evidence quality: This is very subjective. The core PA faculty review each student on several parameters that reflect the professional qualities a PA student should possess. Logistics: At the end of each session, each student is reviewed. Following the review, the student is instructed to meet with his/her advisor for review. Criteria: The students are rated on a rubric with 3 (average) being a preferred score and 1 or 5 at the extremes. Uses: The results of this tool help in the professional development of the student, provides guidance in counseling, and clearly defines expectations. Professional evaluations after first 15 weeks (Beginning with the 2009 class, the evaluative tool was changed, making it difficult to assess the results in the below, approaching, meets, and exceeds category. The new tool is designed to assess extremes in behavior where a three out of five is the preferred score.) Cohort Below 2008B Assessment level: Developmental Physician Assistant Clinical Knowledge Rating Assessment Tool (PACKRAT)

3 Evidence of quality: This is a shelf multiple choice exam of about 225 questions that is purchased. The exam has been in existence for over 25 years, but has never been validated. Several studies by more established PA programs have been done in an attempt to correlate the results with success on the Physician Assistant National Certification Examination (PANCE), the certification exam that all PAs must pass in order to obtain licensure to practice. While it has been shown to have some predictive value, the greatest benefit of this tool is to define areas of weakness by clinical specialties and tasks. This then can be used to first, alert the student to areas of weaknesses, and second, alert the faculty as well on areas that need stronger instruction. Data to include the overall score and scores in each specialty and task is collected and placed in the data base offering an additional tool in which to evaluate the effectiveness of teaching. Logistics: The PACKRAT is administered to students prior to beginning their clinical rotations. Criteria: Students are given an overall raw score. Based on prior studies, a raw score below 120 is predicted as failing the PANCE. This score will be used as approaching expectations. Uses: There are multiple uses to this examination. As explained above, it provides feedback to the student and the faculty. It also helps to track whether changes in the curriculum, modalities of instruction, or the admissions process have impacted on the formative process of the students. PACKRAT #1 scores (National average across all three years is 134.8, highest score achievable is 225) Cohort Below (<120) ( ) ( ) (>140) 2008B Transitional Practicum Evidence of quality: This is a cumulative course the students take prior to going on the clinical rotations. It is taught in a seminar style with student directed learning. While it is interactive, and tends to be subjective in grading, the students do receive a grade. Efforts to improve this course as an evaluative tool to learning outcomes would be a rubric and the development of a portfolio that would reflect the learning objectives in the syllabus. Logistics: The course is given in the fourth and last didactic session. Criteria: This course is a pass/fail. Uses: This course incorporates clinical assessment and knowledge skills, ethical decision making, professional concerns, and the ability to access information and critically evaluate it. It prepares the student for the clinical rotations and identifies weaknesses the student may possess that may impair their progress. All students met expectations.

4 Assessment Level: Mastery Summative Examinations to include the following: PACKRAT (second administration of the shelf exam but a new version) Comprehensive Examination Evidence of quality: The PACKRAT, as previously discussed and the comprehensive examination have not been validated. The comprehensive examination is an outgrowth of a collaborative effort between several programs on the use of an additional tool that can be statistically reviewed, to evaluate the clinical knowledge of the graduating student. Logistics: The examination is given two months prior to completion of the program. This allows for time for remediation. Criteria: The raw score of 120 is considered passing. This is based on two recent studies that correlated that score with successfully passing the PANCE. Uses: The results will aid in identifying subject matter in the rotations that is weak. It will also aid in reviewing the end of rotation examinations and determining if the students are being challenged in the areas pertinent on this shelf exam as well as on the PANCE. PACKRAT #2 scores (National average across all three years is 134.3, highest score achievable is 225) Cohort Below (<120) ( ) ( ) (>140) 2008B Not available Preceptor evaluations and clinical write ups /Indirect Evidence of quality: At the end of each rotation, the student s preceptor grades the student on clinical and professional expertise using a 5 point Likert scale. The preceptor then assigns a grade to the student which does not necessarily reflect a summation of the points. Four parameters from the preceptor evaluation form were selected to assess the student learning outcomes. They were: 1. The student can perform a problem based history 2. The student demonstrates proficiency in performing a problem oriented physical examination 3. The student assesses the usefulness of different diagnostic studies in relation to the disease process 4. The student formulates a therapeutic plan that is consistent with acceptable medical practices Regarding the clinical write-ups, the student submits a long clinical write-up of a patient the student saw in the clinic. At the end of the piece, the student must provide a discussion regarding the case, using three references. The submission is graded on whether the assignment contained certain key items. The following four parameters from the grading were selected to assess the student learning outcomes. There were:

5 1. Able to list potential diagnoses related to chief complaint 2. Includes patient education 3. Includes follow-up 4. Discussion of diagnosis is complete and shows evidenced based thought The Community Medicine rotation was not included as some did not do a clinical writeup but a proposal for the Master paper. Logistics: This is completed at the end of each clinical rotation. Criteria: expectations/accomplished Uses: This tool is used to assess the performance of the students in critical areas along with their ability to adequately document their findings. Preceptor evaluations Rotation Parameter Below (1,2) (3) (4) (5) 2008B B B B 2009 Adult Medicine Family Medicine Emergency Medicine OB/GYN Pediatric Surgery Community Medicine

6 Rotation Parameters Accomplished Not Accomplished 2008B B 2009 Adult Medicine Family Medicine Emergency Medicine OB/GYN Pediatrics Surgery Assessment level: Alumni Physician Assistant National Certification Examination (PANCE) Evidence of quality: This is a national certification examination sponsored by the National Commission on Certification of Physician Assistants (NCCPA) that was originally administered by the National Board of Medical Examiners. It is now administered by the NCCPA. The examination has been in existence since the 1970 s and successful passing is required for licensure in all states. Logistics: The graduate can take this examination no less than seven days after completion of the program. The graduate schedules when and where s/he would like to take it. Criteria: The passing score is a raw score of 350 and iss essentially a pass/fail exam. Uses: While the main use is to determine if the graduate has a knowledge and skills base sufficient to practice medicine, the report that is generated reflects raw scores in the same specialties and tasks that are reported out in the PACKRAT. This is a valuable tool in aiding the program in identifying strengths and weaknesses.

7 Cohort Below (<350) (350- ( ) (>450) 399) 2008B Measure2 Graduate surveys Assessment type: Indirect Evidence of quality: This is a survey, based on a 1-5 Likert scale, that goes to the graduates at least 6 months after completing the program. It contains questions regarding how well s/he feels the program academically and professionally prepared the graduate to function as a health care provider. This is purely subjective, and can be tainted by the relief of finishing the program, passing the PANCE, and being employed with an excellent salary. Logistics: In the past the survey was mailed which yielded a very low response rate. The next survey will be sent out via . Criteria: The Likert scale ranges from 1 (poor) to 5 (high). Uses: The surveys are used during the self study that is required for continuing accreditation by the Accreditation Review Commission on Physician Assistant Education (ARC-PA). There is no data as the measurement tool is in the process of development.

8 Program Goal #2: Think critically to make evidence-informed decisions and evaluate conclusions. Assessment level: Baseline Physical Diagnosis 2 Evidence of quality: This is a problem based, small group learning course that is faculty directed. The student begins the process of developing a differential diagnosis based on what s/he discovers in the history and physical. The student is graded on the ability to assimilate and integrate information in an organized fashion. This course has been taught for 5 classes and, while not quantified, may be predictive on success in the clinical rotations. Logistics: Students are given several practical tests throughout the course. Criteria: The grade of 80% is passing. Because this course is critical for the success of the students, the grade of 85-90% would be considered as meeting expectations. Uses: While this course is graded, it is most useful in identifying problems a student may have in the critical thinking process at a point in his/her education where it can be addressed. Cohort Below (<80%) (80- (86-90%) (>90%) 85%) 2008B Biostatics and Research Methodology Evidence of quality: A graded course that teaches basic statistical principles, it exposes the student to the process of critically evaluating medical studies and sets the groundwork for evidence based medicine. Logistics: The course is given midway through the didactic portion of the program. Criteria: The grade of 70% is passing. However, because this is master level course, a higher grade would be expected as meeting expectations. Uses: This is a preparatory course for the Transitional Practicum and Epidemiology courses and mastery is reflected in the clinical rotation clinical write-ups. Cohort Below (<70%) (70- (80-89%) (>90%) 79%) 2008B This course has undergone numerous transitions. For each cohort, a different instructor, course structure and syllabus was employed. For the 2008B cohort, a clinician taught the course, for the 2009, a statistician, and for the 2010 course, an anatomist.

9 Assessment level: Developmental Transitional Practicum- similar to SLO #1 Epidemiology Evidence of quality: This course utilizes the background information taught in Biostatistics and Research Methodology in a practical fashion and discusses studies done in such areas as obesity, the metabolic syndrome, diabetes, osteoporosis, and bio-terrorism. It challenges the student to assess the literature and develop conclusions. Logistics: This course is taught in the final session of the didactic year. Criteria: The grade of 70% meets expectations. Uses: This course evaluates the student s ability to independently seek information and utilize it to formulate a conclusion. Cohort Below (<70%) (70- (80-89%) (>90%) 79%) 2008B Assessment level: Mastery Clinical write-ups Evidence of quality: At the end of each clinical rotation, the student must submit a long history and physical of a patient s/he saw. At the end of the long write-up, the student must discuss the case using at least three sources of reference, two of which must be from a peer reviewed article. The discussion must lead the reader to understand why the diagnosis was selected and justify the plan. Logistics: The clinical write-up is submitted at the end of each required clinical rotation. Criteria: A grade of 70% meets expectations. Uses: This evaluative tool is indicative of the student s ability to take care of patients in an evidence based medical approach. The data is similar to SLO #1