Jody Frost, PT, DPT, PhD, FNAP American Physical Therapy Association

Transcription

1 Jody Frost, PT, DPT, PhD, FNAP American Physical Therapy Association

2 Story of one profession s 20-year journey to develop a profession-embraced voluntary and standardized student performance assessment for use across programs, practice settings, and clinical experiences. Fundamental principles incorporated in designing a profession-embraced, psychometrically sound, and legally defensible performance instrument. Shared lessons learned in navigating through this journey and their application to Nursing.

3 Assumption that there is only way to approach development of student performance assessments for any profession. Detailed presentation of the specific psychometrics of the Physical Therapy Clinical Performance Instruments (Physical Therapist [PT] CPI and Physical Therapist Assistant [PTA] CPI).

4 Why develop consistent clinical performance assessments for PT and PTA students?

5

6 Needs of and demands on practitioners No standardized training in performance assessments Health care changes Risk of Losing Practice Sites for Clinical Education Litigation and academic grievances Curriculum requirements and changes Proliferation of student assessments

7 Changing health care and higher education landscapes Workforce issues in education and practice Do much more with even less! Scopes of practice Innovative models of practice Innovative models of education Litigation and defensibility Interprofessional education and collaborative practice

8

9 What would it take for Nursing to have one consistent performance assessment system for students during clinical practicum?

10 Turn to your colleagues at your table. Discuss and capture 3 of the most important elements that would be required to make this happen in Nursing! Be prepared for some tables to share the single most important element with the your colleagues in the room.

11 Why is creating an open and inviting culture to enable a culture shift around assessment so important?

12 Expectations when seeking feedback and comments that they will be used! Opportunities for participation to enhance buy in. Ensure that individuals have been heard through active listening. Important to welcome different perspectives to generate new ideas. Best outcomes result from shared invested partners. Provide open forums with opportunities for teachable moments!

13 What are the fundamental components of performance evaluation that ANY student performance assessment system should incorporate?

14 1. Decisions in experiential education are based on objective evaluation and professional judgment. 2. Objective evaluation of student performance drives decision making. 3. Ongoing process consisting of two complementary components: formative and summative evaluations.

15 4. Being an effective evaluator requires skill in 3 competency areas: Evaluation Supervision Communication 5. Experiential education is NOT an evaluative process. Rather, evaluation is a tool used in experiential education.

16 What approach will the profession use to assess students clinical performance?

17 Series of specific competencies or tasks at varied levels of taxonomy. Assesses performance relative to the set of specific defined task analyses. Sum of all competencies is equivalent to the expected performance. May complete different competencies during different clinical practicums. Once all competencies have been completed the learner should be prepared to enter practice.

18 Evaluates performance based on the highest level (taxonomy) of expected performance outcome (cognitive, affective, psychomotor). Typically provides expected performance criteria that collectively represent entry-level practice. Provides an illustrative or exhaustive list of expected observable behaviors, that when demonstrated, enable the learner to meet the outcome performance criterion. Tends to be more global in overall ratings with an assumption that individual tasks are being met.

19

20 1. Provide feedback during practice on performance in in cognitive, psychomotor, and affective domains. 2. Determine whether a student is competent in a procedure or process. 3. Determine how students are changing in performance. 4. Identify additional work needed for clinical competence and mastery.

21 5. Evaluate achievement of specific learning objectives. 6. Assist the development of self-assessment skills. 7. Collect data for determining grades, program progression, and program completion. 8. Evaluate overall effectiveness of an experiential course.

22 Formative Evaluation and Summative Evaluation

23 Formative Evaluation Summative Evaluation PURPOSE Process Outcome USE TIMING Provides learning During the experience Grading Program progression At the end of the experience AUDIENCE Internal External FUNCTION Predicts summative Sets standard for formative

24 Valid Reliable Enhanced through standardized training Acceptable Educational effect Motivates those using the assessment in an educational manner Feasible

25 Provides useful information about contemporary student clinical performance. Addresses legal issues Early Warning System (EWS) for early problem identification and remediation. Assessment is not arbitrary or capricious. Follows good principles of evaluation.

26 1. Professional judgment is an integral part of evaluation. 2. Ratings are based on definite observations and established performance criteria. 3. Ratings are based on typical and frequent performance (not isolated instances). 4. Education is provided about rater errors to maintain objectivity (eg, halo, central tendency, initial impressions, diversity, etc) * Watts N. Handbook of Clinical Teaching. New York, NY: Churchill Livingstone Inc., 1990.

27

28 Adapted from a presentation by Steve Milam, Senior Counsel and Assistant Attorney General, Health Sciences and Medical Centers, University of Washington, Seattle, WA on Legal Context for Evaluating and Dismissing Physical Therapy Students

29

30 1. Used to identify problems 2. Define means to resolve problems 3. Define performance criteria expectations 4. Define timelines for improvement 5. Requires that a student sees his/her documentation to know the specific problem.

31 Student performance improves! Student denies having problems! Student tries to improve, but fails!

32 The real tragedy and perhaps unfairness to the student was the school s failure to dismiss the student earlier when there was ample evidence of his inability to meet scholastic standards. Manangen v. Board of Regents, University of Washington, King County Cause #832518, Wash. Superior Court, Nn 1977, Seattle, Washington (unreported and not appealed)

33 1. Educators have legal as well as ethical responsibilities to learners, the academic institution, the clinical facility, and patients. 2. Documentation of the experiential education process must be sufficiently clear, accurate, and thorough to demonstrate the quality and consistency of the educational experience.

34 The court will not overturn academic decisions. When judges are asked to review the substance of a genuinely academic decision...they should show great respect for the faculty s professional judgment. Plainly, they may not override it unless it is such a substantial departure from accepted academic norms as to demonstrate that the person or committee responsible did not actually exercise professional judgment. Univ. of Michigan v. Ewing, 106 S. Ct. 507 at 513 (1985)

35 Faculty (academic/clinical) are encouraged to uphold high academic standards to candidly and critically evaluate a student. Arbitrary and capricious action is willful and unreasoning action without consideration and in disregard of facts or circumstances. When there is room for two opinions, an action is not arbitrary or capricious when exercised honestly and upon due consideration even though it may be believed that an erroneous conclusion has been reached. McDonald v. Hogness, 92 Wn.431, at 437 (1979)

36 3. Faculty are encouraged to identify students with problems early and to provide assistance. 4. Faculty are encouraged to candidly evaluate a student s performance and to dismiss a student when dismissal is warranted. 5. Rules governing student s performance must be communicated to the student prior to the performance evaluation. 6. The student must be treated fairly and given feedback regarding his/her performance.

37 7. Preceptors must show they have not acted arbitrarily or capriciously. 8. The problem-solving process should be documented, as well as the student s behavior and response to feedback. 9. All parties (student, preceptor, academic program) should be involved in contract negotiations and decision-making.

38

39 : Task Force on Clinical Education Regional student performance assessments PT Bachelor and Masters Degrees PTA Associate two-year degree : 10-person Task Force on PT/PTA Student Clinical Performance Instruments 1997: PT Student CPI available 1998: PTA Student CPI available

40 Phase I: Development First Drafts Review of literature Review of assessments Target behaviors Scoring system Instructional protocol Phase II: Conduct Pilot Studies of 2 nd Draft Feedback from 50 content experts Hosted 10 regional forums US and Canada (Feb-June 96) Conduct pilot study to refine CPIs (350 PT, 350 PTA students) Phase III: Field Study 3 rd Drafts 3 rd Drafts reviewed by users Conducted field study to include validity, reliability, factor analysis, feasibility, and user satisfaction Phase IV: Modified 3 rd Drafts Modified PT/PTA CPIs based on outcomes of field study APTA Board of Directors approval Draft publication

41 Credibility: Composition of the 10-person expert task force. Acknowledgment: Review of literature and all existing student performance assessments for commonalities, distinctions, and gaps. Outcomes Performance: Determination of assessment design. Buy-in: Involvement of all key stakeholders at multiple phases in the development of the instruments.

42 Ownership/Investment: Many opportunities for users comments, feedback, and revisions. Pilot and Field Studies: Provide instrument credibility with evidence for changes. Managing Tension: Importance of balancing the needs of the profession with psychometric properties of the instrument. Legal Issues: Include an early warning system and tested for instrument objectivity. Dissemination: Publish research findings as a benchmark for future investigations.

43 : New 5-person PT CPI Task Force Revise PT CPI for professional DPT degree 2006: Request for Proposal Develop web-based PT/PTA CPIs Create web-based PT CPI course on Learning Management System (LMS) : New 5-person PTA CPI Task Force Develop web-based PTA CPI course on LMS Develop PTA CPI : Develop PT CPI Web : Develop PTA CPI Web

44 Phase I: Review of Literature Review of CPI literature Identify research issues Integrate current relevant APTA documents Phase II: Conduct Survey Survey stakeholder users to identify strengths and deficiencies Revise CPI in response to deficiencies Phase III: CPI Study Develop online training program Develop webbased CPI Conduct study with 66 US and 4 Canadian PT programs Phase IV: PT CPI Availability Create online training program in APTA LMS Develop, test, implement PT CPI Web available for purchase through registration Draft publication

45 Multidimensional system with internal triangulation Outcome Performance Criteria PT: 18 performance criteria with 5 red flag items PTA: 14 outcomes performance criteria with 5 red flag items; 5- items include specific skills assessment (performed, observed, NA) Sample behaviors Qualitative comments provided for performance criterion 5 defined performance dimensions Rating scale PT: 6 well-defined anchors with 5 intervals PTA: 5 well defined anchors with 5 intervals; with distinction box Significant concerns indicator Summative comments

46 Selection of Performance Assessment CPI Web Registration PT CPI voluntary use by 93% of PT programs Annual program registration fee for faculty, students, and clinicians to access the system Complete Required Standardized Web-based Training (.2 CEUs) Applied concepts using practice questions and case scenarios to rate student performance >80,000 individuals completed training Preceptor completes CPI on Student Student completes CPI Self-Assessment Notifications: Access to Performance Assessment by Program/Clinic following sign-off System documentation available entire internship Summative mid- and final evaluations System includes Learning Objectives, Weekly Planning Forms, Critical Incident Reports, and Learning Contracts Retrieval and Data Management Individual: Student and preceptor data Program: Aggregate reports for curriculum assessment, accreditation, and research

47 Staged Revisions: Staged enhancements for PT and PTA resulted in different CPIs. Used two different task forces. Redesigned PT CPI first which helped to inform PTA CPI development. Training How to Use the CPI: Enhanced instrument reliability through required standardized, accessible web-based training. Psychometrics: Determined to be valid and reliable.

48 Feasibility: Widespread voluntary use of PT/PTA CPI Web Ongoing system enhancements in response to users feedback Incentives: Required training by users to access PT/PTA CPIs Web-based course is free Users earn.20 CEUs! Legal Issues: Includes EWS objective ratings supported by comments sign off to protect release of student and preceptor records.

49 Technology: Conversion from paper to webbased system reduced copying costs eliminated measuring VAS compiled data and reports addressed prior known copyright violations CPI Web Training Management: User training and system help managed by vendor Practical Tips: Transition to web-based system required user helpful hints.

50 Data and Reporting: Responsive to program needs for data and reports to: determine student grades and program progression identify at-risk students early scan cohort performance at mid- and final evaluations make curricular decisions provide reports for accreditation National Data: Assured program data managed by external vendor Assured only aggregate data available Dissemination: Published research findings

51 Roach KE, Frost JS, Francis NJ, Giles S, Nordrom JT, Delitto A. Validation of the revised physical therapist clinical performance instrument (PT CPI): Version Phys Ther. 2012; 92(3): Roach KR, Gandy JS, Deusinger SS, et al: The Development and Testing of APTA Clinical Performance Instruments. Phys Ther. 2002; 82(4):

52 Use of CPI Web PT CPI (92%); 202/225 accredited programs PTA CPI (56%); 184/327 accredited programs CPI Training >80,000 PT faculty, students, clinical instructors/preceptors, and center coordinators of clinical education have completed the training Annual CPI Web Registration ( ) PT Programs = $1,050 PTA Programs = $775

53

54 1. Establish an open culture to solicit comments; listen and affirm feedback without defensiveness. 2. Debate is integral to the process as stakeholders view student performance assessment differently. 3. Need to balance stakeholder buy in, psychometric rigor, and legal documentation requirements. 4. Enhance credibility by incorporating ALL of the elements of effective assessment. 5. Use of academic jargon in performance assessment intended for use by clinicians is problematic.

55 6. Standardized training is essential and empowers all users. 7. CPI has been upheld with student grievances, including situations with solely affective performance issues! 8. Support needed for academic and clinical faculty transition to new performance assessments. 9. Enabling programs to voluntarily use the CPI increased its overall use.

56 10. Performance assessment systems must be affordable, accessible, evidence-based, and costeffective. 11. Performance assessments must be dynamic to ensure consistency with entry-level practice and changes in education and health care. 12. CPI has been included in some state licensure regulations requiring supervised practice for foreign-educated PTs curricular rubrics and classroom performance measures 13. CPI has been translated into several languages.

57 Development of a voluntary student clinical performance assessment was time consuming, an investment by all users, had its challenges, required debate to achieve agreement, required humility, and was a learning process for all. Resources invested and effort expended were worthwhile in determining graduates readiness for practice and advancing physical therapy education and the profession.

58

59 Given everything shared today, is this the right time to create standardized student clinical practicum assessments for Nursing? What is the single most important consideration you heard described in this presentation that would need to be incorporated into your process?