American Academy of Physician Assistants. Self-Assessment Design Guide for Live CME Activities

Transcription

1 American Academy of Physician Assistants Self-Assessment Design Guide for Live CME Activities American Academy of Physician Assistants 11/13/2014

2 Table of Contents Considerations and Requirements... 2 Best Practices... 2 Guidelines for Writing Self-Assessment Questions... 3 General Guidelines for Question Writing... 4 General Guidelines for Writing Multiple-Choice Questions... 5 General Guidelines for Developing Clinical Vignettes... 6 Acknowledgement Appendix Appendix 1. General Category 1 CME Eligibility Criteria Appendix 2. General Self-Assessment Eligibility Criteria Appendix 3. NCCPA Content Blueprint Physician Assistant Practice Task Areas and Sample Diseases/Disorders by Organ Systems Appendix 4. Important Information Regarding Plagiarism and Copyright Infringement

3 Considerations and Requirements 1. A live, in-person self-assessment session must comply with all General Eligibility Criteria for Category 1 CME (Appendix 1) and General Eligibility Criteria for Self-assessment activities (Appendix 2). 2. A live, in-person self-assessment session must provide learners with a structured opportunity to systematically assess their knowledge and skills in a defined area of practice. 3. Systematically assessing knowledge with regards to a particular case, disease, disorder or organ system implies the opportunity to assess knowledge across the spectrum of NCCPA Content Blueprint Physician Assistant Task Areas (Appendix 3). 4. The self-assessment questions should serve as a stimulus for learning and any remarks from the moderator should address the rationale for the right answer. 5. The presence of questions in the context of a lecture alone does not make a lecture qualify as selfassessment. 6. Collecting responses: A live self-assessment CME session should be participatory and afford individuals the opportunity to assess their knowledge. As such, some form of mechanism should be used to engage learners. Mechanisms include: a. Audience Response System (e.g. b. Colored Cards: Learners will receive different colored cards where each color represents a response (e.g. A, B, C or D) and as the moderator reads the questions, learners are encouraged to use these cards to respond to the question. c. Graded Answer Sheets: This is a two-part activity, where moderators will first administer self-assessment questions, grade them, and provide feedback during the second part of the activity. d. Small Group Discussions: Moderators provide time for small group discussion regarding selfassessment questions and allow groups to report out their responses. 7. Because self-assessment is intended to guide development of a PAs future learning, some form of tangible takeaway must be available for learners. a. Example: Answer sheet for learners' to record their responses and track their areas for improvement to help direct their future learning. b. Example: Audience Response System report provided digitally to the participant to help direct their future learning. Best Practices Allow participants to discuss the question and possible answer in small groups or pairs if time and space permits In addition to the live self-assessment activities, include the following which could be eligible for Category 1 CME credits: Pre-work in the form of answering questions in advance of the session, and Post-work in the form of recording individual answers after the session. Note this usually requires the use of some sort of learning management system to facilitate the pre-work and post work, although the provider may opt for manual methods of scoring if desired. 2

4 Guidelines for Writing Self-Assessment Questions The purpose of self-assessment is to allow learners to systematically review their knowledge and skill in a given area of practice, identify gaps, and to remediate the identified gaps through the use of a rationale or other materials provided as part of the learning activity. Although self-assessment can take a number of forms, the most common format involves a series of multiple-choice questions supported by a rationale for the correct answer and other supporting information. Learners generally engage in self-assessment in an open book format. Self-assessment activities usually focus on assessing a particular area of medicine or practice and cover knowledge and skills that are considered foundational to successful practice. Developers of self-assessment activities should be guided by the Competencies for the PA Profession in selecting material for assessment to ensure that self-assessment is relevant to PA practice and the regulatory and requirements for Certification Maintenance, state licensing, and institutional privileging. More information on competencies and competency assessment may be found here: Self-assessment is NOT intended to be a test of the ability to recall trivial information, nor should it focus on the recall of factual information alone. At its best, self-assessment should help a PA to assess his or her ability to apply basic concepts from a particular area of medicine or surgery in a realistic clinical context. While a good fund of knowledge is foundational, it is the ability to use what one knows productively that matters most. Self-assessment test developers may feel the need to test for the most important foundational knowledge. However, they are urged to assess the application of knowledge whenever possible. A self-assessment question generally consists of the following parts: A Stem (the question) The correct answer Alternative answers A detailed rationale References to other supporting literature Most self-assessment questions are preceded by a clinical vignette or case. The vignette helps to place the issue into a meaningful clinical context. The added clinical context also aids the question writer in ensuring that the question involves the use of knowledge rather than simple recall of facts. When preparing self-assessment questions it is important to avoid plagiarism and other practices that might infringe on the rights of others. When developing self-assessment questions, be sure that the end product is clearly your work rather than a lightly edited borrowing of the work belonging to someone else. Appendix 4 helps clarify what constitutes plagiarism and/or copyright infringement. 3

5 General Guidelines for Question Writing Identifying Content Before a writer begins to develop questions, he or she must first determine what is important in the given area. It is useful to develop an outline. The outline may follow any number of themes such as the following: Diseases or conditions Diagnostic criteria for diseases or conditions Treatment of diseases or conditions Important research Practice guidelines Patient safety issues The following are criteria that are helpful in evaluating material to be used in a self-assessment question: Is it important to PA practice in this area? Is the topic realistic and relevant? Is the topic noncontroversial? Is the topic of universal concern in this area of practice? Is the topic related to one or more of the PA Competencies? Is the information so new that it has not yet been fully vetted by the medical community? Is the topic now so old as to be considered obsolete or of marginal interest in this area of practice? Recall and Reasoning Skills When possible, questions should test reasoning rather than mere recall of isolated facts. It is helpful to write a brief vignette (1-5 sentences). Questions associated with vignettes are more likely to test reasoning and are more likely to approximate the kinds of problems that occur during practice. It is recommended that vignettes include some or all of the following information: Type of patient(s) Type of clinical or administrative setting Specific clinical or administrative problem Relevant principles, regulations or statutes Initial steps, results or complications (if any) A question that asks for the most appropriate solution or response 4

6 General Guidelines for Multiple-Choice Questions 1. Focus on a single, clearly defined topic. 2. Exclude irrelevant information (avoid window dressing, red herrings, and teaching statements ). 3. Test reasoning, not mere recall of isolated facts. Do not quote from textbooks. 4. Do not use abbreviations, eponyms, and unnecessary complexity. 5. Do not use may and may happen. 6. Do not use indefinite quantifiers, such as: frequently, often. 7. Do not use absolutes, such as: never, none, no, all, always, every. Traditional Questions A traditional question consists of a stem, the correct answer, and three or four options. A well-written question can be answered before reading the options. Sample: Which of the following cranial nerves emerges from the brain stem anteriorly between the pons and medulla oblongata? *(A) Abducens (B) Accessory (C) Facial (D) Oculomotor (E) Trigeminal One useful technique for writing questions takes the following steps: 1. Choose a topic 2. Write an affirmative statement about the topic 3. Turn the statement into a question 4. Supply the answer 5. Supply alternative answers that are plausible, similar in length, and grammatically parallel Here is how it would look applied to the above question. Topic: Cranial nerves Statement: The abducens emerges from the brain anteriorly between the pons and medulla oblongata. Question: Which of the following cranial nerves emerges from the brain stem anteriorly between the pons and medulla oblongata? Answer: Abducens Alternatives: Accessory Facial Oculomotor Trigeminal 5

7 General Guidelines for Developing Clinical Vignettes A typical clinical vignette includes the patient s chief complaint, age, gender, some initial history, physical examination, and/or laboratory findings. The vignette should be realistic as well as concise (excluding irrelevant information). The vignette is usually followed by a question such as those listed below: Which of the following is the most likely diagnosis? Which of the following is the most appropriate next step? Which of the following is the best explanation for these findings? Which of the following is the most appropriate next diagnostic study? Which of the following drugs is most likely to be effective in managing his/her symptoms? Which of the following is the most appropriate next step in managing this patient? Which of the following is the most important additional question to ask the patient? Recall Versus Application of Knowledge Recall items require learners to remember, identify or recognize isolated facts. Application of knowledge items requires learners to recall relevant information, interpret the recalled information and apply the information to a new situation. Application of knowledge items are more likely to assess what is required for practice and differentiate between experts and novices. Components of a Clinical Vignette For items dealing with clinical issues, it is most useful to include a brief vignette (one to five sentences). It is recommended that clinical vignettes be structured as case presentations, including some or all of the following information: Patient age and gender (e.g., a 45-year-old man) Site of care (e.g., comes to the emergency department) Presenting problem and its history; signs and symptoms and their duration (e.g., because he/she has a headache) Duration (e.g., that has continued for the past two days) Relevant medical history Results of physical examination and/or diagnostic studies Initial treatment, subsequent findings, etc.... Vignettes should be brief and include only a subset of this information in the order specified above. There may be situations in which a longer vignette is more appropriate. Vignettes are often followed by questions about: The most likely diagnosis The most appropriate test or procedure The most likely additional finding The most appropriate first or next step in management The prognosis 6

8 Recall Item: Which of the following areas is supplied with blood by the mesenteric artery? Application of Knowledge Item: A 70-year-old man with atrial fibrillation had sudden onset of severe abdominal pain four hours ago. Physical examination shows abdominal distention and tenderness. Hemoglobin level is 8.7 g/dl, white blood cell count is 30,000/mm3 with a shift to the left, and serum amylase level is 500 U/L. Which of the following is the most likely diagnosis? Vignette Worksheet 1. Patient s age and gender 2. Site of care 3. Chief complaint/presenting symptoms (please include nature of onset) 4. Duration of symptoms (please include changing nature of symptoms) 5. Pertinent history (personal/family) 6. Examination findings 7. Pertinent labs 8. Lead-in statement 9. Option set with answer indicated (A) (B) (C) (D) Guidelines for Question Stems 1. Ask a clear, focused question. 2. Do not write multiple true-false questions, such as: Which of the following statements about X is true? 3. Use superlatives, such as most or best. 4. Use positively-worded stems. Guidelines for Answer Options 1. Create grammatically parallel options. 2. Ensure one correct answer. 3. Use options similar in length and complexity. 4. Do not use all of the above. 5. Do not use none of the above. 7

9 6. Do not ask examinees to make finer distinctions than required in practice. 7. Use mutually exclusive options. 8. Lengthen the stem to shorten the options. 9. Use common errors to create distracters. 10. Create plausible distracters. 11. Adjust the homogeneity of options to increase or decrease difficulty. 12. Use single rather than double phrases. Avoiding Clues 1. Avoid including clues in the stem (key words in the stem that are also in the answer). 2. Avoid pairs of similar options. 3. Avoid opposite options. 4. Avoid other logical clues. 5. Avoid convergence clues. Using a Rationale to Support Learning The point of self-assessment to provide a format where learners are able to a) assess their knowledge in a given area of practice and identify gaps or areas of weakness, and b) remediate identified areas of weakness by engaging in additional learning. Providing a strong rationale is an essential element in facilitating learning through self-assessment. A good rationale has the following elements: An explanation of why the correct answer is correct. Summarize any pertinent factual information or research that supports the correct solution and the original rationale for asking the question (why is the issue relevant and important). Explain why the incorrect answers are wrong, taking care to address common errors References for further reading should the learner feel that he or she needs to engage in learning related to the topic that goes beyond what the rationale provides Some Considerations for Administering Self-Assessment Developers of Self-Assessment CME activities have a number of options to consider in how they will administer the activity. The following are among the options: Distribute as a physical publication: Many providers distribute self-assessment materials in a booklet. Sometimes the answers and rationale are on the same page as the question, sometimes they are provided in a separate booklet. To receive credit, most providers require that the participant submit an evaluation form, and sometimes the answer sheet. Some providers do not provide the answers and rationale until the answer sheet has been returned and graded, and generally allow for a 2 nd attempt if a minimum passing grade is required. While individual providers has some discretion in making these choices, AAPA recommends the open book approach where learners have immediate access to the answer key and rationale. Facilitate as an online publication: Many providers administer self-assessment in an online format using a testing engine. This allows for immediate scoring and provision of feedback and rationale to the learner. If the provider requires a minimum passing score, usually more than one attempt is allowed before recording a final answer. 8

10 Facilitate in the context of a live meeting: Self-assessment can be successfully incorporated into the context of a live meeting. This introduces a social learning environment that draws upon the knowledge of a group of people. To make this work, the provider needs a good bank of questions, and a moderator who is familiar with the entire question bank and the rationale for each question. These sessions are most effective when an audience response system is used. However, providers do not need to record individual scores. Documentation of attendance is sufficient for awarding credit, and credit will be designated according to the time assigned for the session on the activity agenda. 9

11 Acknowledgement This guide is adapted from the NCCPA Item Writing Guidelines issued in September 2012 with their permission. 10

12 Appendixes Appendix 1. General Category 1 CME Eligibility Criteria 1. Content must have a direct bearing on patient care 2. Have a direct bearing on PA s ability to deliver patient care 3. Activities that improve PA core competencies a. Medical knowledge b. Interpersonal and communications skills c. Patient care d. Professionalism e. Practice based learning and improvement f. Systems based care 4. Relate to selected non-clinical topics, including, but not limited to the following: a. Teaching, ethical, medico-legal and/or social responsibilities of the PA b. Faculty development c. Professional and/or academic leadership d. Provider-patient relations e. Prepare physician assistants for their role as patient advocates to include advocacy on patient-centered topics like access to care. Appendix 2. General Self-assessment Eligibility Criteria 1. Must comply with AAPA CME Standards and Standards for Commercial Support of CME Activities. 2. Meaningful - Must have a useful purpose or important goal that relates specifically to PA practice. 3. Practical - Should improve PAs' ability to be effective in their practice. 4. Where possible, should be harmonized with other boards of medicine. 5. Where possible, should reinforce and allow for team-based collaboration with physicians and other members of the healthcare team. 6. Self-assessment CME activities will be evaluated based on effort rather than the number of questions in recognition that the amount of effort, thought and analysis is more meaningful than the number of questions alone. Questions can vary tremendously in complexity; therefore, the number of questions is a minor part of the criteria. 7. As a method of formative assessment, self-assessments may be taken as often as the individual finds this beneficial to learning. This is not to be confused with tests that can only be taken one time. Appendix 3. NCCPA Content Blueprint Physician Assistant Practice Task Areas and Sample Diseases/Disorders by Organ System Tasks: 1. History Taking & Performing Physical Examinations 2. Using Laboratory & Diagnostic Studies 3. Formulating Most Likely Diagnosis 11

13 4. Health Maintenance 5. Clinical Intervention 6. Pharmaceutical Therapeutics 7. Applying Basic Science Concepts Organ System: 1. Cardiovascular 2. Dermatologic 3. EENT(Eyes, Ears, Nose and Throat) 4. Endocrine 5. Gastrointestinal/Nutritional 6. Genitourinary 7. Hematologic 8. Infectious Diseases 9. Musculoskeletal 10. Neurologic System 11. Psychiatry/Behavioral 12. Pulmonary 13. Reproductive Appendix 4. Important Information Regarding Plagiarism and Copyright Infringement Plagiarism The Merriam-Webster Dictionary defines the word plagiarize as follows: to steal and pass off (the ideas or words of another) as one s own to use (another s production) without crediting the source to commit literary theft to present as new and original an idea or product derived from an existing source. (Definition: plagiarize. Merriam-Webster, Inc. Accessed June 19, 2014.) According to Plagiarism.org, all of the following are considered plagiarism: turning in someone else s work as your own copying words or ideas from someone else without giving credit failing to put a quotation in quotation marks giving incorrect information about the source of a quotation changing words but copying the sentence structure of a source without giving credit copying so many words or ideas from a source that it makes up the majority of your work, whether you give credit or not (Resources: what is plagiarism? iparadigms, LLC. Accessed June 19, 2014.) 12

14 Copyright Infringement The following information is from the United States Copyright Office: Who is an author? Under the copyright law, the creator of the original expression in a work is its author. The author is also the owner of copyright unless there is a written agreement by which the author assigns the copyright to another person or entity, such as a publisher. In cases of works made for hire, the employer or commissioning party is considered to be the author. What is copyright infringement? As a general matter, copyright infringement occurs when a copyrighted work is reproduced, distributed, performed, publicly displayed, or made into a derivative work without the permission of the copyright owner. (Frequently asked questions about copyright. U.S. Copyright Office. Accessed June 19, 2014.) Example of Plagiarism and Copyright Infringement Below is an example of what is considered plagiarism and copyright infringement. The following vignette was submitted by a test committee member: A 58 year old smoker is found to have malignant squamous cells in a sputum cytology specimen. Chest x-ray (CXR), computer tomography (CT) scans and bronchoscopy are all normal. Random biopsies of all segmental orifices reveal squamous cell carcinoma at the origin of the anterior segment of the right upper lobe. A right thoracotomy is performed, but no palpable tumor can be found. What is the next most appropriate course of action? A very similar question was found in The Annals of Thoracic Surgery, the official journal of The Society of Thoracic Surgeons and the Southern Thoracic Surgical Association: A 55-year-old smoker is found to have malignant squamous cells in a sputum cytology specimen. Chest roentgenogram, tomograms of both lungs, and bronchoscopy are all normal. Random biopsies of all segmental orifices reveal squamous cell carcinoma at the origin of the anterior segment of the right upper lobe. A right thoracotomy is done, but no palpable tumor can be found. The appropriate treatment would be: (Thoracic Surgery Quiz. Ann Thorac Surg. 1982;33: Published April Accessed February 5, 2012.) In short, changing the words of someone else s work does not make it your own and is against the law. 13