Tools for Assessing Course Learning Outcomes: Essay Prompts & Rubrics

Transcription

1 Tools for Assessing Course Learning Outcomes: Essay Prompts & Rubrics Amal BouZeineddine Associate Director Center for Teaching and learning (CTL) August 6, 2012

2 Rubrics Segment 2

3 The Learning Outcomes of this Session Segment We will : Describe a rubric; Analyze the different types of rubrics (descriptive, scoring, analytic, holistic); Design rubrics based on learning outcomes.

4 What is a Rubric? A rubric is a printed set of scoring guidelines (criteria) for evaluating work (a or a product) and for giving feedback. Generally, rubrics are put in the form of a chart with an x and y axis of criteria and an evaluative range or scale. Rubrics help guide the evaluator in giving feedback that is descriptive as well as evaluative.

5 Why are rubrics used? A good rubric tells the performer (student) and evaluator (teacher): By what criteria the work will be judged; What the difference is between good work and weaker work; How we can make sure our judgments (or scores) are valid and reliable; How both, performers and judges, can focus their preparation on excellence. Rubrics may help eliminate comments such as I wasn't sure what you wanted. or I don't see why I got a "B' instead of a 'A'.

6 Rubric Template Rubric Template (Describe here the task or that this rubric is designed to evaluate.) Beginning 1 Developing 2 Accomplished 3 Exemplary 4 Score Stated Objective or Performance Stated Objective or Performance Stated Objective or Performance Stated Objective or Performance Description of identifiable characteristics reflecting a beginning level of. Description of identifiable characteristics reflecting a beginning level of. Description of identifiable characteristics reflecting a beginning level of. Description of identifiable characteristics reflecting a beginning level of. Description of identifiable characteristics reflecting development and movement toward mastery of. Description of identifiable characteristics reflecting development and movement toward mastery of. Description of identifiable characteristics reflecting development and movement toward mastery of. Description of identifiable characteristics reflecting development and movement toward Description of identifiable characteristics reflecting mastery of. Description of identifiable characteristics reflecting mastery of. Description of identifiable characteristics reflecting mastery of. Description of identifiable characteristics reflecting mastery of. Description of identifiable characteristics reflecting the highest level of. Description of identifiable characteristics reflecting the highest level of. Description of identifiable characteristics reflecting the highest level of. Description of identifiable characteristics reflecting the highest level of.

7 Sample Levels

8 Example: Computer Programming Grading Rubric Trait Exceptional Acceptable Amateur Unsatisfactory Specifications The program works and meets all of the specifications. The program works and produces the correct results and displays them correctly. It also meets most of the other specifications. The program produces correct results but does not display them correctly. The program is producing incorrect results. Readability The code is exceptionally well organized and very easy to follow. The code is fairly easy to read. The code is readable only by someone who knows what it is supposed to be doing. The code is poorly organized and very difficult to read. Reusability The code could be reused as a whole or each routine could be reused. Most of the code could be reused in other programs. Some parts of the code could be reused in other programs. The code is not organized for reusability. Documentation The documentation is well written and clearly explains what the code is accomplishing and how. The documentation consists of embedded comment and some simple header documentation that is somewhat useful in understanding the code. The documentation is simply comments embedded in the code with some simple header comments separating routines. The documentation is simply comments embedded in the code and does not help the reader understand the code. Delivery The program was delivered on time. The program was delivered within a week of the due date. The code was within 2 weeks of the due date. The code was more than 2 weeks overdue. Efficiency The code is extremely efficient without sacrificing readability and understanding. The code is fairly efficient without sacrificing readability and understanding. The code is brute force and unnecessarily long. The code is huge and appears to be patched together.

9 Example: Rubric for Class Participation

10 Exercise 1 Let s design our first Rubric draft: Select a course learning outcome (CLO) Identify the traits/dimensions/ criteria in the CLO which the rubric will be assessing Draw the Rubric X axis and Y axis Fill out the Criteria column Fill out the evaluation levels/range Choose one trait/dimension/ criterion Fill out the description cells in the evaluation range for the selected criterion Make sure the description is observable and measurable Share your description with a colleague at your table

11 What are the types of Rubrics? (1) Rubrics used for grading are often called "scoring rubrics" and can be considered a form of summative assessment. Rubrics can also be formative assessment. They can be very effective in improving self-assessment competence and skills such as peer-editing where consistency of judgment is critical.

12 Example 1 SCALE = 1 point - NOVICE 2 - APPRENTICE 3 - PROFICIENT 4 - DISTINGUISHED elements are in this column descriptors for lowest levels will be in this column this column represents what is the highest level you expect to see sample CLARITY & COHERENCE Sentence structure, word choice, lack of transitions and/or sequencing of ideas make reading and understanding difficult. Sentence structure and/or word choice sometimes interfere with clarity. Needs to improve sequencing of ideas within paragraphs and transitions between paragraphs to make the writing easy to follow. Sentences are structured and words are chosen to communicate ideas clearly. Sequencing of ideas within paragraphs and transitions between paragraphs make the writer s points easy to follow. In addition to meeting the requirements for a 3, writing flows smoothly from one idea to another. The writer has taken pains to assist the reader in following the logic of the ideas expressed.

13 Example 2

14 Example 3 Clinical Simulation Initial Steps 1. Prepares for patient contact ppe/handwashing 2. Introduces him/herself to the patient. 3 identifies his/herself to bedside nurse/doctor Advanced 4 pts Advanced Performs all initial steps without difficulty that are required in the skill listed. Proficient 3 pts Proficient Performs more than three initial steps required in the skill llisted. Basic 2 pts Basic Performs less than two initial steps required in the skill listed. Below Basic 1 pts Below Basic Does not perform any of the initial steps required in the skill listed Patient Assessment 1. Obtain information regarding patients history/ entrance complaint 2. Observe the patients vital signs displayed on the monitor. 3. check position and patencey of EET 4. Chest auscultation Advanced Recognizes, verbalizes, and demonstrates all assessment steps in a logical manner consistently without difficulty. Proficient Recognizes, verbalizes, and demonstrates the assessment steps in a logical manner 85% of the time during this check off. Basic Able to recognize, verbalize, and demonstrate the assessment steps in a logical manner 75% of the time during this check off. Below Basic Fails to recognize, verbalize, and/or demonstrate the assessment steps in a logical manner less than 50% of the time during this check off. Rationale Advanced Proficient Basic Below Basic Verbalizes relevant information relating to the skill being checked off on. Answers instructor's questions regarding the skill and its importance plus the impact to the client without difficulty. Able to verbalize relevant information relating to the designatedt skill that is being checked off on. Able to answer more than four questions regarding the skill and its importance re garding the skill and its importance plus the impact to the client. Limited ability to verbalize relevant information relating to the designated skill that is being checked off on. Able to answer less than three questions regarding the skill and its importance plus the impact to the client. Fails to verbalize relevant information relating to the skill being checked off on. Unable to answer instructor's questions regarding the skill and its importance and/or impact to the client. Final Steps & Lab Area Advanced Proficient Basic Below Basic Able to do all of the following: a.verbalize final steps. b. mention documentation of skill. c. mention client response to skill performed. d.clean up after the One of the following occurs: Fails to: a. verbalize final steps. b. mention documentation of skill. c.mention client response to skill performed. Two or more of the following occurs: Fails to: a.verbalize final steps. b.mention documentation of skill. c. mention client response to skill performed. Three or more of the following occurs: Fails to: a.verbalize final steps. b.mention documentation of skill. c.mention client response to skill performed.

15 What are the types of Rubrics? (2) Types Purpose/Distinction* Focal Use Holistic Analytic provide a single score based on an overall impression of learner achievement on a task. provide specific feedback along several dimensions To provide overall evaluation guidelines that clarify how grades relate to /achievement, such as in course grades To break assignments or scores down into separate components for grading (description, analysis, grammar, references, etc.)

16 Analytic Rubrics Analytic rubrics identify and assess components of a finished product. It divides a student s into essential dimensions (traits), and each dimension is judged separately. A separate score is given for each dimension or trait considered important for the assessed. PLO-GE/AUB: Apply drafting and editing strategies to compose work that reflects critical thinking and appropriate understanding of conventions of genre and usage.

17 Ideas Example criteria Exemplary Strong Satisfactory Needs Improvement Organization and Development The essay develops fresh insight that challenges the readers thinking. The essay demonstrates mastery of the subject, and uses appropriate, relevant and compelling content that shows the writers understanding and shaping of the work. The thesis or focus of the essay is made clear to readers. Evidence is relevant, accurate, well developed, and well integrated. Essay demonstrates detailed attention to successful execution of conventions, including organization, style and format of academic research. The essay uses appropriate and relevant content in most parts to illustrate good command of the subject. The essay demonstrates diligent work and solid preparation, but not outstanding originality. The thesis or focus of the essay is made clear to the readers. Evidence is relevant, accurate, well developed in most parts of the essay. Attention to execution of conventions, including organization, style, and formatting of academic research may be inconsistent. Main idea of essay may be obvious or unimaginative. Thesis or focus is somewhat vague. Essay demonstrates some knowledge of the subject and uses appropriate and relevant content inconsistently. Evidence is either irrelevant or loosely related to the thesis. It may be partially inaccurate, underdeveloped or not integrated. Essay demonstrates acceptable attention to conventions of academic research, such as organization, style, or format. Essay uses simplistic or irrelevant or inappropriate content in most parts of the work. Lacks originality and simply dwells on what is already common knowledge to readers. Essay lacks focus or purpose. Evidence is not relevant or not developed. Essay demonstrates little attention to conventions of academic research, such as organization, style, or format. Style Process Essay uses academic English skillfully to communicate meaning to readers with clarity, grace, and concision, in prose that is virtually errorfree. Research project as a whole (process and product) shows evidence and demonstrates excellent understanding of how to use composing strategies, such as drafting, revising, Essay uses academic English skillfully to communicate meaning to readers with clarity, though may include some errors. The research project as a whole (process and product) shows some evidence and demonstrates good understanding of how to use composing strategies, such Essay uses acceptable academic English to communicate meaning to readers with clarity, though includes many errors. Research project as a whole (process and product) shows little evidence and demonstrates basic understanding of how to use composing strategies, such as drafting, revising, and Essay uses language that many times impedes meaning, because of errors in usage. Research project as a whole (process and product) shows little or no evidence of development through revising and editing.

18 Rubric from Nursing

19 Rubric from Medical Lab Medical Assistant clinicl Lab 60 % Assessment of basic clinical skills Unsatisfactory 60 pts Does not meet requirements - Needs much more attention to detail Satisfactory 70 pts meets requirements Good 85 pts Level of expected Excellent 100 pts above level of expected Infection Control 10 pts Standard precautions in patient care Unsatisfactory Fails to wash hands <2minutes prior to 1st patient, clear all surfaces of contaminated materials, place contaminated material in waste can Satisfactory wash hands <2minutes prior to 1st patient, clear all surfaces of contaminated materials, place contaminated material in waste can Good Perform 2minute hand wash before 1st patient, patients, use of gloves in potential biohazards, clean all surfaces and dispose of contaminated materials, wash hands 1minute after patient exam Excellent Check all common surfaces upon arrival, perform 2 minute hand wash before seeing 1st patient-, use of proper PPE in potential biohazards, disinfect all surfaces and proper disposal of all contaminated materials and wash hands 1minute after patient exam Injections 15 pts Demonstrate each of the steps required in administering injections Unsatisfactory Failed to accurately calculate prescribed dosage. unable to site the six rights for drug administration. Did not follow safety procedures when handling equipment. Satisfactory calculated prescribed dosage accurately. Able to site the six rights for drug administration. Followed all safety procedures when handling equipment. Good calculated prescribed dosage accurately. Able to site the six rights for drug administration. Followed all safety procedures when handling equipment. Inserted needle at the correct angle for each of the injections demonstrated. Excellent calculated prescribed dosage accurately. Able to site the six rights for drug administration. Followed all safety procedures when handling equipment. Inserted needle at the correct angle for each of the injections demonstrated. Documents injection administration. CPR Skills 15 pts all procedure do with accordance with the guidelines of the American Heart Association Unsatisfactory Failed to perform all the Skills required to do Adult, child and infant CPR. failed the Written CPR Test Satisfactory Adequately performed all the required skills to do Adult, child and Infant CPR. Passed CPR written test with 85% or higher. Good Performed all the required skills to do Adult, Child and Infant CPR with 85% Accuracy. Passed CPR written test with 85% or higher Excellent Performed all the required skills to do Adult, Child and Infant CPR with 100% Accuracy. Passed CPR written test with 100%. Electrocardiogram 20 pts Demonstrate each step required in obtaining a standard 12 lead ECG Unsatisfactory unable to demonstrate the steps to obtain a standard 12 lead ECG. Could not distinguish any artifacts or how to correct them. Satisfactory Adequately performed :Checked provider's orders, prepared patient,demonstrated the steps to obtain a standard 12 lead ECG, corrected any artifacts, alerted Good Checked provider's orders, prepared patient,demonstrated the steps to obtain a standard 12 lead ECG, corrected any artifacts, alerted provider if any abnormalities were Excellent Checked provider's orders, prepared patient,demonstrated the steps to obtain a standard 12 lead ECG, corrected any artifacts, alerted provider if any abnormalities were

20 Holistic Rubrics The focus of a score reported using a holistic rubric is on the overall quality, proficiency, or understanding of the specific content and skills; it involves assessment on a unidimensional level (Mertler, 2001). Use of holistic rubrics can result in a somewhat quicker scoring process than use of analytic rubrics (Nitko, 2001). This is basically due to the fact that the teacher is required to read through or examine the student s product or only once, in order to get an "overall" sense of what the student was able to accomplish (Mertler, 2001). Only limited feedback is provided to the student as a result of scoring tasks in this manner. A template for holistic scoring rubrics is presented in Table 1.

21 Table 1: Template for Holistic Rubrics Score Description Holistic Rubric Template Demonstrates complete understanding of the problem. All requirements of task are included in response. Demonstrates considerable understanding of the problem. All requirements of task are included. Demonstrates partial understanding of the problem. Most requirements of task are included. 2 Demonstrates little understanding of the problem. Many requirements of task are missing. 1 Demonstrates no understanding of the problem. 0 No response/task not attempted.

22 Example Holistic Rubric for Essay Questions Response Criteria Rating Exemplary Competent Minor Flaws Nearly Satisfactory Fails to complete Unable to begin effectively Clarity of thought, Complete. Shows understanding of all processes, reasonable hypothesis or thoughtful questions, conclusions supportable by data, shows creativity, some graphic representation of data or concepts. Clarity of thought, shows understanding of major processes, includes good hypothesis or questions, draws acceptable inferences and conclusions, may have graphic representations. Completes the assignment, but explanations may be slightly ambiguous or unclear, may contain some incompleteness, inappropriateness, or unclearness in representation, hypothesis, understanding of processes, or conclusions. Begins successfully, but omits significant parts or fails to complete, may misuse scientific terms, representations may be incorrect or omitted, incorrect or incomplete in analysis, inferences and conclusions. Assignment and explanation is unclear, or major flaws in concept mastery, incorrect use of scientific terms, inappropriate or omitted hypothesis. Product does not reflect the assignment, does not distinguish what information is needed, restates the question without making an attempt at a solution. No attempt Does not begin assignment

23 Exercise 2 Let s go back to the draft rubric you developed in Exercise 1. As a group at each table, select one draft rubric from exercise 1 which the whole group will work on Select another criterion from the CLO selected in Exercise 1 Assign scores for the range/levels you have developed. As a group, fill out the description cells Put up the group s rubric on one of the room walls Gallery Walk: Move around and check out the other developed draft rubrics. Paste a star sticker on the rubric which is clear and assesses the CLO accurately.

24 Exercise 3 Let s develop a holistic rubric. Share your Rubric with a colleague at your table.

25 Summary: Developing a Rubric

26 Further Rubric Samples

27 Sample Rubrics

28 Further Rubric Samples

29 Some Rubrics Websites Rubric Samples Create Your Own Rubrics Online: RubiStar References and Guides to Rubric Development: For Guides to developing Rubrics in support of teaching, learning and self reflection: This Rubric reference page - Flashlight Resources - TLT Group assessment, evaluation and survey tools Flashlight Rubric Samples - Flashlight Online An online survey tool that supports rubric construction and assessment Scoring Rubrics Definitions & Construction Rubric Basics Definitions, types, purposes, learner involvement, learning enhancement and rubric use (Schreyer Institute for Innovation in Learning). Rubric Builder An interactive web page rubric that can score and give item specific feedback Rubistar Helps you construct online rubrics Roobrix Converts your rubric scores into percentages. Waypoint online, interactive rubrics that let you create tailored narrative feedback for students based on your rubric and, on a larger scale (multiple classes, programs, institution-wide) collect and analyze longitudinal data on student

30 Some of the Consulted References e_design_courses.pdf SearchBox&first=11&FORM=PERE 20Kinzie,%20McCormick%20&%20Nelson%20Laird.pdf an+ge_capstones+assessment.pdf

31 End of Segment 2

32