THE COLLEGE OF ST. CATHERINE MASTER S DEGREE PROGRAM NURSE EDUCATOR CONCENTRATION

1 THE COLLEGE OF ST. CATHERINE MASTER S DEGREE PROGRAM NURSE EDUCATOR CONCENTRATION NURS 699B/760: EVALUATION and EDUCATIONAL MEASUREMENT Winter Semester, 2005 I. COURSE NUMBER NURS 699/760 (with LINK to course description in academic catalog) II. III. COURSE TITLE: Evaluation and Educational Measurement COURSE CREDITS: 4 credits (3 Theory, 1 Practicum) IV. PLACEMENT IN CURRICULUM: Level III, Winter 2005 V. CLASS TIME: Wednesday, 5-9 PM VI. PLACE: Whitby 4 VII. VIII. COURSE FACULTY: Name, Office location, Phone #, email address COURSE DESCRIPTION This course examines curricular, program, course and individual student/patient evaluation. Test items that are free of ethnic and gender bias and constructed in a style understood by individuals whose native language is not English are studied. Assessment, evaluation and constructive feedback to students and colleagues are investigated. Evaluation concepts will be linked to Core Competencies for Nurse Educators (NLN, 2005). IX. COURSE OBJECTIVES: Upon completion of this course, students will be able to: 1. Evaluate the consistency of the personal philosophy of teaching learning with measurement strategies employed to assess learning outcomes (Critical Thinking, Role). 2. Reconstruct trends in evaluation and educational measurement and their impact on current nursing education (Critical Thinking). 3. Design measures to assess knowledge, psychomotor skills, clinical performance for diverse groups of learners (Phenomena of Nursing, Critical Thinking). 4. Analyze assessment measures for reliability and validity (Critical Thinking, Phenomena of Nursing). 5. Evaluate test blueprints for different levels of students (Critical Thinking, Therapeutic Nursing Interventions). 6. Evaluate classroom assessment techniques for utility and effectiveness (Critical thinking, role). 7. Propose a plan for assessing own teaching effectiveness that reflects personal philosophy of teaching/learning, is consistent with expected group of learners, and demonstrates a commitment to scholarship of teaching (Communication, Role). 8. Evaluate Evaluation Models for utility and effectiveness for educational programs for nurses (nursing) using criteria that reflect goals for evaluation (Critical Thinking, Systems). 9. Evaluate the strengths and limitations of research evidence for evaluation and educational

2 measurement in nursing education programs (Role). 10. Display attitudes of being a critically reflective nurse educator evaluator (Role, Critical Thinking). X. COURSE OUTLINE Class 1 February 2 nd No Class: Reflective assignments and interview 1. Write your personal philosophy of evaluation. This should be a 1-2 page paper. Here are some questions to consider when developing your philosophy. What is evaluation? When does it occur? Why is it done (purpose)? How does it differ from assessment? Is there a relationship between assessment and evaluation? If so, what is the relationship? Does evaluation facilitate: a) student learning and/or b) instructor performance? How and why? What do instructors do to help students learn through evaluation? Include in your philosophy statement, your beliefs about students, instructors, the relationship between the instructor and the student as adult learner. At the end of the course, you will submit your revised philosophy of evaluation with a brief discussion of how your philosophy has changed or developed during this course. As part of your revised paper, include any references that you have found helpful, insightful or meaningful in the development of your philosophy statement. 2. Interview an experienced nurse educator (academic, staff development, patient educator, etc) about the trends in evaluation that she/he has observed or experienced during her/his career as a nurse educator. Here are some questions you may wish to pose: Who does your interviewee evaluate? What does your interviewee evaluate? How has evaluation changed? Has there been a change in the emphasis, focus, or purpose of evaluation? As part of this assignment, you will give a brief 5-6 minute in-class presentation on your interview. Include historical perspective and changes in evaluation as experienced by your interviewee. Read one article on trends in educational evaluation and reflect upon how your interview results compare to the literature. During class, we will look for common or recurring themes and trends in the class presentations. We will also compare these trends to those reported in the literature. (week of 2/9) 3. Skim chapter 15: Evaluation and Healthcare Education in Barnstable (2003) Nurse as Educator. Write a short reflective paper addressing two questions: 1. What concepts in this chapter did you find useful for the course? or (What concepts in this chapter do you think will be useful in the course?) 2. What was missing in the chapter that you expected to find? Do you want this content covered in this course? 4. Read, Chapter 1: Advancing Evidenced-Based Teaching in Stevens, K. & Cassidy, V. (eds.). Evidence-Based Teaching: Current Research in Nursing Education. What evidence is available related to effective teaching? 5. Review standards for baccalaureate and associate degree education for behaviors expected for program graduates. Review ANA standards for behaviors expected for

3 professional nurses. Reflect on the implication of these standards for evaluation and measurement in nursing education and practice. Class 2 February 9 th Trends in evaluation and measurement in nursing education and nursing practice Overview of the course and rationale for course design Orientation to Evaluation Project Reports of reflections on Chapter 15 in Barnstable Reports of interviews with faculty Identification of key concepts in evaluation for course (i.e., criterion referenced, norm referenced, mastery learning, interrater reliability) Identification of standards that guide evaluation and educational measurement in nursing. Class 3 February 16 th Learning Theories and Implications for Evaluation Assumption to be critiqued in class: Learning theories inform how learning outcomes should be evaluated. 1. Skim Chapter 3 Applying Learning Theories to Healthcare Practice, by Braungart and Braungart in Barnstable. 2. Review Table 3-3 Summary of Learning Theories. Note assumptions about the learner for each of these theories and the educator s task. Note also the material related to transfer of learning. 3. Which of these learning theories are appropriate for associate and baccalaureate nursing education? For staff education? Are some more appropriate, and others, less appropriate? 4. Reflect on appropriate strategies to assess student learning as well as ability to transfer the learning to the clinical setting (student or practitioner) with each of these learning theories. Discuss assumptions about the learning, educator s task, and transfer of learning for the various theories in Chapter 3 of Barnstable (Table 3-3 Summary of Learning Theories). Discuss appropriate strategies to assess student learning and strategies to facilitate transfer of learning to the clinical setting (student or practitioner) with each of these learning theories. Consider the validity of the assumption that learning theories inform how learning outcomes should be evaluated and discuss this assumption. Class 4 February 23 rd Evaluation of Knowledge Readings: Read a recent article on cognitive testing. There was a series of articles in Nurse Educator, 24 (1999) written by A.Tomey and other nurse educators that you may wish to read. Bosher, S. (2003). Barriers to creating a more culturally diverse nursing profession: Linguistic bias in multiple-choice nursing exams. Nursing Education Perspectives, 24 (1), 25-34. Morrison, S, and Free, K. (2001). Writing multiple-choice test items that promote and measure critical thinking. Journal of Nursing Education, 40, (1), 17-32. Discuss and critique methods for evaluating knowledge, such as: true-false, multiple choice, multiple response, short answer or restricted response essay, matching, etc. Discuss the process of test plan development and review the NCLEX-RN test plan and look at other examples of test plans

4 Review, write and critique test items. Other topics for this session include: reliability and validity of test items, item analysis and use of the essay (or formal paper) as evaluation method. Discussion questions: Should students be given a detailed copy of the test blue print? Should students be given the opportunity to develop and use cheat sheets during exams? Bring to class: 1. Sample of lecture/course objectives and three to four test items that will evaluate these objectives. 2.example of a written assignment and answer key, if you have one. Class 5 March 2 nd Evaluation of psychomotor skills Discuss the process of skills evaluation, development of evaluation tools, critical components, interrater reliability, use of grading rubrics. Critique selected skills checklists. Develop a skills checklist for evaluating performance in either the skills lab environment or the professional work environment. Bring to class: a skills checklist or description of a skill that you teach, a copy of a skills grading criteria (rubric) that you have used. Class 6 March 9 th Evaluation of affective domain Readings: Martin, B. (1989). A checklist for designing instruction in the affective domain. Educational Technology, 29 (8), 7-15. Read one of the following: Cook, P. and Cullen, J. (2003). Caring as an imperative for nursing education. Nursing Education Perspectives, 24 (4), 192-197. Cook, P. and Cullen, J. (2000). Diversity as a value in undergraduate nursing education. Nursing Education Perspectives, 21 (4), 178-183. Identify the behaviors/values/beliefs that may be included in the affective domain. Discuss Krathwohl, Bloom and Masia s taxonomy of the affective domain and focus on questions related to evaluation of this domain. Review and discuss the following professional documents for affective behaviors: Code of Ethics for Nurses with Interpretive Statements, Nursing: Scope and Standards of Practice, Core Competencies of Nurse Educators. Discussion question: How do we develop inter-rater reliability in evaluating the affective domain? Class 7 March 16 th Evaluation of clinical Readings: Read on clinical evaluation. Kathleen N. Bondy has written several articles on this topic. Wooley, G., Bryan, M., and Davis, J. (1998). A comprehensive approach to clinical evaluation. Journal of Nursing Education, 37 (8), 361-366. Discuss what is included in the process of clinical evaluation. Analyze several clinical evaluation tools and identify similarities and differences among tools. What are there underlying assumptions in these tools? Discussion questions: Should clinical evaluation yield a grade (A, B, C, etc) or S/U? What role does student self-evaluation play in clinical evaluation?

5 Is the amount of detail included in a clinical evaluation tool worth the faculty effort? Does the perspective of the facility personnel (nursing staff, preceptor, physician, other health professionals) have a role or impact on the evaluation of student s clinical performance? Bring to class: a clinical evaluation tool for discussion (can be one discovered in the literature or one used in clinical) March 23 rd SPRING BREAK Class 8 March 30 th Ethical and Legal Aspects of Evaluation Select one of the following topics related to evaluation of students. Review the literature on the issue (3-5 references). Develop a 5-minute oral presentation on the issue. In your presentation, discuss what is the problem, who is harmed and why, what are the consequences. Establish a philosophical link to models of evaluation (principles) and at least one learning theory. Topics: Academic misconduct: Cheating on exams Plagarism Unsafe clinical performance: Use of alcohol, drugs in clinical setting Unsafe patient care Non-professional behavior: In the classroom In the clinical setting In interactions with faculty, peers Dress Violations of HIPPA Student privacy (Family Educational Rights and Privacy Act) Student appeal process and due process Cultural bias: In test construction In evaluation of clinical performance Gender bias: In evaluation of clinical performance Grading policies: Grade inflation Presentations on ethical and legal aspects Discussion of ethical and legal aspects Class 9 April 6 th Evaluation of Teaching Select one of the following evaluation techniques or another technique from the literature or from your personal experience and develop a 5-minute oral presentation on the technique. Review the extant literature, including any research on the technique. In your presentation, describe the technique and advantages and disadvantages. Give at least one example of its use in nursing education. Discuss how the technique reflects learning theory (ies). Muddiest point 1-minute paper Poster

6 Student self-evaluation Pre-test/post-test Videotaping Journaling Concept mapping Written assignment Role play Pro/Con grid Debate Investigate tools used by students and faculty to evaluate a faculty member s teaching. Reflect on the following questions: 1. What criteria are used to develop questions to evaluate the effectiveness of a faculty members teaching? Are the criteria the same for students and faculty? 2. How does a faculty member assess the effectiveness of his/her teaching? What is the purpose of peer evaluation? 3. How can data from self-reflection, students, and peers be used to improve one s teaching? 4. Is there a standard for acceptable teaching (i.e., acceptable mean scores)? Presentations of Classroom Assessment Techniques Discuss forms for Student Evaluation of Faculty Teaching (classroom and clinical) Discussion of peer evaluation of teaching Discussion of how to use data to improve teaching Class 10 April 13 th No Class-Clinical Practicum Class 11 April 20 th Course Evaluation During this class evaluation of courses by students and faculty will be discussed. Bring to class examples of course syllabi and tools used to evaluate courses by both students and faculty. Reflect on the following questions: 1. What must a student demonstrate to pass the course? If there are different components of the course that must be passed (i.e., theory and clinical), what is the rationale for demonstrating satisfactory completion of all components? 2. Is there a standard for students to pass a course at the College? If so, what is the rationale for that standard? If not, what is the rationale for having different standards? Can different courses within a department (program) have different passing levels? If so, why? If not, why not? 3. What course components are best evaluated by students? Which are best evaluated by faculty? 4. What is the value of having a standardized course evaluation form for all courses? Should there be a standard (i.e., mean value score on certain or all questions) for a satisfactory course at the department or college? Discuss components of course evaluation Discuss value of student and faculty input Discuss value of standardized formats for course evaluation Using data to improve course delivery Discuss value of student and faculty input Discuss value of standardized formats for course evaluation Using data to improve course delivery Explore the following assumption: Different performances can be added and averaged to yield final results in a course or program. Discussion question: What is considered a passing grade in a course and why? Class 12 April 27 th Curriculum Evaluation

7 During this class, strategies to evaluate the curriculum will be discussed. Students will examine reports that provide data about how students perform on various aspects of the curriculum in comparison to other groups of students. Examples of these reports are results from the HESI Exit Exam, and the NCLEX Program Reports. Select two three readings from the following list: 1. Launcher, K., Newman, M., & Britt, R. Predicting licensure success with a computerized comprehensive nursing exam: The HESI Exit Exam. Computers in Nursing, 17 (3), pp. 120-125. 2. Newman, M., Britt, R, & Lauchner, K. Predictive accuracy of the HESI Exit Exam: A follow-up study. Computers in Nursing 18(3), pp. 132-136. 3. Morrison, S., Free, K., & Newman, M. Do progression and remediation policies improve NCLEX-RN pass rates? Nurse Educator 27(2), pp 94-96. 4. Nibert, A. & Young, A. A third study on predicting NCLEX success with the HESI Exit Exam. Computers in Nursing 19 (4), pp. 172-178. 5. Nibert, A. Young, A., & Adamson, C. Predicting NCLEX success with the HESI Exit Exam: Fourth annual validity study. Computers in Nursing, 20 (6), pp. 261-267. 6. Morrison, S. Recommendations for improving NCLEX-RN pass rate. (handout). Reflect on the following questions: 1. Review HESI Exit Exam and NCLEX Program Reports. What are the components evaluated in these standardized reports? Are there similarities and differences in the components between HESI Exit Exam and NCLEX Program Reports? What is the value of having standardized reports (parameters) to evaluate curriculum? Are the data from these reports reliable and valid? 2. Reflect on how data from these reports would be used to make changes in the curriculum? What would be the best process to use to make curricular changes based on this data? Discuss standardized program reports for evaluating curriculum Evaluate reliability and validity of standardized reports Discuss how to use data from standardized reports to improve curriculum Discussion question: Can different courses in a curriculum have different standards for passing a course? Can one course s C be a higher-level course s D? Class 13 May 4 th Program Evaluation Program evaluation will be the focus of this class. Bring a copy of a program evaluation plan to class. Read the following: 1. Chapter 2: Evaluation s Basic Purpose, Uses, and Conceptual Distinctions in Worthem, B., Sanders, J., & Fitzpatrick, J. (1997), Program Evaluation: Alternative Approaches and Practical Guidelines (2 nd ed.). New York, NY: Longman. 2. Association of American Colleges (1992). Program Review and Educational Quality in the Major: A Faculty Handbook. 3. Chapter 4: Intended Process Uses In Utilization-focused Evaluation (3 rd ed.). Thousand Oaks, CA: Sage. 4. Minnesota Board of Nursing Rules, 2003. Reflect on the following questions:

8 1. What is the purpose of engaging in program evaluation? Why is program evaluation performed? What is the value of using a model for program evaluation? What are important components of a useful model? What would be criteria to use in adapting a model for a program/department? 2. Review an existing plan for program evaluation. Was a model used for that plan? What are the components of the program evaluation plan? Why were these components selected? 3. How would the utility, effectiveness, and efficiency of an evaluation plan be assessed/evaluated? Discussion of Models for Program Evaluation Donabedian, 1966 (Structure, Process, Outcome) Stufflebeam s CIPP Model (Context, Input, Process, and Product) Evaluate program evaluation plan(s) Class 14 May 11 th Presentations of Evaluation Projects XI. TEACHING STRATEGIES/REQUIRED LEARNING ACTIVITIES: Teaching-learning activities in NURS 699/760 include ª discussion ª collaborative learning in small groups ª critical reflection and dialogue ª individually written assignments ª practicum Please refer to the College of St. Catherine (CSC) Department of Nursing Graduate Student Handbook, Expectations of Faculty and Student Regarding the Teaching-Learning Process for additional information. XII COURSE ASSIGNMENTS AND GRADING Weekly Journaling assignment The course faculty members have two assumptions about evaluation that we hope to validate or refute during the semester. Assumption 1: Evaluation/assessment is a critical component of the teaching role of the educator that influences the beliefs and actions of the educator. Assumption 2: Reading, discussing and reflecting upon the process and experience of evaluation will clarify the evaluative role of the educator. You are expected to write a 1-2 page weekly journal entry focusing on some aspect of evaluation. The trigger for this writing may be something that you read, a classroom experience, an interaction with a student, a discussion with a colleague, your professional work or any other trigger that causes you to think about evaluation. For your first entry, you may use your comments on Chapter 15 in Barnstable as a trigger.

9 Week 9 Journaling assignment The topic for this week s journal is self-evaluation. Here are several questions to guide your reflective writing on the topic. What criteria do you use to evaluate your own performance as a teacher? Do you compare yourself to peers (norm-referenced), to published college/department standards (criterion-referenced), to the Core Competencies (criterion-referenced) or some other comparison? What impact do student evaluations, classroom assessments, peer evaluations, or course evaluation results have on your self-evaluation? Evolving Course Bibliography One of the products to be developed in the course is a useful bibliography on evaluation. We would like you to routinely submit an APA formatted reference of articles, books, etc that you read or viewed in preparation for class. Please indicate the focus of the article on your reference, i.e. clinical evaluation, essay construction/evaluation, etc. Any annotations will be helpful but are not necessary. The faculty will keep a running list of the references that will be distributed to the course participants. Evaluation Project There are four components to this evaluation project. Please submit all components as one item during the last class of the course. The total project is worth 50 points. Component Pts Comments/Instructions 1. Summary of interview with nurse educator 5 This is your written summary of the interview you presented in class during Week 2. Include the trends that you identified from the class discussion and how your interviewee s experiences agreed or disagreed with the trends. Limit your summary to 2-4 pages. Include your APA 2. Ethical or Legal issue in evaluation 3. Classroom assessment technique 4. Independent evaluation project formatted reference(s). 5 This is your written summary of the ethical or legal issue that you presented in class during Week 8. Limit your summary to 2-4 pages. Include your APA formatted references. 5 This is your written summary of the classroom assessment technique that you presented in class during Week 9. Limit your summary to 2-4 pages. Include your APA formatted references 35 Design, implement and critique an evaluation of student/learner learning. This project could be the development of a test plan with test questions; a case study with questions, grading rubric and ideal response; a skill evaluation form with grading rubric; a clinical assessment tool or some other evaluation tool. You may wish to begin by consulting with a teaching colleague, or supervisor about the existence of a current evaluation activity that needs to be developed, revised or refined. Once you have designed the tool to your satisfaction, test it out on a sufficient number of subjects to be able to evaluate the tool s performance. Analyze the results of your evaluation tool. Oral presentation of your project will occur during the final session of the course.

10 Component Pts Comments/Instructions Compon Points Comments ent Design 10 Incorporate the following in design: relevant course content, purpose of the tool, who will be evaluated by it, description of tool, directions for use, etc. What do you hope to accomplish by developing this evaluation tool? What do you want to measure? Impleme ntation 7 Describe conditions of use, group evaluated, time to complete evaluation, Analysis 10 Analyze the results. How effective was your tool. What would you change? Was the tool appropriate for the level of ability of the user? What impact will the use of this tool have on: a. other components in course; b. other courses in curriculum; c. faculty use of time/workload; d. other. Referenc es 3 Include relevant references. Use APA format in citations and bibliography. Oral presentat ion 5 You will have 20 minutes to present. You may develop handouts, PowerPoint or use any other supporting material for your presentation. Allow time for questions. Total points 50 Students will present their evaluation projects during the last class period. Name of Assignment Points % Theory Assignments: Reflective Journal 25 points 25% Philosophy of Evaluation 25 points 25% Evaluation Project** 50 points 50% Total Points Possible 100 points 100% ª The Evaluation Project will be used as a formative measure for Level III in the Nurse Educator Concentration to evaluate student achievement of level objectives for all curricular threads. Grading Policy: All assignments must be completed satisfactorily to pass the course. Grading Scale: The grading scale for NURS 699/760 is found in Academic Procedures: Grades in LeGuide. Grades are determined as follows: A 93 and over B+ 87-89 C+ 77-79 D+ 68-69 A- 90-92 B 83-86 C 73-76 D 66-67 B- 80-82 C- 70-72 F 65 or below

11 XIII. COURSE POLICIES The following college policies relate to this course: Academic Integrity: The Academic Integrity Policy in the Graduate Academic Catalog: 2004-2006 (pp. 22-23). Attendance: Class attendance is expected. Please refer to the College of St. Catherine (CSC) Department of Nursing Graduate Student Handbook for further information. Accommodations: LeGuide: Services for Students with Disabilities XIV. PROFESSIONAL STANDARDS, ABILITIES, AND COLLEGE REQUIREMENTS Professional Standards: In the Nurse Educator Concentration of the, these professional standards include: ª Code of Ethics for Nurses with Interpretive Statements (ANA, 2001) ª Accreditation Manual for Post Secondary and Higher Degree Programs in Nursing and Interpretive Guidelines by Program Type (NLNAC, 2004) ª Nursing s Social Policy Statement, 2nd Edition (ANA, 2003) ª Core Competencies of Nurse Educators (NLN, 2003 [Proposed]) ª The Essentials of Master s Education for Advanced Practice Nursing (AACN, 1996 [Graduate Core Curriculum Elements]) ª Nursing: Scope and Standards of Practice (ANA, 2004; including Nursing Role Specialist measurement criteria) ª The essentials of baccalaureate education for professional nursing practice (AACN, 1998). ª Educational competencies for graduates of associate degree nursing programs. (NLN, 2000). In NURS 699/760, the following standards and abilities are specifically addressed in required learning activities and course assignments: Core Competencies of Nurse Educators The Essentials of Master s Education for Advanced Practice Nursing (Graduate Core Curriculum Elements): Nursing: Scope and Standards of Practice (including Nursing Role Specialist measurement criteria): XV. TEXTBOOKS AND COURSE MATERIALS Required Textbooks/Readings: Bastable, S. (2003). Nurse As Educator, 2 nd Edition. Boston: Jones & Bartlett Association of American Colleges (1992). Program Review and Educational Quality in the Major: A Faculty Handbook. NLN, (2005). Core competencies for nurse educators. New York: Author. Recommended Textbooks: Angelo, R. A. & Cross, K. P. (1993). Classroom Assessment Techniques: A Handbook for College Teachers (2 nd ed.). San Francisco: Jossey-Bass.