Quality Matters Distance Learning Course Review Process

Quality Matters Distance Learning Course Review Process I. CNM Distance Learning Course Review II. CNM: Quality Matters Distance Learning Course Review Process III. Team Roles and Responsibilities IV. Instructor Worksheet V. QM Rubric Template VI. Planned Course Improvements Worksheet

I. CNM Distance Learning Course Review Objective: The CNM Distance Learning (DL) course review process is designed to facilitate a standardized approach to the evaluation of distance learning courses at CNM using a peer reviewer model. In addition, the peer review process can assist faculty professional development and improve teaching. The QM peer review process will create an opportunity for constructive feedback and stimulate ideas for improvement in the online classroom. Discussions, before and after the evaluation, will emphasize the sharing of information and ideas to facilitate the ongoing improvement of distance learning courses. Procedure: DL staff/faculty who are certified in the Quality Matters Peer Review Process will conduct the evaluation using the Quality Matters Rubric as the evaluation tool. Peer Evaluators will review the course and complete the QM evaluation form and provide feedback to the DL faculty member. Constructive feedback, including instructional enhancement suggestions and/or recommendations will be communicated to the instructor. A score of 81/95 is necessary to meet the standards of the QM Rubric. Revisions to meet this minimum score are to be completed within 90 days from receiving QM score and feedback. The evaluator will invite faculty s comments, and request faculty s signature and date. Completed forms will be reviewed and signed by Dean or designee and documents will be filed with the faculty s home school. Process: Peer Review Evaluators consist of Distance Learning Staff/Faculty who are Quality Matters (QM) certified and have online teaching experience. Following the course review, the feedback that is generated is provided to the instructor with the following two components 1) scoring of the rubric that indicates the standards that were met and also the standards that were not met 2) feedback that provides the instructor with the strengths of the course, areas for improvement and our overall recommendations. Note: Each school is required to review "content" for rigor prior to or during the review, if peer reviewers have any questions, they will be designated to a Faculty member/chair to address the content question. If the course does not meet the Quality Matters score, the instructor would then be expected to address these concerns in order to teach or continue to teach the online course. Instructors may discuss review findings with the peer review team to gain greater clarity on needed revisions. The time frame for completing the revisions will be discussed between the instructor, peer reviewer(s) and chair. Depending on what needs to be completed the time frame can range from two weeks to 2 months. Once the revisions have been completed these areas will then be re-reviewed by the peer reviewer(s). Revised: January 29, 2013 2

Once a course passes the QM benchmark, the course should be re-reviewed every 2-3 years to be sure the course still meets quality standards. Quality Matters standards are revised every two years this is based on keeping standards up to date with current research in online education. The rubrics, along with the revisions will be signed by the peer reviewer(s), the instructor, and the Chair. The signed documents will be filed with the faculty s home school. Quality Matters Rubric There are eight general review standards 1) Course Overview and Introduction 2) Learning Objectives 3) Assessment and Measurement 4) Resources and Materials 5) Learner Interaction 6) Course Technology 7) Learner Support 8) Accessibility. Within these eight general standards there are 41 specific review standards. General review standards 2-6 are key components that must align (elements work together to ensure that students achieve the desired learning outcomes). For a course to meet quality expectations the course must receive a Yes on all 21 of the essential 3 point review standards. A minimum of 81 out of 95 points (85%) must be accumulated. 21 Standards are Essential each valued at 3 points for each standard 12 Standards are Very Important each valued at 2 points for each standard 8 Standards are Important each valued at 1 point for each standard Guidelines: Information about review and review process will be provided to DL faculty prior to initial review Identification of courses to be reviewed: rotational basis of each college/school (develop an observation cycle) Evaluators will provide objective feedback on DL course enhancement strategies, which focus on development. Revised: January 29, 2013 3

II. CNM: Quality Matters Distance Learning Course Review Process The following steps illustrate the main steps of the course review process. 1. Select Course Institutions decide to examine an online or blended course as part of a peer review. Since institutions make a significant investment in time, resources, and funds during a course review, QM suggests reviewing "mature" courses to maximize this investment. A mature course is one that has been held two or three times. However, the QM Rubric and tools also can be effectively used to review courses under development. As a suggestion, it is recommended that the academic schools begin by reviewing online courses that have been taught for at two or more semesters. Triggers for subsequent reviews may include: Faculty request Administration request New instructor Pending professional or accreditation review 2. Complete Online Instructor Worksheet The Instructor Worksheet is the voice of the course instructor and introduces the peer review team to his/her course philosophy, organization, technologies, and concerns. [Note: Instructor Worksheet will be provided to instructor prior to review.] 3. Master Reviewer/Team Chair Selects Review Team A member of DL Staff will be appointed as a Master Reviewer (MR) who serves as the Team Chair. The MR then selects the review team from a database of trained, available online faculty. The team chair is responsible for organizing all communication between members of the review team, including the faculty developer, and for keeping the review on track. 4. Hold Pre-Review Discussion The pre-review conference call between all members of the review team and the faculty developer is the best way to set expectations, coordinate, and communicate. The team agrees on the schedule for the review and learns how to access the course to be reviewed. 5. Individual Reviewers Complete Reviews Individual reviewers use Quality Matters Rubric as the tool to make decisions about whether this course meets standards at about the 85% level. 6. Conduct Post-Review Team Discussion The post-review conference between the review team members is necessary if a course does not meet expectations. These discussions are conducted electronically. 7. Final Review Report Submitted Once each peer reviewer completes his/her review and the team has met for a discussion, the report is submitted to the appropriate school. 8. Revise the Course if it Does Not Yet Meet the 85% Standard The faculty author works independently or with an DL Specialist to make the recommended changes. The Master Reviewer/Team Chair and the DL Specialist review the changes and determine if the course now meets the standards. Revised: January 29, 2013 4

III. Quality Matters Review Team Roles Faculty Course Developer Roles and Responsibilities Explanations Keep in contact 1. You are now part of a team! Communicate with your review team members as needed. Contact your Review Chair if you have questions, concerns, or suggestions. Set up a calendar 2. Work with your review team to develop a calendar for your peer course review. Complete Instructor's Worksheet Enroll Peer Course Review team members in your course Send login directions to your team 3. QM will email the Instructor Worksheet to you when your course is selected for review. Carefully fill out this Worksheet this is your opportunity to introduce your course to the review team. Send the completed Instructor Worksheet to QM Course Review Coordinator. 4. As the Faculty Developer, you must arrange for access by the Peer Review Team. You have some options, including: Copy your course (or ask the tech specialists at your institution to do so) so that the peer reviewers work in a copy rather than the "live" course. Enroll the peer reviewers in your course and begin the review before the end of the current semester. Enroll the peer reviewers in your course and begin the review after the semester ends. Use a completed section of the course from the previous semester if it's substantially the same as your current course. You can purge the specific student information and let the peer review team have free reign for the duration of the review period. 5. Send complete and easy-to-understand login directions to your Review Team (some of whom may not have used your particular course management system). Remember to send the course URL, the username and the password. Report any concerns 6. If your college is experiencing any technical problems that might make it difficult to access your course, please notify the Review Chair and team members. Also, if you notice any unexpected changes in your course or are concerned for any reason, please discuss it with your Review Chair. Dis-enroll Review Team members 7. On the agreed-upon date (or when access is no longer needed), please end access for your Peer Review Team to your course. Submit your Exit Interview 8. We really want to know about your experience as a Faculty Developer and your suggestions to improve our process. Submit your Faculty Response Form by the due date Thank You! 9. This form ensures that you know outcome of your course review and that you have input into the next steps. The peer review process isn't complete until we receive your form. 10. Thank you for participating in a Quality Matters Peer Course Review. Revised: January 29, 2013 5

Master Reviewer/Team Chair Master Reviewers (MRs) serve as authoritative guides to a course review team during the Quality Matters peer review process in the following ways: Roles and Responsibilities Ensure that the Quality Matters (QM) process is correctly implemented. Answer questions relating to the process and the application of the rubric. Anticipate problems during the review process and try to resolve them in an effective and timely manner. Know how to get help and information when it's needed. Explanations Master Reviewers will typically serve on teams for institutions that are just beginning the Quality Matters (QM) process and that don t have access to experienced Peer Reviewers from within their institutions. You will be bringing your experience and training as a QM Peer Reviewer to these institutions to ensure that the process works smoothly and is consistent with the QM principles. As a Master Reviewer, you can expect that you will be the most experienced member of the team. You will be the expert who can provide answers, guidance and know when and how to get needed help. Since the members of your team are inexperienced, questions about how to apply the standards and carry out the QM process are likely. You will guide the team through the various aspects of the review, striving for thoroughness and balance. While the master reviewer/team chair will guide the team through the QM process, Melissa Poole, QM s course review coordinator, can be reached at mpoole@qualitymatters.org to provide advice to the master reviewer/team chair. You will find information and documentation about the QM process on the Master Reviewer Administrative web site: http://qminstitute.org/home/vo/mr/default.aspx Keep in contact. Submit a team calendar. Solve technical problems immediately. Observe confidentiality. Do not copy, change, delete, or add anything. Meet deadlines. You are now the leader of a team and responsible for coordinating team communication. Keep in touch with your other team members and schedule team discussions (electronic or by audio conference) as needed. Convene a meeting (electronic or by audio conference) with the Peer Reviewers and Faculty Developer to discuss process and materials. Develop a calendar for your peer review team. Share calendar with all team members and the QM Project Coordinator. If you have difficulty in accessing or navigating the course, contact a distance learning specialist. Please communicate only with the members of your team about what you find in the course you are reviewing. You are an invited guest; resist the urge to make any changes or copy anything without the faculty developer s permission. Please complete your individual course peer review by the established due date. Revised: January 29, 2013 6

Roles and Responsibilities Compile Team results. Explanations Access your team s compiled reviews online and review the Final Review Report. Convene a post-review discussion with the reviewers on your team if needed. Submit the Final Report electronically by the established due date. Note: A preliminary discussion between the team chair and the faculty developer is mandatory; a preliminary discussion with the entire team is strongly recommended. Evaluate your experience. Review revisions if needed. Note: A post-review discussion is recommended if course met at least 15 or 16 out of the 17 essential 3-point standards and/or there are major discrepancies in any scores or comments. Complete an electronic Exit Interview and submit it by the review deadline. Once any revisions necessary for a course to meet missing standards are completed, you may be asked to review the amendment and confirm if they meet the appropriate review standard. Peer Reviewer Roles/Responsibilities Keep in contact. Set up a calendar. Solve technical problems immediately. Observe confidentiality. Do not copy, change, delete, or add anything. Meet deadlines. Explanations You are now part of a team! Communicate with your review team members as needed. Work with your Review Chair to develop a calendar for your review team. If you have difficulty in accessing or navigating the course, contact the Review Chair, other team members, the faculty developer, or a distance learning specialist Please communicate only with the members of your team about what you find in the course you are reviewing. You are an invited guest; resist the urge to make any changes or copy anything without the faculty developer s permission. Please complete your course peer review by the established due date. Revised: January 29, 2013 7

IV. Quality Matters Rubric Instructor Worksheet Instructor Name: Course Name and Number: 1. Is the course supplemented with other tools for online study? If so, please list them: such as Smarter Thinking, Smarter Measures etc. 2. Does the course use support materials (other than the audiovisual) provided by the textbook publisher? YES NO If YES, please describe or explain: 3. Does the course use any synchronous instructor/student interaction? YES NO If YES, please describe or explain: If YES, is the synchronous component optional or is it mandatory? 4. Is any course content or instructor communication done by individual emails? YES NO If YES, please describe or explain: 5. Are any course materials or activities located outside the course website? YES NO If YES, please explain what they are and how to access them: 6. Identify any particular areas and/or issues in this course that you want to ask for feedback from the reviewers 7. Please provide any other information you want to communicate to the Review Team about your course. Revised: January 29, 2013 8

Course Overview & Introduction V. Quality Matters Rubric Score Max Pts 1.1 Instructions make clear how to get started and where to find various course components. 1.2 Students are introduced to the purpose and structure of the course. 0 / 2 1.3 Etiquette expectations (sometimes called netiquette) for online discussions, email, and other forms of communication are stated clearly. 1.4 Course and/or institutional policies with which the student is expected to comply are clearly stated, or a link to current policies is provided. 1.5 Prerequisite knowledge in the discipline and/or any required competencies are clearly stated. 0 / 2 0 / 1 1.6 Minimum technical skills expected of the student are clearly stated. 0 / 1 1.7 The self-introduction by the instructor is appropriate and available online. 0 / 1 1.8 Students are asked to introduce themselves to the class. 0 / 1 Learning Objectives 2.1 The course learning objectives describe outcomes that are measurable. 2.2 The module/unit learning objectives describe outcomes that are measurable and consistent with the course-level objectives. 2.3 All learning objectives are stated clearly and written from the students' perspective 2.4 Instructions to students on how to meet the learning objectives are adequate and stated clearly. 2.5 The learning objectives are appropriately designed for the level of the course. Assessment & Measurement 3.1 The types of assessment selected measure the stated learning objectives and are consistent with course activities and resources. 3.2 The course grading policy is stated. 3.3 Specific and descriptive criteria are provided for the evaluation of students' work and participation and are tied to the course grading policy. 3.4 The assessment instruments selected are sequenced, varied, and appropriate to the content being assessed. 0 / 2 3.5 Students have multiple opportunities to measure their own learning progress. 0 / 2

Resources & Materials 4.1 The instructional materials contribute to the achievement of the stated course and module/unit learning objectives. 4.2 The relationship between the instructional materials and how the materials are to be used for learning activities are clearly explained. 4.3 All resources and materials used in the course are appropriately cited. 0 / 2 4.4 The instructional materials are current. 0 / 2 4.5 The instructional materials present a variety of perspectives on the course content. 0 / 1 4.6 The distinction between required and optional materials is clearly explained. 0 / 1 Learner Engagement 5.1 The learning activities promote the achievement of the stated learning objectives. 5.2 Learning activities provide opportunities for interaction that support active learning. 5.3 The instructor s plan for classroom response time and feedback on assignments is clearly stated. 5.4 The requirements for student interaction are clearly articulated. 0 / 2 Course Technology 6.1 The tools and media support the course learning objectives. 6.2 Course tools and media support student engagement and guide the student to become an active learner. 6.3 Navigation throughout the online components of the course is logical, consistent, and efficient. 6.4 Students have ready access to the technologies required in the course. 0 / 2 6.5 The course technologies are current. 0 / 1 Learner Support 7.1 The course instructions articulate or link to clear descriptions of the technical support offered and how to access it. 7.2 Course instructions articulate or link to the institution accessibility policies and services. 7.3 Course instructions articulate or link to an explanation of how the institution's academic support services and resources can help students succeed in the course and how students can access the services. 0 / 2 Revised: January 29, 2013 10

7.4 Course instructions articulate or link to an explanation of how the institution's student support services can help students succeed and how students can access the services. 0 / 1 Accessibility 8.1 The course employs accessible technologies and provides guidance on how to obtain accommodation. 8.2 The course contains equivalent alternatives to auditory and visual content. 0 / 2 8.3 The course design facilitates readability and minimizes distractions. 0 / 2 8.4 The course design accommodates the use of assistive technologies. 0 / 2 Overall feedback: Total: 0 / 95 Revised: January 29, 2013 11

VI. Planned Course Improvements As a result of this Quality Matters review, I plan to improve my online course and/or my teaching in the following ways. I plan to make the following improvements to my course: I can implement these improvements by myself: I need help implementing these improvements: The date by which I will implement these changes: Instructor Name: Academic School: Date: Revised: January 29, 2013 12