NHS Shared Learning Quality Assurance Checklists for Evaluating Learning Objects and Online Courses February 2009 Page 1
Note This document provides an outline of the Resource workflow within NHS Shared Learning and also provides the checklists of evaluating learning objects and online courses before they are uploaded and published within NHS Shared Learning. To provide high quality learning resources, all resources in NHS Shared Learning (NSL) should be subject to a quality assurance process requiring that they meet the majority of the evaluation checklist criteria outlined below. The following checklists are built upon the peer review checklist and development guidelines produced by the e-learning Alliance NHS Special Interest Group and the evaluation criteria produced by the Southern Regional Education Board (www.sreb.org). They aim to cover the technical and educational quality of e-learning resources. Page 2
Table of Content Resource Workflow...4 Who does what?...7 Contributor s Quality Assurance Checklist for evaluating learning objects...8 Standard A: Content quality...9 Standard B: Learning objective alignment...10 Standard C: Feedback...11 Standard D: Motivation...11 Standard E: Design and Usability...12 Standard F: Accessibility...13 Standard G: Reusability and standards compliance...14 Standard H: Intellectual property and copyright...15 Contributor s Quality Assurance Checklist for evaluating online courses...16 Standard A: Content...16 Standard B: Instructional design...17 Standard C: Assessment...19 Standard D: Technology...20 Contributor s Quality Assurance Checklist for resources during the upload process within Intralibrary...21 How can Review Panel help in Peer review?...22 Page 3
Resource Workflow A basic workflow * for a resource before it is published within NHS Shared Learning will be as follows: Step 1 Upload resource Contributor (whether organization or individual) uploads the resource after completing the basic Metadata and assigning license rights. The resource is not published at this point. The Contributor is asked to indicate in the description field whether the below QA checklist(s) have been applied. Step 2 Catalogue and Publish The cataloguer completes the full metadata record and publishes the resource to NHS Shared Learning where it can be retrieved by searching and browsing by users. Communication between Contributor and Cataloguer There are 2 instances when a contributor will receive an email alert from the cataloguer 1. A resource has been published to NHS Shared Learning After the contributor has successfully completed the upload process, the Cataloguer will receive an email alert indicating that a resource is available to be published. The cataloguer completes the full metadata record, checks the validity of the resource as per the cataloguer checklist and publishes the resource to the portal. At this point the contributor will receive an email alert stating that their resource has been published. * A step by step guide to contributing resources to NHS Shared Learning can be found here Page 4
2. A resource has been returned for clarification or the resource has been found unsuitable * for NHS Shared Learning When a cataloguer receives an email alert about a resource being uploaded and begin to complete the metadata record, he/she might find that some important information about the resource is missing. In such a scenario the contributor will receive an email alert with the appropriate query/ comment. There after the contributor will follow the below steps to fix the issue: Contributor logs into Intralibrary to view the comments sent by the Cataloguer Contributor then adds an appropriate reply/ comment to the resource after the appropriate change has been made to the resource. At this point the cataloguer will again be indicated with an email alert with the comment added by the Contributor. * Please refer the Collection policy to view the Resource types and categories that may be contributed to NHS Shared Learning Page 5
A flowchart illustrating the above process is given below Contributor UPLOADS resource via Intralibrary Contributor ADDS basic Metadata information and License rights to complete the upload process Email alert to Cataloguer Email alert to Contributor Cataloguer RESERVES the resource within Intralibrary Cataloguer VALIDATES the resource as per the cataloguer checklist Email alert to Contributor Resource COMPLIES as per checklist No Yes Cataloguer completes full METADATA PUBLISH to NHS Shared Learning Page 6
Who does what? It is the responsibility of both the Contributor and Cataloguer to perform Quality assurance as per the resource workflow mentioned above and the roles defined below. The checklists defined in the following pages have been designed to assist contributors in determining the quality and effectiveness of learning objects. This evaluation is essential to ensure that learning objects placed in NHS Shared Learning are of high quality. Contributors (whether organization or individual) 1. Every contributor will apply the essential elements of the Checklist for evaluating Learning Objects or Checklist for evaluating online courses before uploading resources to NSL. 2. Contributors are responsible for ensuring that both technical and educational quality assurance, and, where appropriate, quality assurance from a clinical or subject perspective is carried out before uploading the resource to NSL. 3. Contributors are responsible for indicating in the description field of the metadata the nature of the quality assurance carried out. Please refer Contributor s Quality Assurance Checklist for resources during the upload process within Intralibrary for more details. 4. Boards have the option of using the Review Panel tool to carry out local peer review and consultation on e-learning resources during the development stage, including use of the QA checklists. NOTE: If the contributor themselves are not qualified to perform the Quality Assurance, it is the responsibility of the Contributor to ensure that it is done by an appropriately qualified evaluator before uploading it to NHS Shared Learning. Cataloguer 1. Every Cataloguer will check to ensure that all essential items in the metadata record has been filled before publishing the resource to NSL 2. The cataloguer will apply a metadata value within the catalogue record to indicate that the Quality Assurance process has been applied to the resource. This information will then be apparent in the record as retrieved by searching and browsing of NHS Shared Learning. NOTE: If no evidence is provided by the contributor of carrying out the quality check then the cataloguer can request this information from the Contributor. Page 7
Contributor s Quality Assurance Checklist for evaluating learning objects The checklist given below will be applied by Contributors before uploading resources to NHS Shared Learning portal via the Intralibrary tool. The IEEE (2002) defines Learning Objects as any entity, digital or non-digital, which can be used, re-used or referenced during technology supported learning. Learning objects exist and interoperate at different levels of granularity. The simplest level is content, information or knowledge object. This could be a simple text document, a photograph, a video clip, a two- or three- dimensional image A single topic(s) with a learning objective an online course (s) Standards B E may not be appropriate for simple files e.g. images. Page 8
Standard A: Content quality - The content is accurate and grammatically correct, and the scope is sufficient for the intended use. 1. Content is accurate 2. Content is fair and unbiased 3. The Learning Object is appropriate for the scope it aims to cover. Fully Partiall NA y More information Desirable criteria 4. Content is sequenced logically and effectively 5. Learning experiences are sufficient in number and scope to support the targeted outcomes. 6. Content has been reviewed by a subject-matter expert. 7. Content and learning activities are clearly aligned with learning outcomes. Page 9
Standard B: Learning objective alignment - Learning goals and objectives are provided to outline learning expectations and are applicable and relevant to the subject matter and the audience. 1. Learning objectives * are appropriate to learning outcomes *. 2. Learning objectives are clearly stated. Fully Partially NA More information * Learning objectives: Statements about what a student will gain from a course or activity. These are specific statements about exactly what a student should know, be able to do, or value as a result of accomplishing a learning goal. Learning objectives form the basis for curriculum and course development as well as testing. * Learning outcome: A concrete action that a student demonstrates as a result of learning. A learning outcome can be a demonstration of knowledge, a skill, or a value. Generally, learning outcomes are assessed at the course and/or program levels. Page 10
Standard C: Feedback - Learners are provided with constructive, relevant and frequent feedback based on their activities within the learning object. 1. Feedback supports positive learning outcomes. 2. Learners are provided with timely responses and feedback when asked to answer questions or provide information. 3. Feedback compares learner performance with the relevant criteria and explains how performance can be improved. Desirable criteria 4. Feedback uses language that encourages learners. Fully Partiall NA y More information Standard D: Motivation - The learning environment is engaging, interactive and relevant to the intended learner. 1. The learning environment provides appropriate and engaging learning opportunities. Fully Partiall NA y More information Page 11
2. The learning object provides true-tolife learning activities and interactivity whenever possible. 3. The learning object defines realistic expectations and standards for success. 4. Learners are given adequate directions and support to engage in the learning object activities. Standard E: Design and Usability 1. Design is clear, consistent and provides appropriate instructions for navigation and interaction. 2. Design and interface is consistent and predictable. 3. Navigation provides users with a way to return to the start menu, navigate within and exit from the learning object. 4. If audio and video components are present, user control is allowed. 5. Hyperlinks and buttons function effectively. Desirable criteria 6. Type and typeface are appropriate. 7. Images and graphics are used appropriately. Fully Partially NA More information Page 12
8. Colour is used appropriately throughout. 9. Plain English is used as far as possible. Standard F: Accessibility The learning object provides accommodation for learners with sensory and/or motor disabilities. 1. Learning objects should comply with W3C AA accessibility standards. Fully Partiall NA y More information Desirable criteria 1. Provide information for learners to change their default settings to make the application accessible * * Refer www.saifscotland.org.uk for guidance on how to provide accessible information and how to ensure that hardcopy and electronic documents are accessible to disabled people. Page 13
Standard G: Reusability and standards compliance - The learning object can be used in varying learning contexts with learners from diverse backgrounds and supports international standards and specifications. 1. The learning object consists of one or more self-contained learning units, each addressing a single topic or learning objective, and structured as a standalone resource. 2. Hardware and software requirements are defined. 3. Learning objects are constructed in compliance with technical interoperability standards allowing sharing of content and assessments among different learning management systems / virtual learning environments. Content should be SCORM 1 1.2 or IMS Content Packaging 2 1.1.2 compliant at a minimum. Assessments should be IMS QTI conformant 3. Fully Partiall NA y More information 1 SCORM, the Sharable Content Object Reference Model, is a technical specification that governs how online training (or "e-learning") is created and delivered to learners. For more details visit http://www.adlnet.gov/about/index.aspx 2 The IMS Content Packaging Information Model describes data structures that are used to provide interoperability of Internet based content with content creation tools, learning management systems (LMS), and run time environments. For more details visit http://www.imsglobal.org 3 The IMS Question & Test Interoperability (QTI) specification describes a data model for the representation of question (assessmentitem) and their corresponding results reports. For more details visit http://www.imsglobal.org Page 14
Standard H: Intellectual property and copyright - The learning object metadata address the rights of the owner and the conditions for use. 1. Issues associated with copyright or IPR of learning materials are clearly addressed. Fully Partiall NA y More information Desirable criteria 2. If the content is developed and owned by the individual or organization submitting the learning object, a Creative Commons * license or similar is attached. 3. All quoted materials are cited correctly by adhering to one of the standard citation formats. * Refer www.creativecommons.org for more details Page 15
Contributor s Quality Assurance Checklist for evaluating online courses Standard A: Content 1. Course objectives are measurable and state clearly what participants will know or be able to do at end of the course. 2. A clear, complete course overview or syllabus is provided. 3. Objectives are matched to content requirements and to the grade and skill levels of the intended audience. 4. Course requirements (e.g. timeframes, expectations for communication, activities and assignments, and assessments) are consistent with course objectives and are clearly stated. 5. Issues associated with copyright or IPR of learning materials are clearly addressed. 6. Data protection policies are clearly stated where required e.g. if course requires user identification. 7. Assessment of learning is included and answers and explanations are available. Fully Partially NA More information
8. The course content is accurate, current and free of bias. This includes Equality and Diversity impact assessment. Desirable criteria 9. Generic information literacy and communication skills are incorporated as an integral part of the course. 10. Learning resources are made available to help learners acquire prerequisite knowledge and skills before starting the course 11. Notes and resources for instructors are included. Standard B: Instructional design 1. Course design reflects a clear understanding of learner needs, and incorporates varied ways to learn and multiple levels of mastery (novice to expert) 2. The course is organized into units and lessons. 3. The course unit overview describes the objectives, activities, and resources within the unit. 4. Each lesson includes a lesson overview, concepts and activities, Fully Partiall NA y More information
assignments and assessments, providing multiple learning opportunities for students to master the content. 5. The course engages learners in activities that address a variety of learning styles and preferences. 6. Readability levels, written assignments and numeracy requirements are appropriate for the course content and the learners. Desirable criteria 7. The course is designed to teach concepts and skills that learners will retain over time - i.e. they are relevant to the real world in which the learning is to be applied. Meaningful and authentic learning experiences are provided that will help learners to apply course concepts. 8. Course instruction includes activities that engage learners in active learning. 9. Instruction provides learners with alternative learning pathways to master the content, based on learner needs. 10. The course provides opportunities for learners to engage in higher-order thinking, including critical reasoning, problem solving, developing mental models, forming opinions. 11. Instructional design enables the course to be adapted to accommodate special learner needs e.g. learners with physical disabilities; low literacy
or numeracy skills. 12. Course design provides appropriate opportunities for instructor-learner interaction, including timely feedback about student progress. 13. Learners have access to information and learning materials that enrich the course content. 14. Course offers opportunities for students to give feedback on course content Standard C: Assessment 1. Assessment strategies are consistent with course objectives and are clearly stated. 2. The course structure includes adequate and appropriate methods and procedures to assess learners mastery of learning objectives. Fully Partiall NA y More information Desirable criteria 3. Ongoing assessments are conducted to verify the learner s readiness for the next lesson. 4. Assessment strategies make the learner aware of his/her progress. 5. Assessment strategies are sufficiently flexible to assess learners in
a variety of ways. 6. Grading schemes and models are available to tutors assessing learners. Standard D: Technology 1. Course architecture permits tutors to add content, activities and assessments to extend learning opportunities. 2. The course is constructed in compliance with technical interoperability standards allowing sharing of content and assessments among different learning management systems / virtual learning environments. Content should be SCORM 1.2 or IMS Content Packaging 1.1.2 compliant at a minimum. Assessments should be IMS QTI conformant. 3. Course adheres to W3C AA accessibility standards. 4. Hardware, web browser and software requirements are specified. 5. Prerequisite technology skills are explicitly stated. Desirable criteria 1. The course is easy to navigate. 2. The course provider offers Fully Partially NA More information
assistance with technical and course management. Contributor s Quality Assurance Checklist for resources during the upload process within Intralibrary These checklists applies only when the contributor uploads the resource * to NHS Shared Learning portal and adds metadata via the Intralibrary system 1. Check if the resource uploaded through Intralibrary has an appropriate title, description and keywords 2. Check if the description field indicates what QA process has been applied. This should include the checklists presented in this document (which cover technical and educational QA) plus any quality assurance conducted from a subject/clinical perspective. E.g., you can state in the description, this resource has been Educationally/ Technically Quality assured. 3. Check if the description field indicates what KSF mapping applies to the resource, if applicable 4. Check if you have indicated within the description field when the resource was created or published, if known. If the exact date is unknown you can also state the year of creation/publication 5. Check if you have indicated the author and the publisher of the resource if the you yourself/organization has not authored or published the resource 6. Check if you have selected the appropriate License rights associated with the resource while uploading the resource * A step by step guide to Contributing resources to NHS Shared Learning can be found here
How can Review Panel help in Peer review? Review Panel is a space where you can share your work with others and invite everyone involved in your project to make contributions and have their say. It offers you a collaborative online forum where anybody with a vested interest in your work can assess and review it quickly and easily. Reviewers can help to guide the development of your project. You can track, preview and feedback on: PDFs Multimedia Websites e-learning materials Training courses Word documents And much more! Here are the basic steps involved to invite reviewers to your project 1. Login Review Panel using your ATHENS details 2. Load the project you would like to be reviewed i. You can also create a new project that will allow you to upload a totally new project 3. Select the potential reviewers by ticking the check boxes next to their name. i. Note: Your reviewer will have to login once for you to be able to see their names ii. An email will be sent to your reviewers when you select/add them to the project. * A step by step guide to Reviewing resources online can be found here
How to add comments and suggestions? 1. Load up the project you wish to review by clicking on its name. 2. Select the item you wish to comment on by clicking its name. 3. Type your comment into the space provided on left hand side and click Add. 4. Your comment will appear along with your name and the correct date. Your projects screen will let you know how many reviewers have been assigned to your project and how many comments have been posted. The reviewer can then indicate via email of telephone that the review has been completed for the project owner to analyze the comments and address them.