Embedded Assessments: Building Knowledge Checks, Surveys and Other Assessments into Learning Materials

Transcription

1 Questionmark White Paper Embedded Assessments: Building Knowledge Checks, Surveys and Other Assessments into Learning Materials The ability to place assessments within many different contexts embedding them in wikis, blogs and portals, for instance changes their potential uses and brings them directly into the learning process. This paper will provide an overview of embedded assessments, explain their value to learners and instructors, share examples of where such assessments can be used and consider how they might help shape the future of learning. This white paper is an update of a paper prepared by the authors for presentation at the 2010 Computer Assisted Assessment (CAA) Conference at the University of Southampton in England. Authors: John Kleeman Eric Shepherd Joan Phaup

2 1. Introduction Until recently, online assessments have been presented in isolation, occupying an entire computer screen and presumably the student s undivided attention! Such assessments still play a vital role, particularly as higher-stakes tests. But today, just as videos are now embedded within a Web page alongside other content, so too are quizzes and surveys. Such assessments can be found in many settings and contexts, including portals, wikis, blogs, social networking sites, and even within Microsoft Word, Excel, and PowerPoint files. They are right at home within learning mashups that combine text, videos, podcasts, graphics and other materials for students. Smart phones and other mobile devices are also being used to deliver assessments. The ability to place assessments within many different contexts changes their potential uses. They become an integral part of the learning process. Students can respond to surveys immediately while they are viewing a page, or take a quick knowledge-check quiz relating to the page s content. In his Research-to-Practice Report, The Learning Benefit of Questions, Dr. Will Thalheimer provides powerful evidence that learning with questions is far superior to learning without questions. He states that the empirical evidence is overwhelming. Questions are one of the most powerful tools for building learning environments and promoting successful performance. Research quoted by him indicates that the retrieval practice, feedback and repetition afforded by well-constructed assessments motivate learners to study, and have been shown to boost the effectiveness of learning from 40% to 150%. Since the early days of the Web age, it has always been possible to link assessments from learning content, but it is now easy to embed assessments using standard tools and technologies into blogs, wikis and other kinds of learning material and Web pages. This creates stimulating and powerful learning environments that are easy to maintain. Distributing a single assessment to multiple and dispersed Web sites, wikis, and blogs increases its availability and increases the likelihood that students will find and use it. Embedding assessments adds interactivity to Web pages, blogs, portals and wikis, and it does away with the conception that assessments should always be treated as separate learning activities. This paper will provide an overview of embedded assessments, share examples of where they can be used and consider how they might help shape the future :38:34 Copyright Questionmark 2010 Page 2 of 12

3 2. Overview of Embedded Assessments Embedded assessments could be regarded as the next step in blended delivery, providing yet another option in addition to printing and scanning paper-based tests, integrating with enterprise systems, and delivering via test centers, CDs, the Internet, intranets and mobile devices. This progress stems from the ability of an assessment management system to auto-sense the requirements of different delivery environments be they iframes, smart phones or other devices and then auto-size assessment content to suit each situation. This makes it possible to deliver a single assessment to various devices to produce a single set of results, as shown in the following illustration: The most typical way of distributing assessments on the Internet is via a fixed browser application that requires a Web site, wiki or blog to link to that assessment. Links can be provided to these standalone Web applications and by other Web pages, wikis, blogs and RSS feeds. Linking is appropriate for high-stakes exams and tests where the measurements of knowledge, skills and abilities demand highly consistent presentations. However, the objectives of low-stakes quizzes, and attitude and opinion surveys, can be better served by making them easily accessible from within other Web resources. For example, the cognitive impact of linking back and forth between learning materials and a formative quiz can detract from its learning impact. Embedding the assessment within the learning material, whether that be a Web page, wiki, blog, or other Web-based learning content, enables the learner to use the assessment as a learning aid rather than a cognitive burden. So, learning mashups can improve learner experiences and learning outcomes :38:34 Copyright Questionmark 2010 Page 3 of 12

4 In the same way, post-event satisfaction surveys can be provided within the material, efficiently harvesting valuable data on how participants felt about an event. They can respond to questions without getting distracted and possibly lost by needing to click on links. And there is well-established evidence that asking for student satisfaction on learning provides useful and effective input to improve the learning. For educators, being able to create and maintain an assessment in one place, distributing it far and wide and then processing the results centrally, means considerable savings in time and effort. Documentation, learning content and assessments are often available throughout an institution s infrastructure but can be challenging to find they are especially difficult to find in context. Wikis, blogs and Web pages can be used to expose existing resources, to wrap context, to provide a meaningful learning and/or assessment experience. For assessment software providers, the key requirement is that the software is able to sense the display characteristics of the device or window in which the assessment is being delivered and display the questions appropriately. For example, buttons to move to the next question might say Next question if there is plenty of space and just > if there is not. Also, the size of headers and footers will vary depending on the window size. On mobile devices which do not support Flash, the assessment software should choose an alternative display mechanism that will display on the mobile device :38:34 Copyright Questionmark 2010 Page 4 of 12

5 3. Examples Wikis, Web Pages and Blogs Wikis, Web pages and blogs lend themselves effectively to delivering learning content. They give people the opportunity to contribute, discuss and consume knowledge. Embedding assessments into these resources provides information as well as a knowledge check to reinforce learning. It also helps reduce the forgetting curve. Implementing solutions like this can improve the user s experience and create an effective, readily available learning environment. Often, embedding assessments in wikis, Web pages and blogs can help to improve the user s learning experience by: Maintaining a single place to learn and assess oneself Personalizing the process of learning and assessing to ensure the relevance to the learner Focusing the learner s attention Providing feedback to assist the learner to correct misconceptions, to practice memory recall, and to provide confidence in what the learner does actually know Creating intrigue and identifying what the learner needs to learn before accessing the learning material. The assessment feedback can even direct learners to the relevant material in the wiki, Web page, or blog Identifying what the learner has successfully learnt/recalled from the learning material used Other benefits of embedding an assessment into these platforms include: Providing a seamless user experience Giving more focused knowledge-sharing based on results :38:34 Copyright Questionmark 2010 Page 5 of 12

6 Combining other media within the assessment engine or within the Web resource, for example using a YouTube video with the wiki, Web page or blog and/or within the actual assessment at the same time. Educators can effectively use embedded assessments throughout the learning process. Here, for instance, is an example of how an assessment template has been customized and delivered as part of a blog post in a seamless way. This particular example shows how an assessment in the form of a quiz has been incorporated in a blog post: :38:34 Copyright Questionmark 2010 Page 6 of 12

7 SharePoint and Other Portals As a platform used by thousands of corporate, government and academic organizations, Microsoft SharePoint, is increasingly used to manage and provide access to learning content. SharePoint Portals make it possible to provide a single sign-on for various personalized productivity tools and information systems. Users can customize portals to present information in digestible dashboard views and easy access to the systems that are most appropriate for that user. There are several ways to use SharePoint in connection with assessments: Embed a quiz or survey in a SharePoint portal page, wiki or blog. Provide single sign-on access to assessments allowing students to view and launch assessments assigned to them from within SharePoint. Following are several reasons for embedding assessments within SharePoint: Organizations using SharePoint as a content management system may want to assess the knowledge of the people who use it People in an organization who use SharePoint as part of their learning activities can be assessed on what they already know or what they have learned from training Embedding an assessment within a SharePoint portal page, wiki or blog perhaps a quiz within a page helps to focus the learning that the user is undertaking Below is an illustration of an embedded assessment within SharePoint: In similar fashion, assessments can be utilized in other portals or even in course management systems such as Moodle or Blackboard. Again, by including the assessment within the page rather than as a separate link, it becomes more accessible and more relevant :38:34 Copyright Questionmark 2010 Page 7 of 12

8 Productivity Tools such as Microsoft Word, Excel and PowerPoint Someone studying a Word document could encounter a quick quiz covering the subject matter or perhaps a feedback survey about it. Or perhaps in the course of learning, a student will need to perform an exercise using Excel maybe a scientific question. An Excel file could be used in working out the problem. Quick assessments within that file could serve as learning aids. Having an assessment readily available can help the learner master the information required to perform the task. Moreover, assessments within PowerPoint can keep learners on task, providing knowledge checks and giving opportunities for feedback. Social Networking Sites and Communication Tools It is even possible to run assessments within social networking sites such as Ning, Facebook and Social Go (used by some people in the same way as wikis) as well communication tools like igoogle, Google Wave and Live Meeting. Again, the principle is that including questions and assessments where learning is being delivered and where the student is makes it possible to be immediate and relevant with the questions. This also provides an easy way to include short satisfaction or opinion surveys :38:34 Copyright Questionmark 2010 Page 8 of 12

9 Mobile Devices Learners use smart phones and other mobile devices as much or more than Web environments, and thanks to auto-sensing and auto-sizing technologies, these smaller devices can deliver assessments to students on the move. In order for delivery on mobile devices to be useful in most organisations, the same assessment must work on a variety of common mobile devices, since learners will typically have a wide variety of devices. The same assessment needs to be deliverable on a PC or Mac and on a mobile device. By making the same assessment deliverable through different media, the assessment becomes widely accessible and results more comparable. 4. Future Developments Now that look-and-feel-alignment technologies have enabled assessments to take their place among other learning resources, assessments will be used more often and for a wider array of purposes. They may be shorter, serving as quick job or learning aids. Assessments will proliferate not only on blogs, wikis, portals and social networks but also on mobile devices. We are seeing an increase in IT department support for standardization on portals, wikis, blogs, data warehouses and reporting systems. This indicates that there could be a revolution within learning environments, which seem poised to become more aligned with standard and supported IT systems. Such a change would help institutions rapidly deploy powerful, integrated, and scalable environments for learners. Once upon a time, videos were specialist tools that only a few could use, but now they are everyday and included everywhere. The same can be true for assessments. Research shows clearly that by including questions with learning, you increase the power of the learning and reduce forgetting. With :38:34 Copyright Questionmark 2010 Page 9 of 12

10 technology now allowing us to include assessments within pages, not just on their own pages, we can truly make computerized assessments widespread and allow everyone to use them, not just the few. The technology to allow embedded assessments also potentially allows the same assessment to adapt itself according to its delivery environment. So, we may see an increase in the use of assessment syndication, the controlled delivery of the same assessments for participants in multiple Internet locations, with each assessment matching the destination s context. Whether publishing inside an organization, or to/via third parties, assessment syndication can reach larger and more targeted audiences more easily, with participants benefiting from accessing highquality up-to-date assessments more easily. Improved access controls could bring about the use of syndicated assessments for higher-stakes situations. Assessment syndication will allow the wide publication of assessments, in multiple contexts, without complex systems integrations. Creating an assessment once and using it in different contexts is likely to become more commonplace. We may also see that central technologies or servers allow assessment aggregation as technologies such as Web pages, Web services and blogs, continue to proliferate and audiences become more fragmented. By having assessment technologies work in sympathy with Web technologies, assessments will be deployed more widely, and thereby be more valuable to both the producer and the consumer. Aggregating students from multiple locations to a centralized assessment server will provide greater and easier access to students with a simple update and publishing model. Increasingly, students will learn within their preferred context, using the Web, and take embedded assessments at the right time, in the right place and within the right context without having to click through links in order to access them. Assessments presented in the places learners frequent will mean more frequent use of assessments, and when combined with the right kind of learning, this will mean better learning and better remembering! :38:34 Copyright Questionmark 2010 Page 10 of 12

11 5. Resources Metcalf, David S. (2009). Mobile Learning: Transforming the Delivery of Education and Training. Amherst, MA: HRD Press. Questionmark blog posts: how to embed assessments in different environments Questionmark Web site: Auto-sensing, auto-sizing, mobile delivery and embedded assessments Questionmark White Paper: Assessments through the Learning Process Questionmark s Web site for developers Thalheimer, W. (2003, January). The Learning Benefits of Questions. Thalheimer, W. 2008, May. Providing Learners with Feedback: Parts 1 and 2 All names, product names, and trade names are trademarks or registered trademarks of their respective holders :38:34 Copyright Questionmark 2010 Page 11 of 12

12 About Questionmark: Questionmark provides technologies and services that enable organizations to measure knowledge, skills and attitudes securely for certification, regulatory compliance and improved learning outcomes. Questionmark solutions enable reliable, valid and defensible assessments by empowering subject matter experts through collaborative authoring, accommodating participant needs with blended and multilingual delivery and informing stakeholders through timely reporting and meaningful analytics. The Questionmark Perception assessment management system, available as a licensed, hosted or subscription-based solution, enables organizations to create, administer and report on surveys, quizzes, tests and exams. Complete details are available at Questionmark 535 Connecticut Avenue Suite 100 Norwalk, CT Tel: (800) (203) Fax: (800) info@questionmark.com Questionmark 4th Floor, Hill House Highgate Hill London N19 5NA United Kingdom Tel: +44 (0) Fax: +44 (0) info@questionmark.co.uk Questionmark Parc d Activités Economiques Avenue Léon Champagne, 2 B 1480 Tubize (Saintes) Belgium Tel: +32 (0) Fax: +32 (0) info@questionmark.be Legal note This document is copyright Questionmark Corporation (Questionmark) Although Questionmark has used all reasonable care in writing this document, Questionmark makes no representations about the suitability of the information contained in this and related documents for any purpose. The document may include technical inaccuracies or typographical errors, and changes may be periodically made to the document or to the software referenced. This document is provided as is without warranty of any kind. See your Perception support contract for further information. Company and product names are trademarks of their respective owners. Mention of these companies in this document does not imply any warranty by these companies or approval by them of this guide or its recommendations :38:34 Copyright Questionmark 2010 Page 12 of 12