Introduction to Questionmark s Assessment Management Technologies

Transcription

1 Introduction to Questionmark s Assessment Management Technologies Brian McNamara Marketing Director, Questionmark All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

2 Agenda Assessment Solutions enabled by Questionmark Assessment Management System Collaborative Authoring Demonstration Blended Delivery Demonstration Reporting and Analytics Demonstration Integration Q&A Slide 2

3 To ask questions, use the Q&A feature Slide 3

4 About Questionmark Background Founded in 1988 Offices in US, UK, Canada; Distribution partners in 27 countries Assessment technologies and portal solutions that enable organizations to measure knowledge, skills, and attitudes securely for certification, regulatory compliance and improved learning outcomes. Customers 1,000 current enterprise customers worldwide Corporate and Government (Learning/HR function); Higher Education; Awarding Bodies and Test Publishers

5 About Questionmark Products and Services Assessment Management System Questionmark OnDemand (SaaS/cloud) Questionmark Perception (installable software) Multilingual Localized for dozens of different languages and cultures Global 24/7 Support

6 Questionmark Conference 2016 SAVE THE DATE Miami April Get vital info and training on the latest assessment technologies and best practices Network with fellow assessment and learning professionals Learn about the Questionmark product roadmap w w w. q u e s t i o n m a r k. c o m /g o /c o n fe r e n c e Slide 6

7 Types of assessments Quizzes, Tests, Exams Questions and scoring algorithms Scores and feedback at question, topic and assessment level Surveys Questions relate to a person s opinions on a topic Conclusions drawn based on analyzing frequency of responses Examples include course evaluations, employee attitude surveys Observational Assessments Someone assesses or rates a person s performance, knowledge, skills, and abilities Slide 7

8 How our customers use assessments Measure and Enhance Learning Measure knowledge, skills and attitudes before, during and after learning Correct misconceptions and increase knowledge retention Qualification and Certification Deliver reliable, defensible tests to certify employees Ensure sound personnel decisions Regulatory Compliance Ensure knowledge of essential laws, regulations and health/safety guidelines Demonstrate compliance and reduce risk Sales Channel Verification Ensure competence and product knowledge of sales professionals Increase revenues and customer satisfaction Technical Channel Verification Ensure competence and expertise of technical and service professionals Reduce costs and improve customer satisfaction Credentialing Efficiently create and maintain valid, reliable tests Generate revenue from exams and practice tests

9 Assessment Management System Evaluate results and inform stakeholders with timely, meaningful reports Collaborative, secure authoring of surveys, quizzes, tests, and exams Blended delivery to support a broad range of devices and security needs

10 Deployment Options Cloud/SaaS Installable Software

11 Questionmark OnDemand Audited data centers in US and Europe State-of-the-art data centers SaaS / Rapid Deployment 24/7 Monitoring Global 24/7 support available 99.9+% Uptime Scalable to meet highvolume, peak delivery needs Always have the latest version Turnkey service - no hardware to buy and maintain

12 Slide 12

13 Authoring Options: OnDemand Authoring module Authoring module within Assessment Management System Browser-based Create items and assessments Manage item bank Publish assessments for delivery Questionmark Live Access via Questionmark website Ideal for SMEs / content harvesting Browser based Create items and assessments Export and/or as Qpack Admin then imports via Authoring Module Slide 13

14 Authoring Options: Perception Authoring Manager Windows-based application Create items and assessments Manage item bank Questionmark Live Access via Questionmark website Ideal for SMEs / content harvesting Browser based Create items and assessments Export and/or as Qpack Admin then imports via Authoring Module Slide 14

15 Many item types to choose from Multiple choice Multiple Response (Dichotomous) Multiple Response (Score per choice) Numeric questions Pull-Down List Ranking Select-a-blank True/False Word response/text match Job Task Analysis Drag-and-Drop Essay question Explanation screens File Upload Fill-in-the-blank Hotspot Knowledge Matrix Likert scale Matching Yes/No Slide 15

16 Demonstration Slide 16

17 Authoring Process Questionmark Live Authoring Module QPack Items and Assessment Content Item Bank / Assessment Content Published Assessments Results from Delivery Results Warehouse

18 Publish for Delivery Questionmark Live QPack Authoring Module Publish Content that is being created & edited separated from Published content Protects Live assessments from inadvertent changes You control how updates occur Items and Assessment Content Item Bank / Assessment Content Published Assessments Results from Delivery Results Warehouse

19 Roles & Permission Management Provide users role-based access to content

20 Item version history and rollback Revision History Audit trail of edits and option to rollback Easily track changes made in collaborative authoring or review Slide 20

21 Metatags Define Metatags Define as mandatory or not Assign to items Slide 21

22 Item Search Search item bank to find items that contain certain words and/or tags Search description, stimulus, choices, status, meta tags, dates and more

23 Multilingual Assessments

24 Managing Multilingual Assessments Translate within Questionmark Export/Import for third-party translation On the fly Translation management system (TMS) Set up project using TMS Export to XLIFF (XML) Translate using third-party tools 3 Rd Party App Manages translation process.does not do translation Import from XLIFF (XML)

25 Authoring: Summary of Key Points Item Authoring 20+ question types Collaborative, browser-based and secure Tool that SMEs can use directly Include video, audio, equations and more Item versioning and version history Search questions to identify those that need update Assessment Authoring Random selection of questions from item bank Random ordering of questions and choices Set a pass score including topic pre-requisites Set assessment time limit (and override if required) Topic scoring and feedback Slide 25

26 Slide 26

27 Author Once Deliver Anywhere Author once Deliver to any device in context Available Securely Single results set Slide 27

28 Responsive Design, Blended Delivery One Assessment many devices and browsers

29 Demonstration Slide 30

30 Observational Assessment Observer rates the participant Example Applications Sales Training Certification Medical and Dental (OSCE) Smartphones/tablets ideal for observational assessment Equipment operation

31 Levels of Monitoring Delivery Stakes and Devices Diagnostic Tests Low Stakes Tests Medium Stakes Exams High Stakes Exams Formative Quizzes Course Evaluations Stakes of Assessment

32 Hope for the best; prepare for the worst Results saved at the server as you go so IT failures don t lose data Slide 33

33 Questionmark Secure One Setting Multiple Platforms Secure browser for high-stakes assessments Reduce content theft, cheating; improve exam integrity Prevents. Windows Mac ipad Printing Screen capture Task switching Control/system keys Right-click / context menu options View source Accidentally exiting exam Slide 34

34 Require Proctoring Option to require monitor to confirm participant identity Online proctoring solutions available Slide 35

35 Text sizing, contrast controls to enhance accessibility Also support for - Non-mouse navigation; alternate pointing devices - Screen readers

36 Delivery via Integration Launch and Track API / Data Integration Connectors for Embedded / iframe / Portals AICC SCORM QMWISe OData LTI HR-XML SAP Blackboard Moodle SharePoint Wikis Blogs Portals SharePoint Slide 37

37 Publish to LMS

38 Slide 39

39 Reporting and Analytics What can we learn from assessments? Did they like it? Did they learn it? Can they do it? Can we prove it? Slide 40

40 What you can learn from Questionmark reporting and analytics How are our Assessments doing? How are our Participants doing? How did one group of test takers compare to another? Did the learning investment impact performance? Where are the skills and knowledge gaps in my organization? How do I coach participants to improve performance? Are my questions and tests valid and reliable? What did participants learn from the training or course? What did participants think of the learning experience?

41 Types of reports available Item and Assessment Analysis Item analysis report Test Analysis Report Score list report Test center analysis report Assessment completion time report Question frequency analysis report Demographic report Question type report Assessment Overview report Assessment results over time report Participant-focused reports Score list report Coaching Report Transcript Report Grade Book Report Survey Reports Survey Frequency Analysis Job-task-analysis (JTA) JTA Summary JTA dimension by demographic Course Evaluation Course Summary Report Instructor Summary Report Class Summary Report Class Detail Report Slide 42

42 Demonstration

43 Score List Report Slide 44

44 Correlation - Discrimination Difficulty/ P-Value Slide 45

45 Coaching Report Compare against benchmarks or other participants Participant Topic level Overall Slide 46

46 Job Task Analysis (JTA) Reports Slide 47

47 Four Course Evaluation Reports: Course Summary Instructor Summary Class Summary Class Detail Slide 48

48 Big Data Applications: Results API Built-in Reports Custom Reports and Dashboards Microsoft Excel PowerPivot BI Tools R e s u l t s W a r e h o u s e Slide 49

49 Consulting Services Deployment Planning Installation Integration Authoring Assessments Importing items Templates / Look & feel Reporting Item reports Other custom reports Solutions & Integration Testing and Assessment Portals People Data syncing Results Data syncing Self Registration Auto Scheduling Assessment Clustering Certificate Generation Distributed Administration Assessment Catalogs Slide 50

50 Candidate Portals

51 Summary of key benefits Collaborative Authoring Secure Browser-based Available globally Responsive Design, Blended Delivery Latest browsers Tablets/smartphones Accessibility Secure delivery for high stakes exams Extensive reporting and analytics How are my participants doing? How are my tests doing? Results API Multilingual Translation Management Screens/Controls in 36 languages Right-to-left languages Integration with Enterprise Systems Supports industry standards APIs Connectors

52 Questions?

53 Recommended White Papers Slide 54

54 Ways to Evaluate Questionmark OnDemand Complimentary technical support while you try Slide 55

55 Keep in Touch blog.questionmark.com twitter.com/questionmark youtube.com/questionmarkvideo facebook.com/questionmarkgroup plus.google.com/+questionmarkcorp/ Slide 56

56 2016 Users Conference SAVE THE DATE Miami April Get vital info and training on the latest assessment technologies and best practices Network with fellow assessment and learning professionals Learn about the Questionmark product roadmap w w w. q u e s t i o n m a r k. c o m /g o /c o n fe r e n c e Slide 57

57 Thank you for attending! Brian McNamara Marketing Director, Questionmark All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

58 Slide 59

59 Item Analysis Difficulty index (aka P value ) What it measures: % answered correctly How it is expressed: Number from 0.0 to 1.0 How to interpret: Too high = too easy Too low could mean Too difficult Confusing/ambiguous Miscoded Item correlation discrimination: What it measures: Correlation between item score and test score How it is expressed: Number between -1.0 and How to interpret: Generally < 0.2 low > 0.3 good

60 Correlation - Discrimination Difficulty/ P-Value Slide 61

61 Slide 62