Introduction to Questionmark s Assessment Management Technologies Brian McNamara Marketing Director, Questionmark All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Agenda Assessment Solutions enabled by Questionmark Assessment Management System Collaborative Authoring Demonstration Blended Delivery Demonstration Reporting and Analytics Demonstration Integration Q&A Slide 2
To ask questions, use the Q&A feature Slide 3
About Questionmark Background Founded in 1988 Offices in US, UK, Canada; Distribution partners in 27 countries Assessment technologies and portal solutions that enable organizations to measure knowledge, skills, and attitudes securely for certification, regulatory compliance and improved learning outcomes. Customers 1,000 current enterprise customers worldwide Corporate and Government (Learning/HR function); Higher Education; Awarding Bodies and Test Publishers
About Questionmark Products and Services Assessment Management System Questionmark OnDemand (SaaS/cloud) Questionmark Perception (installable software) Multilingual Localized for dozens of different languages and cultures Global 24/7 Support
Questionmark Conference 2016 SAVE THE DATE Miami April 12-15 Get vital info and training on the latest assessment technologies and best practices Network with fellow assessment and learning professionals Learn about the Questionmark product roadmap w w w. q u e s t i o n m a r k. c o m /g o /c o n fe r e n c e Slide 6
Types of assessments Quizzes, Tests, Exams Questions and scoring algorithms Scores and feedback at question, topic and assessment level Surveys Questions relate to a person s opinions on a topic Conclusions drawn based on analyzing frequency of responses Examples include course evaluations, employee attitude surveys Observational Assessments Someone assesses or rates a person s performance, knowledge, skills, and abilities Slide 7
How our customers use assessments Measure and Enhance Learning Measure knowledge, skills and attitudes before, during and after learning Correct misconceptions and increase knowledge retention Qualification and Certification Deliver reliable, defensible tests to certify employees Ensure sound personnel decisions Regulatory Compliance Ensure knowledge of essential laws, regulations and health/safety guidelines Demonstrate compliance and reduce risk Sales Channel Verification Ensure competence and product knowledge of sales professionals Increase revenues and customer satisfaction Technical Channel Verification Ensure competence and expertise of technical and service professionals Reduce costs and improve customer satisfaction Credentialing Efficiently create and maintain valid, reliable tests Generate revenue from exams and practice tests
Assessment Management System Evaluate results and inform stakeholders with timely, meaningful reports Collaborative, secure authoring of surveys, quizzes, tests, and exams Blended delivery to support a broad range of devices and security needs
Deployment Options Cloud/SaaS Installable Software
Questionmark OnDemand Audited data centers in US and Europe State-of-the-art data centers SaaS / Rapid Deployment 24/7 Monitoring Global 24/7 support available 99.9+% Uptime Scalable to meet highvolume, peak delivery needs Always have the latest version Turnkey service - no hardware to buy and maintain
Slide 12
Authoring Options: OnDemand Authoring module Authoring module within Assessment Management System Browser-based Create items and assessments Manage item bank Publish assessments for delivery Questionmark Live Access via Questionmark website Ideal for SMEs / content harvesting Browser based Create items and assessments Export and/or Email as Qpack Admin then imports via Authoring Module Slide 13
Authoring Options: Perception Authoring Manager Windows-based application Create items and assessments Manage item bank Questionmark Live Access via Questionmark website Ideal for SMEs / content harvesting Browser based Create items and assessments Export and/or Email as Qpack Admin then imports via Authoring Module Slide 14
Many item types to choose from Multiple choice Multiple Response (Dichotomous) Multiple Response (Score per choice) Numeric questions Pull-Down List Ranking Select-a-blank True/False Word response/text match Job Task Analysis Drag-and-Drop Essay question Explanation screens File Upload Fill-in-the-blank Hotspot Knowledge Matrix Likert scale Matching Yes/No Slide 15
Demonstration Slide 16
Authoring Process Questionmark Live Authoring Module QPack Items and Assessment Content Item Bank / Assessment Content Published Assessments Results from Delivery Results Warehouse
Publish for Delivery Questionmark Live QPack Authoring Module Publish Content that is being created & edited separated from Published content Protects Live assessments from inadvertent changes You control how updates occur Items and Assessment Content Item Bank / Assessment Content Published Assessments Results from Delivery Results Warehouse
Roles & Permission Management Provide users role-based access to content
Item version history and rollback Revision History Audit trail of edits and option to rollback Easily track changes made in collaborative authoring or review Slide 20
Metatags Define Metatags Define as mandatory or not Assign to items Slide 21
Item Search Search item bank to find items that contain certain words and/or tags Search description, stimulus, choices, status, meta tags, dates and more
Multilingual Assessments
Managing Multilingual Assessments Translate within Questionmark Export/Import for third-party translation On the fly Translation management system (TMS) Set up project using TMS Export to XLIFF (XML) Translate using third-party tools 3 Rd Party App Manages translation process.does not do translation Import from XLIFF (XML)
Authoring: Summary of Key Points Item Authoring 20+ question types Collaborative, browser-based and secure Tool that SMEs can use directly Include video, audio, equations and more Item versioning and version history Search questions to identify those that need update Assessment Authoring Random selection of questions from item bank Random ordering of questions and choices Set a pass score including topic pre-requisites Set assessment time limit (and override if required) Topic scoring and feedback Slide 25
Slide 26
Author Once Deliver Anywhere Author once Deliver to any device in context Available Securely Single results set Slide 27
Responsive Design, Blended Delivery One Assessment many devices and browsers
Demonstration Slide 30
Observational Assessment Observer rates the participant Example Applications Sales Training Certification Medical and Dental (OSCE) Smartphones/tablets ideal for observational assessment Equipment operation
Levels of Monitoring Delivery Stakes and Devices Diagnostic Tests Low Stakes Tests Medium Stakes Exams High Stakes Exams Formative Quizzes Course Evaluations Stakes of Assessment
Hope for the best; prepare for the worst Results saved at the server as you go so IT failures don t lose data Slide 33
Questionmark Secure One Setting Multiple Platforms Secure browser for high-stakes assessments Reduce content theft, cheating; improve exam integrity Prevents. Windows Mac ipad Printing Screen capture Task switching Control/system keys Right-click / context menu options View source Accidentally exiting exam Slide 34
Require Proctoring Option to require monitor to confirm participant identity Online proctoring solutions available Slide 35
Text sizing, contrast controls to enhance accessibility Also support for - Non-mouse navigation; alternate pointing devices - Screen readers
Delivery via Integration Launch and Track API / Data Integration Connectors for Embedded / iframe / Portals AICC SCORM QMWISe OData LTI HR-XML SAP Blackboard Moodle SharePoint Wikis Blogs Portals SharePoint Slide 37
Publish to LMS
Slide 39
Reporting and Analytics What can we learn from assessments? Did they like it? Did they learn it? Can they do it? Can we prove it? Slide 40
What you can learn from Questionmark reporting and analytics How are our Assessments doing? How are our Participants doing? How did one group of test takers compare to another? Did the learning investment impact performance? Where are the skills and knowledge gaps in my organization? How do I coach participants to improve performance? Are my questions and tests valid and reliable? What did participants learn from the training or course? What did participants think of the learning experience?
Types of reports available Item and Assessment Analysis Item analysis report Test Analysis Report Score list report Test center analysis report Assessment completion time report Question frequency analysis report Demographic report Question type report Assessment Overview report Assessment results over time report Participant-focused reports Score list report Coaching Report Transcript Report Grade Book Report Survey Reports Survey Frequency Analysis Job-task-analysis (JTA) JTA Summary JTA dimension by demographic Course Evaluation Course Summary Report Instructor Summary Report Class Summary Report Class Detail Report Slide 42
Demonstration
Score List Report Slide 44
Correlation - Discrimination Difficulty/ P-Value Slide 45
Coaching Report Compare against benchmarks or other participants Participant Topic level Overall Slide 46
Job Task Analysis (JTA) Reports Slide 47
Four Course Evaluation Reports: Course Summary Instructor Summary Class Summary Class Detail Slide 48
Big Data Applications: Results API Built-in Reports Custom Reports and Dashboards Microsoft Excel PowerPivot BI Tools R e s u l t s W a r e h o u s e Slide 49
Consulting Services Deployment Planning Installation Integration Authoring Assessments Importing items Templates / Look & feel Reporting Item reports Other custom reports Solutions & Integration Testing and Assessment Portals People Data syncing Results Data syncing Self Registration Auto Scheduling Assessment Clustering Certificate Generation Distributed Administration Assessment Catalogs Slide 50
Candidate Portals
Summary of key benefits Collaborative Authoring Secure Browser-based Available globally Responsive Design, Blended Delivery Latest browsers Tablets/smartphones Accessibility Secure delivery for high stakes exams Extensive reporting and analytics How are my participants doing? How are my tests doing? Results API Multilingual Translation Management Screens/Controls in 36 languages Right-to-left languages Integration with Enterprise Systems Supports industry standards APIs Connectors
Questions?
Recommended White Papers www.questionmark.com/go/whitepapers Slide 54
Ways to Evaluate Questionmark OnDemand www.questionmark.com/go/od30us Complimentary technical support while you try Slide 55
Keep in Touch blog.questionmark.com twitter.com/questionmark youtube.com/questionmarkvideo https://www.linkedin.com/grp/home?gid=1931441 facebook.com/questionmarkgroup plus.google.com/+questionmarkcorp/ Slide 56
2016 Users Conference SAVE THE DATE Miami April 12-15 Get vital info and training on the latest assessment technologies and best practices Network with fellow assessment and learning professionals Learn about the Questionmark product roadmap w w w. q u e s t i o n m a r k. c o m /g o /c o n fe r e n c e Slide 57
Thank you for attending! Brian McNamara Marketing Director, Questionmark All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Slide 59
Item Analysis Difficulty index (aka P value ) What it measures: % answered correctly How it is expressed: Number from 0.0 to 1.0 How to interpret: Too high = too easy Too low could mean Too difficult Confusing/ambiguous Miscoded Item correlation discrimination: What it measures: Correlation between item score and test score How it is expressed: Number between -1.0 and +1.0. How to interpret: Generally < 0.2 low > 0.3 good
Correlation - Discrimination Difficulty/ P-Value Slide 61
Slide 62