Case Study: Implementing Level 1 Online Evaluations

Transcription

1 0 Case Study: Implementing Level Online Evaluations Alexandra Halem & Kathy Bohnenberger, ADP December 7 & 8, 006 Produced by Designing & Developing Online Assessments & Evaluations

2 Designing & Developing Online Assessments and Evaluations December 7 & 8, 006 Case Study: Implementing Level Online Evaluations Alexandra Halem and Kathy Bohnenberger, Automatic Data Processing, Inc. (ADP) Objectives In this session, you will learn: The value of using an online tool to deploy evaluations Best practices for tool selection, design, and deployment How to roll out online evaluations in your organization Session 0 Case Study: Implementing Level Online Evaluations, Alexandra Halem & Kathy Bohnenberger, ADP Page

3 Designing & Developing Online Assessments and Evaluations December 7 & 8, 006 ADP s Evaluation History ADP trains over 90,000 learners a year. We were unable to generate reports on training evaluation results because the information was not consolidated and not all regions were using the same evaluation. Paper evaluations - must manually key responses into a database or spreadsheet to view consolidated results 3 Why Use an Online Tool? Single deployment method for all classes Easily collect evaluation data Identify trends quickly Share reports with ADP employees Numerous online tools on the market 4 Session 0 Case Study: Implementing Level Online Evaluations, Alexandra Halem & Kathy Bohnenberger, ADP Page

4 Designing & Developing Online Assessments and Evaluations December 7 & 8, 006 Brainstorm Imagine your manager approached you to say your company needs to begin using online evaluations. How would you approach this project? Type your answers in the Chat pod. 5 Our Approach Used the ADDIE instructional design model, just like any training development Analyzed the needs for an online tool Designed an improved Level evaluation Developed and tested the online evaluation Implemented the online evaluation with a pilot group, and then with all regions Evaluating the effectiveness of online evaluations through feedback Handout: Sample Project Schedule 6 Session 0 Case Study: Implementing Level Online Evaluations, Alexandra Halem & Kathy Bohnenberger, ADP Page 3

5 Designing & Developing Online Assessments and Evaluations December 7 & 8, 006 Brainstorm Who or what would you need to consider before moving to online evaluations in your organization? Type your answers in the Chat pod. 7 ADP s Considerations Stakeholders interested in evaluation results Learners who are completing the evaluations How many responses we anticipate in a given time frame Vendor that can support such a large organization 8 Session 0 Case Study: Implementing Level Online Evaluations, Alexandra Halem & Kathy Bohnenberger, ADP Page 4

6 Designing & Developing Online Assessments and Evaluations December 7 & 8, 006 ADP s Analysis Process. Identified the audience requirements Learners, managers, stakeholders, sales, and our training department. Documented and prioritized all requirements 3. Sent out Request For Proposal (RFP) 4. Narrowed down to vendors that met the majority of requirements or were willing to provide enhancements 5. Tested various tools 9 Brainstorm If you re deploying online evaluations in your organization, what are some characteristics you d look for in a tool or vendor? Type your answers in the Chat pod. 0 Session 0 Case Study: Implementing Level Online Evaluations, Alexandra Halem & Kathy Bohnenberger, ADP Page 5

7 Designing & Developing Online Assessments and Evaluations December 7 & 8, 006 Best Practices for Tool Selection Have all stakeholders identify and prioritize their requirements Handout: Sample Requirements List Research vendors online; find those who seem to match many requirements Handout: Sample Tool Analysis Spreadsheet Send RFP to various vendors Handout: Sample Request For Proposal (RFP) Deploy to a pilot group to ensure it meets all stakeholders needs Sample Evaluation To view an example of our online training evaluation, click here. Handout: Sample Training Evaluation Session 0 Case Study: Implementing Level Online Evaluations, Alexandra Halem & Kathy Bohnenberger, ADP Page 6

8 Designing & Developing Online Assessments and Evaluations December 7 & 8, 006 Rollout Strategy and Communication Worked with vendor on security and pricing Fully documented procedures centrally located Numerous communications to gain buy-in from field Excel macros to support field s use of evaluation Training and demonstrations to make field comfortable 3 Training Session We trained all training managers using our virtual training platform, Centra Symposium. Training included: Overview of the evaluation tool Types of reports available Hands-on practice with reports 4 Session 0 Case Study: Implementing Level Online Evaluations, Alexandra Halem & Kathy Bohnenberger, ADP Page 7

9 Designing & Developing Online Assessments and Evaluations December 7 & 8, 006 Internal Processes Supporting/managing 0 training regions Evaluation champions in each training region Evaluation champions on our team to support the regional champions Monthly statistics report for management 5 Success So Far It It took less time and I I got got more in in the the way of of survey comments than I I did did with the the paper forms. Trainer Going very smoothly! This is is so so cool! Manager Filtering with the the macro worked very well. Nice job! Director 6 Session 0 Case Study: Implementing Level Online Evaluations, Alexandra Halem & Kathy Bohnenberger, ADP Page 8

10 Designing & Developing Online Assessments and Evaluations December 7 & 8, 006 Success So Far Awesome macro. The ability to to easily collect, Thank you and the the entire store, and report on on team for for the the detailed work this data will will be be very you ve done to to make this valuable in in cross project effective and for for referencing our our other being so so responsive to to our our surveys. needs! Manager Director 7 Success So Far Managers and trainers can easily identify client satisfaction and the success of our courses. Managers are using the data for trainer assessments/score cards. 8 Session 0 Case Study: Implementing Level Online Evaluations, Alexandra Halem & Kathy Bohnenberger, ADP Page 9

11 Designing & Developing Online Assessments and Evaluations December 7 & 8, 006 Brainstorm What other uses do you see for online evaluation data in your company? Type your answers in the Chat pod. 9 Next Steps Using data to predict success Comparing data with other online data such as assessments and other ADP surveys Forming a stakeholders group Reviewing revenue reports to see how much revenue the sales leads have brought in Integrating data with other ADP systems that house client data 0 Session 0 Case Study: Implementing Level Online Evaluations, Alexandra Halem & Kathy Bohnenberger, ADP Page 0

12 Designing & Developing Online Assessments and Evaluations December 7 & 8, 006 Lessons Learned Excitement continuing to grow So well-received, getting lots of other requests for other applications of the online tool Difficult to maintain control of all the requests Recap of Considerations What hardware/software will respondents have? Who will need access to evaluation results? What types of reporting or data analysis tools will they need? What other requirements will respondents, managers, and others have? How many responses do you anticipate receiving? What is your budget? Session 0 Case Study: Implementing Level Online Evaluations, Alexandra Halem & Kathy Bohnenberger, ADP Page

13 Designing & Developing Online Assessments and Evaluations December 7 & 8, 006 Questions? 3 Session 0 Case Study: Implementing Level Online Evaluations, Alexandra Halem & Kathy Bohnenberger, ADP Page

14 Sample Project Schedule ID Task Name Duration Start Finish Predecessors Room Readiness 5.33 days Mon /3/06 Mon /0/06 determine terminal server and eval impact 30 mins Mon /3/06 Mon /3/06 3 draft whats coming communication to field training managers hr Tue /4/06 Thu /6/06 4 review communication hr Thu /6/06 Fri /7/ send communication 0.5 hrs Mon /0/06 Mon /0/ Room requirements communicated 0 days Mon /0/06 Mon /0/ Identify Tools.33 days Mon /9/06 Tue /7/06 9 Tool Analysis & Requirements 4 days Mon /9/06 Thu /6/06 0 create tool requirements document 5.33 days Mon /9/06 Mon /6/06 compile tool criteria 3.5 hrs Mon /9/06 Mon /6/06 list tool requirements 3.5 hrs Mon /9/06 Mon /6/06 FF 3 review findings document and prioritize 3.5 hrs Mon /9/06 Mon /6/06 FF 4 meet with managers 0.75 hrs Mon /3/06 Mon /3/06 5 meet with LAN group 0.5 hrs Mon /9/06 Mon /9/06 SS 6 notify tools team 0.5 hrs Mon /9/06 Mon /9/06 SS 7 create numbers analysis document 5 days Fri /0/06 Thu /6/06 8 identify courses to support 3.75 hrs Fri /0/06 Thu /6/06 9 identify issues/concerns 3.75 hrs Fri /0/06 Thu /6/06 8FF 0 Tool analysis complete 0 days Thu /6/06 Thu /6/06 9 Tool Selection 6 days Mon /6/06 Tue /7/06 select tools to evaluate based on criteria 3.8 hrs Mon /6/06 Tue /4/ create RFP hr Tue /4/06 Wed /5/ send RFP to vendors 0 days Wed /5/06 Wed /5/ RFP responses received 7 days Wed /5/06 Fri /3/ update tools team proposal & spreadsheet hr Tue /4/06 Thu //06 7FF 7 collect RFP data hr Thu /6/06 Thu //06 5FF 8 review proposal & spreadsheet with team 0.5 hrs Tue /7/06 Tue /7/06 6FS+ day 9 review proposal & spreadsheet with mgrs/tools team 0.5 hrs Mon /6/06 Mon /6/ Tool selection complete 0 days Tue /7/06 Tue /7/ Tools identified for testing 0 days Tue /7/06 Tue /7/ Tool Testing (internal) days Fri /7/06 Wed 3/5/06 34 create dummy homepage and test URL 3.75 hrs Fri /7/06 Thu //06 35 Evaluation Tool # 0.08 days Tue /7/06 Tue //06 36 contact vendor for licenses 0.75 hrs Tue /7/06 Wed /8/ install licenses 0.75 hrs Wed /8/06 Thu /9/ enter evaluation 0.75 days Thu /9/06 Mon /3/ run reports (Web and Excel) 0.75 hrs Mon /3/06 Tue /4/ modify evaluations 0.5 days Mon /3/06 Mon /3/ export data 0.5 hrs Mon /3/06 Mon /3/ share evaluations 0.5 hrs Mon /3/06 Mon /3/ end evaluation 0.5 hrs Mon /3/06 Mon /3/ merge evaluation data w/existing 0.75 hrs Mon /3/06 Tue /4/ compile User Acceptance Testing (UAT) tasks.5 hrs Wed /8/06 Tue /4/06 44FF 46 update tools team spreadsheet hrs Tue /4/06 Mon /0/ review findings with team 0.5 hrs Mon /0/06 Mon /0/ Internal tool # testing complete 0 days Tue //06 Tue //06 47SF+ day 49 Evaluation Tool # 0.08 days Tue /7/06 Tue //06 50 contact vendor for licenses 0.75 hrs Tue /7/06 Wed /8/06 5 install licenses 0.75 hrs Wed /8/06 Thu /9/ enter evaluation 0.75 days Thu /9/06 Mon /3/ run reports (Web and Excel) 0.75 hrs Mon /3/06 Tue /4/ modify evaluations 0.5 days Mon /3/06 Mon /3/ export data 0.5 hrs Mon /3/06 Mon /3/ share evaluations 0.5 hrs Mon /3/06 Mon /3/ end evaluation 0.5 hrs Mon /3/06 Mon /3/ merge evaluation data w/existing 0.75 hrs Mon /3/06 Tue /4/ compile UAT tasks.5 hrs Wed /8/06 Tue /4/06 58FF 60 update tools team spreadsheet hrs Tue /4/06 Mon /0/ review findings with team 0.5 hrs Mon /0/06 Mon /0/ Internal tool # testing complete 0 days Tue //06 Tue //06 6SF+ day 63 User Acceptance Testing (UAT) Readiness 6 days Tue /7/06 Wed 3/5/06 64 identify UAT regions 0.5 hrs Tue /7/06 Tue /7/ create UAT plan 4.33 days Tue /7/06 Mon 3/3/06 66 communication 5 hrs Tue /7/06 Fri /7/ support.5 hrs Fri /0/06 Mon 3/3/06 66 (c) 006 ADP, Inc. Page

15 Sample Project Schedule ID Task Name Duration Start Finish Predecessors 68 goals.5 hrs Fri /0/06 Mon 3/3/ create testing scripts 3.75 hrs Tue //06 Wed 3// test scripts hr Wed 3//06 Thu 3// determine testing dates 0.75 hrs Tue /7/06 Thu /9/ secure field testing resources 0.75 hrs Thu /9/06 Fri /0/ demo both tools 0.75 hrs Mon 3/3/06 Mon 3/3/06 70SF+8 days 74 meet with field to review UAT process 0.75 hrs Mon 3/3/06 Mon 3/3/06 73FF 75 send req docs to field 0.75 hrs Tue 3/4/06 Tue 3/4/ add URLs to lockdown as needed 0.5 hrs Wed 3/5/06 Wed 3/5/ UAT readiness complete 0 days Wed 3/5/06 Wed 3/5/ Tool Testing - UAT 6 days Thu 3/6/06 Fri 3/4/06 80 Evaluation Tool # 6 days Thu 3/6/06 Fri 3/4/06 8 enter evaluation 0.5 hrs Thu 3/6/06 Fri 3/7/ enter data for UAT.5 hrs Thu 3/6/06 Mon 3/0/06 8SS 83 field evaluation starts 0 days Mon 3/0/06 Mon 3/0/ UAT Evaluation Tool # 3.5 hrs Mon 3/0/06 Fri 3/4/06 83SF+ day 85 Evaluation Tool # 6 days Thu 3/6/06 Fri 3/4/06 86 enter evaluation 0.5 hrs Thu 3/6/06 Fri 3/7/06 87 enter data for UAT.5 hrs Thu 3/6/06 Mon 3/0/06 86SS 88 field evaluation starts 0 days Mon 3/0/06 Mon 3/0/ UAT Evaluation Tool # 3.5 hrs Mon 3/0/06 Fri 3/4/06 88SF+ day 90 Post UAT Tasks 6.5 days Fri 3/4/06 Tue 4/4/06 9 compile, document and review results.5 hrs Fri 3/4/06 Wed 3/9/ review findings with team 0.75 hrs Thu 3/30/06 Thu 3/30/ review findings with managers 0.5 hrs Fri 3/3/06 Fri 3/3/ review findings with field 0.75 hrs Fri 3/3/06 Mon 4/3/ Determine Evaluation Tool to use 0.5 days Mon 4/3/06 Tue 4/4/ UAT complete 0 days Fri 3/3/06 Fri 3/3/ Update L Evaluation days Thu /6/06 Thu 3/6/06 99 Analysis 4.67 days Thu /6/06 Wed //06 00 kickoff meeting 0.5 hrs Thu /6/06 Thu /6/06 0 meet with managers 0.75 hrs Fri /7/06 Mon /30/06 00FS+ day 0 review existing documents on L hr Thu /6/06 Fri /7/ Review current L evaluation questions hr Mon /30/06 Tue /3/ review findings document hr Tue /3/06 Wed // L analysis complete 0 days Wed //06 Wed // Design/Dev 5 days Fri /3/06 Thu /9/06 07 rewrite L questions based on findings 3.75 hrs Fri /3/06 Thu /9/ Review 3.83 days Tue /4/06 Fri 3/3/06 09 review new questions with team 5 days Tue /4/06 Mon /0/06 07SF+8 days 0 review new questions with mgrs 5 days Wed /5/06 Thu /3/06 09 update questions 8 days Thu /6/06 Fri 3/3/06 0 Review 8.83 days Fri 3/3/06 Thu 3/6/06 3 review new questions with team 0.75 hrs Fri 3/3/06 Mon 3/6/06 4 review new questions with mgrs.5 hrs Mon 3/6/06 Fri 3/0/ update questions hrs Mon 3/3/06 Wed 3/5/ review new L with field 0.75 hrs Wed 3/5/06 Thu 3/6/ L evaluation update complete 0 days Thu 3/6/06 Thu 3/6/ Extended pilot tasks 76 days Thu 9//05 Thu 5/4/06 0 determine method/strategy for extended pilot hrs Mon 3/0/06 Thu 3/3/06 83 update evaluation homepage 0 hrs Thu 3/3/06 Tue 4//06 0 test evaluation homepage updates 0.75 hrs Tue 4//06 Wed 4//06 3 identify TPS evaluation champions 7.5 hrs Wed 3/5/06 Mon 4/7/ purchase additonal licenses 3.75 hrs Wed 3/5/06 Wed 3// Communications Plan 5.33 days Thu /9/06 Mon 4/4/06 6 Create area for posting on TPS Web/toolkit 0 days Mon 3/0/06 Mon 4/7/ create & send whats new/whats coming hrs Mon 4/7/06 Thu 4/0/ start to get on field meeting agendas to present evaluation & d 34 hrs Thu /9/06 Mon 4/4/06 9 create & send TPS communication 5 hrs Fri 3/3/06 Tue 4// commuincate with field 5 hrs Fri 3/3/06 Tue 4// communication complete 0 days Tue 4//06 Tue 4// document processes 4 days Fri 3/3/06 Thu 5/4/06 33 external processes 9.75 days Fri 3/3/06 Fri 4/4/06 34 evaluation creation 5 hrs Fri 3/3/06 Tue 4//06 96 (c) 006 ADP, Inc. Page

16 Sample Project Schedule ID Task Name Duration Start Finish Predecessors 35 run reports 5 hrs Fri 3/3/06 Tue 4// job aids 5 hrs Fri 3/3/06 Tue 4// backup/recovery 6.4 days Mon 4/3/06 Wed 4// support model hr Mon 4/3/06 Wed 4// non-electronic eval options 0.75 hrs Mon 4/3/06 Tue 4// post docs to TPSWeb hrs Tue 4//06 Fri 4/4/06 36,39 4 external processes documented & posted 0 days Fri 4/4/06 Fri 4/4/ internal processes 4 days Fri 3/3/06 Thu 5/4/06 43 evaluation champion role 5 hrs Fri 3/3/06 Tue 5// evaluation update method 5 hrs Mon 4/4/06 Tue 5//06 45 reporting process hr Tue 4/4/06 Wed 5/3/ evaluation creation standards hr Tue 4/4/06 Thu 5/4/ internal processes documented 0 days Thu 5/4/06 Thu 5/4/06 46,44 48 Checkpoints 76 days Thu 9//05 Thu 5/4/06 49 contact Implementation with info/agenda hr Wed 3//06 Thu 3// meet with managers hr Thu 4/3/06 Thu 5/4/06 4,47 5 meet with BES hr Thu /9/06 Fri /0/ meet with field - demo tool prior to rollout 0.5 hrs Mon 4/7/06 Mon 4/7/06 4FS+ day 53 Evaluation Champion Tasks days Thu 9//05 Fri 9//05 54 meet with development team hr Thu 9//05 Fri 9//05 55 create evaluations day Thu 9//05 Thu 9//05 56 edit evalutaions day Fri 9//05 Fri 9// Pre-GO LIVE days Wed 4/9/06 Fri 4//06 59 update live evaluation homepage 0.5 hrs Fri 4//06 Fri 4//06 60 test links from lockdown hr Wed 4/9/06 Fri 4//06 59FF 6 pre go-live tasks complete 0 days Fri 4//06 Fri 4// GO LIVE 0 days Mon 4/4/06 Mon 4/4/06 63 Deploy new Evaluation tool (GO LIVE) 0 days Mon 4/4/06 Mon 4/4/06 6FS+ day 64 Post-GO LIVE 35 days Mon 4/4/06 Mon 6//06 65 Monitor delivery & reporting 30 days Mon 4/4/06 Mon 6/5/ Conduct feedback sessions (TPS/Field) hr Mon 5/8/06 Tue 5/9/06 65SS+0 day 67 Incorporate necessy changes / adjustments 3.75 hrs Mon 6/5/06 Mon 6//06 65 (c) 006 ADP, Inc. Page 3

17 Sample Tool Requirements List # Required Feature Priority Notes. Easy to use, intuitive software interface that requires minimal effort to achieve proficiency. Easily reuse or copy existing evaluations Evaluation is web-based and provides a URL link that can be launched from another application, including messages or a Web page Supports the use of various fonts, size, colors, text type, question types and functionality Ability to post a draft version of the evaluation for editing, sharing, and testing Ability to view individual responses by date (and time) stamp in a Web or Excel report Ability for both the evaluation creator and other users to download the evaluation data and reports to Excel Provide a highly visual, graphical interface for reports. Readability of results should be at a glance (bar charts, colors, etc) Provides online ad-hoc reporting capability for all users. This includes the ability to create and share report templates and centrally run saved reports Ability for all users to filter or narrow down data on Web reports by a selected criteria or response Ability to update a live evaluation without impact to data. For example, update the text in a question, add responses to a drop list, delete or reorder response choices.. Ability to hide results from respondents 3. Ability to into the back end and make changes for a response, without changing the date & time stamp of the response 4. Provides security for data 5. Provides excellent, free support via telephone, , and online help 6. Vendor hosted 7. Supports flexibility in assigning and creating user group rights and roles. For example, multiple users can create and edit evaluations, others can only run reports and view data, etc. 006 ADP, Inc. - Page of Session 0 - Case Study: Implementing Level Online Evaluations, Alexandra Halem & Kathy Bohnenberger

18 Sample Tool Requirements List # Required Feature Priority Notes 8. Ability to select individual questions to view online or download 9. Ability to allow multiple people to edit an evaluation Ability to easily reuse or copy an existing question within an evaluation Ability to keep evaluations available indefinitely or expire them Ability for all users to create custom Web and Excel reports easily. No queries required or knowledge of advanced reporting. Merges existing results or data from old to revised evaluation. For example, ability to edit an evaluation (change one question) and when reporting, still see the old data with the new. Copy and paste evaluation text from other applications (for example, Word) into the evaluation with minimal tweaking Provides data integration. Ability to send data to homegrown and off-the-shelf products. Ability to link to another application and send data into fields. Integration with SABA. Supports a library of evaluation templates that can be added to and reused 3 7. Provides a spell check feature 3 8. Ability to embed a hyperlink anywhere in the evaluation Imports addresses from another application into the evaluation application for mass ing of evaluations Provides reminders in user-specified timeframes to complete the evaluation Ability to re-brand the site to company standards (i.e. addresses, UI colors and fonts, images, templates, etc.) Allows mass update changes to an evaluation ADP, Inc. - Page of Session 0 - Case Study: Implementing Level Online Evaluations, Alexandra Halem & Kathy Bohnenberger

19 Sample Tool Analysis Spreadsheet # Questions Vendor # Vendor # Pricing and Licensing Is an evaluation copy of the tool available? Is the tool available for purchase or under license? 3 Is the license transferable from user to user? 4 What is the price per user, if applicable? 5 What is the price per server, if applicable? 6 Does sliding-scale pricing apply for volume usage? 7 What is the annual cost of maintenance/support? 8 Is licensing based on concurrent usage, named users, PC-based, or server-based? 9 Is other software required to use this tool either to develop or deploy surveys/evaluations? 0 Does the tool offer both vendor-hosted and self-hosted options? What is the vendor s industry reputation? How long has the vendor been in business? Is the tool new? If not, how many versions of the tool have there been thus far? What information is available about upcoming versions? 3 Are there journal or trade magazine articles available that provide information helpful to your analysis? Distinguishing Features of Tool (Capabilities/Benefits) 4 What are the primary capabilities of tool? 5 What are the primary benefits of the tool? 6 How would you describe the overall ease of use of this tool? Simple or complex? How much time is required to attain basic competence? To acquire expertise? Tool Requirements 7 What are the hardware requirements (including RAM, OS, disk space and peripherals) of the tool? 8 Does the tool require another platform to publish? For example, the need to publish to a server? 9 Does the tool require additional software or plug-ins to create evaluations? 0 Can the output file of the tool be delivered from a server? Can the output file of the tool be launched via hyperlink? Can the output file launch out (to another Web site) via hyperlink? 3 Does the tool require additional software or plug-ins to run? Delivery Environment Requirements 4 What type of output file does the tool produce? 5 Is the tool compatible with other tools currently in use? With commonly used file types (such as MP3, for example)? 6 Is the output of the tool compatible with supported operating systems and browsers? Session 0 - Case Study: Implementing Level Online Evaluations, Alexandra Halem & Kathy Bohnenberger

20 7 Does the tool require the end user to install a player or other software? 8 What is the size range of a typical output file? 9 Are there any bandwidth or performance considerations? Intended Audience 30 Can the tool be used for client training? 3 Can the tool be used for associate training? Linkage Integration/Interoperability with Other Products and Delivery Systems 3 Does the tool have integration/compatibility with standard operating systems such as Windows XP? 33 With databases such as Oracle or SQL Server? 34 With Web browsers, such as a specific version of Internet Explorer or Netscape? 35 With corporate e-learning systems such as Centra, Saba, or other enterprise LMS? Interactivity Features/Product Simulation/User Interface Features 36 Can authors create all evaluations/surveys through a visual interface? 37 Are scripting or programming required? Available? 38 If scripting or programming is available, are they required for basic development or only for advanced or custom development? 39 Is a code editor available? 40 Are predefined templates or preprogrammed objects provided? Can they be used as a basis for customization? Can customized templates be stored and reused? Evaluation Creation Capabilities 4 Does the tool support a variety of test question types? 4 Are layout (pre-built) templates available? 43 How do learners take the evaluation and what happens after they complete it? 44 What data collection capabilities does the tool have? 45 Is the tool AICC/SCORM compliant? (i.e., can tool transfer test results to LMS via communication standard?) 46 Does the tool support reporting features, such as online viewing of results, generating standard or custom reports, and exporting report data? Session 0 - Case Study: Implementing Level Online Evaluations, Alexandra Halem & Kathy Bohnenberger

21 COMPANY LOGO REQUEST FOR PROPOSAL (RFP) ISSUE DATE: RESPONSES DUE: SCHEDULE OF PROCESS AND PROJECT TIMELINE Advertisement of RFP: Responses due by: Follow up calls by: Testing Scheduled: Tool Selected by: DIRECT QUESTIONS TO ENTER CONTACT INFORMATION REPLY TO: ENTER CONTACT INFORMATION 006 ADP, Inc. Page /30/006

22 BACKGROUND BACKGROUND OF YOUR COMPANY, YOUR DEPARTMENT, AND ANTICIPATED NUMBER OF LEARNERS/RESPONDENTS IN A YEAR. DESCRIPTION Our goal is to locate and test a tool suitable for deployment of electronic after-class evaluations nationwide by MONTH/YEAR. Requirements for this tool have been gathered, analyzed and prioritized. These are included in this RFP (see Tool Requirements section). CONSIDERATION All companies in receipt of the RFP are asked to review the attached requirements and indicate their compliance with them, providing additional information and attachments where necessary. Responses will be reviewed, and vendors whose products can provide the required functionality will be contacted for follow-up based on their responses to this RFP. The company that most clearly demonstrates alignment with these requirements will be asked to provide a short demonstration of the tool, its features and functionality. The winning bidder, in addition to being the primary tool used for COMPANY NAME training evaluation deployment, will also be considered as a solution to internal and external evaluation needs in other divisions in COMPANY NAME. LICENSE and COST STRUCTURE In addition to the following requirements, all responses to this request must include:. A comprehensive description of the tool s licensing structure. For example, does the structure require the purchasing of named licenses or can groups of licenses be purchased for use?. A breakdown of the cost associated with the product. This should include, based on the numbers above, costs for licenses, maintenance, upgrades, servers and responses as applicable. 3. The ability for COMPANY NAME to obtain a free, limited-duration, full-feature license or licenses to be used for internal testing. 006 ADP, Inc. Page /30/006

23 TOOL REQUIREMENTS Priority: = Critical Requirement (Must have); = Important Requirement (Should have); 3= Not a Critical Requirement (Nice to have) # Required Feature Priority Available? (Y/N) Notes. Easy to use, intuitive software interface that requires minimal effort to achieve proficiency. Easily reuse or copy existing evaluations 3. Evaluation is web-based and provides a URL link that can be launched from another application, including messages or a Web page Supports the use of various fonts, size, colors, text type, question types and functionality Ability to post a draft version of the evaluation for editing, sharing, and testing Ability to view individual responses by date (and time) stamp in a Web or Excel report Ability for both the evaluation creator and other users to download the evaluation data and reports to Excel 8. Provide a highly visual, graphical interface for reports. Readability of results should be at a glance (bar charts, colors, etc) 9. Provides online ad-hoc reporting capability for all users. This includes the ability to create and share report templates and centrally run saved reports 0.. Ability for all users to filter or narrow down data on Web reports by a selected criteria or response Ability to update a live evaluation without impact to data. For example, update the text in a question, add responses to a drop list, delete or reorder response choices.. Ability to hide results from respondents 3. Ability to into the back end and make changes for a response, without changing the date & time stamp of the response 4. Provides security for data 006 ADP, Inc. Page 3 /30/006

24 # Required Feature Priority Available? (Y/N) Notes 5. Provides excellent, free support via telephone, , and online help 6. Vendor hosted Supports flexibility in assigning and creating user group rights and roles. For example, multiple users can create and edit evaluations, others can only run reports and view data, etc. Ability to select individual questions to view online or download 9. Ability to allow multiple people to edit an evaluation Ability to easily reuse or copy an existing question within an evaluation Ability to keep evaluations available indefinitely or expire them Ability for all users to create custom Web and Excel reports easily. No queries required or knowledge of advanced reporting. Merges existing results or data from old to revised evaluation. For example, ability to edit an evaluation (change one question) and when reporting, still see the old data with the new. Copy and paste evaluation text from other applications (for example, Word) into the evaluation with minimal tweaking Provides data integration. Ability to send data to home-grown and off-the-shelf products. Ability to link to another application and send data into fields. Integration with SABA. Supports a library of evaluation templates that can be added to and reused 3 7. Provides a spell check feature Ability to embed a hyperlink anywhere in the evaluation Imports addresses from another application into the evaluation application for mass ing of evaluations ADP, Inc. Page 4 /30/006

25 # Required Feature Priority Available? (Y/N) Notes Provides reminders in user-specified timeframes to complete the evaluation Ability to re-brand the site to company standards (i.e. addresses, UI colors and fonts, images, templates, etc.) Allows mass update changes to an evaluation ADP, Inc. Page 5 /30/006

26 Sample Training Evaluation <Enter Complete Course Name> First Name: Last Name: Company Name: Instructor: Please respond to each of the following items. Your feedback will help us improve future offerings of this training.. Did the prerequisite material prepare you for this training? Yes No I did not take the prerequisites. There were no prerequisites.. Rate your confidence in performing the following tasks: High Medium Low <Objective > <Objective > <Objective 3> <Objective 4> <Objective 5> 3. Rate your satisfaction with how well the sequence of topics helped you learn the content. Very satisfied Satisfied Somewhat satisfied Somewhat dissatisfied Dissatisfied Very dissatisfied 4. Rate the length of time required to cover the content presented. Just right Too long Too short 006 ADP, Inc. <VMMDDYYCOURSECODEVERSION>

27 <COURSE NAME> TRAINING EVALUATION 5. Rate your satisfaction with how well the following class activities and training materials helped you learn the content. Very satisfied Satisfied Somewhat satisfied Somewhat dissatisfied Dissatisfied Very dissatisfied N/A Demonstrations Practices Group exercises Knowledge checks Illustrations 6. Rate your satisfaction with these capabilities of the instructor: Very satisfied Satisfied Somewhat satisfied Somewhat dissatisfied Dissatisfied Very Dissatisfied Explained the purpose of the training Explained the content Provided responses that were relevant to the questions asked Provided ample opportunities for participation Gave directions for participating in activities Created an environment conducive to learning Demonstrated product knowledge 7. Please provide additional comments you have about the instructor. 006 ADP, Inc.

28 <COURSE NAME> TRAINING EVALUATION 8. Rate your overall satisfaction with the training. Very satisfied Satisfied Somewhat satisfied Somewhat dissatisfied Dissatisfied Very dissatisfied 9. Did the class meet your expectations? Yes No 0. Please provide any additional comments you have about the training. 006 ADP, Inc. 3