ONC HIT Certification Program

Transcription

1 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 07-Aug-2015 ONC HIT Certification Program Test Results Summary for 2014 Edition EHR Certification Part 1: Product and Developer Information 1.1 Certified Product Information Product Name: Greenway Intergy Meaningful Use Edition Product Version: v9.30 Domain: Ambulatory Test Type: Complete EHR 1.2 Developer/Vendor Information Developer/Vendor Name: Greenway Health, LLC Address: 4301 West Boy Scout Boulevard, Suite 800 Tampa FL Website: Phone: (813) Developer/Vendor Contact: Danny Shipman Page 1 of 12

2 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 07-Aug-2015 Part 2: ONC-Authorized Certification Body Information 2.1 ONC-Authorized Certification Body Information ONC-ACB Name: Drummond Group Address: North Hwy 183, Ste B , Austin, TX Website: Phone: ONC-ACB Contact: Bill Smith This test results summary is approved for public release by the following ONC-Authorized Certification Body Representative: Bill Smith ONC-ACB Authorized Representative Certification Body Manager Function/Title Signature and Date 8/28/ Gap Certification The following identifies criterion or criteria certified via gap certification (a)(1) (a)(19) (d)(6) (h)(1) (a)(6) (a)(20) (d)(8) (h)(2) (a)(7) (b)(5)* (d)(9) (h)(3) (a)(17) (d)(1) (f)(1) (a)(18) (d)(5) (f)(7)** *Gap certification allowed for Inpatient setting only **Gap certification allowed for Ambulatory setting only x No gap certification Page 2 of 12

3 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 07-Aug Inherited Certification The following identifies criterion or criteria certified via inherited certification x (a)(1) (a)(16) Inpt. only x (c)(2) x (f)(2) x (a)(2) (a)(17) Inpt. only x (c)(3) x (f)(3) x (a)(3) (a)(18) x (d)(1) (f)(4) Inpt. only x (a)(4) (a)(19) x (d)(2) x (a)(5) (a)(20) x (d)(3) x (a)(6) x (b)(1) x (d)(4) x (a)(7) x (b)(2) x (d)(5) x (a)(8) x (b)(3) x (d)(6) (f)(7) x (a)(9) x (b)(4) x (d)(7) (g)(1) x (a)(10) x (b)(5) x (d)(8) x (g)(2) x (a)(11) (b)(6) Inpt. only x (d)(9) Optional x (g)(3) x (a)(12) x (b)(7) x (e)(1) x (g)(4) x (a)(13) (b)(8) x (e)(2) Amb. only (h)(1) x (a)(14) (b)(9) x (e)(3) Amb. only (h)(2) x (a)(15) x (c)(1) x (f)(1) (h)(3) (f)(5) Amb. only (f)(6) Amb. only No inherited certification Page 3 of 12

4 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 07-Aug-2015 Part 3: NVLAP-Accredited Testing Laboratory Information Report Number: SG Test Date(s): N/A 3.1 NVLAP-Accredited Testing Laboratory Information ATL Name: Drummond Group EHR Test Lab Accreditation Number: NVLAP Lab Code Address: North Hwy 183, Ste B , Austin, TX Website: Phone: ATL Contact: Kyle Meadors For more information on scope of accreditation, please reference NVLAP site. Part 3 of this test results summary is approved for public release by the following Accredited Testing Laboratory Representative: Sonia Galvan Test Proctor ATL Authorized Representative Function/Title Signature and Date 8/28/2015 Houston, TX Location Where Test Conducted 3.2 Test Information Additional Software Relied Upon for Certification Additional Software Applicable Criteria Functionality provided by Additional Software Kryptiq b.1, 2 HISP No additional software required Page 4 of 12

5 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 07-Aug Test Tools Test Tool Version x Cypress 2.6 x eprescribing Validation Tool HL7 CDA Cancer Registry Reporting Validation Tool HL7 v2 Electronic Laboratory Reporting (ELR) Validation Tool x HL7 v2 Immunization Information System (IIS) Reporting Validation Tool x HL7 v2 Laboratory Results Interface (LRI) Validation Tool x HL7 v2 Syndromic Surveillance Reporting Validation Tool x Transport Testing Tool 181 x Direct Certificate Discovery Tool Edge Testing Tool No test tools required Test Data Alteration (customization) to the test data was necessary and is described in Appendix [insert appendix letter] x No alteration (customization) to the test data was necessary Standards Multiple Standards Permitted The following identifies the standard(s) that has been successfully tested where more than one standard is permitted Criterion # Standard Successfully Tested (a)(8)(ii)(a)(2) (a)(13) x (b)(1) HL7 Version 3 Implementation Guide: URL-Based Implementations of the Context-Aware Information Retrieval (Infobutton) Domain x (a)(3) IHTSDO SNOMED CT International Release July 2012 and US Extension to SNOMED CT March 2012 Release (b)(2) HL7 Version 3 Implementation Guide: Context-Aware Knowledge Retrieval (Infobutton) Service-Oriented Architecture Implementation Guide (j) HL7 Version 3 Standard: Clinical Genomics; Pedigree Page 5 of 12

6 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 07-Aug-2015 Criterion # (a)(15)(i) x (b)(1) HL7 Version 3 Implementation Guide: URL-Based Implementations of the Context-Aware Information Retrieval (Infobutton) Domain Standard Successfully Tested (b)(2) HL7 Version 3 Implementation Guide: Context-Aware Knowledge Retrieval (Infobutton) Service-Oriented Architecture Implementation Guide (a)(16)(ii) (b)(2)(i)(a) (b)(7)(i) (g) Network Time Protocol Version 3 (RFC 1305) (i) The code set specified at 45 CFR (c)(2) (ICD-10- CM) for the indicated conditions (i) The code set specified at 45 CFR (c)(2) (ICD-10- CM) for the indicated conditions (g) Network Time Protocol Version 4 (RFC 5905) x (a)(3) IHTSDO SNOMED CT International Release July 2012 and US Extension to SNOMED CT March 2012 Release x (a)(3) IHTSDO SNOMED CT International Release July 2012 and US Extension to SNOMED CT March 2012 Release (b)(8)(i) (e)(1)(i) (e)(1)(ii)(a)(2) (e)(3)(ii) (i) The code set specified at 45 CFR (c)(2) (ICD-10- CM) for the indicated conditions Annex A of the FIPS Publication [list encryption and hashing algorithms] AES-256 SHA (g) Network Time Protocol Version 3 (RFC 1305) Annex A of the FIPS Publication [list encryption and hashing algorithms] AES-256 SHA (a)(3) IHTSDO SNOMED CT International Release July 2012 and US Extension to SNOMED CT March 2012 Release x (g) Network Time Protocol Version 4 (RFC 5905) Common MU Data Set (15) x (a)(3) IHTSDO SNOMED CT International Release July 2012 and US Extension to SNOMED CT March 2012 Release (b)(2) The code set specified at 45 CFR (a)(5) (HCPCS and CPT-4) Page 6 of 12

7 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 07-Aug-2015 Criterion # Standard Successfully Tested None of the criteria and corresponding standards listed above are applicable Newer Versions of Standards The following identifies the newer version of a minimum standard(s) that has been successfully tested Newer Version Applicable Criteria No newer version of a minimum standard was tested Optional Functionality Criterion # x (a)(4)(iii) (b)(1)(i)(b) (b)(1)(i)(c) (b)(2)(ii)(b) (b)(2)(ii)(c) (e)(1) x (f)(3) (f)(7) Common MU Data Set (15) Common MU Data Set (15) Optional Functionality Successfully Tested Plot and display growth charts Receive summary care record using the standards specified at (a) and (b) (Direct and XDM Validation) Receive summary care record using the standards specified at (b) and (c) (SOAP Protocols) Transmit health information to a Third Party using the standards specified at (a) and (b) (Direct and XDM Validation) Transmit health information to a Third Party using the standards specified at (b) and (c) (SOAP Protocols) View, download and transmit data to a third party utilizing the Edge Protocol IG version 1.1 Ambulatory setting only Create syndrome-based public health surveillance information for transmission using the standard specified at (d)(3) (urgent care visit scenario) Ambulatory setting only transmission to public health agencies syndromic surveillance - Create Data Elements Express Procedures according to the standard specified at (b)(3) (45 CFR (a)(4): Code on Dental Procedures and Nomenclature) Express Procedures according to the standard specified at (b)(4) (45 CFR (c)(3): ICD-10-PCS) No optional functionality tested Page 7 of 12

8 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 07-Aug Edition Certification Criteria* Successfully Tested Criteria # Version Version Criteria # TP** TD*** TP TD (a)(1) (c)(3) (a)(2) 1.2 (d)(1) 1.2 (a)(3) (d)(2) 1.6 (a)(4) (d)(3) 1.3 (a)(5) (d)(4) 1.3 (a)(6) (d)(5) 1.2 (a)(7) (d)(6) 1.2 (a)(8) 1.3 (d)(7) 1.2 (a)(9) (d)(8) 1.2 (a)(10) (d)(9) Optional 1.2 (a)(11) 1.3 (e)(1) (a)(12) 1.3 (e)(2) Amb. only (a)(13) 1.2 (e)(3) Amb. only 1.3 (a)(14) 1.2 (f)(1) (a)(15) 1.5 (f)(2) (a)(16) Inpt. only (f)(3) (a)(17) Inpt. only 1.2 (f)(4) Inpt. only (a)(18) (a)(19) (a)(20) (b)(1) (f)(5) Amb. only (f)(6) Amb. only (b)(2) (f)(7) Amb. only 1.1 (b)(3) (g)(1) (b)(4) (g)(2) (b)(5) (g)(3) 1.4 (b)(6) Inpt. only (g)(4) 1.2 (b)(7) (h)(1) 1.1 (b)(8) (h)(2) 1.1 (b)(9) (h)(3) 1.1 (c)(1) (c)(2) Page 8 of 12

9 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 07-Aug-2015 Criteria # x No criteria tested Version Version Criteria # TP** TD*** TP TD *For a list of the 2014 Edition Certification Criteria, please reference (navigation: 2014 Edition Test Method) **Indicates the version number for the Test Procedure (TP) ***Indicates the version number for the Test Data (TD) Page 9 of 12

10 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 07-Aug Clinical Quality Measures* Type of Clinical Quality Measures Successfully Tested: x Ambulatory Inpatient No CQMs tested *For a list of the 2014 Clinical Quality Measures, please the CMS ecqm Library (Navigation: June 2014 and April 2014 Updates) Ambulatory CQMs CMS ID Version CMS ID Version CMS ID Version CMS ID Version x 2 v3 x 90 v3 x 136 v3 x 155 v2 x 22 v2 x 117 v2 x 137 v2 x 156 v2 x 50 v2 x 122 v2 x 138 v2 x 157 v2 x 52 v2 x 123 v2 x 139 v2 x 158 v2 x 56 v2 x 124 v2 x 140 v2 x 159 v2 x 61 v3 x 125 v2 x 141 v3 x 160 v2 x 62 v2 x 126 v2 x 142 v2 x 161 v2 x 64 v3 x 127 v2 x 143 v2 x 163 v2 x 65 v3 x 128 v2 x 144 v2 x 164 v2 x 66 v2 x 129 v3 x 145 v2 x 165 v2 x 68 v3 x 130 v2 x 146 v2 x 166 v3 x 69 v2 x 131 v2 x 147 v2 x 167 v2 x 74 v3 x 132 v2 x 148 v2 x 169 v2 x 75 v2 x 133 v2 x 149 v2 x 177 v2 x 77 v2 x 134 v2 x 153 v2 x 179 v2 x 82 v1 x 135 v2 x 154 v2 x 182 v3 Inpatient CQMs CMS ID Version CMS ID Version CMS ID Version CMS ID Version Page 10 of 12

11 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 07-Aug Automated Numerator Recording and Measure Calculation Automated Numerator Recording Automated Numerator Recording Successfully Tested (a)(1) (a)(11) (a)(18) (b)(6) (a)(3) (a)(12) (a)(19) (b)(8) (a)(4) (a)(13) (a)(20) (b)(9) (a)(5) (a)(14) (b)(2) (e)(1) (a)(6) (a)(15) (b)(3) (e)(2) (a)(7) (a)(16) (b)(4) (e)(3) (a)(9) (a)(17) (b)(5) x Automated Numerator Recording was not tested Automated Measure Calculation Automated Measure Calculation Successfully Tested x (a)(1) x (a)(11) (a)(18) (b)(6) x (a)(3) x (a)(12) (a)(19) (b)(8) x (a)(4) x (a)(13) (a)(20) (b)(9) x (a)(5) x (a)(14) x (b)(2) x (e)(1) x (a)(6) x (a)(15) x (b)(3) x (e)(2) x (a)(7) (a)(16) x (b)(4) x (e)(3) x (a)(9) (a)(17) x (b)(5) Automated Measure Calculation was not tested Attestation Attestation Forms (as applicable) x Safety-Enhanced Design* x Quality Management System** x Privacy and Security Appendix A B C 3.3 Appendices Attached below. *Required if any of the following were tested: (a)(1), (a)(2), (a)(6), (a)(7), (a)(8), (a)(16), (a)(18), (a)(19), (a)(20), (b)(3), (b)(4), (b)(9). **Required for every EHR product Page 11 of 12

12 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 07-Aug-2015 Test Results Summary Change History Test Report ID Description of Change Date 2014 Edition Test Report Summary Page 12 of 12

13 USER CENTER DESIGN REPORT TEST REPORT UPDATE This test report was updated in December 2015 to satisfy User Center Design Report specifications by ONC. The new Test Report ID is amended as follows: Part 3: NVLAP-Accredited Testing Laboratory Information: Report Number plus the suffix _Dec2015.

14

15 ONC-ATL EHR Testing Safety-Enhanced Design Checklist 2014 Edition Prepared & Administered by: DRUMMOND GROUP INC. Rev.: 20Mar2013 EHR Test-120 Page 1 of 7 Copyright 2013 Drummond Group Inc. The information contained in this document is strictly held confidential and shall not be disclosed in any manner or form, directly or indirectly, to any person or entity under any circumstances, without prior approval.

16 Safety-Enhanced Design Evaluation Guide Scope and Purpose This document provides a dual purpose. It serves both a means for the Test Proctor to more quickly evaluate the submitted usability report and also a checklist for the Vendor to confirm all aspects of the report have been submitted. The User-Centered Design Process is just a question of what user-centered design process was used in this product-version submitted for testing. Vendor should answer that question in this document which will be returned to the DGI Test Proctor. The Usability Report Checklist section is just that. A section where the Vendor fills in the page numbers in the report corresponding to the specific information that the Test Proctor must check off to confirm completion of report. Some additional CONFIRM actions are included to assist both Vendor and Test Proctor in checking for specific areas of evaluation listed in the test procedure. The 314.g.3 Submission Checklist is a checklist of items to turn in for completion of testing of 314.g.3. These items should be submitted no later than 2 weeks prior to your test event date. Turning these items in later than 2 weeks out will result in delay of processing test results. 314.g.3 Submission Checklist Vendors must turn in the following items to be tested for certification in Safety- Enhanced Design (314.g.3): A completed usability report of their product-version using the NISTIR 7742 Customized Common Industry Format template ( The report does not have to follow NISTIR 7742 exactly but it should cover the sections required in that template which are checked in the Usability Report Checklist section below. A signed letter from a representative of the Vendor to the veracity and authenticity of the usability report. The signed letter may be included in the report itself. This DGI Safety-Enhanced Design Checklist documented completed with page numbers referencing where the required usability information is provided. Rev.: 20Mar2013 EHR Test-120 Page 2 of 7 Copyright 2013 Drummond Group Inc. The information contained in this document is strictly held confidential and shall not be disclosed in any manner or form, directly or indirectly, to any person or entity under any circumstances, without prior approval.

17 User-Centered Design Process Describe you user-centered design (UCD) process. Was it based on a UCD industry standard (for example; ISO , ISO , ISO 13407, ISO 16982, and ISO/IEC 62366) or a non-industry standard process? The UCD process used was based on NISTIR 7741 and the NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records found at this link: Usability Report Checklist For the submitted Usability Report, please record the report page numbers where each required section can be found as well as any other brief notes needed to understand this section of the report. To clarify, do NOT record the answers below but only include the page number reference to them in the submitted report for easier evaluation. Name and version of the product: P. 3 Data and location of the usability test: P. 1 Test Environment: P. 6 Description of intended users: P. 3 Total number of participants: P. 3 Description of participants: their experience and demographic characteristics: P. 4 CONFIRM: Participants match the description of the intended users. Description of the user tasks that were tested: P. 5 CONFIRM: User tasks are prioritized in accordance with risk associated with user errors. Rev.: 20Mar2013 EHR Test-120 Page 3 of 7 Copyright 2013 Drummond Group Inc. The information contained in this document is strictly held confidential and shall not be disclosed in any manner or form, directly or indirectly, to any person or entity under any circumstances, without prior approval.

18 CONFIRM: All test scenarios for each of the EHR technology capabilities submitted for testing. CPOE (314.a.1) Record Medication Order Change Medication Order Access Medication Order Record Laboratory Order Change Laboratory Order Access Laboratory Order Record Radiology/imaging Order Change Radiology/imaging Order Access Radiology/imaging Order Drug-drug, drug-allergy interactions checks (314.a.2) Create drug-drug and drug-allergy interventions prior to CPOE completion Adjustment of severity level of drug-drug interventions (may be an admin type function) Medication list (314.a.6) Record Medication List Change Medication List Access Medication List Medication allergy list (314.a.7) Record Medication List Change Medication List Access Medication List Clinical decision support (314.a.8) Problem List Interventions Medication List Interventions Medication Allergy List Interventions Demographics Interventions Lab Tests and Results Interventions Vital Signs Interventions Identify User Diagnostic and Therapeutic Reference Information Configuration of CDS interventions by user (may be an admin type function) Rev.: 20Mar2013 EHR Test-120 Page 4 of 7 Copyright 2013 Drummond Group Inc. The information contained in this document is strictly held confidential and shall not be disclosed in any manner or form, directly or indirectly, to any person or entity under any circumstances, without prior approval.

19 emar (314.a.16) Using assistive technology, verify the right patient, medication, dose, route and time. Electronic prescribing (314.b.3) Create prescriptions Clinical information reconciliation (314.b.4) Reconcile patient s active medication list with another source Reconcile patient s active problem list with another source Reconcile patient s active medication allergy list with another source List of the specific metrics captured during the testing for effectiveness, efficiency and satisfaction: P. 7 CONFIRM: Effectiveness and efficiency were evaluated. Data scoring: P. 8 Results of the test and data analysis: P CONFIRM: Test results provided a risk analysis of the use, tested performance and error rates. Major test findings: P Identified area(s) of improvement(s): P Rev.: 20Mar2013 EHR Test-120 Page 5 of 7 Copyright 2013 Drummond Group Inc. The information contained in this document is strictly held confidential and shall not be disclosed in any manner or form, directly or indirectly, to any person or entity under any circumstances, without prior approval.

20 Change Log Revision Change Description 20-Mar-2013 Added instructions for verifying admin functions drug-drug severity adjustment and CDS interventions per user are tested. 18-Feb-2013 Initial Release. Rev.: 20Mar2013 EHR Test-120 Page 6 of 7 Copyright 2013 Drummond Group Inc. The information contained in this document is strictly held confidential and shall not be disclosed in any manner or form, directly or indirectly, to any person or entity under any circumstances, without prior approval.

21 About Drummond Group Inc. Drummond Group Inc. is a global software test and certification lab that serves a wide range of vertical industries. In healthcare, Drummond Group tests and certifies Controlled Substance Ordering Systems (CSOS), Electronic Prescription of Controlled Substances (EPCS) software and processes, and Electronic Health Records (EHRs) designating the trusted test lab as the only third-party certifier of all three initiatives designed to move the industry toward a digital future. Founded in 1999, and accredited for the Office of the National Coordinator HIT Certification Program as an Authorized Certification Body (ACB) and an Authorized Test Lab (ATL), Drummond Group continues to build upon its deep experience and expertise necessary to deliver reliable and cost-effective services. For more information, please visit or [email protected] Rev.: 20Mar2013 EHR Test-120 Page 7 of 7 Copyright 2013 Drummond Group Inc. The information contained in this document is strictly held confidential and shall not be disclosed in any manner or form, directly or indirectly, to any person or entity under any circumstances, without prior approval.

22 USABILITY TESTING: VITERA INTERGY EHR Usability Tests: September 12-14, 2012 Usability Test Location: Orlando, Florida Report Date: June 7, 2013 Report Prepared by: Vitera Healthcare Solutions 1 P age

23 TABLE OF CONTENTS EXECUTIVE SUMMARY... 1 INTRODUCTION... 3 METHOD... 3 DESCRIPTION OF INTENDED USERS... 3 PARTICIPANTS... 3 TOTAL NUMBER AND DESCRIPTION OF PARTICIPANTS... 4 STUDY DESIGN... 5 TASKS... 5 PROCEDURES... 6 TEST LOCATION AND ENVIRONMENT... 6 TEST FORMS AND TOOLS... 7 PARTICIPANT INSTRUCTIONS... 7 USABILITY METRICS... 7 DATA SCORING... 8 RESULTS... 9 DATA ANALYSIS AND REPORTING... 9 DISCUSSION OF THE FINDINGS... 9 EFFECTIVENESS... 9 EFFICIENCY SATISFACTION MAJOR FINDINGS AREAS FOR IMPROVEMENT APPENDICES i P age Vitera Healthcare Solutions

24 Usability Testing: Vitera Intergy EHR EXECUTIVE SUMMARY A usability test of Vitera Intergy EHR, an ambulatory electronic health record (EHR) covering multiple specialties was conducted on September 12-14, 2012 in Orlando, Florida by Vitera Healthcare Solutions. The purpose of this study was to test and validate the usability of the current user interface and provide evidence of usability in Vitera Intergy EHR. During the usability test, twelve healthcare providers matching the target demographic criteria served as participants and used Vitera Intergy EHR in simulated, but representative tasks. The study collected performance data on 16 tasks typically conducted on an EHR. During the 60-minute, one-on-one usability test, each participant was greeted by the administrator and asked to review and sign a recording consent form (included in Appendix 2: Recording Consent Form). Participants had varying levels of prior experience with the Vitera Intergy EHR ranging from 1-5 years. The administrator introduced the test and instructed participants to complete a series of tasks (given one at a time) using Vitera Intergy EHR. During the testing, the administrator timed the test and, along with the data loggers recorded participant performance data on paper and electronically. The participants interactions with screens and their faces were video/audio recorded for subsequent analysis. The following types of data were collected for each participant: Number of tasks successfully completed within the allotted time Time to complete the tasks Number and types of errors Path deviations Participants verbalizations Participants familiarity and ease-of-use and satisfaction ratings with the system and system components All participant data was de-identified no correspondence could be made from the identity of the participant to the data collected. Following the conclusion of the testing, participants were asked to complete a post-test questionnaire and were compensated with small branded items pens and USB drives for their time. Various metrics recommended in the NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records were used to evaluate the usability of Vitera Intergy EHR. Below is a summary of the performance and rating data collected on Vitera Intergy EHR. 1 P age Vitera Healthcare Solutions

25 Usability Testing: Vitera Intergy EHR Overall Ease of Use and Satisfaction of Tasks Performed During the Study Ease of Use Based on a scale of 1-5, where 1 = not at all satisfied and 5 = very satisfied: The overall subjective ease-of-use mean was 4.33 of 5. The overall percent ease of use was 87%. Satisfaction The overall subjective satisfaction mean was 4.45 of 5. The overall percent satisfaction was 89%. Major Findings In addition to the performance data, the following qualitative observations were made: Overall, the new user interface (UI) was very well received. Participants were able to adapt quickly to the new UI, and they appreciated being able to see more data on one screen, thus reducing the back and forth actions required between screens to see the same data in previous versions. All participants were very satisfied with the new design of the Drug Utilization Review (DUR) display. Several of the participants admitted to not paying much attention to the DUR in previous versions of the software, and they commented on the vast improvement of the clean, concise design, making priority alerts more prominent and reducing alert noise. The redesigned Lab Orders system tested extremely well with much faster and less scrolling as the general feedback. Several providers said they would start ordering patient lab tests rather than having support staff handle submitting the orders. This design improvement and resulting workflow change gains enormous efficiencies for the practices, since a step in the previous lab ordering workflow is completely eliminated. Another feature that received accolades from participants was the ability to import discrete clinical data from a Continuity of Care Document (CCD). Although this feature is currently underutilized by practices, the Meaningful Use 2 requirement to provide clinical summaries to patients and other interoperability requirements may propel clinical document exchange features into greater use. For some areas of the application, such as Allergies and Medications, there are multiple ways to complete a task. This is beneficial as healthcare providers adapt their unique workflows in the EHR, but in some cases, as with Allergies, there was minor confusion about what actions could be taken on an allergy in a given area of the application. There is no risk to patient safety, but the study revealed a reduced efficiency issue until the slight learning curve is overcome. Health Management, the integrated clinical decision support subsystem, also presented some challenges in terms of learnability, especially in terms of setup and customization. The system poses no inherent safety risks; however, the user interface for the Reminders Browser would benefit from a usability driven update. 2 P age Vitera Healthcare Solutions

26 Usability Testing: Vitera Intergy EHR INTRODUCTION The electronic health record application tested for this study was Vitera Intergy EHR, Version 9.0, an application used in an ambulatory setting. Designed to present medical information to healthcare providers in an outpatient setting for various specialties, Vitera Intergy EHR consists of a comprehensive electronic clinical system used to create, store, and retrieve patient data. Vitera Intergy EHR is fully integrated with the Vitera Intergy Practice Management System. The usability testing attempted to represent realistic exercises and conditions that occur in a typical medical practice environment. The purpose of this study was to test and validate the usability of the current user interface and provide evidence of usability in Vitera Intergy EHR. To this end, measures of effectiveness, efficiency, and user satisfaction, such as task success, time on task, and task group ratings, were captured during the usability testing. METHOD DESCRIPTION OF INTENDED USERS Intended users of Vitera Intergy EHR are doctors, nurses, physician s assistants, medical assistants, and anyone entering or accessing clinical data at an ambulatory medical practice. PARTICIPANTS A total of 12 participants were tested on Vitera Intergy EHR. Participants in the test were physicians and allied healthcare providers in ambulatory settings, ranging in years of experience with Vitera Intergy EHR and varied in specialty including Family Practice, Internal Medicine, Pediatrics, and others. Participants were recruited by a Certified Usability Analyst at Vitera Healthcare Solutions, and they were compensated with small branded items for their time pens and USB drives. In addition, participants had no direct connection to the development organization producing Vitera Intergy EHR. Participants were not from the testing or supplier organization. Participants were given the opportunity to have the same orientation and level of training as the actual end users would have received. The interface being tested was new to some of the participants, changes having been made prior to Version 9. A 5-minute training video used for customer training was shown to those who did not have prior experience using the new interface. The video explained general navigation of the system. For the test purposes, end-user characteristics were identified and potential participants were solicited. An example of the invitations sent to potential participants is provided in Appendix 1: Recruiting Methodology. 3 P age Vitera Healthcare Solutions

27 Usability Testing: Vitera Intergy EHR Recruited participants had a mix of backgrounds and demographic characteristics. The following is a table of participants by characteristics, including demographics, professional experience, computing experience and user needs for assistive technology. Participant names were replaced with Participant IDs so that an individual s data cannot be tied back to individual identities. TOTAL NUMBER AND DESCRIPTION OF PARTICIPANTS The total number of participants was 12. Their descriptions are indicated in the table below: ID Age Education Occupation/ Role Professional Experience (Practice Specialty, # yrs. practicing) Computer Experience (Beginner, Intermediate, Advanced) Product Experience (# years using Intergy EHR) Assistive Technology Needs (for vision, etc.) 1 37 D.O., Board Certified Physician Family Practice, 8 years Advanced 1 year Glasses Family Medicine 2 55 M.D. Physician Internal Medicine, 27 years Intermediate 1 year Reading Glasses 3 32 D.O. Post Graduate Physician Family Practice Family Practice, 2.5 years Advanced (did some programming in college) 2.5 years None 4 52 M.D. Pediatrician Pediatrics, 22 years Intermediate 5 years None 5 48 M.D. at University of MD Internal Medicine, 19 Intermediate, not a 5 years None AL School of Medicine years novice, but not a programmer 6 54 M.D. Primary Care Physician 7 48 M.D. Physician/ Medical Director Internal Medicine, 18 years General Pediatrics, Pediatric Hematology, 14 years 8 46 M.D. Physician Internal Medicine, 15 years 9 57 M.D. Family Physician Practice Owner Intermediate 5 years None Family Practice, 28 years Upper intermediate Has learned to setup Intergy for practice needs. Intermediate 5 years Glasses/contacts Advanced 1 year Eye Glasses - Bifocal 5 years None University of Iowa Physician Family Medicine, 24 years Intermediate 4 years None M.D HS degree, some Medical Assistant Family Practice, 18 years Advanced 1.5 years None college M.D. U of Tennessee Physician Family Practice, 30 years Intermediate 1 year None Table 1: Participants 4 P age Vitera Healthcare Solutions

28 Usability Testing: Vitera Intergy EHR Thirteen participants (matching the demographics in the section on Participants) were recruited, and twelve participated in the usability test. One participant failed to show for the study. Participants were scheduled for 60-minute sessions with 15 minutes in between each session for debriefing by the administrators and data loggers and to reset systems to proper test conditions. A spreadsheet was used to keep track of the participant schedule. STUDY DESIGN Overall, the objective of this test was to uncover areas where the application performed well that is, effectively, efficiently, and with satisfaction and areas where the application failed to meet the needs of the participants. The data from this study may serve as a baseline for future tests with an updated version of Vitera Intergy EHR. In short, this testing serves as both a means to record or benchmark current usability, but also to identify areas where improvements must be made. During the usability test, participants interacted with Vitera Intergy EHR. Each participant used the system in the same location and was provided with the same instructions. The system was evaluated for effectiveness, efficiency, and satisfaction as defined by measures collected and analyzed for each participant: Number of tasks successfully completed within the allotted time Time to complete the tasks Number and types of errors Path deviations Participants verbalizations (comments) Participants satisfaction ratings of the system Additional information about the various measures can be found in the Usability Metrics section of this report. TASKS A number of tasks were constructed that would be realistic and representative of the kinds of activities a user might perform with Vitera Intergy EHR. Tasks were selected to assess the usability of the changes made to the application navigation and new features added in this version. The tasks in the Usability Protocol used for this usability test were prioritized in accordance with the risks associated with user errors (see Appendix 3: Usability Protocol). 5 P age Vitera Healthcare Solutions

29 Usability Testing: Vitera Intergy EHR PROCEDURES Upon arrival, participants were greeted; their identity was verified and matched with a name on the participant schedule. Participants were then assigned a Participant ID. Each participant reviewed and signed a recording consent form (see Appendix 2: Recording Consent Form). A representative from the test team witnessed the participant s signature. To ensure that the test ran smoothly, two staff members participated in this test, the usability test administrator and the data logger. The usability testing staff conducting the test were experienced usability practitioners. The administrator moderated the session including administering instructions and tasks. The administrator also monitored task times, obtained post-task rating data, and took notes on participant comments. A second person served as the data logger and took notes on task success, task time, path deviations, number and type of errors, and comments. Participants were instructed to perform the tasks (see specific instructions below): As quickly as possible making as few errors and deviations as possible. Without assistance; administrators were allowed to give immaterial guidance and clarification on tasks, but not instructions on use. Without using a think-aloud technique. Task timing began once the administrator finished reading the question and said Begin now. The task time was stopped once the participant indicated he or she had successfully completed the task. Following the session, the administrator gave the participant the post-test questionnaire. Participants' demographic information, task success rate, time on task, errors, deviations, verbal responses, and post-test questionnaire were recorded into a spreadsheet. Scoring is discussed below in the Data Scoring section. Participants were thanked for their time and compensated. TEST LOCATION AND ENVIRONMENT Vitera Intergy EHR would be typically be used in an ambulatory healthcare setting. In this instance, testing was conducted in usability testing booths at VIBE 2012, Vitera Healthcare Solutions annual user conference held in Orlando, Florida. The booths contained a table, a computer for the participant, and a computer for the data logger. The administrator sat next to the participant to facilitate the test, and the data logger sat across the table from the participant where he could see the participant s face as well as the laptop screen to view the participant s interactions with Vitera Intergy EHR. For testing, Dell Latitude E6410 laptops running Windows 7, at 1024x768 resolution were configured with a 19-inch monitor attached to each computer. The participants used the 19-inch monitor, a mouse, and a keyboard when interacting with Vitera Intergy EHR. 6 P age Vitera Healthcare Solutions

30 Usability Testing: Vitera Intergy EHR The Vitera Intergy EHR software was set up by Vitera s in-house Information Technology resources according to the product documentation describing the Intergy EHR system setup and preparation. The application was running using a demonstration database. The system performance was representative of what actual users would experience in a field implementation. TEST FORMS AND TOOLS During the usability test, various documents and instruments were used, including: 1. Recording Consent Form 2. Usability Protocol, including the Post-test Questionnaire 3. Techsmith Morae Recorder software 4. Logitech Orbit AF Webcam and Audio Recorder Examples of the above documents can be found in Appendices 2 and 3, respectively. The Usability Protocol was devised so as to be able to capture required data. The participants interaction with Vitera Intergy EHR was captured and recorded digitally with screen capture software running on the test machine. A camera recorded each participant s facial expressions synced with the screen capture and verbal comments were recorded with a microphone. Each test session was observed by the data logger. PARTICIPANT INSTRUCTIONS The participant instructions were read from the Usability Protocol, which is available in Appendix 3: Usability Protocol. USABILITY METRICS According to the NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records, EHRs should support a process that provides a high level of usability for all users. The goal is for users to interact with the system effectively, efficiently, and with an acceptable level of satisfaction. To this end, metrics for effectiveness, efficiency, and user satisfaction were captured during the usability testing. The goals of the test were to assess: 1. Efficiency of Vitera Intergy EHR by measuring the average task time and path deviations 2. Effectiveness of Vitera Intergy EHR by measuring participant success rates and errors 3. Satisfaction with Vitera Intergy EHR by measuring ease-of-use ratings 7 P age Vitera Healthcare Solutions

31 Usability Testing: Vitera Intergy EHR DATA SCORING The table below details how tasks were scored, errors evaluated, and the time data analyzed. Rationale and Scoring Effectiveness: Task Success Task success was determined by assigning numeric weights for various levels of task success, as follows: Complete success (without assistance) = 1.0 Partial success = 0.5 Gives up or wrong answer = 0.0 A success score for each task was calculated by averaging the scores for each task. The results are provided as a percentage. Task times were recorded for successes. Observed task times divided by the optimal time for each task were calculated as a measure of optimal efficiency. Optimal task performance time, as benchmarked by expert performance under realistic conditions, was recorded when constructing tasks. Target task times were operationally derived by multiplying a benchmarked expert performance by a factor of 1.5, allowing for some buffer because participants were not trained to expert performance and the user interface had been recently redesigned. When a task had multiple valid paths to successful outcome, each task path was timed and the average was multiplied by the buffer to obtain the target task time. Effectiveness: Task Failures Efficiency: Task Deviations Efficiency: Task Time Satisfaction: Task Rating If the participant abandoned the task, did not reach the correct answer, or performed it incorrectly, the task was counted as a failure. No task times for failed tasks or tasks that exceeded the target task time were used in calculations. A qualitative account of the observed errors and error types was collected. The participant s navigation path through the application was recorded. Deviations occur if the participant, for example, went to a wrong screen, clicked on an incorrect menu item, followed an incorrect link, or interacted incorrectly with an on-screen control. This path was compared to the optimal path. Path deviations are reported on a qualitative level for use in recommendations for improvement. Each task was timed from when the administrator said Begin until the participant said, Done. If the participant failed to say Done, the time was stopped when the participant ceased performing the task. Only task times for tasks that were successfully completed and tasks that were completed at or under the target time were included in the average task time analysis. Average time per task and variance measures were calculated for each task for use in the results analysis. Participant s subjective impression of the ease of use of the application was measured by administering both a simple question on completion of each task group and a post-session questionnaire. After each task group, the participant was asked to rate Overall, these tasks were: on a scale of 1 (Very Difficult) to 5 (Very Easy). These data are averaged across participants. Table 2: Details of how observed data were scored. 8 P age Vitera Healthcare Solutions

32 Usability Testing: Vitera Intergy EHR RESULTS DATA ANALYSIS AND REPORTING The results of the usability test were calculated according to the methods specified in the Usability Metrics section above. Participants who failed to follow session and task instructions had their data excluded from the analyses. Of note, four tasks were inadvertently skipped by test administrators, so the number of participants was adjusted accordingly for those tasks. The usability testing results for Vitera Intergy EHR are detailed below. The results should be seen in light of the objectives and goals outlined in the Study Design section. The data yielded actionable results that, when corrected, will yield a material, positive impact on user performance. DISCUSSION OF THE FINDINGS The major areas of the usability test in light of the findings are discussed below. EFFECTIVENESS Review Active Allergies The task success score was 100%. Participants were easily able to review their active allergies using one of three valid paths for completing this task. Edit Allergies One participant failed the task due to adding a new allergy instead of editing the existing allergy. The participant initiated the allergy edit task from the Patient Summary and became confused when unable to find on the Allergy menu an option to edit allergies. The difficulty may be attributed to inexperience with the redesigned user interface. The Orientation Video shown to participants provided a general overview of the new user interface, but did not specifically cover the allergy edit workflow. Add Allergies The success score for this task was 100%. Participants were easily able to add patient allergies. Review Inactive Allergies The success score for this task was 92%, due to two participants requiring hints on how to complete the task. All participants were ultimately able to review the patient s active allergies and complete the task. Reactivate Inactive Allergies All of the participants were able to successfully change the status of an inactive allergy to Active, yielding a task success score of 100%. 9 P age Vitera Healthcare Solutions

33 Usability Testing: Vitera Intergy EHR Locate Health Reminders The task success score was 100%. All participants were able to locate patient health reminders. Identify Overdue Reminders The success score for this task was 100%. All participants were able to identify the overdue reminders in patient health reminders lists. Identify Guideline for Foot Exam The task success score was 95%. One participant required a hint to find the guideline health reminders list in the Patient Summary and the task was inadvertently skipped for another participant. Mark Reminder as Satisfied The success score for this task was 88% due to three participants requiring a hint to complete the task. Two participants were given a hint about right clicking a reminder to find more options, and one participant was given a hint to satisfying reminders from the Reminders Browser. Locate Reference At 75% success, this task was the least successfully performed of all in the study, with four participants requiring a hint to complete the task and one participant failing to complete the task due to giving up. All hints were about accessing the Reminders Browser to find the reference information. Review Current Medications The success score for this task was 100%. All participants successfully reviewed the patient s current medications. Review Previous Medications One participant failed the task, due to selecting the filter option that showed lapsed medications, but not all previous medications. The participant did not realize the error and indicated completion of the task. Discontinue Medications All of the participants successfully discontinued a medication, per the task instructions. Order an X-Ray All participants were able to successfully add the barium x-ray to the patient s orders. Order a Medication All participants were able to successfully order a medication using Electronic Transmission as the transmission method. 10 P age Vitera Healthcare Solutions

34 Usability Testing: Vitera Intergy EHR Renew a Prescription All but one participant was able to complete the prescription renewal task successfully. The participant who failed the task entered a new prescription for the medication instead of issuing a renewal. However, the correct medication was issued, so any safety issue posed by the error is negligible. Order a Laboratory Test The user interface and workflow for this task had been redesigned, and no training was provided to participants purposely in order to test the usability of the new UI. Given their lack of training, the task success score was still 92%, with 11 out of 12 participants successfully completing the task. The participant who failed the test neglected to enter a diagnosis for the lab test per the task instructions. Several participants indicated that they preferred a faster workflow in which they indicated diagnoses and tests for the lab order and then sent the information to a designated practice user for completion. The software supports this workflow, but that workflow was not tested as part of this usability test. Overall, participants were able to complete the lab order and indicated that the design for electronic lab orders was a vast improvement. Identify High Priority Drug Interactions The success score for this task was 100%. Participants were easily able to identify high priority drug interactions on the Drug Utilization Review Report window. View Full Drug Details The success score for this task was 100%. All participants located the monograph in the full drug details of the Drug Utilization Review Report window successfully. Review Other Interactions The success score for this task was 92%, with one participant failing the task. The participant incorrectly indicated the Warning Details as the other interactions. Describe Response to Interaction Checks By reviewing the drug interactions presented in the Drug Utilization Review, participants were easily able to describe how they would alter the patient s medications based on the information presented. The task success score was 100%. Locate Retrieved Continuity of Care Document (CCD) This task score was 96% with one participant requiring a hint to complete the task. The hint was about the location of navigation object that opened the list of patient CCDs, which was labeled Exchange. 11 P age Vitera Healthcare Solutions

35 Usability Testing: Vitera Intergy EHR Import CCD into Chart The success score for this task was 100%. All participants were about to successfully import clinical data from the CCD into the Chart. Review Health Reminders The task success score was 100%. All participants successfully reviewed the patient s health reminders after importing clinical data from the CCD into the patient s chart. EFFICIENCY Review Active Allergies One participant exceeded the target time for this task, due to difficulty finding the button to open the Allergies List in the Chart Navigation Bar. As this task was the first of the usability test tasks, this error is likely attributed to the participant s unfamiliarity with the new user interface design and the button order not being customized to the participant s preference, which is a feature of the software. Edit Allergies Two participants exceeded the target task time due to searching in the menu for an allergy list or exploring other controls for an edit option, rather than selecting or right clicking the existing allergy. Both participants initiated the allergy edit task from the Allergies List. Their difficulty may be attributed to inexperience with the redesigned user interface. The Orientation Video shown to participants provided a general overview of the new user interface, but did not specifically cover the editing an allergy workflow. Add Allergies All participants completed the task at or under the target task time. Participants were easily able to add patient allergies. The workflow for adding an allergy is available from any of the three valid paths for working with patient allergy lists. Review Inactive Allergies Five participants exceeded the target time for this task. While there are three valid paths for working with patient allergy lists, only one of those paths supported the ability to review inactive allergies. The five participants who used one of the other two paths to work with allergy lists took some time to realize that the path did not support the review of allergy lists before switching to the valid path for the task. Reactivate Inactive Allergies Three participants exceeded the target time for this task. They all completed the task successfully; it just took them longer than others. The difficulty for those participants centered on finding the option to change the status. Of note, the three participants who had difficulty with this task declined watching the Orientation Video. 12 P age Vitera Healthcare Solutions

36 Usability Testing: Vitera Intergy EHR Locate Health Reminders The participant who exceeded the target time for this task looked on several windows before finding the right one. Participants located the patient s health reminder either on the Patient Summary (40%) or by opening the Health List (60%). Identify Overdue Reminders All participants were able to locate the overdue reminders within the target time using one of three possible paths. No errors or path deviations were observed. Participants successfully located the overdue reminders from the Patient Summary (30%), the Health List (60%), or the Health Reminders Browser (10%). Identify Guideline for Foot Exam One participant exceeded the target time, due to looking for the information on the Patient Summary. The participant was the only non-physician participant in the study, and cited unfamiliarity with the system as the reason for the delay. Participants successfully located the guideline for the foot exam from the Health Reminders Browser (11%), Reminders List (33%), Reminders List Hover (22%), Patient Summary Hover (33%). Mark Reminder as Satisfied Three participants did not complete the task within the target task time. Two of these participants had selected the reminder from the Patient Summary and then had difficulty because the Reminders Browser that opened did not display the Work Pane by default. The other participants selected the reminder in the Reminders List and then moved to the Reminders Browser to mark the reminder as satisfied, but had difficulty because the lists were not synchronized. This workflow was problematic for five participants overall. Locate Reference Eight participants did not complete the task within the target task time. The participants who completed the task within the target time all started from the Reminders Browser, which was the only location from which the guideline reference was accessible. Despite the number of participants who had trouble locating the guideline reference, one participant commented that the reference was perfectly placed. The wording of the test protocol was another source of confusion in that the task was for participants to locate reference information for the Depression Screening reminder ; however, the label for the reference link on the user interface was Knowledge Link. Review Current Medications All participants were able to complete the review current medications task successfully and within the target time. No errors or path deviations were observed. Review Previous Medications One participant exceeded the target time, due to selecting the filter option that showed lapsed medications, but not all previous medications. The participant realized the error and corrected it with some difficulty finding the option. 13 P age Vitera Healthcare Solutions

37 Usability Testing: Vitera Intergy EHR Of the three valid paths in the system for reviewing previous medications, participants who viewed the patient s medications list from the Patient Summary or Info List had to switch to the Medications List to view the previous meds, making the task times naturally longer for those participants. Participants with the shorter task times were already viewing the patient s medications from the Medications List. Discontinue Medications Three participants did not complete the task within the target task time. The difficulty for those participants centered on finding the option to discontinue the medication. Also, adjusting the medications list filter added to one participant s time. The participants lack of familiarity with the new user interface design is the likely cause of their time overruns. Order an X-Ray Two participants exceeded the target time for the task. Both participants had indicated a moderate to low familiarity with the CPOE features respectively familiarities of 3 and 2 (of 5). Both participants had difficulty finding the order, selecting multiple order tabs to find the X-ray order. Also, one participant incorrectly attributed the order to the wrong diagnosis, but quickly corrected the error. Order a Medication Three participants exceeded the target time for the task. All three participants entered a new drug instead of using a favorite prescription. They also had difficulty entering the drug name in the search field due to misspelling or typographical error. In the test data, the prescription that the participants were asked to add was on a favorite prescriptions list with the correct diagnosis. Only 6 of the 12 participants checked for the favorite and used it to order the medication. The participants who used the favorite had faster task times. However, when using their office systems, it is likely that providers are more familiar with the prescription favorites that are available. Several participants navigated to the Medications List to order the medication, indicating a preference for using the Medications List for entering prescriptions. Seven participants encountered difficulty due to misspelling the drug name or typographical errors when entering the drug name into the Search field. Renew a Prescription Of the two participants who exceeded the target time for the task, one had difficulty finding the patient renewals on the Rx Orders screen and the other navigated to the Medications List and issued the renewal there. Both participants who exceeded the task time had indicated a familiarity with the CPOE features of 0 out of 5. Order a Laboratory Test One participant exceeded the target time for the task, an acceptable result considering the user interface and workflow for the task had been redesigned and no training was provided to participants purposely in order to test the new UI usability. 14 P age Vitera Healthcare Solutions

38 Usability Testing: Vitera Intergy EHR Path deviations and errors observed for this task included diagnosis and lab test selection errors and searching in the wrong lists for the lab test. These path deviations and errors can all be attributed to unfamiliarity with the redesigned user interface and workflow. Excluding the participant who failed the task, all other participants recognized and corrected their selection errors instantly and issued the lab order with the correct selections. Several participants indicated that they preferred a faster workflow in which they indicated diagnoses and tests for the lab order and then sent the information to a designated practice user for completion. The software supports this workflow, but that workflow was not tested as part of this usability study. Overall, participants indicated that the new design for electronic lab orders was a vast improvement. Identify High Priority Drug Interactions All participants completed the task at or under the target time. Several participants commented that the redesigned user interface for the Drug Utilization Review Report window was intuitive and easy to use. Two participants suggested that high level drug warnings should be in red or otherwise made more prominent on the screen. No path deviations or errors were observed for this task. View Full Drug Details One participant exceeded the target time for completing the task. The participant was the only Medical Assistant tested in the study, and was not authorized to issue prescriptions. In an office environment, the practice has the option to set up their system to prohibit unauthorized users from issuing prescriptions. The system also supports a supervised provider workflow for prescriptions. Review Other Interactions Excluding the one participant who failed the task, all other participants were able to complete the task at or under the target task time. No path deviations or errors were observed with these participants. Describe Response to Interaction Checks All participants completed the task at or under the target task time. The interaction with the system for this task was simply viewing the screen, so path deviations were not a factor in this task. Locate Retrieved CCD The three participants who exceeded the target time for this task had difficulty finding the navigation object that opened the patient s exchange documents. Import CCD into Chart Three participants exceeded the target task time due to thoroughly reviewing the CCD prior to completing the import. One participant read the instructions and talked to moderators about the details of the data in the CCD. Another participant initially 15 P age Vitera Healthcare Solutions

39 Usability Testing: Vitera Intergy EHR selected the Retrieve button, but then quickly realized that selecting the Import button was the correct path. Review Health Reminders Two participants exceeded the target time for this task. Both were confused about the location of the Reminders List. One participant indicated that he was looking for something that said reminders, whereas the navigation object for the Reminders List is labeled Health. Interestingly enough, both participants had completed the Locate Health Reminders task in the fastest time. SATISFACTION After the completion of each task group, participants rated the ease or difficulty of completing the tasks in the group. The 5-point rating scale ranged from 1 (Very difficult to use) to 5 (Very easy to use). Overall ease-of-use and satisfaction ratings were given. Allergies All participants agreed that it was easy to work with patient allergies (mean agreement rating = 4.25). Participants giving the highest scores indicated that they were familiar with the system or that the new user interface was intuitive. One participant suggested that right-clicking might not be intuitive for other providers; however, a participant who was new to the system indicated that it was very intuitive to try right clicking. Four participants indicated that their performance would have been better if they had been more familiar with the new user interface by using it daily in their office environment. One participant indicated unfamiliarity with allergy entry because the Medical Assistants at the participant s practice typically entered the patient allergies and the practice does not track environmental allergies. Clinical Decision Support Reminders Ease of use and satisfaction for the health reminders task group was the lowest score of all task groups (mean agreement rating = 4.00 or 80% satisfaction). Two tasks in this group had low average task success rates at 88% and 75%. Completion times for CDS reminder tasks were some of the lengthiest in the study. At the start of the usability test, participants indicated a low familiarity with the feature (mean familiarity rating = 2.82 or 56% familiarity), second only to their lack of familiarity with CCD. Participants struggle to complete some tasks combined with their lack of familiarity with the Health Reminders system contributed to the relatively low ease-of-use and satisfaction score for this task group. Several participants commented on the importance of incorporating clinical decision support into their practice workflow, and almost all participants had suggestions for making the system more usable. Medications Ease of use and satisfaction for the medications task group was the highest score of all task groups (mean agreement rating = 4.88 or 97.5% satisfaction). Participants indicated that it was easy and intuitive. They were familiar with the workflow (mean familiarity rating = 4.82 or 96% familiarity) and liked the new user interface. User interface features 16 P age Vitera Healthcare Solutions

40 Usability Testing: Vitera Intergy EHR that were specifically mentioned were easy to find menus, menus easy to understand, right click functionality, and no extra clicks. For the lower scores that were given, one participant indicated a desire to be able to search by drug type and another participant indicated having to get used to different filter options on the patient medications list introduced in the redesigned user interface. Two participants indicated that prescriptions were typically entered by Medical Assistants at their practice. Computerized Physician Order Entry Participants agreed it was easy to complete tasks successfully in the Computer Physician Order Entry (CPOE) task group (mean agreement rating = 4.13 or 83% satisfaction), even though they indicated a moderate familiarity with those features (mean familiarity rating = 3.27 for orders and 3.55 for lab orders). Two participants indicated that they do not use the CPOE features and another participant indicated recent adoption of CPOE features, having previously written orders on the encounter form for the nurse to fulfill. One of these participants stated, everything was exactly where I expected, even though this is the first time using this. Four participants indicated that the new user interface for electronic lab orders was easier. Three participants expressed a preference for having their Medical Assistants complete the lab orders for them, a workflow that is supported by the system. One participant liked that diagnoses entered for the patient in other areas of the system were then prominently displayed for selection in orders. Several participants indicated a strong preference for entering prescriptions from the Chart Medications List, as opposed to entering them in Orders. Participants who gave the lowest scores said their reason was that the system was new. Drug Utilization Review Despite indicating a moderate familiarity with the drug interaction warning features of the system (mean familiarity rating = 3.18), participants awarded a high ease-of-use and satisfaction rating to the Drug Utilization Review task group (mean agreement rating = 4.67). All participants indicated that the warnings on the redesigned DUR screen were much easier to read, with comments such as, dramatically improved, much more concise, this one is usable totally usable, and much cleaner DUR screen. Two participants indicated that they do not typically look at the DUR. Two participants indicated that they use alternate methods for getting drug interaction warnings (phone or third-party software); however, one of these participants considered discontinuing use of the alternate method and using the system DUR. Two participants suggested that allergy and drug-drug interactions be displayed in red or more prominently. One participant suggested that the high level warnings be prominently displayed throughout the system, especially in patient medication displays. Continuity of Care Document (CCD) Despite assigning the lowest familiarity rating of this system (mean familiarity rating = 0.91 or 18% familiarity), participants gave CCD a favorable ease-of-use and satisfaction rating (mean agreement rating = 4.29 or 86% satisfaction). A number of participants indicated that their practice was not familiar with the exchange document functionality 17 P age Vitera Healthcare Solutions

41 Usability Testing: Vitera Intergy EHR in the system because they were simply never provided with CCDs by other doctors or healthcare providers. Participants particularly liked the ability to import discrete clinical data from the CCD into the chart, and the fact that the system indicated the clinical data on the CCD that was already in the chart. One participant discussed the current difficulty with incorporating data from outside providers into the electronic chart, such as faxed or scanned charts from another provider. He complained that the faxed or scanned charts could be quite large, and were accessible from the chart images instead of the encounter notes. This participant was very impressed after viewing the import and seeing the discrete data being added to the appropriate places in the chart with notation of the data source. Overall Ease of Use Participants agreed that the system was easy to use (mean agreement rating = 4.33 or 87% satisfaction). Four participants commented on being unfamiliar with the new interface. Features that participants mentioned specifically as liking included the redesigned Drug Utilization Review, standardization within the new overall user interface, the Home Page for the Chart. Two participants indicated minor concerns with the redesigned lab orders window. One provider cited general dislike of computerized electronic health records and a preference for using paper superbills. Overall Satisfaction Participants were satisfied with the system (mean agreement rating = 4.45 or 89% satisfaction). Participant s comments about overall satisfaction were very positive, such as: Better than the other system (older version). One of the reasons I came to the conference to see new version. Going in the right direction keep it up! At the conclusion of the test, many participants discussed other features of the system that they would like to see improved and gave many ideas for improvement. 18 P age Vitera Healthcare Solutions

42 Usability Testing: Vitera Intergy EHR MAJOR FINDINGS A discussion of the interpretation of the quantitative findings and verbal comments of the participants is provided below: New Navigation Positive Results Generally, participants were able to navigate the new interface with ease. Comments from participants: It s much more intuitive and easier to find things. The formatting and the way things are displayed are more standard. All the categories look the same, so it's easy to figure out. (P1) I like the easy to find drop-down menus. They are easy to understand. (P1) I m used to my current system, but I can see that when I m familiar with this new interface, this will be great. This is better than the previous version. (P2) I like the way that I can see things. I don t have to click back and forth, back and forth. (P4) I've been doing it a long time, so even though it's in a different spot, it's pretty much the same thing. (P4) I like the new look and navigation is easier than I expected it to be. I like that I can go back to the home page. (P7) Things were easy to read. My only hesitation was a struggle to find things that were in new spots. The icons make it so you are not flipping whole pages. It seems like it's quicker. (P9) There s no place like home. (P12) in reference to the new Home button. Usability Issues/Concerns There was an initial, slight learning curve, but once but once the participants understood how the new user interface was structured, they found navigation to be easy. Allergies Positive Results Allergy information is prominently displayed and easily accessible. There is more than one way to add and edit allergies which is beneficial as healthcare providers are able to adapt their unique workflows in Vitera Intergy EHR. 19 P age Vitera Healthcare Solutions

43 Usability Testing: Vitera Intergy EHR Usability Issues/Concerns Several users had difficulty finding the inactive allergies for the patient, which required a drop-down menu to be accessed and the Include Inactive view to be selected. There was minor confusion about what actions could be taken on an allergy in a given area of the application. There is no risk to patient safety, but the study revealed a reduced efficiency issue until the slight learning curve is overcome. Clinical Decision Support Reminders Positive Results Two participants commented on the necessity of implementing clinical decision support into their medical practice. Comment from participant: Health Maintenance - that's something we HAVE to implement. (P1) One participant liked that the health reminders displayed on the Patient Summary. Comment from participant: I like having this page (points to Patient Summary). I like having health reminders here, because before I honestly don't know where they were. And that was aggravating, because you've got all these things you need to be doing, but trying to find them was not as easy. And here they're stuck in front of you, so you've got to deal with them. So, I like this system much better. You're definitely going in the right direction - keep it up. (P12) Most participants indicated that the system was intuitive or easy to use, but also cited unfamiliarity with the system as the reason for errors in completing the tasks. Comment from participant: I think it's intuitive enough that you can navigate through without having seen it before. (P6) Usability Issues/Concerns Two participants commented that the Health Reminders browser was busy. Comment from participant: I just think that the screen up here that shows the reminders is somewhat busy takes some getting used to. It has a lot of great information in it. (P1) Five participants expected that the Reminders List and the list of reminders in the Reminders Browser would be synchronized so that when they selected a reminder in one list, the same reminder was selected in the other. One participant commented about wanting more automation in the system, so that patient health reminders would be automatically marked as satisfied in response to certain data being entered or received. The participant was unaware 20 P age Vitera Healthcare Solutions

44 Usability Testing: Vitera Intergy EHR of the available automation features for satisfying reminders and how to set them up. Medication Order Positive Results Medication orders were executed swiftly, particularly when participants used the Favorites by Diagnosis option. There are multiple paths for working with medication orders, which is beneficial as healthcare providers are able to adapt their unique workflows in Vitera Intergy EHR. Usability Issues/Concerns The learnability issues presented for Allergies would be the same for Medications; however, once participants resolved the issues for allergies, they were able to apply that knowledge to medications. In fact, two participants cited the standardization of the interface as a benefit of the new system design. New Lab Orders System Positive Results Participants were able to navigate the new Lab Order interface very easily. Comments from participants: This is the first time seeing this lab ordering screen, but everything was exactly where I expected, even though this is the first time using it. (P1) There was no scrolling, and it was all drop-downs. The other thing I liked is that I wasn't back and forth between the mouse and the keyboard. So, I can keep my hand on the mouse and keep talking to the patient. If I can stay with one mode of entry, it makes it much smoother so I can just click away while I'm talking and listening to my patient. (P1) Participants were able to complete lab orders faster than in the previous version. Several participants who currently do not order the lab tests themselves (have an assistant do it by simply creating an order) said they would definitely now order the lab tests themselves directly since it was so simple. Comment from participant: I like this! It s so easy there will be no more tasks to the nurse for lab orders. (P8) 21 P age Vitera Healthcare Solutions

45 Usability Testing: Vitera Intergy EHR Usability Issues / Concerns Inconsistency with same feature, but different functionalities based on location in system. There was some confusion about how to execute tasks based on the ability to perform them in multiple areas of the system, and not all areas contained the same functionality. Comment from participant: I like the new Lab Ordering screen, but still won't do it myself. Still easier to fill out the superbill. (P2) Redesigned Drug Utilization Review (DUR) Report Window Positive Results Participants responded very favorably to the new interface of the DUR. Comments from participants: The new DUR is like night and day compared to the previous version. It s much more concise. The order of the interactions is better. It s much clearer. It s dramatically improved. It s totally usable. This is what we were looking for. (P1) I like this DUR format much better. (P2) This DUR went from a 1 to a 4 from previous version to this version. (P8) This is a much cleaner DUR screen. (P10) This is a lot better. Nobody pays attention to the current DUR. Previously, nothing was highlighted, and there was a lot of noise. (P12) Continuity of Care Documents (CCD) Positive Results Participants commented that the interface was intuitive and stressed a lack of familiarity with the system as the reason for errors in completing tasks. Comments from participants: The ability to import records is dramatically impressive. (P1) As far as being intuitive what is it and where to click it's easy to use. (P5) It's intuitive. You can look around and see where you're supposed to go. If you don't know, you can right click or left click, and then usually get there. (P6) It shows you what's already in the chart and it makes it easy to add new things by just checking it. (P7) I like being able to incorporate this (pointing to CCD) into the record. (P10) Two participants commented that they liked that unimported clinical data was sorted to the top of each data type list, making the data easier to find and import and suggested further refinements. 22 P age Vitera Healthcare Solutions

46 Usability Testing: Vitera Intergy EHR Several participants commented that they liked that the CCD Import window clearly indicates the CCD data that is already in the chart. Usability Issues/Concerns Participants had difficulty understanding the difference between generated and retrieved CCDs. The icons for the generated and retrieved CCDs are not intuitively meaningful. Participant comment: I don't know the difference between (points to icons) one of these has blue icons and the other one has green icons. (P1) Some participants found the terminology used for interoperability documents confusing. The navigation object and lists are labeled Exchange or Exchange Documents, but the document is typically referred to by its type, such as CCD, CCA, or CRS. One participant suggested adding an indicator when a newly retrieved CCD is available in the patient s chart and an indicator when the CCD has unimported clinical data on it. 23 P age Vitera Healthcare Solutions

47 Usability Testing: Vitera Intergy EHR AREAS FOR IMPROVEMENT Below is a discussion of the interpretation of the findings, based on the results and participant comments. Edit Allergies Consider adding a graphical option for allergy edit. Provide training and documentation to orient users to the new UI. Review Inactive Allergies Provide a mechanism for users to access inactive allergies from all patient allergy list displays and analyze other differentials. Reactivate Inactive Allergies Consider adding a quick selection option for reactivating an inactive allergy. Provide more options for editing allergies from the Patient Summary. Overall Clinical Decision Support Reminders Synchronize the selection of a reminder on the Reminders List and Reminders Browser list. Redesign the Health Reminders user interface to be less busy. Add a way for providers to view the history of each health reminder for the patient, such as what occurred to satisfy the health reminder in past visits. Change the label of the button for accessing the health reminders from Health to Reminders. Mark Reminder as Satisfied Redesign the work pane for the Reminders Browser to be more compact so that providers can see more of the reminders list. Allow users to satisfy multiple reminders at once. Provide more functionality for satisfying reminders automatically, based on creation of orders, entry of data, or return of results or upon scanning an image that contains specific text in the title or attributes. Improve the training because participants were requesting features that are already available in the system. Move the Save button on the work pane to the right side because that position is standard. When a health reminder is selected in the Patient Summary, open the Reminders Browser with the Work Pane displayed by default. 24 P age Vitera Healthcare Solutions

48 Usability Testing: Vitera Intergy EHR Locate Reference Make the guideline reference more prominent in the display. Standardize the terminology for the guideline reference. Make the guideline reference accessible from other locations where health reminders are displayed. Review Previous Medications Ensure that training and documentation cover the medication list filters. Order an X-Ray Consider providing a search for orders. Perform further usability studies on CPOE features of the system. Order a Medication Clear the drug search field automatically after the medication order is added. Consider adding a way to direct the provider to the prescription favorites list if a diagnosis for the patient has a matching prescription favorite. Provide more intelligence- and analytics-based options for displaying quick selection lists for prescriptions. Renew a Prescription Prior to issuing a new prescription for a patient, check to see if the patient has a matching prescription that is eligible for renewal. Prompt the user to issue a renewal instead of a new prescription if one is found. Order a Laboratory Test Make the search fields in the Lab Orders window more prominent, as only one participant used the Search field for finding lab tests. Provide training and documentation for the new lab order user interface. Online help and computer-based training is a standard deliverable with the software. Identify High Priority Drug Interactions Two participants suggested making high-level drug warnings more prominent on the screen. All participants were able to locate high priority warnings on the Drug Utilization Review Report window, so no safety issue was detected with the current display. However, it is still a good idea. View Full Drug Details Conduct further usability studies focused on prescription writing by nonphysician providers and supervising provider workflows. 25 P age Vitera Healthcare Solutions

49 Usability Testing: Vitera Intergy EHR Continuity of Care (CCD) Consider changing the wording of retrieve and import because they are similar or provide more guidance on the intended use of those buttons. Redesign the icons for generate CCD and retrieve CCD to be more intuitive. Add a setting to the CCD import that would filter the CCD inventory to show only unimported findings. Add a select / deselect all for the unimported findings on the CCD inventory. Add an indicator when a newly retrieved CCD is available for the patient. Participants suggested a red exclamation mark on the Exchange button or highlight around the Exchange button. Add an indicator when a CCD has unimported data. Participant suggested finding a way to light up the Import button. 26 P age Vitera Healthcare Solutions

50 Usability Testing: Vitera Intergy EHR APPENDICES The following appendices include supplemental data for this usability test report. Following is a list of the appendices provided: 1: Recruiting Methodology 2: Recording Consent Form 3: Usability Protocol 27 P age Vitera Healthcare Solutions

51 Usability Testing: Vitera Intergy EHR Appendix 1: Recruiting Methodology Vitera Healthcare Solutions customers that were registered for Vitera s Annual User Conference, VIBE 2012, were contacted about participating in a usability study for Vitera Intergy EHR. Only registered customers in a clinical role were contacted. The following was sent to prospective participants, and those who responded were scheduled on a first-come-first-serve basis until all desired slots were taken (only one of the 14 slots was not taken). Several responders who wanted to attend a usability session were not able to find a time slot to fit their schedules. Dear Potential Participant, This year at the VIBE 2012 Conference, we will have a Usability Lab setup, and I would like to invite you to participate in a Vitera Intergy EHR Usability Testing session in the St. Augustine D room. We will be running Usability Tests on Vitera Intergy EHR to help improve the product. In the session, you will be asked to perform several tasks in IEHR on computers in the Usability Testing Lab and give feedback. Each test session will last approximately one hour. Rest assured, we will be testing the software and not the participants! If you are interested in helping improve the product, please reply with your 1 st, 2 nd and 3 rd time preferences from the options A - G shown below (for example F, C, A). Then I will send a follow up confirming your appointment and provide more details. Please note that the Usability Testing sessions do overlap with the other sessions at the conference except for the Free Time - 5:30 pm - 6:30 pm slot on Thursday and the Breakfast - 7:30 am 8:30 am slot on Friday. From the table below, please select 3 time options in order of 1 st choice, 2 nd choice, 3 rd choice. For example F, C, A. (scratched out slots are already booked): Those who participated last year really enjoyed the opportunity to give feedback on Vitera Intergy EHR. Please let me know if you have any questions. 28 P age Vitera Healthcare Solutions

52 Usability Testing: Vitera Intergy EHR Appendix 2: Recording Consent Form Recording Consent Form Thank you for participating in our usability research. We will be recording your session to allow Vitera Healthcare Solutions, LLC staff members who are unable to be here today to observe your session and benefit from your comments. Please read the statement below and sign where indicated I understand that my usability test session will be recorded. I grant Vitera Healthcare Solutions, LLC permission to use this recording for internal use only, for the purpose of improving the products being tested. Signature: Print your name: Date: 29 P age Vitera Healthcare Solutions

53 Usability Testing: Vitera Intergy EHR Appendix 3: Usability Protocol Vitera Intergy EHR Usability Protocol VIBE September 2012 Participant #: 30 P age Vitera Healthcare Solutions

54 Background and Introduction [5 minutes] Thank you for agreeing to participate in this usability study. Today you will be helping us evaluate workflows pertaining to Meaningful Use phase 2 requirements. In a moment, I ll be asking you to complete these workflows using Intergy EHR. I ll be taking notes and so will <note taker>. As you go through the workflows, please keep in mind that it is Intergy EHR under review here, not you. You may arrive at a point where you re not sure what you are supposed to do. Just take your best guess, and if you get really stuck, let me know. We really want to observe how you would expect to use the system if we weren t here. Also, I want to ask you to hold your questions until the end since the workflows are being timed. Please keep in mind that the data populated in the EHR may not be clinically accurate, and we understand that it may not be applicable to your specialty. I would like to video record this session to help me later with my report. Therefore, to help with this data collection, we are seeking your consent to video record this session. <Ask them to sign the consent form.> Do you have any questions before we get started? <BEGIN NOW recording the session.> 31 P age Vitera Healthcare Solutions

55 User Profile and Vitera System Questions [5-10 minutes] 1. Please tell me a little bit about yourself your job title and role description/responsibilities, and how long you ve been using Intergy EHR. 2. Which features do you use most frequently? Please list your top 2 or 3. (e.g. orders, prescriptions, labs, notes, flowsheets, tasks, etc. 3. What are your top 2 main likes/dislikes about Intergy EHR? 4. On a scale of 1 to 5 with 5 being most familiar, how would you rate your level of familiarity with these workflows in Intergy EHR: a. Maintaining Allergies b. Reviewing and Satisfying Health Reminders c. Maintaining Medications d. Creating Orders (using the Orders System) e. Ordering Lab Tests f. Reviewing the DUR g. Reviewing and Retrieving CCDs 5. What version of Intergy are you currently using at your practice? a. If earlier than the GA version, ask how long. i. If not GA, or less than a week on GA, then ask if they ve seen the presentation Intergy EHR Introduction this week at VIBE. a. If not on GA, and If haven t seen the presentation, then say, We are going to show you a video that describes the changes, so you can be properly prepared for the tasks you will be performing. <If applicable, show the 5-minute video from the Learn tab in My Day.> 32 P age Vitera Healthcare Solutions

56 EHR [20-30 minutes] What you will be looking at today is Vitera Intergy EHR populated with a mock patient s information. I will be asking you to perform some workflows that a clinician might perform with Intergy EHR. I will ask you to start when I say BEGIN NOW, and please tell me when you think you ve completed the workflow by saying DONE. Scenario 1: Jeff Adams has come in today for a Follow-up visit for his hyperlipidemia, but he also complains of stomach pain. Allergies 1. Open Jeff Adams chart so you can review his allergies with him. 2. Locate and review Jeff s Active allergies BEGIN NOW 3. Jeff says that his reactions to penicillin are not correct. He gets shortness of breath, not hives when he takes penicillin. a. Update the allergy to change the adverse reaction to shortness of breath. BEGIN NOW 4. Jeff also states that he gets hives when he takes Sulfa Drugs. a. Add Sulfa Drugs as a new allergy for Jeff, and note hives as the adverse reaction. BEGIN NOW 5. Jeff indicates his seasonal allergies have started again a. Locate Jeff s Inactive allergies BEGIN NOW b. Re-activate Jeff s Pollen Allergy BEGIN NOW Facilitator Comments Ease of Use: On a scale of 1 to 5 with 5 being very easy to use how would you rate the ease of use with the system when completing these tasks? Why? P age Vitera Healthcare Solutions

57 Scenario 2: Having updated Jeff s allergies condition over time, you are ready to proceed with reviewing Jeff s Health reminders. NOTE: The data used for the usability study triggered interventions in Vitera Intergy EHR Health Management for patient problems, medications, medication allergies, demographics, lab tests and results, and vital signs. Note that reference information can be accessed the same way for each kind of intervention. Reminders 1. Locate Jeff s Health Reminders. BEGIN NOW 2. Tell me which reminders are overdue BEGIN NOW 3. Tell me which guideline is triggering the Foot Exam reminder. BEGIN NOW 4. You have asked Jeff about some of these reminders, and he says that he already had an Eye Exam done last week, a. Mark the Eye Exam reminder as satisfied. BEGIN NOW 5. You are not sure that a Depression Screening needs to be done so frequently for a patient of this age. Look for reference information for the Depression Screening reminder. BEGIN NOW Facilitator Comments Ease of Use: On a scale of 1 to 5 with 5 being very easy to use how would you rate the ease of use with the system when completing these tasks? Why? 34 P age Vitera Healthcare Solutions

58 Scenario 3: After reviewing the reminders, you are now ready to review Jeff s meds with him. Meds 1. Locate and review Jeff s current meds. BEGIN NOW 2. You notice Jeff is taking Lipitor, you want to see if he has ever been on any other medications for hyperlipidemia a. Review his past meds to see what other medications have been given to Jeff BEGIN NOW 3. Jeff states that he is no longer taking Ambien. a. Discontinue Ambien. BEGIN NOW Facilitator Comments Ease of Use: On a scale of 1 to 5 with 5 being very easy to use how would you rate the ease of use with the system when completing these tasks? Why? 35 P age Vitera Healthcare Solutions

59 Scenario 4: You have finished examining Jeff, and you are now ready to create Orders. You suspect Jeff has a Peptic Ulcer, and you are continuing treatment of his hyperlipidemia. You ll be creating 4 orders, but please stop before you Save the Orders CPOE (Computerized Physician Order Entry) For this exercise, we are skipping the encounter note. 1. For Jeff s ulcer: a. Order a Barium Xray. BEGIN NOW b. Electronically prescribe the drug Amoxicill-Clarithro-Lansopraz for his Peptic Ulcer for twice a day for 30 days. BEGIN NOW 2. For Jeff s hyperlipidemia: a. Renew the existing prescription for Lipitor BEGIN NOW b. Order a Lipid Panel (when you order the lab, we d like you to select the Lab Order button in the Labs tab so we can test the real-time lab ordering workflow.) BEGIN NOW. We re not connected to a printer, so just close the window and answer Yes to the message. Facilitator Comments Ease of Use: On a scale of 1 to 5 with 5 being very easy to use how would you rate the ease of use with the system when completing these tasks? Why? 36 P age Vitera Healthcare Solutions

60 Scenario 5: Now we will perform the DUR test scenario. DUR 1. Select the Save option and the system will display a DUR report. 2. Locate the highest priority warnings and describe them to me. BEGIN NOW 3. Navigate to the full monograph of the first Drug Interaction warning. BEGIN NOW 4. Review all other warnings. BEGIN NOW 5. Describe how you would respond to the higher priority warnings. BEGIN NOW Facilitator Comments Ease of Use: On a scale of 1 to 5 with 5 being very easy to use how would you rate the ease of use with the system when completing these tasks? Why? 37 P age Vitera Healthcare Solutions

61 Scenario 6: Now we are going to talk about Interoperability. Part of the Meaningful Use measures is about interoperability and exchanging clinical data. NOTE: CCD data used for the usability study included medications, problems, and medication allergies. The Import CCD screen allows comparison of chart data and CCD data, enabling the participant to reconcile the two sets of medications, problems, and medication allergies as part of the import process. This process includes the ability to bring in data from the CCD and to remove data from the chart. CCD Do you know what a CCD document is? If not, then explain. A CCD is a Continuity of Care document used in the Interoperability initiative where doctors can exchange clinical info. with each other. A document would be generated by one doctor from clinical information in his system and then shared with another doctor, either sent directly to the doctor, or given to the patient to share with the doctor. The document is not just text but discrete data that can be imported into the chart. Your next patient, Steven Carlson, has arrived for his annual exam, and he brought a CCD from another provider. Your medical assistant has already retrieved the CCD document from the patient s flash drive and put it into Steven s chart as an Exchange document. 1. Open Steven Carlson s chart. BEGIN NOW 2. Locate and review the newly retrieved CCD exchange document. BEGIN NOW 3. Now bring the information from the CCD into Steven s Chart that is not already in his chart BEGIN NOW This test is about finding the reminders again, not about identifying a new reminder: 4. One of the meaningful use measures is that the reminders would auto update once the CCD has been imported. So, show me where you would go to review the Health Reminders to see if anything new was triggered for the patient. BEGIN NOW Facilitator Comments Ease of Use: On a scale of 1 to 5, how would you rate the ease of use with the system when completing these tasks? Not at all easy to use Very easy to use Why? 38 P age Vitera Healthcare Solutions

62 Wrap Up [5minutes] 1. On a scale of 1-5, where 1 = not at all easy to use and 5 = very easy to use, overall, how would you rate the ease of use of the features you used within the EHR today? Not at all easy to use Very Easy to use Why did you give this rating? 2. On a scale of 1-5, where 1 = not at all satisfied and 5 = very satisfied, how would you rate your level of satisfaction of the features you used within the EHR today? Not at all satisfied Very satisfied Why did you give this rating? 3. If time permits: What would you like to see incorporated in the EHR? Probe: For user requirements (features or capabilities) that are currently not in the product? a. Would this significantly improve your ability to do your job? Why or why not? 4. Are there any further comments that you d like to add? THANK YOU FOR YOUR TIME TODAY. Task: Run batch file to restore db. 39 P age Vitera Healthcare Solutions

63

64

65

66

67

68

69