ONC HIT Certification Program

Transcription

1 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 07-Aug-2015 ONC HIT Certification Program Test Results Summary for 2014 Edition EHR Certification Part 1: Product and Developer Information 1.1 Certified Product Information Product Name: Greenway Intergy Meaningful Use Edition Product Version: v9.30 Domain: Ambulatory Test Type: Complete EHR 1.2 Developer/Vendor Information Developer/Vendor Name: Greenway Health, LLC Address: 4301 West Boy Scout Boulevard, Suite 800 Tampa FL Website: Phone: (813) Developer/Vendor Contact: Danny Shipman Page 1 of 12

2 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 07-Aug-2015 Part 2: ONC-Authorized Certification Body Information 2.1 ONC-Authorized Certification Body Information ONC-ACB Name: Drummond Group Address: North Hwy 183, Ste B , Austin, TX Website: Phone: ONC-ACB Contact: Bill Smith This test results summary is approved for public release by the following ONC-Authorized Certification Body Representative: Bill Smith ONC-ACB Authorized Representative Certification Body Manager Function/Title Signature and Date 8/28/ Gap Certification The following identifies criterion or criteria certified via gap certification (a)(1) (a)(19) (d)(6) (h)(1) (a)(6) (a)(20) (d)(8) (h)(2) (a)(7) (b)(5)* (d)(9) (h)(3) (a)(17) (d)(1) (f)(1) (a)(18) (d)(5) (f)(7)** *Gap certification allowed for Inpatient setting only **Gap certification allowed for Ambulatory setting only x No gap certification Page 2 of 12

3 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 07-Aug Inherited Certification The following identifies criterion or criteria certified via inherited certification x (a)(1) (a)(16) Inpt. only x (c)(2) x (f)(2) x (a)(2) (a)(17) Inpt. only x (c)(3) x (f)(3) x (a)(3) (a)(18) x (d)(1) (f)(4) Inpt. only x (a)(4) (a)(19) x (d)(2) x (a)(5) (a)(20) x (d)(3) x (a)(6) x (b)(1) x (d)(4) x (a)(7) x (b)(2) x (d)(5) x (a)(8) x (b)(3) x (d)(6) (f)(7) x (a)(9) x (b)(4) x (d)(7) (g)(1) x (a)(10) x (b)(5) x (d)(8) x (g)(2) x (a)(11) (b)(6) Inpt. only x (d)(9) Optional x (g)(3) x (a)(12) x (b)(7) x (e)(1) x (g)(4) x (a)(13) (b)(8) x (e)(2) Amb. only (h)(1) x (a)(14) (b)(9) x (e)(3) Amb. only (h)(2) x (a)(15) x (c)(1) x (f)(1) (h)(3) (f)(5) Amb. only (f)(6) Amb. only No inherited certification Page 3 of 12

4 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 07-Aug-2015 Part 3: NVLAP-Accredited Testing Laboratory Information Report Number: SG Test Date(s): N/A 3.1 NVLAP-Accredited Testing Laboratory Information ATL Name: Drummond Group EHR Test Lab Accreditation Number: NVLAP Lab Code Address: North Hwy 183, Ste B , Austin, TX Website: Phone: ATL Contact: Kyle Meadors For more information on scope of accreditation, please reference NVLAP site. Part 3 of this test results summary is approved for public release by the following Accredited Testing Laboratory Representative: Sonia Galvan Test Proctor ATL Authorized Representative Function/Title Signature and Date 8/28/2015 Houston, TX Location Where Test Conducted 3.2 Test Information Additional Software Relied Upon for Certification Additional Software Applicable Criteria Functionality provided by Additional Software Kryptiq b.1, 2 HISP No additional software required Page 4 of 12

5 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 07-Aug Test Tools Test Tool Version x Cypress 2.6 x eprescribing Validation Tool HL7 CDA Cancer Registry Reporting Validation Tool HL7 v2 Electronic Laboratory Reporting (ELR) Validation Tool x HL7 v2 Immunization Information System (IIS) Reporting Validation Tool x HL7 v2 Laboratory Results Interface (LRI) Validation Tool x HL7 v2 Syndromic Surveillance Reporting Validation Tool x Transport Testing Tool 181 x Direct Certificate Discovery Tool Edge Testing Tool No test tools required Test Data Alteration (customization) to the test data was necessary and is described in Appendix [insert appendix letter] x No alteration (customization) to the test data was necessary Standards Multiple Standards Permitted The following identifies the standard(s) that has been successfully tested where more than one standard is permitted Criterion # Standard Successfully Tested (a)(8)(ii)(a)(2) (a)(13) x (b)(1) HL7 Version 3 Implementation Guide: URL-Based Implementations of the Context-Aware Information Retrieval (Infobutton) Domain x (a)(3) IHTSDO SNOMED CT International Release July 2012 and US Extension to SNOMED CT March 2012 Release (b)(2) HL7 Version 3 Implementation Guide: Context-Aware Knowledge Retrieval (Infobutton) Service-Oriented Architecture Implementation Guide (j) HL7 Version 3 Standard: Clinical Genomics; Pedigree Page 5 of 12

6 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 07-Aug-2015 Criterion # (a)(15)(i) x (b)(1) HL7 Version 3 Implementation Guide: URL-Based Implementations of the Context-Aware Information Retrieval (Infobutton) Domain Standard Successfully Tested (b)(2) HL7 Version 3 Implementation Guide: Context-Aware Knowledge Retrieval (Infobutton) Service-Oriented Architecture Implementation Guide (a)(16)(ii) (b)(2)(i)(a) (b)(7)(i) (g) Network Time Protocol Version 3 (RFC 1305) (i) The code set specified at 45 CFR (c)(2) (ICD-10- CM) for the indicated conditions (i) The code set specified at 45 CFR (c)(2) (ICD-10- CM) for the indicated conditions (g) Network Time Protocol Version 4 (RFC 5905) x (a)(3) IHTSDO SNOMED CT International Release July 2012 and US Extension to SNOMED CT March 2012 Release x (a)(3) IHTSDO SNOMED CT International Release July 2012 and US Extension to SNOMED CT March 2012 Release (b)(8)(i) (e)(1)(i) (e)(1)(ii)(a)(2) (e)(3)(ii) (i) The code set specified at 45 CFR (c)(2) (ICD-10- CM) for the indicated conditions Annex A of the FIPS Publication [list encryption and hashing algorithms] AES-256 SHA (g) Network Time Protocol Version 3 (RFC 1305) Annex A of the FIPS Publication [list encryption and hashing algorithms] AES-256 SHA (a)(3) IHTSDO SNOMED CT International Release July 2012 and US Extension to SNOMED CT March 2012 Release x (g) Network Time Protocol Version 4 (RFC 5905) Common MU Data Set (15) x (a)(3) IHTSDO SNOMED CT International Release July 2012 and US Extension to SNOMED CT March 2012 Release (b)(2) The code set specified at 45 CFR (a)(5) (HCPCS and CPT-4) Page 6 of 12

7 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 07-Aug-2015 Criterion # Standard Successfully Tested None of the criteria and corresponding standards listed above are applicable Newer Versions of Standards The following identifies the newer version of a minimum standard(s) that has been successfully tested Newer Version Applicable Criteria No newer version of a minimum standard was tested Optional Functionality Criterion # x (a)(4)(iii) (b)(1)(i)(b) (b)(1)(i)(c) (b)(2)(ii)(b) (b)(2)(ii)(c) (e)(1) x (f)(3) (f)(7) Common MU Data Set (15) Common MU Data Set (15) Optional Functionality Successfully Tested Plot and display growth charts Receive summary care record using the standards specified at (a) and (b) (Direct and XDM Validation) Receive summary care record using the standards specified at (b) and (c) (SOAP Protocols) Transmit health information to a Third Party using the standards specified at (a) and (b) (Direct and XDM Validation) Transmit health information to a Third Party using the standards specified at (b) and (c) (SOAP Protocols) View, download and transmit data to a third party utilizing the Edge Protocol IG version 1.1 Ambulatory setting only Create syndrome-based public health surveillance information for transmission using the standard specified at (d)(3) (urgent care visit scenario) Ambulatory setting only transmission to public health agencies syndromic surveillance - Create Data Elements Express Procedures according to the standard specified at (b)(3) (45 CFR (a)(4): Code on Dental Procedures and Nomenclature) Express Procedures according to the standard specified at (b)(4) (45 CFR (c)(3): ICD-10-PCS) No optional functionality tested Page 7 of 12

8 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 07-Aug Edition Certification Criteria* Successfully Tested Criteria # Version Version Criteria # TP** TD*** TP TD (a)(1) (c)(3) (a)(2) 1.2 (d)(1) 1.2 (a)(3) (d)(2) 1.6 (a)(4) (d)(3) 1.3 (a)(5) (d)(4) 1.3 (a)(6) (d)(5) 1.2 (a)(7) (d)(6) 1.2 (a)(8) 1.3 (d)(7) 1.2 (a)(9) (d)(8) 1.2 (a)(10) (d)(9) Optional 1.2 (a)(11) 1.3 (e)(1) (a)(12) 1.3 (e)(2) Amb. only (a)(13) 1.2 (e)(3) Amb. only 1.3 (a)(14) 1.2 (f)(1) (a)(15) 1.5 (f)(2) (a)(16) Inpt. only (f)(3) (a)(17) Inpt. only 1.2 (f)(4) Inpt. only (a)(18) (a)(19) (a)(20) (b)(1) (f)(5) Amb. only (f)(6) Amb. only (b)(2) (f)(7) Amb. only 1.1 (b)(3) (g)(1) (b)(4) (g)(2) (b)(5) (g)(3) 1.4 (b)(6) Inpt. only (g)(4) 1.2 (b)(7) (h)(1) 1.1 (b)(8) (h)(2) 1.1 (b)(9) (h)(3) 1.1 (c)(1) (c)(2) Page 8 of 12

9 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 07-Aug-2015 Criteria # x No criteria tested Version Version Criteria # TP** TD*** TP TD *For a list of the 2014 Edition Certification Criteria, please reference (navigation: 2014 Edition Test Method) **Indicates the version number for the Test Procedure (TP) ***Indicates the version number for the Test Data (TD) Page 9 of 12

10 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 07-Aug Clinical Quality Measures* Type of Clinical Quality Measures Successfully Tested: x Ambulatory Inpatient No CQMs tested *For a list of the 2014 Clinical Quality Measures, please the CMS ecqm Library (Navigation: June 2014 and April 2014 Updates) Ambulatory CQMs CMS ID Version CMS ID Version CMS ID Version CMS ID Version x 2 v3 x 90 v3 x 136 v3 x 155 v2 x 22 v2 x 117 v2 x 137 v2 x 156 v2 x 50 v2 x 122 v2 x 138 v2 x 157 v2 x 52 v2 x 123 v2 x 139 v2 x 158 v2 x 56 v2 x 124 v2 x 140 v2 x 159 v2 x 61 v3 x 125 v2 x 141 v3 x 160 v2 x 62 v2 x 126 v2 x 142 v2 x 161 v2 x 64 v3 x 127 v2 x 143 v2 x 163 v2 x 65 v3 x 128 v2 x 144 v2 x 164 v2 x 66 v2 x 129 v3 x 145 v2 x 165 v2 x 68 v3 x 130 v2 x 146 v2 x 166 v3 x 69 v2 x 131 v2 x 147 v2 x 167 v2 x 74 v3 x 132 v2 x 148 v2 x 169 v2 x 75 v2 x 133 v2 x 149 v2 x 177 v2 x 77 v2 x 134 v2 x 153 v2 x 179 v2 x 82 v1 x 135 v2 x 154 v2 x 182 v3 Inpatient CQMs CMS ID Version CMS ID Version CMS ID Version CMS ID Version Page 10 of 12

11 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 07-Aug Automated Numerator Recording and Measure Calculation Automated Numerator Recording Automated Numerator Recording Successfully Tested (a)(1) (a)(11) (a)(18) (b)(6) (a)(3) (a)(12) (a)(19) (b)(8) (a)(4) (a)(13) (a)(20) (b)(9) (a)(5) (a)(14) (b)(2) (e)(1) (a)(6) (a)(15) (b)(3) (e)(2) (a)(7) (a)(16) (b)(4) (e)(3) (a)(9) (a)(17) (b)(5) x Automated Numerator Recording was not tested Automated Measure Calculation Automated Measure Calculation Successfully Tested x (a)(1) x (a)(11) (a)(18) (b)(6) x (a)(3) x (a)(12) (a)(19) (b)(8) x (a)(4) x (a)(13) (a)(20) (b)(9) x (a)(5) x (a)(14) x (b)(2) x (e)(1) x (a)(6) x (a)(15) x (b)(3) x (e)(2) x (a)(7) (a)(16) x (b)(4) x (e)(3) x (a)(9) (a)(17) x (b)(5) Automated Measure Calculation was not tested Attestation Attestation Forms (as applicable) x Safety-Enhanced Design* x Quality Management System** x Privacy and Security Appendix A B C 3.3 Appendices Attached below. *Required if any of the following were tested: (a)(1), (a)(2), (a)(6), (a)(7), (a)(8), (a)(16), (a)(18), (a)(19), (a)(20), (b)(3), (b)(4), (b)(9). **Required for every EHR product Page 11 of 12

12 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 07-Aug-2015 Test Results Summary Change History Test Report ID Description of Change Date 2014 Edition Test Report Summary Page 12 of 12

13 USER CENTER DESIGN REPORT TEST REPORT UPDATE This test report was updated in December 2015 to satisfy User Center Design Report specifications by ONC. The new Test Report ID is amended as follows: Part 3: NVLAP-Accredited Testing Laboratory Information: Report Number plus the suffix _Dec2015.

14

15 ONC-ATL EHR Testing Safety-Enhanced Design Checklist 2014 Edition Prepared & Administered by: DRUMMOND GROUP INC. Rev.: 20Mar2013 EHR Test-120 Page 1 of 7 Copyright 2013 Drummond Group Inc. The information contained in this document is strictly held confidential and shall not be disclosed in any manner or form, directly or indirectly, to any person or entity under any circumstances, without prior approval.

16 Safety-Enhanced Design Evaluation Guide Scope and Purpose This document provides a dual purpose. It serves both a means for the Test Proctor to more quickly evaluate the submitted usability report and also a checklist for the Vendor to confirm all aspects of the report have been submitted. The User-Centered Design Process is just a question of what user-centered design process was used in this product-version submitted for testing. Vendor should answer that question in this document which will be returned to the DGI Test Proctor. The Usability Report Checklist section is just that. A section where the Vendor fills in the page numbers in the report corresponding to the specific information that the Test Proctor must check off to confirm completion of report. Some additional CONFIRM actions are included to assist both Vendor and Test Proctor in checking for specific areas of evaluation listed in the test procedure. The 314.g.3 Submission Checklist is a checklist of items to turn in for completion of testing of 314.g.3. These items should be submitted no later than 2 weeks prior to your test event date. Turning these items in later than 2 weeks out will result in delay of processing test results. 314.g.3 Submission Checklist Vendors must turn in the following items to be tested for certification in Safety- Enhanced Design (314.g.3): A completed usability report of their product-version using the NISTIR 7742 Customized Common Industry Format template ( The report does not have to follow NISTIR 7742 exactly but it should cover the sections required in that template which are checked in the Usability Report Checklist section below. A signed letter from a representative of the Vendor to the veracity and authenticity of the usability report. The signed letter may be included in the report itself. This DGI Safety-Enhanced Design Checklist documented completed with page numbers referencing where the required usability information is provided. Rev.: 20Mar2013 EHR Test-120 Page 2 of 7 Copyright 2013 Drummond Group Inc. The information contained in this document is strictly held confidential and shall not be disclosed in any manner or form, directly or indirectly, to any person or entity under any circumstances, without prior approval.

17 User-Centered Design Process Describe you user-centered design (UCD) process. Was it based on a UCD industry standard (for example; ISO , ISO , ISO 13407, ISO 16982, and ISO/IEC 62366) or a non-industry standard process? The UCD process used was based on NISTIR 7741 and the NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records found at this link: Usability Report Checklist For the submitted Usability Report, please record the report page numbers where each required section can be found as well as any other brief notes needed to understand this section of the report. To clarify, do NOT record the answers below but only include the page number reference to them in the submitted report for easier evaluation. Name and version of the product: P. 3 Data and location of the usability test: P. 1 Test Environment: P. 6 Description of intended users: P. 3 Total number of participants: P. 3 Description of participants: their experience and demographic characteristics: P. 4 CONFIRM: Participants match the description of the intended users. Description of the user tasks that were tested: P. 5 CONFIRM: User tasks are prioritized in accordance with risk associated with user errors. Rev.: 20Mar2013 EHR Test-120 Page 3 of 7 Copyright 2013 Drummond Group Inc. The information contained in this document is strictly held confidential and shall not be disclosed in any manner or form, directly or indirectly, to any person or entity under any circumstances, without prior approval.

18 CONFIRM: All test scenarios for each of the EHR technology capabilities submitted for testing. CPOE (314.a.1) Record Medication Order Change Medication Order Access Medication Order Record Laboratory Order Change Laboratory Order Access Laboratory Order Record Radiology/imaging Order Change Radiology/imaging Order Access Radiology/imaging Order Drug-drug, drug-allergy interactions checks (314.a.2) Create drug-drug and drug-allergy interventions prior to CPOE completion Adjustment of severity level of drug-drug interventions (may be an admin type function) Medication list (314.a.6) Record Medication List Change Medication List Access Medication List Medication allergy list (314.a.7) Record Medication List Change Medication List Access Medication List Clinical decision support (314.a.8) Problem List Interventions Medication List Interventions Medication Allergy List Interventions Demographics Interventions Lab Tests and Results Interventions Vital Signs Interventions Identify User Diagnostic and Therapeutic Reference Information Configuration of CDS interventions by user (may be an admin type function) Rev.: 20Mar2013 EHR Test-120 Page 4 of 7 Copyright 2013 Drummond Group Inc. The information contained in this document is strictly held confidential and shall not be disclosed in any manner or form, directly or indirectly, to any person or entity under any circumstances, without prior approval.

19 emar (314.a.16) Using assistive technology, verify the right patient, medication, dose, route and time. Electronic prescribing (314.b.3) Create prescriptions Clinical information reconciliation (314.b.4) Reconcile patient s active medication list with another source Reconcile patient s active problem list with another source Reconcile patient s active medication allergy list with another source List of the specific metrics captured during the testing for effectiveness, efficiency and satisfaction: P. 7 CONFIRM: Effectiveness and efficiency were evaluated. Data scoring: P. 8 Results of the test and data analysis: P CONFIRM: Test results provided a risk analysis of the use, tested performance and error rates. Major test findings: P Identified area(s) of improvement(s): P Rev.: 20Mar2013 EHR Test-120 Page 5 of 7 Copyright 2013 Drummond Group Inc. The information contained in this document is strictly held confidential and shall not be disclosed in any manner or form, directly or indirectly, to any person or entity under any circumstances, without prior approval.

20 Change Log Revision Change Description 20-Mar-2013 Added instructions for verifying admin functions drug-drug severity adjustment and CDS interventions per user are tested. 18-Feb-2013 Initial Release. Rev.: 20Mar2013 EHR Test-120 Page 6 of 7 Copyright 2013 Drummond Group Inc. The information contained in this document is strictly held confidential and shall not be disclosed in any manner or form, directly or indirectly, to any person or entity under any circumstances, without prior approval.

21 About Drummond Group Inc. Drummond Group Inc. is a global software test and certification lab that serves a wide range of vertical industries. In healthcare, Drummond Group tests and certifies Controlled Substance Ordering Systems (CSOS), Electronic Prescription of Controlled Substances (EPCS) software and processes, and Electronic Health Records (EHRs) designating the trusted test lab as the only third-party certifier of all three initiatives designed to move the industry toward a digital future. Founded in 1999, and accredited for the Office of the National Coordinator HIT Certification Program as an Authorized Certification Body (ACB) and an Authorized Test Lab (ATL), Drummond Group continues to build upon its deep experience and expertise necessary to deliver reliable and cost-effective services. For more information, please visit or DGI@drummondgroup.com Rev.: 20Mar2013 EHR Test-120 Page 7 of 7 Copyright 2013 Drummond Group Inc. The information contained in this document is strictly held confidential and shall not be disclosed in any manner or form, directly or indirectly, to any person or entity under any circumstances, without prior approval.

22 USABILITY TESTING: VITERA INTERGY EHR Usability Tests: September 12-14, 2012 Usability Test Location: Orlando, Florida Report Date: June 7, 2013 Report Prepared by: Vitera Healthcare Solutions 1 P age

23 TABLE OF CONTENTS EXECUTIVE SUMMARY... 1 INTRODUCTION... 3 METHOD... 3 DESCRIPTION OF INTENDED USERS... 3 PARTICIPANTS... 3 TOTAL NUMBER AND DESCRIPTION OF PARTICIPANTS... 4 STUDY DESIGN... 5 TASKS... 5 PROCEDURES... 6 TEST LOCATION AND ENVIRONMENT... 6 TEST FORMS AND TOOLS... 7 PARTICIPANT INSTRUCTIONS... 7 USABILITY METRICS... 7 DATA SCORING... 8 RESULTS... 9 DATA ANALYSIS AND REPORTING... 9 DISCUSSION OF THE FINDINGS... 9 EFFECTIVENESS... 9 EFFICIENCY SATISFACTION MAJOR FINDINGS AREAS FOR IMPROVEMENT APPENDICES i P age Vitera Healthcare Solutions

24 Usability Testing: Vitera Intergy EHR EXECUTIVE SUMMARY A usability test of Vitera Intergy EHR, an ambulatory electronic health record (EHR) covering multiple specialties was conducted on September 12-14, 2012 in Orlando, Florida by Vitera Healthcare Solutions. The purpose of this study was to test and validate the usability of the current user interface and provide evidence of usability in Vitera Intergy EHR. During the usability test, twelve healthcare providers matching the target demographic criteria served as participants and used Vitera Intergy EHR in simulated, but representative tasks. The study collected performance data on 16 tasks typically conducted on an EHR. During the 60-minute, one-on-one usability test, each participant was greeted by the administrator and asked to review and sign a recording consent form (included in Appendix 2: Recording Consent Form). Participants had varying levels of prior experience with the Vitera Intergy EHR ranging from 1-5 years. The administrator introduced the test and instructed participants to complete a series of tasks (given one at a time) using Vitera Intergy EHR. During the testing, the administrator timed the test and, along with the data loggers recorded participant performance data on paper and electronically. The participants interactions with screens and their faces were video/audio recorded for subsequent analysis. The following types of data were collected for each participant: Number of tasks successfully completed within the allotted time Time to complete the tasks Number and types of errors Path deviations Participants verbalizations Participants familiarity and ease-of-use and satisfaction ratings with the system and system components All participant data was de-identified no correspondence could be made from the identity of the participant to the data collected. Following the conclusion of the testing, participants were asked to complete a post-test questionnaire and were compensated with small branded items pens and USB drives for their time. Various metrics recommended in the NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records were used to evaluate the usability of Vitera Intergy EHR. Below is a summary of the performance and rating data collected on Vitera Intergy EHR. 1 P age Vitera Healthcare Solutions

25 Usability Testing: Vitera Intergy EHR Overall Ease of Use and Satisfaction of Tasks Performed During the Study Ease of Use Based on a scale of 1-5, where 1 = not at all satisfied and 5 = very satisfied: The overall subjective ease-of-use mean was 4.33 of 5. The overall percent ease of use was 87%. Satisfaction The overall subjective satisfaction mean was 4.45 of 5. The overall percent satisfaction was 89%. Major Findings In addition to the performance data, the following qualitative observations were made: Overall, the new user interface (UI) was very well received. Participants were able to adapt quickly to the new UI, and they appreciated being able to see more data on one screen, thus reducing the back and forth actions required between screens to see the same data in previous versions. All participants were very satisfied with the new design of the Drug Utilization Review (DUR) display. Several of the participants admitted to not paying much attention to the DUR in previous versions of the software, and they commented on the vast improvement of the clean, concise design, making priority alerts more prominent and reducing alert noise. The redesigned Lab Orders system tested extremely well with much faster and less scrolling as the general feedback. Several providers said they would start ordering patient lab tests rather than having support staff handle submitting the orders. This design improvement and resulting workflow change gains enormous efficiencies for the practices, since a step in the previous lab ordering workflow is completely eliminated. Another feature that received accolades from participants was the ability to import discrete clinical data from a Continuity of Care Document (CCD). Although this feature is currently underutilized by practices, the Meaningful Use 2 requirement to provide clinical summaries to patients and other interoperability requirements may propel clinical document exchange features into greater use. For some areas of the application, such as Allergies and Medications, there are multiple ways to complete a task. This is beneficial as healthcare providers adapt their unique workflows in the EHR, but in some cases, as with Allergies, there was minor confusion about what actions could be taken on an allergy in a given area of the application. There is no risk to patient safety, but the study revealed a reduced efficiency issue until the slight learning curve is overcome. Health Management, the integrated clinical decision support subsystem, also presented some challenges in terms of learnability, especially in terms of setup and customization. The system poses no inherent safety risks; however, the user interface for the Reminders Browser would benefit from a usability driven update. 2 P age Vitera Healthcare Solutions

26 Usability Testing: Vitera Intergy EHR INTRODUCTION The electronic health record application tested for this study was Vitera Intergy EHR, Version 9.0, an application used in an ambulatory setting. Designed to present medical information to healthcare providers in an outpatient setting for various specialties, Vitera Intergy EHR consists of a comprehensive electronic clinical system used to create, store, and retrieve patient data. Vitera Intergy EHR is fully integrated with the Vitera Intergy Practice Management System. The usability testing attempted to represent realistic exercises and conditions that occur in a typical medical practice environment. The purpose of this study was to test and validate the usability of the current user interface and provide evidence of usability in Vitera Intergy EHR. To this end, measures of effectiveness, efficiency, and user satisfaction, such as task success, time on task, and task group ratings, were captured during the usability testing. METHOD DESCRIPTION OF INTENDED USERS Intended users of Vitera Intergy EHR are doctors, nurses, physician s assistants, medical assistants, and anyone entering or accessing clinical data at an ambulatory medical practice. PARTICIPANTS A total of 12 participants were tested on Vitera Intergy EHR. Participants in the test were physicians and allied healthcare providers in ambulatory settings, ranging in years of experience with Vitera Intergy EHR and varied in specialty including Family Practice, Internal Medicine, Pediatrics, and others. Participants were recruited by a Certified Usability Analyst at Vitera Healthcare Solutions, and they were compensated with small branded items for their time pens and USB drives. In addition, participants had no direct connection to the development organization producing Vitera Intergy EHR. Participants were not from the testing or supplier organization. Participants were given the opportunity to have the same orientation and level of training as the actual end users would have received. The interface being tested was new to some of the participants, changes having been made prior to Version 9. A 5-minute training video used for customer training was shown to those who did not have prior experience using the new interface. The video explained general navigation of the system. For the test purposes, end-user characteristics were identified and potential participants were solicited. An example of the invitations sent to potential participants is provided in Appendix 1: Recruiting Methodology. 3 P age Vitera Healthcare Solutions

27 Usability Testing: Vitera Intergy EHR Recruited participants had a mix of backgrounds and demographic characteristics. The following is a table of participants by characteristics, including demographics, professional experience, computing experience and user needs for assistive technology. Participant names were replaced with Participant IDs so that an individual s data cannot be tied back to individual identities. TOTAL NUMBER AND DESCRIPTION OF PARTICIPANTS The total number of participants was 12. Their descriptions are indicated in the table below: ID Age Education Occupation/ Role Professional Experience (Practice Specialty, # yrs. practicing) Computer Experience (Beginner, Intermediate, Advanced) Product Experience (# years using Intergy EHR) Assistive Technology Needs (for vision, etc.) 1 37 D.O., Board Certified Physician Family Practice, 8 years Advanced 1 year Glasses Family Medicine 2 55 M.D. Physician Internal Medicine, 27 years Intermediate 1 year Reading Glasses 3 32 D.O. Post Graduate Physician Family Practice Family Practice, 2.5 years Advanced (did some programming in college) 2.5 years None 4 52 M.D. Pediatrician Pediatrics, 22 years Intermediate 5 years None 5 48 M.D. at University of MD Internal Medicine, 19 Intermediate, not a 5 years None AL School of Medicine years novice, but not a programmer 6 54 M.D. Primary Care Physician 7 48 M.D. Physician/ Medical Director Internal Medicine, 18 years General Pediatrics, Pediatric Hematology, 14 years 8 46 M.D. Physician Internal Medicine, 15 years 9 57 M.D. Family Physician Practice Owner Intermediate 5 years None Family Practice, 28 years Upper intermediate Has learned to setup Intergy for practice needs. Intermediate 5 years Glasses/contacts Advanced 1 year Eye Glasses - Bifocal 5 years None University of Iowa Physician Family Medicine, 24 years Intermediate 4 years None M.D HS degree, some Medical Assistant Family Practice, 18 years Advanced 1.5 years None college M.D. U of Tennessee Physician Family Practice, 30 years Intermediate 1 year None Table 1: Participants 4 P age Vitera Healthcare Solutions

28 Usability Testing: Vitera Intergy EHR Thirteen participants (matching the demographics in the section on Participants) were recruited, and twelve participated in the usability test. One participant failed to show for the study. Participants were scheduled for 60-minute sessions with 15 minutes in between each session for debriefing by the administrators and data loggers and to reset systems to proper test conditions. A spreadsheet was used to keep track of the participant schedule. STUDY DESIGN Overall, the objective of this test was to uncover areas where the application performed well that is, effectively, efficiently, and with satisfaction and areas where the application failed to meet the needs of the participants. The data from this study may serve as a baseline for future tests with an updated version of Vitera Intergy EHR. In short, this testing serves as both a means to record or benchmark current usability, but also to identify areas where improvements must be made. During the usability test, participants interacted with Vitera Intergy EHR. Each participant used the system in the same location and was provided with the same instructions. The system was evaluated for effectiveness, efficiency, and satisfaction as defined by measures collected and analyzed for each participant: Number of tasks successfully completed within the allotted time Time to complete the tasks Number and types of errors Path deviations Participants verbalizations (comments) Participants satisfaction ratings of the system Additional information about the various measures can be found in the Usability Metrics section of this report. TASKS A number of tasks were constructed that would be realistic and representative of the kinds of activities a user might perform with Vitera Intergy EHR. Tasks were selected to assess the usability of the changes made to the application navigation and new features added in this version. The tasks in the Usability Protocol used for this usability test were prioritized in accordance with the risks associated with user errors (see Appendix 3: Usability Protocol). 5 P age Vitera Healthcare Solutions

29 Usability Testing: Vitera Intergy EHR PROCEDURES Upon arrival, participants were greeted; their identity was verified and matched with a name on the participant schedule. Participants were then assigned a Participant ID. Each participant reviewed and signed a recording consent form (see Appendix 2: Recording Consent Form). A representative from the test team witnessed the participant s signature. To ensure that the test ran smoothly, two staff members participated in this test, the usability test administrator and the data logger. The usability testing staff conducting the test were experienced usability practitioners. The administrator moderated the session including administering instructions and tasks. The administrator also monitored task times, obtained post-task rating data, and took notes on participant comments. A second person served as the data logger and took notes on task success, task time, path deviations, number and type of errors, and comments. Participants were instructed to perform the tasks (see specific instructions below): As quickly as possible making as few errors and deviations as possible. Without assistance; administrators were allowed to give immaterial guidance and clarification on tasks, but not instructions on use. Without using a think-aloud technique. Task timing began once the administrator finished reading the question and said Begin now. The task time was stopped once the participant indicated he or she had successfully completed the task. Following the session, the administrator gave the participant the post-test questionnaire. Participants' demographic information, task success rate, time on task, errors, deviations, verbal responses, and post-test questionnaire were recorded into a spreadsheet. Scoring is discussed below in the Data Scoring section. Participants were thanked for their time and compensated. TEST LOCATION AND ENVIRONMENT Vitera Intergy EHR would be typically be used in an ambulatory healthcare setting. In this instance, testing was conducted in usability testing booths at VIBE 2012, Vitera Healthcare Solutions annual user conference held in Orlando, Florida. The booths contained a table, a computer for the participant, and a computer for the data logger. The administrator sat next to the participant to facilitate the test, and the data logger sat across the table from the participant where he could see the participant s face as well as the laptop screen to view the participant s interactions with Vitera Intergy EHR. For testing, Dell Latitude E6410 laptops running Windows 7, at 1024x768 resolution were configured with a 19-inch monitor attached to each computer. The participants used the 19-inch monitor, a mouse, and a keyboard when interacting with Vitera Intergy EHR. 6 P age Vitera Healthcare Solutions

30 Usability Testing: Vitera Intergy EHR The Vitera Intergy EHR software was set up by Vitera s in-house Information Technology resources according to the product documentation describing the Intergy EHR system setup and preparation. The application was running using a demonstration database. The system performance was representative of what actual users would experience in a field implementation. TEST FORMS AND TOOLS During the usability test, various documents and instruments were used, including: 1. Recording Consent Form 2. Usability Protocol, including the Post-test Questionnaire 3. Techsmith Morae Recorder software 4. Logitech Orbit AF Webcam and Audio Recorder Examples of the above documents can be found in Appendices 2 and 3, respectively. The Usability Protocol was devised so as to be able to capture required data. The participants interaction with Vitera Intergy EHR was captured and recorded digitally with screen capture software running on the test machine. A camera recorded each participant s facial expressions synced with the screen capture and verbal comments were recorded with a microphone. Each test session was observed by the data logger. PARTICIPANT INSTRUCTIONS The participant instructions were read from the Usability Protocol, which is available in Appendix 3: Usability Protocol. USABILITY METRICS According to the NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records, EHRs should support a process that provides a high level of usability for all users. The goal is for users to interact with the system effectively, efficiently, and with an acceptable level of satisfaction. To this end, metrics for effectiveness, efficiency, and user satisfaction were captured during the usability testing. The goals of the test were to assess: 1. Efficiency of Vitera Intergy EHR by measuring the average task time and path deviations 2. Effectiveness of Vitera Intergy EHR by measuring participant success rates and errors 3. Satisfaction with Vitera Intergy EHR by measuring ease-of-use ratings 7 P age Vitera Healthcare Solutions

31 Usability Testing: Vitera Intergy EHR DATA SCORING The table below details how tasks were scored, errors evaluated, and the time data analyzed. Rationale and Scoring Effectiveness: Task Success Task success was determined by assigning numeric weights for various levels of task success, as follows: Complete success (without assistance) = 1.0 Partial success = 0.5 Gives up or wrong answer = 0.0 A success score for each task was calculated by averaging the scores for each task. The results are provided as a percentage. Task times were recorded for successes. Observed task times divided by the optimal time for each task were calculated as a measure of optimal efficiency. Optimal task performance time, as benchmarked by expert performance under realistic conditions, was recorded when constructing tasks. Target task times were operationally derived by multiplying a benchmarked expert performance by a factor of 1.5, allowing for some buffer because participants were not trained to expert performance and the user interface had been recently redesigned. When a task had multiple valid paths to successful outcome, each task path was timed and the average was multiplied by the buffer to obtain the target task time. Effectiveness: Task Failures Efficiency: Task Deviations Efficiency: Task Time Satisfaction: Task Rating If the participant abandoned the task, did not reach the correct answer, or performed it incorrectly, the task was counted as a failure. No task times for failed tasks or tasks that exceeded the target task time were used in calculations. A qualitative account of the observed errors and error types was collected. The participant s navigation path through the application was recorded. Deviations occur if the participant, for example, went to a wrong screen, clicked on an incorrect menu item, followed an incorrect link, or interacted incorrectly with an on-screen control. This path was compared to the optimal path. Path deviations are reported on a qualitative level for use in recommendations for improvement. Each task was timed from when the administrator said Begin until the participant said, Done. If the participant failed to say Done, the time was stopped when the participant ceased performing the task. Only task times for tasks that were successfully completed and tasks that were completed at or under the target time were included in the average task time analysis. Average time per task and variance measures were calculated for each task for use in the results analysis. Participant s subjective impression of the ease of use of the application was measured by administering both a simple question on completion of each task group and a post-session questionnaire. After each task group, the participant was asked to rate Overall, these tasks were: on a scale of 1 (Very Difficult) to 5 (Very Easy). These data are averaged across participants. Table 2: Details of how observed data were scored. 8 P age Vitera Healthcare Solutions

32 Usability Testing: Vitera Intergy EHR RESULTS DATA ANALYSIS AND REPORTING The results of the usability test were calculated according to the methods specified in the Usability Metrics section above. Participants who failed to follow session and task instructions had their data excluded from the analyses. Of note, four tasks were inadvertently skipped by test administrators, so the number of participants was adjusted accordingly for those tasks. The usability testing results for Vitera Intergy EHR are detailed below. The results should be seen in light of the objectives and goals outlined in the Study Design section. The data yielded actionable results that, when corrected, will yield a material, positive impact on user performance. DISCUSSION OF THE FINDINGS The major areas of the usability test in light of the findings are discussed below. EFFECTIVENESS Review Active Allergies The task success score was 100%. Participants were easily able to review their active allergies using one of three valid paths for completing this task. Edit Allergies One participant failed the task due to adding a new allergy instead of editing the existing allergy. The participant initiated the allergy edit task from the Patient Summary and became confused when unable to find on the Allergy menu an option to edit allergies. The difficulty may be attributed to inexperience with the redesigned user interface. The Orientation Video shown to participants provided a general overview of the new user interface, but did not specifically cover the allergy edit workflow. Add Allergies The success score for this task was 100%. Participants were easily able to add patient allergies. Review Inactive Allergies The success score for this task was 92%, due to two participants requiring hints on how to complete the task. All participants were ultimately able to review the patient s active allergies and complete the task. Reactivate Inactive Allergies All of the participants were able to successfully change the status of an inactive allergy to Active, yielding a task success score of 100%. 9 P age Vitera Healthcare Solutions

33 Usability Testing: Vitera Intergy EHR Locate Health Reminders The task success score was 100%. All participants were able to locate patient health reminders. Identify Overdue Reminders The success score for this task was 100%. All participants were able to identify the overdue reminders in patient health reminders lists. Identify Guideline for Foot Exam The task success score was 95%. One participant required a hint to find the guideline health reminders list in the Patient Summary and the task was inadvertently skipped for another participant. Mark Reminder as Satisfied The success score for this task was 88% due to three participants requiring a hint to complete the task. Two participants were given a hint about right clicking a reminder to find more options, and one participant was given a hint to satisfying reminders from the Reminders Browser. Locate Reference At 75% success, this task was the least successfully performed of all in the study, with four participants requiring a hint to complete the task and one participant failing to complete the task due to giving up. All hints were about accessing the Reminders Browser to find the reference information. Review Current Medications The success score for this task was 100%. All participants successfully reviewed the patient s current medications. Review Previous Medications One participant failed the task, due to selecting the filter option that showed lapsed medications, but not all previous medications. The participant did not realize the error and indicated completion of the task. Discontinue Medications All of the participants successfully discontinued a medication, per the task instructions. Order an X-Ray All participants were able to successfully add the barium x-ray to the patient s orders. Order a Medication All participants were able to successfully order a medication using Electronic Transmission as the transmission method. 10 P age Vitera Healthcare Solutions

34 Usability Testing: Vitera Intergy EHR Renew a Prescription All but one participant was able to complete the prescription renewal task successfully. The participant who failed the task entered a new prescription for the medication instead of issuing a renewal. However, the correct medication was issued, so any safety issue posed by the error is negligible. Order a Laboratory Test The user interface and workflow for this task had been redesigned, and no training was provided to participants purposely in order to test the usability of the new UI. Given their lack of training, the task success score was still 92%, with 11 out of 12 participants successfully completing the task. The participant who failed the test neglected to enter a diagnosis for the lab test per the task instructions. Several participants indicated that they preferred a faster workflow in which they indicated diagnoses and tests for the lab order and then sent the information to a designated practice user for completion. The software supports this workflow, but that workflow was not tested as part of this usability test. Overall, participants were able to complete the lab order and indicated that the design for electronic lab orders was a vast improvement. Identify High Priority Drug Interactions The success score for this task was 100%. Participants were easily able to identify high priority drug interactions on the Drug Utilization Review Report window. View Full Drug Details The success score for this task was 100%. All participants located the monograph in the full drug details of the Drug Utilization Review Report window successfully. Review Other Interactions The success score for this task was 92%, with one participant failing the task. The participant incorrectly indicated the Warning Details as the other interactions. Describe Response to Interaction Checks By reviewing the drug interactions presented in the Drug Utilization Review, participants were easily able to describe how they would alter the patient s medications based on the information presented. The task success score was 100%. Locate Retrieved Continuity of Care Document (CCD) This task score was 96% with one participant requiring a hint to complete the task. The hint was about the location of navigation object that opened the list of patient CCDs, which was labeled Exchange. 11 P age Vitera Healthcare Solutions

35 Usability Testing: Vitera Intergy EHR Import CCD into Chart The success score for this task was 100%. All participants were about to successfully import clinical data from the CCD into the Chart. Review Health Reminders The task success score was 100%. All participants successfully reviewed the patient s health reminders after importing clinical data from the CCD into the patient s chart. EFFICIENCY Review Active Allergies One participant exceeded the target time for this task, due to difficulty finding the button to open the Allergies List in the Chart Navigation Bar. As this task was the first of the usability test tasks, this error is likely attributed to the participant s unfamiliarity with the new user interface design and the button order not being customized to the participant s preference, which is a feature of the software. Edit Allergies Two participants exceeded the target task time due to searching in the menu for an allergy list or exploring other controls for an edit option, rather than selecting or right clicking the existing allergy. Both participants initiated the allergy edit task from the Allergies List. Their difficulty may be attributed to inexperience with the redesigned user interface. The Orientation Video shown to participants provided a general overview of the new user interface, but did not specifically cover the editing an allergy workflow. Add Allergies All participants completed the task at or under the target task time. Participants were easily able to add patient allergies. The workflow for adding an allergy is available from any of the three valid paths for working with patient allergy lists. Review Inactive Allergies Five participants exceeded the target time for this task. While there are three valid paths for working with patient allergy lists, only one of those paths supported the ability to review inactive allergies. The five participants who used one of the other two paths to work with allergy lists took some time to realize that the path did not support the review of allergy lists before switching to the valid path for the task. Reactivate Inactive Allergies Three participants exceeded the target time for this task. They all completed the task successfully; it just took them longer than others. The difficulty for those participants centered on finding the option to change the status. Of note, the three participants who had difficulty with this task declined watching the Orientation Video. 12 P age Vitera Healthcare Solutions