ONC HIT Certification Program

Transcription

1 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 07-Aug-2015 ONC HIT Certification Program Test Results Summary for 2014 Edition EHR Certification Part 1: Product and Developer Information 1.1 Certified Product Information Product Name: PCC EHR Product Version: V7 Domain: Ambulatory Test Type: Complete EHR 1.2 Developer/Vendor Information Developer/Vendor Name: Physicians Computer Company Address: 20 Winooski Falls Way Winooski VT Website: Phone: Developer/Vendor Contact: William VanDeventer Page 1 of 12

2 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 07-Aug-2015 Part 2: ONC-Authorized Certification Body Information 2.1 ONC-Authorized Certification Body Information ONC-ACB Name: Drummond Group Address: North Hwy 183, Ste B , Austin, TX Website: Phone: ONC-ACB Contact: Bill Smith This test results summary is approved for public release by the following ONC-Authorized Certification Body Representative: Bill Smith ONC-ACB Authorized Representative Certification Body Manager Function/Title Signature and Date 9/25/ Gap Certification The following identifies criterion or criteria certified via gap certification (a)(1) (a)(19) (d)(6) (h)(1) (a)(6) (a)(20) (d)(8) (h)(2) (a)(7) (b)(5)* (d)(9) (h)(3) (a)(17) (d)(1) (f)(1) (a)(18) (d)(5) (f)(7)** *Gap certification allowed for Inpatient setting only **Gap certification allowed for Ambulatory setting only x No gap certification Page 2 of 12

3 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 07-Aug Inherited Certification The following identifies criterion or criteria certified via inherited certification x (a)(1) (a)(16) Inpt. only (c)(2) (f)(2) x (a)(2) (a)(17) Inpt. only (c)(3) (f)(3) (a)(3) (a)(18) x (d)(1) (f)(4) Inpt. only (a)(4) (a)(19) (d)(2) (a)(5) (a)(20) (d)(3) (f)(5) Amb. only x (a)(6) (b)(1) (d)(4) x (a)(7) (b)(2) x (d)(5) (f)(6) Amb. only (a)(8) (b)(3) x (d)(6) (f)(7) (a)(9) (b)(4) (d)(7) (g)(1) x (a)(10) (b)(5) x (d)(8) (g)(2) (a)(11) (b)(6) Inpt. only (d)(9) Optional (g)(3) (a)(12) (b)(7) (e)(1) x (g)(4) (a)(13) (b)(8) (e)(2) Amb. only (h)(1) (a)(14) (b)(9) (e)(3) Amb. only (h)(2) (a)(15) (c)(1) (f)(1) (h)(3) No inherited certification Page 3 of 12

4 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 07-Aug-2015 Part 3: NVLAP-Accredited Testing Laboratory Information Report Number: SG Test Date(s): 6/24/2015, 9/21/ NVLAP-Accredited Testing Laboratory Information ATL Name: Drummond Group EHR Test Lab Accreditation Number: NVLAP Lab Code Address: North Hwy 183, Ste B , Austin, TX Website: Phone: ATL Contact: Kyle Meadors For more information on scope of accreditation, please reference NVLAP site. Part 3 of this test results summary is approved for public release by the following Accredited Testing Laboratory Representative: Sonia Galvan Test Proctor ATL Authorized Representative Function/Title Signature and Date 9/25/2015 Houston, TX Location Where Test Conducted 3.2 Test Information Additional Software Relied Upon for Certification Additional Software Applicable Criteria Functionality provided by Additional Software Updox Direct b.1, 2; e.1 HISP No additional software required Page 4 of 12

5 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 07-Aug Test Tools Test Tool Version x Cypress 2.6 x eprescribing Validation Tool HL7 CDA Cancer Registry Reporting Validation Tool HL7 v2 Electronic Laboratory Reporting (ELR) Validation Tool x HL7 v2 Immunization Information System (IIS) Reporting Validation Tool x HL7 v2 Laboratory Results Interface (LRI) Validation Tool x HL7 v2 Syndromic Surveillance Reporting Validation Tool x Transport Testing Tool 181 x Direct Certificate Discovery Tool Edge Testing Tool No test tools required Test Data Alteration (customization) to the test data was necessary and is described in Appendix [insert appendix letter] x No alteration (customization) to the test data was necessary Standards Multiple Standards Permitted The following identifies the standard(s) that has been successfully tested where more than one standard is permitted Criterion # Standard Successfully Tested (a)(8)(ii)(a)(2) (a)(13) x (b)(1) HL7 Version 3 Implementation Guide: URL-Based Implementations of the Context-Aware Information Retrieval (Infobutton) Domain x (a)(3) IHTSDO SNOMED CT International Release July 2012 and US Extension to SNOMED CT March 2012 Release (b)(2) HL7 Version 3 Implementation Guide: Context-Aware Knowledge Retrieval (Infobutton) Service-Oriented Architecture Implementation Guide (j) HL7 Version 3 Standard: Clinical Genomics; Pedigree Page 5 of 12

6 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 07-Aug-2015 Criterion # (a)(15)(i) x (b)(1) HL7 Version 3 Implementation Guide: URL-Based Implementations of the Context-Aware Information Retrieval (Infobutton) Domain Standard Successfully Tested (b)(2) HL7 Version 3 Implementation Guide: Context-Aware Knowledge Retrieval (Infobutton) Service-Oriented Architecture Implementation Guide (a)(16)(ii) (b)(2)(i)(a) (b)(7)(i) (g) Network Time Protocol Version 3 (RFC 1305) (i) The code set specified at 45 CFR (c)(2) (ICD-10- CM) for the indicated conditions (i) The code set specified at 45 CFR (c)(2) (ICD-10- CM) for the indicated conditions (g) Network Time Protocol Version 4 (RFC 5905) x (a)(3) IHTSDO SNOMED CT International Release July 2012 and US Extension to SNOMED CT March 2012 Release x (a)(3) IHTSDO SNOMED CT International Release July 2012 and US Extension to SNOMED CT March 2012 Release (b)(8)(i) (e)(1)(i) (e)(1)(ii)(a)(2) (e)(3)(ii) (i) The code set specified at 45 CFR (c)(2) (ICD-10- CM) for the indicated conditions Annex A of the FIPS Publication [list encryption and hashing algorithms] AES SHA (g) Network Time Protocol Version 3 (RFC 1305) Annex A of the FIPS Publication [list encryption and hashing algorithms] AES SHA (a)(3) IHTSDO SNOMED CT International Release July 2012 and US Extension to SNOMED CT March 2012 Release x (g) Network Time Protocol Version 4 (RFC 5905) Common MU Data Set (15) (a)(3) IHTSDO SNOMED CT International Release July 2012 and US Extension to SNOMED CT March 2012 Release x (b)(2) The code set specified at 45 CFR (a)(5) (HCPCS and CPT-4) Page 6 of 12

7 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 07-Aug-2015 Criterion # Standard Successfully Tested None of the criteria and corresponding standards listed above are applicable Newer Versions of Standards The following identifies the newer version of a minimum standard(s) that has been successfully tested Newer Version Applicable Criteria No newer version of a minimum standard was tested Optional Functionality Criterion # x (a)(4)(iii) (b)(1)(i)(b) (b)(1)(i)(c) (b)(2)(ii)(b) (b)(2)(ii)(c) (e)(1) (f)(3) (f)(7) Common MU Data Set (15) Common MU Data Set (15) Optional Functionality Successfully Tested Plot and display growth charts Receive summary care record using the standards specified at (a) and (b) (Direct and XDM Validation) Receive summary care record using the standards specified at (b) and (c) (SOAP Protocols) Transmit health information to a Third Party using the standards specified at (a) and (b) (Direct and XDM Validation) Transmit health information to a Third Party using the standards specified at (b) and (c) (SOAP Protocols) View, download and transmit data to a third party utilizing the Edge Protocol IG version 1.1 Ambulatory setting only Create syndrome-based public health surveillance information for transmission using the standard specified at (d)(3) (urgent care visit scenario) Ambulatory setting only transmission to public health agencies syndromic surveillance - Create Data Elements Express Procedures according to the standard specified at (b)(3) (45 CFR (a)(4): Code on Dental Procedures and Nomenclature) Express Procedures according to the standard specified at (b)(4) (45 CFR (c)(3): ICD-10-PCS) No optional functionality tested Page 7 of 12

8 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 07-Aug Edition Certification Criteria* Successfully Tested Criteria # Version Version Criteria # TP** TD*** TP TD (a)(1) x (c)(3) (a)(2) 1.2 (d)(1) 1.2 x (a)(3) x (d)(2) 1.6 x (a)(4) x (d)(3) 1.3 x (a)(5) x (d)(4) 1.3 (a)(6) (d)(5) 1.2 (a)(7) (d)(6) 1.2 x (a)(8) 1.3 x (d)(7) 1.2 x (a)(9) (d)(8) 1.2 (a)(10) (d)(9) Optional 1.2 x (a)(11) 1.3 x (e)(1) x (a)(12) 1.3 x (e)(2) Amb. only x (a)(13) 1.2 x (e)(3) Amb. only 1.3 x (a)(14) 1.2 x (f)(1) x (a)(15) 1.5 x (f)(2) (a)(16) Inpt. only x (f)(3) (a)(17) Inpt. only 1.2 (f)(4) Inpt. only (a)(18) (a)(19) (a)(20) x (b)(1) (f)(5) Amb. only (f)(6) Amb. only x (b)(2) (f)(7) Amb. only 1.1 x (b)(3) (g)(1) x (b)(4) x (g)(2) x (b)(5) x (g)(3) 1.4 (b)(6) Inpt. only (g)(4) 1.2 x (b)(7) (h)(1) 1.1 (b)(8) (h)(2) 1.1 (b)(9) (h)(3) 1.1 x (c)(1) x (c)(2) Page 8 of 12

9 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 07-Aug-2015 Criteria # Version Version Criteria # TP** TD*** TP TD No criteria tested *For a list of the 2014 Edition Certification Criteria, please reference (navigation: 2014 Edition Test Method) **Indicates the version number for the Test Procedure (TP) ***Indicates the version number for the Test Data (TD) Page 9 of 12

10 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 07-Aug Clinical Quality Measures* Type of Clinical Quality Measures Successfully Tested: x Ambulatory Inpatient No CQMs tested *For a list of the 2014 Clinical Quality Measures, please the CMS ecqm Library (Navigation: June 2014 and April 2014 Updates) Ambulatory CQMs CMS ID Version CMS ID Version CMS ID Version CMS ID Version x 2 v4 90 x 136 v4 x 155 v3 22 x 117 v x 126 v x 146 v x 75 v x 153 v x 154 v3 182 Inpatient CQMs CMS ID Version CMS ID Version CMS ID Version CMS ID Version Page 10 of 12

11 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 07-Aug Automated Numerator Recording and Measure Calculation Automated Numerator Recording Automated Numerator Recording Successfully Tested (a)(1) (a)(11) (a)(18) (b)(6) (a)(3) (a)(12) (a)(19) (b)(8) (a)(4) (a)(13) (a)(20) (b)(9) (a)(5) (a)(14) (b)(2) (e)(1) (a)(6) (a)(15) (b)(3) (e)(2) (a)(7) (a)(16) (b)(4) (e)(3) (a)(9) (a)(17) (b)(5) x Automated Numerator Recording was not tested Automated Measure Calculation Automated Measure Calculation Successfully Tested x (a)(1) x (a)(11) (a)(18) (b)(6) x (a)(3) x (a)(12) (a)(19) (b)(8) x (a)(4) x (a)(13) (a)(20) (b)(9) x (a)(5) x (a)(14) x (b)(2) x (e)(1) x (a)(6) x (a)(15) x (b)(3) x (e)(2) x (a)(7) (a)(16) x (b)(4) x (e)(3) x (a)(9) (a)(17) x (b)(5) Automated Measure Calculation was not tested Attestation Attestation Forms (as applicable) x Safety-Enhanced Design* x Quality Management System** x Privacy and Security Appendix A B C 3.3 Appendices Attached below. *Required if any of the following were tested: (a)(1), (a)(2), (a)(6), (a)(7), (a)(8), (a)(16), (a)(18), (a)(19), (a)(20), (b)(3), (b)(4), (b)(9). **Required for every EHR product Page 11 of 12

12 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 07-Aug-2015 Test Results Summary Change History Test Report ID Description of Change Date 2014 Edition Test Report Summary Page 12 of 12

13 pcc.. Pediatric EHR Solutions September 1, 2015 To Whom It May Concern, The Peoples Computer Company, Inc. dba The Physician s Computer Company performed a usability test on selected features of PCC ERR version 7.0 in accordance with the Safety- Enhanced Design requirements outlined in (g) (3). Our report containing the results of this test is true and accurate. Sincerel William an e ter Director of Finance and Operations 2OWinooski Falls Way, Suite 7, Winooski,Vt~ r F TM fax I

14 PCC Pediatric EHR Solutions n EHR Usability Test Report of PCC Version 7.0 Report based on ISO/JEC 25062:2006 Common Industry Format Template for Usability Test Reports (NISTIR 7742) Product: PCC Version 7.0 Date of Usability Test: August 27, 2015 Date of Report: September I, 2015 Report Prepared By: Physicians CoinpuLer Company William VanDeventer, Director of Operations & Finance bill@pcc.com 20 Winooski Falls Way, Suite 7 Winooski, VT Winooski Falls Way, Suite 7, Winooski, VT, contro I You r F Ut U re TM lax

15 pcc Pediatric EHR Solutions Table of Contents 1 Executive Summary 2 Introduction 3 Method 3.1 Participants 3.2 Study Design 3.3 Tasks 3.4 Procedure 3.5 Test Location 3.6 Test Environment 3.7 Test Forms and Tools 3.8 Participant Instructions 3.9 Usability Metrics Results 4.1 Data Analysis and Reporting 4.2 Discussion of the Findings Appendices Appendix 1: Appendix 2: Appendix 3: Appendix 4: Participant Demographics Informed Consent Form Example Moderator s Guide System Usability Scale Questionnaire Winooski Falls Way, Suite 7, Winooski, VI, fax Control Your FutureTM...

16 pcc.. Pediatric EHR Solutions Executive Summary A usability test was performed on selected features of the PCC ERR Version 7.0 as part of the Safety-Enhanced Design requirements outlined in (g)(3) in a conference room at 128 Lakeside Drive, Burlington, Vermont on August 27, The purpose of this study was to test and validate our user-centered design process, and guide the development of our ERR system based on observed task completion, activities and documented user feedback. During the usability test, five healthcare providers, matching the target demographic criteria, served as participants and used our EHR in simulated, but representative tasks. The study collected performance data on four tasks typically conducted on an ERR: Computerized provider order entry Medication Allergy List Clinical Decision Support Clinical Information Reconciliation We conducted 5 one-hour, one-on-one usability tests for this study. Test participants were greeted by the moderator, asked to review and sign an Informed Consent form (included in Appendix 2); they were instructed that they could withdraw at any time. Participants had prior experience with the ERR. The moderator introduced the test, and instructed the participants to complete a series of tasks (given one at a time) using the ERR. During the testing, the moderator observed the participants steps, timed the test, and along with the data logger, recorded user performance data on a laptop computer. The moderator did not give the participants assistance in how to complete the task. 20 Winooski Falls Way, Suite 7, Winooski, V1~ ce I N F r TM fax

17 Pcc.. Pediatric EHR Solutions The following types of data were collected for each participant: Number of tasks successfully completed within the allotted time without assistance. Time to complete tasks Number and of types of errors Path deviations Participant s verbalizations Participants satisfaction ratings of the system All participant data was de-identified no correspondence could be made from the identity of the participants to the data collected. Following the conclusion of the testing, participants were asked to complete a post-test questionnaire (included in Appendix 4). No compensation was provided to test participants. Recommended metrics detailed in the NIST Guide to the Process Approach for Improving the Usability of Electronic Health Records were used to evaluate the usability of PCC Version 7.0 and prepare the following summary of results. Table 1 Performance and Rating Summary CPOE (314.a.1) Task Namel Task Time to Optimal Errors Task # participants Success Complete Path Rating Scenario IA: Record Medication Order (Success Deviations (1=Easy Only) 5=Hard) 5 5/5 (M)1:04 35/ % (SD)18.70 Avg:1.O Winooski Falls Way, Suite 7, Winooski, VT, C I N F TM fax

18 Pediatric EHR Solutions Task Name! Task Time to Optimal Errors Task # participants Success Complete Path Rating Scenario 18: Access & Change Medication Order Scenario 2A: Record Laboratory Order Scenario 28: Access & Change Laboratory Order Scenario 3A: Record Radiology Order Scenario 38: Access & Change Radiology Order Medication Allergy List (314.a.7) Scenario IA: Record a Medication Allergy Scenario IB: Access & Change a Medication Allergy Clinical Decision Support (314.a.8) Scenario IA: Select/Activate Interventions (Success Deviations (1=Easy Only) 5=Hard) 5 5/5 (M)38.6 7/ % (SD)10.28 Avg: /5 (M) 22.4 Avg: % (SD) /5 (M) 26.4 Avg: % (SD) % (M):17 Avg: 1.00 (SD) /5 (M) 18.8 Avg: % (SD) /5 (M) Avg: % (SD) /5 (M) 40.5 Avg: % 80% (SD) Winooski Falls Way, Suite 7, Winooski, VT, r TM fax I

19 Pediatric EHR Solutions with permission Task Name/ Task Time to Optimal Errors Task # participants Success Complete Path Rating Scenario 18: Intervention Activity Log with permission Scenario IC: Select/Activate Interventions without permission Scenario 2A: Problem List Interventions Scenario 2B: Medication List Interventions Scenario 2C: Medication Allergy List Interventions Scenario 2D: Demographics Interventions Scenario 2E: Laboratory Test and value/results Interventions (Success Deviations (1=Easy Only) 5=Hard) 1 1/1 (M) 1:04 Avg: % 1 1/1 (M) :33 Avg: % 1/1 (M) :08 Avg: % 5 5/5 (M) 39.8 Avg: % (SD) /5 (M) 39.8 Avg: % (SD) /5 (M) 39.8 Avg: % (SD) /5 (M) 39.8 Avg: % (SD) /5 (M) 39.8 Avg: % (SD) OWinooski Falls Way, Suite 7, Winooski, VT, fax TM...

20 Pediatric EHR Solutions Task Name! Task Time to Optimal Errors Task # participants Success Complete Path Rating Scenario 2F: Vital Signs Interventions Scenario 2G: Combination Interventions Trigger CDS Diagnostic & The rapeutic Reference Resources Scenario 3A: Problem List Interventions Scenario 38. Medication List Interventions Scenario 3C: Laboratory Test and value/results Interventions Scenario 3D: Combination Interventions Scenario 4A. Configure CDS Interventions by Role Scenario 4B: Trigger CDS Intervention by User Role allowed access (Success Deviations (1=Easy Only) 5=Hard) 5 5/5 (M) 39.8 Avg: % (SD) /5 (M) 39.8 Avg: % (SD) /5 (M) 55.2 Avg: % (SD) /5 (M) 55.2 Avg: % (SD) /5 (M) 55.2 Avg: % (SD) /5 (M) 55.2 Avg: % (SD) /1 (M) :30 Avg: % 1 1/1 (M) :30 Avg: I 2oWinooski Falls Way, Suite 7, Winooski, VT, I r TM fax

21 Pediatric EHR Solutions Task Name/ Task Time to Optimal Errors Task # participants Success Complete Path Rating Scenario 4C: No Trigger CDS Intervention by User Role not allowed access Scenario 4D: Select/Activate CDS Intervention without permission Scenario 5: CDS Intervention Review Source Attributes Clinical Information Reconciliation (314.b.4) Scenario I: Reconcile Patient s Active Medication List with Another Source Scenario 2: Reconcile Patient s Active Problem List with Another Source 100% (Success Deviations (1=Easy Only) 5=Hard) 1/1 (M) :08 Avg: % I 1/1 (M) :08) Avg: % 5 (M) 19.2 Avg: % (SD) (M) Avg: % 80% (SD) /5 (M) 65.2 Avg: 1.06 Scenario 3: Reconcile Patient s Active Medication Allergy List with Another Source 100% (SD) /5 (M) 60.8 Avg: % (SD) OWinooski Falls Way, Suite 7,Winooski,V] I i F r TM I fax

22 pcc.. Pediatric EHR Solutions Our System Usability Scale (SUS) score, based on the performance with theses tasks is: 89.5 In addition to the performance data, the following observations were made: Would be nice to have color in the alert message to denote the urgency (red, yellow, green). Larger font in e-prescribing would be easier to see Clinical alerts is a great tool Would also like a sticky note feature in the chart We transfer records between offices a lot, reconciliation is a great tool. I like the reconciliation feature, seems pretty straight forward, I like that there is permission on the reconcile button. Reconcile tool I had to go back because I wasn t sure which one to keep, could be more clear. Allergies I don t know why I have both, I will put it there, too. Reconcile tool there was a lot of back and forth because I wasn t sure. Reconcile tool A lot of screens, would be nice to just have some kind of linking. I still like paper charting better, would go back if I could. Major Findings Participants achieved consistently high success rates for individual tasks. Task completion times were within the expected ranges with the exception of the two tasks for Medication Allergy. The tester completed this task correctly, with the optimal number of steps, it just took longer than the allotted time. 2oWinooski Falls Way, Suite 7, Winooski,VT, I Y r F r TM i fax I I

23 Pcc.. Pediatric EHR Solutions The third failure recorded was on the new Clinical Reconciliation tool. The tester selected the incorrect problem for the problem list task. However, this tester was successful with the medication and medication allergy tasks. Users rated the ease of use a 1 very easy for most tasks. There was one 4 rating, and the comments given pertained to the e-prescribing part of the task, which is a 3~i~ party software (Dr. First). Area for Improvement Participants recruited from our user community for this study were pleased that we asked them to be part of our usability testing. We will continue to offer our users opportunities to participate in usability reviews going forward. We will continue to review our Clinical Reconciliation tool to improve the ease of use for our users. We gathered good feedback during this test to make some changes that will assist us. We will continue to monitor ease of use and efficiency measures and focus our efforts on improving overall scores. Introduction The Electronic Health Record Under Testing (EHRUT) for this study was PCC EHR Version 7.0. Designed to present medical information to healthcare providers in pediatric ambulatory outpatient practices to record and manage patient charts. These solutions improve revenue cycle management and enhance the accuracy in clinical documentation. The usability testing attempted to represent realistic exercises and conditions. 20 Winooski Falls Way, Suite 7, Winooski, VT, r I ~i r F TM i fax I I...

24 pcc.. Pediatric EHR Solutions The purpose of this study was to test and validate the usability of the current user interface in accordance with the Safety-Enhanced Design requirements for 2014 ONC EHR certification, and to collect data for our ongoing user-centered design and usabilit to use in our product development. Measures of effectiveness, efficiency and user satisfaction were collected as part of this study. Method Participants A total of 5 participants, matching our target demographic, were tested on PCC EHR Version 7.0. Participants in the test were Doctors of Medicine, Registered Nurses, Nurse Practitioners, and Medical Assistants. No participants were compensated for their participation. In addition, test participants had no direct connection to the development of the EHR and are not connected to any PCC employees or contractors. Participants were given the opportunity to have the same orientation and level of training as the actual end users would have received. The following is a table (Table 2) of participants by characteristics, including demographics, professional experience, computer experience and user needs for assistive technology. Participant names were replaced with Participant IDs so that an individuals data cannot be tied back to individual identities. 2OWinooski Falls Way, Suite 7, Winooski, VT, I i r F TM lax I...

25 Pcc.. Pediatric EHR Solutions Table 2 Participant Characteristics Part Gender Age Occupation! Professional Product Assistive ID Role Experience Experience Technology Needs T- 1 M 40 Medical Asst 24 years 5 years None T-2 F 24 Medical Asst 8 years 4 years None T-3 F 31 Nurse 11 years 5 years None T-4 M 35 Physician 12 years 5 years None T-5 F 58 Nurse 38 years 4 years None Practitioner Five participants (matching the demographics in the section on Participants) were recruited and five participated in the usability test. Zero participants failed to show for the study. Participants were scheduled for 60-minute sessions, including the time for debriefing by the moderator and data logger, and to reset systems to proper test conditions. Information on Participant schedule and demographics characteristics was recorded. Study Design Overall, the objective of this test was to uncover areas where the application performed well that is, effectively, efficiently and with satisfaction and areas where the application failed to meet the needs of the participants. The data from this test may serve as a baseline for future tests with an updated version of the same EHR and/or comparison with other EHRs provided the same tasks are used. In short, this testing serves as both a means to record or benchmark current usability, but also to identify areas where improvements must be made. 2oWinooski Falls Way, Suite 7, Winooski,VT, r F r TM fax I

26 Pcc.. Pediatric EHR Solutions During the usability test, participants interacted with one ERR, PCC ERR Version 7.0. Each participant used the system in the same location, and was provided with the same instructions. The system was evaluated for effectiveness, efficiency and satisfaction as defined by the measures collected and analysed for each participant: Number of tasks successfully completed within the allotted time without assistance Time to complete the tasks Number and types of errors Path deviations Participant s verbalizations (comments) Participant s satisfaction ratings of the system Additional information about the various measures can be found in Section 3.9 on Usability Metrics. Tasks A number of tasks were constructed that would be realistic and representative of the kinds of activities a user might do with this ERR, including: 1. Computerized Provider Order Entry (CPOE) 2. Medication Allergy List 3. Clinical Decision Support 4. Clinical Information Reconciliation Representative tasks were designed to exercise ERR functionality for each feature under study. Individual tasks were grouped so that each participant could reasonably complete their assigned tasks in one hour. Tasks were selected based on their frequency of use, critically of function, and those that may be the most troublesome for users. Tasks should always be constructed in light of the study objectives. 20 Winooski Falls Way, Suite 7, Winooski, VT, I i F lax

27 pcc.. Pediatric EHR Solutions Risk Assessment Physician s Computer Company assessed the risk level to patients for all provider workilows included in this test. These workflows were evaluated based on the potential risk for adverse events to the patient and assigned a risk category of high, moderate or low risk. Tasks which included medications, interventions, alerts and/or notifications were determined to be of high risk, while basic order functions and non-medication list reviews were moderate, and accessing reference material was determined to be low. The risk assessment for each individual task can be found in Table 3 below. 2oWinooski Falls Way, Suite 7, Winooski, VT r r F r TM j fax I I I

28 pcc Pediatric EHR Solutions Table 3 Risk Categories by Task a. 1: a. 1: a. 1: a.1: a.7: a.7: a.8: Clinical Decision Support a.8: Clinical Decision Support a.8: Clinical Decision Support a.8: Clinical Decision Support l4.a.8: Clinical Decision Support a.8: Clinical Decision Support a.8: Clinical Decision Support a.8: Clinical Decision Support Reference Interventions a.8: Clinical Decision Support Tasks/Measures CPOE Record a Medication Order CPOE Access and Change a Medication Order CPOE Order, Access and Change a Lab Order CPOE Order, Access and Change a Radiology Order Medication Allergy List: Record a Medication Allergy Medication Allergy List: Access and Change a Medication Allergy Configuration of CDS Interventions Problem List Interventions Medication List Interventions Medication Allergy List Interventions Demographics Interventions Laboratory Tests & Results Interventions Vital Signs Interventions Identify Diagnostic & Therapeutic Review Source Attributes Interventions Risk Assessment High High Moderate Moderate High High High Moderate High High Low Moderate Moderate b.4: Clinical Reconciliation: Reconcile Patient s Active Problem List with Moderate another source b.4: Clinical Reconciliation: Reconcile Patient s Active Medication List Moderate with another source b.4: Clinical Reconciliation: Reconcile Patient s Active Medication Allergy Moderate List with another source Low Low Procedures Upon arrival, participants were greeted; their identity was verified and matched with a name on the participant schedule. Participants were assigned a participant ID. Each 20 Winooski Falls Way, Suite 7, Winooski, VT, fax Control Your FutureTM n

29 Pcc.. Pediatric EHR Solutions participant reviewed and signed an informed consent and release form (See Appendix 2). A representative from the test team witnessed the participant s signature. To ensure the test ran smoothly, two staff members participated in this test, the moderator and the data logger. The moderator conducted the session including administering instructions and tasks. The moderator also took notes on task success, path deviations, number and types of errors, and obtained post-task rating data. A second person served as the data logger and recorded task times, took notes on participant comments, task success, path deviations, and number of errors. Participants were instructed to perform the tasks (see specific instructions below) As quickly as possible making as few errors and deviations as possible. Without assistance; moderators were allowed to give immaterial guidance and clarification on tasks, but not instructions on use Without using a think aloud technique For each task, the moderator read the task. Task timing began once the moderator finished reading the questions and said begin. The task time was stopped once the participant indicated they had successfully completed the task, by saying done, or completing the task and returning to the schedule screen. Scoring is discussed below in Section 3.9. Following the test completion, the moderator gave the participant the post test System Usability Scale Questionnaire (Appendix 4). Asked the participants for any comments and feedback, and thanked them for their participation. Participants demographic information, task success rate, time on task, errors, deviations, verbal responses, and post-test questionnaire were recorded. 20 Winooski Falls Way, Suite 7, Winooski, VI, r I y r F r TM lax I I I

30 pcc.. Pediatric EHR Solutions Test Location The testing facility included a waiting area and a quiet testing conference room with a table, computer for the participant, and recording computer for the data logger. Only the participant, moderator and data logger were in the testing room. To ensure the environment was comfortable for users, noise levels were kept to a minimum with the ambient temperature at a normal range. All of the safety instruction and evacuation procedures were valid, in place, and visible to the participants. Test Environment PCC ERR would typically be used in a healthcare office or facility. In this instance, the testing was conducted in a conference room. For testing, the computer used was a laptop running Windows. The participants used a mouse and keyboard when interacting with the ERR. The ERR was displayed on a 18-inch laptop screen in full color. The application was set up by Physician s Computer Company. The application itself was running on a Windows platform with a wireless connection, using a testing database via a Remote Desktop connection. Technically, the system performance (i.e., response time) was representative to what actual users would experience in a field implementation. Additionally, participants were instructed not to change any of the default system settings (such as control of the font size). Test Forms and Tools During the usability test, various documents and instruments were used, including: 1. Informed Consent 2. Moderator s Guide 3. Post-Test Questionnaire Examples of these documents can be found in Appendices 2-4 respectively. The 2OWinooski Falls Way, Suite 7,Winooski,VT, I Y r F TM fax I n

31 Pcc.. Pediatric EHR Solutions Moderator s Guide was devised so as to be able to communicate tasks to participants in a uniform manner and to facilitate capturing required data. Participant Instructions The moderator reads the following instructions aloud to each participants (also see ful Moderator s Guide in Appendix 3). Thank you for participating in this study. Our session today will last approximately 60 minutes. During that time you will take a look at an elect ronic health record system. This is an alpha version of the software. Some of the data may not make sense as it is testing data. We are testing the software, not you. During this test, we will ask you to conduct several tasks using this syvtem and answer some questions. We are interested in how easy (or how difficult) this system is to use, what features in the software would be useful to you, and how we could improve it. For each task we will read the instructions out loud and then ask you to walk through peiforniing the task. We ask that you complete these tasks on your own trying to do theni as quickly as possible with the fewest possible errors or deviations. Do not do anything more than asked. When you have completed the task, please say Done, and we ll move onto the next task. We cannot answer any questions or help you with anything to do with the system itself If you get stuck dii task and don t know what to do next, just let us know. We ll end the task and move onto the next one. Please save your detailed comments until the end of a task or the end of the session as a whole when we can discuss freely Please be honest with your opinions. Following the procedural instructions, participants were given their tasks to complete. Tasks are listed in the Moderator s guide in Appendix 3. Usability Metrics According to the NJST Guide to the Process Approach for Improving the Usability of Electronic Health Records, EHRs should support a process that provides a high level of usability for all users. The goal is for users to interact with the system effectively, 20 Winooski Falls Way, Suite 7, Winooski, VT, r I V F fax

32 PCC Pediatric EHR Solutions efficiently, and with an acceptable level of satisfaction. To this end, metrics for effectiveness, efficiency and user satisfaction were captured during the usability testing. The goals of the test were to assess: 1. Effectiveness of PCC EHR by measuring participant success rates and errors 2. Efficiency of PCC EHR by measuring task time and path deviations 3. Satisfaction of PCC ERR by measuring ease of use ratings. Data Scoring The scoring table (Table 4) details how tasks were scored, errors evaluated, and time data analyzed. 20WinooskiFaIIsWay,Suite7,Winooski,Vt~ fax Control Your Future TM

33 pcc.. Pediatric EHR Solutions Measures Effectiveness: Task Success Effectiveness: Task Failure Efficiency: Task Deviations Efficiency: Task Time Rationale and Scoring Table 4 Data Scoring Table A task was considered as a Success if the participant was able to achieve the correct outcome, without assistance, within the time allotted on a per task basis. The total number of successes were calculated for each task and then divided by the total number of times that task was attempted. The results are provided as a percentage. Task times were recorded for successes. Optimal task performance time, as bench marked by expert performance under realistic conditions, is recorded when constructing tasks. Target task times used must be operationally defined by taking multiple measures of optimal performance and multiplying by a factor of [1.25] that allows some buffer because the participants are presumably not trained to expert performance. Thus, if expert, optimal performance on a task was [30] seconds then allotted task time performance was [30 X 1.25 = 38] seconds. This ratio should be aggregated across tasks and reported with mean and variance scores. A task was considered as a Failure if the participant abandoned the task, did not reach the correct answer or performed it incorrectly, or reached the end of the allotted time before successful completion. No task times were taken for errors. The total number of errors/failures was calculated for each task and then divided by the total number of times that task was attempted. Not all deviations would be counted as errors. The participants path (i.e. steps) through the application was recorded. Deviations occur if the participant, for example, went to a wrong screen, clicked on an incorrect menu item, followed an incorrect link, or interacted with an on-screen control. This path was compared to the optimal path. The number of steps in the observed path is divided by the number of optimal steps to provide a ratio of path deviation. It is strongly recommended that task deviations be reported. Optimal paths (i.e. procedural steps) should be recorded when constructing tasks. Each task was timed from when the moderator said begin until the participant said done. If he or she failed to say done, the time was stopped when the participant stopped performing the task. Only task times for tasks that were successfully completed were included in the average task time analysis. Average time per task was calculated for each task. Variance measures (standard deviation and mean) were also calculated. Satisfaction: Participant s subjective impression of the ease of use of the application was measured by administenng both a simple post-task question as well as a post-session questionnaire. After each task, the participant was asked to rate Overall, this task Task Rating was: on a scale of I (Very Easy) to 5 (Very Difficult). These data are averaged across participants. Common convention is that average ratings for systems judged easy to use should be 3.3 or below. To measure participants confidence in the likeability of the PCC EHR overall, the testing team administered the System Usability Scale (SUS) questionnaire. See Appendix 4 for the questionnaire. 20Winooski Falls Way, Suite 7, Winooski,Vt, I N r F r TM fax I...

34 Pediatric EHR Solutions Results Data Reporting and Analysis The results of the usability test were calculated according to the methods specified in the Usability Metrics section above. The usability testing results for the PCC EHR Version 7.0 are detailed below (see Table 5). CPOE (314.a.1) Table 5 Testing Results Task Name/ Task Time to Errors Task # participants Success Complete Path Rating Scenario IA: Record Medication Order scenario IB: Access & Change Medication Order (Success Deviations (1=Easy Only) (Observed 5=Hard) /Optimal) Success 1:48 7/7 0 1 Success 1:30 7/7 0 I Success 1:06 7/7 0 I Success 1:04 7/7 0 I Success 1:32 7/ /5 (M)1:04 35/ % (SD)18.70 Avg: I Success :54 7/7 0 I Success :44 7/7 0 1 Success :29 7/7 0 1 Success :32 7/7 0 1 Success :34 7/7 0 I 5 5/5 (M)38.6 7/ oWinooski Falls Way, Suite 7, Winooski,VT, r I Y F r TM fax I I...

35 Pediatric EHR Solutions Task Name/ Task Time to Errors Task # participants Success Complete Path Rating Scenario 2A: Record Laboratory Order Scenario 2B: Access & Change Laboratory Order (Success Deviations (1=Easy Only) (Observed 5=Hard) /Optimal) 100% (SD)10.28 Avg: Success :19 5/5 0 1 Success :25 5/5 0 1 Success :31 5/5 0 1 Success :20 5/5 0 1 Success :17 5/ /5 (M) 22.4 Avg: % (SD) I Success :28 9/8 0 1 Success :14 8/8 0 I Success :40 8/8 0 I Success :20 8/8 0 I Success :30 8/8 0 I 5 5/5 (M) 26.4 Avg: I 22 Scenario 3A: Record Radiology Order 100% (SD) Success :15 5/5 0 1 Success :10 5/5 0 1 Success :29 5/5 0 1 Success :17 5/5 0 1 Success :14 5/ % (M):17 Avg: (SD) OWinooski Falls Way, Suite 7,WinooskL VT, r F r TM fax I...

36 Pediatric EHR Solutions Task Name/ Task Time to Errors Task # participants Success Complete Path Rating Scenario 3B: Access & Change Radiology Order Medication Allergy List (314.a.7) Scenario IA: Record a Medication Allergy Scenario IB: Access & Change a Medication Allergy (Success Deviations (1=Easy Only) (Observed 5=Hard) /Optimal) Success :30 9/8 0 Success :10 8/8 0 Success :20 8/8 0 Success :15 8/8 0 Success :19 8/8 0 5/5 (M) 18.8 Avg: % (SD) 7.39 Success 1:20 10/7 0 2 Failure 1 Success 1:28 11/7 0 2 Success 1:37 7/7 0 4 Success :58 7/ /5 (M) Avg: % % (SD) Success :40 7/7 0 2 Failure Success :54 7/7 0 2 Success :25 7/7 0 4 Success :43 7/ /5 (M) 40.5 Avg: % OWinooski Falls Way, Suite 7,Winooski,VT r I y r r TM fax I

37 Pediatric EHR Solutions Task Name! Task Time to Errors Task # participants Success Complete Path Rating Clinical Decision Support (314.a.8) Scenario IA: Select/Activate Interventions with permission Scenario I B: Intervention Activity Log with permission. Scenario IC: Select/Activate Interventions without permission (Success Deviations (1=Easy Only) (Observed 5=Hard) /Optimal) 80% (SD) Success 1:04 5/5 1/1 (M) 1:04 Avg: % Success :33 3/3 1/1 (M) :33 Avg: % Success :08 2/2 I/i (M) :08 Avg: % Scenario 2A: Problem List Interventions Success :58 4/4 0 1 Scenario 2B: Medication List Interventions Success :34 4/4 0 3 Success :36 4/4 0 1 Success :40 4/4 0 1 Success :31 4/ /5 (M) 39.8 Avg: % (SD) Success :58 4/ oWinooski Falls Way, Suite 7, Winooski, Vt, r I Y r F TM fax I I

38 Pediatric EHR Solutions Task Name! Task Time to Errors Task # participants Success Complete Path Rating Scenario 2C: Medication Allergy List Interventions Scenario 2D: Demographics Interventions (Success Deviations (1=Easy Only) (Observed 5=Hard) /Optimal) Success :34 4/4 0 3 Success : Success :40 4/4 0 1 Success :31 4/ /5 (M) 39.8 Avg: % (SD) Success 4/4 0 1 Success 4/4 0 3 Success 4/4 0 1 Success :40 4/4 0 1 Success :31 4/ /5 (M) 39.8 Avg: % (SD) Success :58 4/4 0 1 Success :34 4/4 0 3 Success :36 4/4 0 1 Success :40 4/4 0 1 Success :31 4/ /5 (M) 39.8 Avg: Scenario 2E: Laboratory Test and value/results Interventions 100% (SD) Success :58 4/ oWinooski Falls Way, Suite 7,Winooski,VT, r r F TM fax I I nn

39 Pediatric EHR Solutions Task Name! Task Time to Errors Task # participants Success Complete Path Rating Scenario 2F: Vital Signs Interventions (Success Deviations (1=Easy Only) (Observed 5=Hard) /Optimal) Success :34 4/4 0 3 Success :36 4/4 0 1 Success :40 4/4 0 1 Success :31 4/ /5 (M) 39.8 Avg: % (SD) Scenario 2G: Combination Interventions Trigger CDS Diagnostic & Therapeutic Reference Resources Scenario 3A: Problem List Interventions Success :58 4/4 0 1 Success :34 4/4 0 3 Success :36 4/4 0 1 Success :40 4/4 0 1 Success :31 4/ /5 (M) 39.8 Avg: % (SD) Success :58 4/4 0 1 Success :34 4/4 0 3 Success :36 4/4 0 1 Success :40 4/4 0 1 Success :31 4/ /5 (M) 39.8 Avg: % (SD) Success :54 5/ Winooski Falls Way, Suite 7, Winooski, VT, I Y F r TM i fax

40 Pediatric EHR Solutions Task Name! Task Time to Errors Task # participants Success Complete Path Rating Scenario 3B: Medication List Interventions Scenario 3C: Laboratory Test and value/results Interventions Scenario 3D: Combination Interventions (Success Deviations (1=Easy Only) (Observed 5=Hard) /Optimal) Success 1:26 5/5 0 3 Success :42 5/5 0 1 Success :55 5/5 0 1 Success :47 5/ /5 (M) 55.2 Avg: % (SD) Success :54 5/5 0 1 Success 1:26 5/5 0 3 Success :42 5/5 0 1 Success :55 5/5 0 1 Success :47 5/ /5 (M) 55.2 Avg: % (SD) Success 5/5 0 1 Success 5/5 0 3 Success 5/5 0 1 Success 5/5 0 1 Success :47 5/ /5 (M) 55.2 Avg: % (SD) Success :54 5/5 0 Success 1:26 5/ oWinooski Falls Way, Suite 7, Winooski,V1~ r I N F r TM fax I n 27

41 pcc.. Pediatric EHR Solutions 28 Task Name! Task Time to Errors Task # participants Success Complete Path Rating Scenario 4A: Configure CDS Interventions by Role Scenario 48: Trigger CDS Intervention by User Role allowed access Scenario 4C: No Trigger CDS Intervention by User Role not allowed access Scenario 4D: SelectlActivate CDS Intervention without permission ScenarioS: CDS Intervention Review Source Attributes (Success Deviations (1=Easy Only) (Observed 5=Hard) /Optimal) Success :42 5/5 0 1 Success :55 5/5 0 1 Success :47 5/ /5 (M) 55.2 Avg: % (SD) Success :30 6/6 1/1 (M) :30 Avg: % Success :30 5/5 1 1/1 (M) :30 Avg: % Success :08 5/5 1/1 (M) :08 Avg: % Success :08 3/3 1 1/1 (M) :08) Avg: % 20 Winooski Falls Way, Suite 7, Winooski, VI, I N r F r TM fax I...

42 Pcc.. Pediatric EHR Solutions Task Name! Task Time to Errors Task # participants Success Complete Path Rating Clinical Information Reconciliation (314.b.4) Scenario 1: Reconcile Patient s Active Medication List with Another Source Scenario 2: Reconcile Patient s Active Problem List with Another Source (Success Deviations (1=Easy Only) (Observed 5=Hard) IOptimal) Success :23 3/3 0 1 Success :20 5/3 0 3 Success :17 3/3 0 1 Success :24 3/3 0 1 Success :12 3/ /5 (M) 19.2 Avg: % (SD) 4.86 Failure 1 Success 1:03 15/ Success :43 13/ Success 1:02 13/ Success 2:05 16/ /5 (M) Avg: % % (SD) Success 13/ Success 1:10 15/ Success :49 13/ Success :59 13/ Success 1:50 15/ /5 (M) 65.2 Avg: oWinooski Falls Way, Suite 7,Winooski,V] I ~( r F TM fax I

43 Pediatric EHR Solutions Task Name/ Task Time to Errors Task # participants Success Complete Path Rating (Success Deviations (1=Easy Only) (Observed 5=Hard) /Optimal) 100% (SD) Scenario 3: Reconcile Patients Active Medication Allergy List with Another Source Success :33 13/ Success 1:05 13/ Success :41 13/ Success 1:00 13/ Success 1:45 13/ /5 (M)60.8 Avg: % (SD)28.00 Discussion of Findings Effectiveness Effectiveness (ease of use) scores were within acceptable ranges of The highest score was due to a tester s unhappiness with the Dr. First e-prescribing software, even though the task was completed successfully. The second highest scores were 1.80 & 1.75 for the less familiar tasks of Clinical Information Reconciliation. There were only three failures during the testing study, two of those were due to the fact that the tester didn t complete the task in the allotted time. The optimal steps were correct. 20Winooski Falls Way, Suite 7,Winooski,VT r F r TM fax I I...

44 Pcc.. Pediatric EHR Solutions Efficiency Participants in the study, for the most part, followed the optimal paths to complete the assigned tasks. The exceptions were mostly with the less familiar tasks of Clinical Information Reconciliation. And, even then, the deviations were to review previous screens to assure they were doing things correctly. Satisfaction Participants were satisfied with the tasks tested in this study. The results from the SUS (System Usability Scale) scored the subjective satisfaction with the system based on performance with these tasks to be Broadly interpreted, scores under 60 represent systems with poor usability; scores over 80 would be considered above average. Within this survey, 100% agreed/strongly agreed that the system was easy to use. And, 100% also agreed/strong agreed they felt very confident using the system. Major Findings Participants achieved consistently high success rates for individual tasks. Task completion times were within the expected ranges with the exception of the two tasks for Medication Allergy. The tester completed this task correctly, with the optimal number of steps, it just took longer than the allotted time. The third failure recorded was on the new Clinical Reconciliation tool. The tester selected the incorrect problem for the problem list task. However, this tester was successful with the medication and medication allergy tasks. Users rated the ease of use a 1 very easy for most tasks. There was one 4 rating, and the comments given pertained to the e-prescribing part of the task, which is a 3t1 party software (Dr. First). 2oWinooski Falls Way, Suite 7, WinooskL VT, I ( F TM fax