ONC HIT Certification Program

Transcription

1 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 2-Oct-2015 ONC HIT Certification Program Test Results Summary for 2014 Edition EHR Certification Part 1: Product and Developer Information 1.1 Certified Product Information Product Name: Infiniti Physicians EMR (ipemr) Product Version: 2.1 Domain: Ambulatory Test Type: Complete EHR 1.2 Developer/Vendor Information Developer/Vendor Name: Physician EMR, LLC. Address: 2500 West Lake Mary Blvd #217 Lake Mary FL Website: Phone: Developer/Vendor Contact: Adnan Malik Page 1 of 12

2 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 2-Oct-2015 Part 2: ONC-Authorized Certification Body Information 2.1 ONC-Authorized Certification Body Information ONC-ACB Name: Drummond Group Address: North Hwy 183, Ste B , Austin, TX Website: Phone: (512) ONC-ACB Contact: Jodi Gonzalez This test results summary is approved for public release by the following ONC-Authorized Certification Body Representative: Jodi Gonzalez ONC-ACB Authorized Representative Certification Body Manager Function/Title Signature and Date 11/9/ Gap Certification The following identifies criterion or criteria certified via gap certification (a)(1) (a)(19) (d)(6) (h)(1) (a)(6) (a)(20) (d)(8) (h)(2) (a)(7) (b)(5)* (d)(9) (h)(3) (a)(17) (d)(1) (f)(1) (a)(18) (d)(5) (f)(7)** *Gap certification allowed for Inpatient setting only **Gap certification allowed for Ambulatory setting only x No gap certification Page 2 of 12

3 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 2-Oct Inherited Certification The following identifies criterion or criteria certified via inherited certification (a)(1) (a)(16) Inpt. only (c)(2) (f)(2) (a)(2) (a)(17) Inpt. only (c)(3) (f)(3) (a)(3) (a)(18) (d)(1) (f)(4) Inpt. only (a)(4) (a)(19) (d)(2) (a)(5) (a)(20) (d)(3) (a)(6) (b)(1) (d)(4) (a)(7) (b)(2) (d)(5) (a)(8) (b)(3) (d)(6) (f)(7) (a)(9) (b)(4) (d)(7) (g)(1) (a)(10) (b)(5) (d)(8) (g)(2) (a)(11) (b)(6) Inpt. only (d)(9) Optional (g)(3) (a)(12) (b)(7) (e)(1) (g)(4) (a)(13) (b)(8) (e)(2) Amb. only (h)(1) (a)(14) (b)(9) (e)(3) Amb. only (h)(2) (a)(15) (c)(1) (f)(1) (h)(3) x No inherited certification (f)(5) Amb. only (f)(6) Amb. only Page 3 of 12

4 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 2-Oct-2015 Part 3: NVLAP-Accredited Testing Laboratory Information Report Number: JPD Test Date(s): 8/18/15, 9/21/15, 10/15/ NVLAP-Accredited Testing Laboratory Information ATL Name: Drummond Group EHR Test Lab Accreditation Number: NVLAP Lab Code Address: North Hwy 183, Ste B , Austin, TX Website: Phone: ATL Contact: Kyle Meadors For more information on scope of accreditation, please reference NVLAP site. Part 3 of this test results summary is approved for public release by the following Accredited Testing Laboratory Representative: Jim Dow Test Proctor ATL Authorized Representative Function/Title 11/9/2015 Vancouver, WA Signature and Date Location Where Test Conducted 3.2 Test Information Additional Software Relied Upon for Certification Additional Software Applicable Criteria Functionality provided by Additional Software Medi-Span a.2, a.6, b.3 Drug DB Lookup EMR Direct phimail b.1, b.2, e.1 Direct No additional software required Page 4 of 12

5 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 2-Oct Test Tools Test Tool Version x Cypress x eprescribing Validation Tool HL7 CDA Cancer Registry Reporting Validation Tool HL7 v2 Electronic Laboratory Reporting (ELR) Validation Tool x HL7 v2 Immunization Information System (IIS) Reporting Validation Tool x HL7 v2 Laboratory Results Interface (LRI) Validation Tool x HL7 v2 Syndromic Surveillance Reporting Validation Tool x Transport Testing Tool 182 x Direct Certificate Discovery Tool Edge Testing Tool No test tools required Test Data Alteration (customization) to the test data was necessary and is described in Appendix [insert appendix letter] x No alteration (customization) to the test data was necessary Standards Multiple Standards Permitted The following identifies the standard(s) that has been successfully tested where more than one standard is permitted Criterion # Standard Successfully Tested (a)(8)(ii)(a)(2) (a)(13) x (b)(1) HL7 Version 3 Implementation Guide: URL-Based Implementations of the Context-Aware Information Retrieval (Infobutton) Domain x (a)(3) IHTSDO SNOMED CT International Release July 2012 and US Extension to SNOMED CT March 2012 Release (b)(2) HL7 Version 3 Implementation Guide: Context-Aware Knowledge Retrieval (Infobutton) Service-Oriented Architecture Implementation Guide (j) HL7 Version 3 Standard: Clinical Genomics; Pedigree Page 5 of 12

6 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 2-Oct-2015 Criterion # (a)(15)(i) x (b)(1) HL7 Version 3 Implementation Guide: URL-Based Implementations of the Context-Aware Information Retrieval (Infobutton) Domain Standard Successfully Tested (b)(2) HL7 Version 3 Implementation Guide: Context-Aware Knowledge Retrieval (Infobutton) Service-Oriented Architecture Implementation Guide (a)(16)(ii) (b)(2)(i)(a) (b)(7)(i) (g) Network Time Protocol Version 3 (RFC 1305) (i) The code set specified at 45 CFR (c)(2) (ICD-10- CM) for the indicated conditions (i) The code set specified at 45 CFR (c)(2) (ICD-10- CM) for the indicated conditions (g) Network Time Protocol Version 4 (RFC 5905) x (a)(3) IHTSDO SNOMED CT International Release July 2012 and US Extension to SNOMED CT March 2012 Release x (a)(3) IHTSDO SNOMED CT International Release July 2012 and US Extension to SNOMED CT March 2012 Release (b)(8)(i) (e)(1)(i) (e)(1)(ii)(a)(2) (e)(3)(ii) (i) The code set specified at 45 CFR (c)(2) (ICD-10- CM) for the indicated conditions Annex A of the FIPS Publication [list encryption and hashing algorithms] AES_128 SHA256 A (g) Network Time Protocol Version 3 (RFC 1305) Annex A of the FIPS Publication [list encryption and hashing algorithms] AES_128 SHA (a)(3) IHTSDO SNOMED CT International Release July 2012 and US Extension to SNOMED CT March 2012 Release S (g) Network Time Protocol Version 4 (RFC 5905) Common MU Data Set (15) (a)(3) IHTSDO SNOMED CT International Release July 2012 and US Extension to SNOMED CT March 2012 Release x (b)(2) The code set specified at 45 CFR (a)(5) (HCPCS and CPT-4) Page 6 of 12

7 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 2-Oct-2015 Criterion # Standard Successfully Tested None of the criteria and corresponding standards listed above are applicable Newer Versions of Standards The following identifies the newer version of a minimum standard(s) that has been successfully tested Newer Version Applicable Criteria No newer version of a minimum standard was tested Optional Functionality Criterion # (a)(4)(iii) (b)(1)(i)(b) (b)(1)(i)(c) (b)(2)(ii)(b) (b)(2)(ii)(c) (e)(1) x (f)(3) (f)(7) Common MU Data Set (15) Common MU Data Set (15) Optional Functionality Successfully Tested Plot and display growth charts Receive summary care record using the standards specified at (a) and (b) (Direct and XDM Validation) Receive summary care record using the standards specified at (b) and (c) (SOAP Protocols) Transmit health information to a Third Party using the standards specified at (a) and (b) (Direct and XDM Validation) Transmit health information to a Third Party using the standards specified at (b) and (c) (SOAP Protocols) View, download and transmit data to a third party utilizing the Edge Protocol IG version 1.1 Ambulatory setting only Create syndrome-based public health surveillance information for transmission using the standard specified at (d)(3) (urgent care visit scenario) Ambulatory setting only transmission to public health agencies syndromic surveillance - Create Data Elements Express Procedures according to the standard specified at (b)(3) (45 CFR (a)(4): Code on Dental Procedures and Nomenclature) Express Procedures according to the standard specified at (b)(4) (45 CFR (c)(3): ICD-10-PCS) No optional functionality tested Page 7 of 12

8 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 2-Oct Edition Certification Criteria* Successfully Tested Criteria # Version Version Criteria # TP** TD*** TP TD x (a)(1) x (c)(3) x (a)(2) 1.2 x (d)(1) 1.2 x (a)(3) x (d)(2) 1.6 x (a)(4) x (d)(3) 1.3 x (a)(5) x (d)(4) 1.3 x (a)(6) x (d)(5) 1.2 x (a)(7) x (d)(6) 1.2 x (a)(8) 1.3 x (d)(7) 1.2 x (a)(9) x (d)(8) 1.2 x (a)(10) (d)(9) Optional 1.2 x (a)(11) 1.3 x (e)(1) x (a)(12) 1.3 x (e)(2) Amb. only x (a)(13) 1.2 x (e)(3) Amb. only 1.3 x (a)(14) 1.2 x (f)(1) x (a)(15) 1.5 x (f)(2) (a)(16) Inpt. only x (f)(3) (a)(17) Inpt. only 1.2 (f)(4) Inpt. only (a)(18) (a)(19) (a)(20) x (b)(1) (f)(5) Amb. only (f)(6) Amb. only x (b)(2) (f)(7) Amb. only 1.1 x (b)(3) (g)(1) x (b)(4) x (g)(2) x (b)(5) x (g)(3) 1.4 (b)(6) Inpt. only x (g)(4) 1.2 x (b)(7) (h)(1) 1.1 (b)(8) (h)(2) 1.1 (b)(9) (h)(3) 1.1 x (c)(1) x (c)(2) Page 8 of 12

9 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 2-Oct-2015 Criteria # No criteria tested Version Version Criteria # TP** TD*** TP TD *For a list of the 2014 Edition Certification Criteria, please reference (navigation: 2014 Edition Test Method) **Indicates the version number for the Test Procedure (TP) ***Indicates the version number for the Test Data (TD) Page 9 of 12

10 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 2-Oct Clinical Quality Measures* Type of Clinical Quality Measures Successfully Tested: x Ambulatory Inpatient No CQMs tested *For a list of the 2014 Clinical Quality Measures, please the CMS ecqm Library (Navigation: June 2014 and April 2014 Updates) Ambulatory CQMs CMS ID Version CMS ID Version CMS ID Version CMS ID Version x 2 v x 156 v3 x 50 v3 122 x 138 v x 127 v x 164 v x 165 v3 x 68 v x 69 v3 131 x 147 v x 182 v4 Inpatient CQMs CMS ID Version CMS ID Version CMS ID Version CMS ID Version Page 10 of 12

11 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 2-Oct Automated Numerator Recording and Measure Calculation Automated Numerator Recording Automated Numerator Recording Successfully Tested (a)(1) (a)(11) (a)(18) (b)(6) (a)(3) (a)(12) (a)(19) (b)(8) (a)(4) (a)(13) (a)(20) (b)(9) (a)(5) (a)(14) (b)(2) (e)(1) (a)(6) (a)(15) (b)(3) (e)(2) (a)(7) (a)(16) (b)(4) (e)(3) (a)(9) (a)(17) (b)(5) x Automated Numerator Recording was not tested Automated Measure Calculation Automated Measure Calculation Successfully Tested x (a)(1) x (a)(11) (a)(18) (b)(6) x (a)(3) x (a)(12) (a)(19) (b)(8) x (a)(4) x (a)(13) (a)(20) (b)(9) x (a)(5) x (a)(14) x (b)(2) x (e)(1) x (a)(6) x (a)(15) x (b)(3) x (e)(2) x (a)(7) (a)(16) x (b)(4) x (e)(3) x (a)(9) (a)(17) x (b)(5) Automated Measure Calculation was not tested Attestation Attestation Forms (as applicable) x Safety-Enhanced Design* x Quality Management System** x Privacy and Security Appendix A B C 3.3 Appendices Attached below. *Required if any of the following were tested: (a)(1), (a)(2), (a)(6), (a)(7), (a)(8), (a)(16), (a)(18), (a)(19), (a)(20), (b)(3), (b)(4), (b)(9). **Required for every EHR product Page 11 of 12

12 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 2-Oct-2015 Test Results Summary Change History Test Report ID Description of Change Date 2014 Edition Test Report Summary Page 12 of 12

13 Dated: Oct 01, 2015 Drummond Group Inc North Hwy 183 Suite B Austin, TX Subject: Attestation of Documentation for g.3.1 I hereby verify that Infiniti Physicians EMR, LLC Version V2.1 conforms to ISO for the following. Sincerely, (a)(1) Computerized provider order entry (a)(2) Drug-drug, drug-allergy interaction checks (a)(6) Medication list (a)(7) Medication allergy list (a)(8) Clinical decision support (b)(3) Electronic prescribing (b)(4) Clinical information reconciliation (a)(18) Computerized provider order entry Medications (a)(19) Computerized provider order entry Laboratory (a)(20) Computerized provider order entry Diagnostic imaging (b)(9) Clinical information reconciliation and incorporation Signature: Signature Block of Authorized Senior Company Representative: Adnan Malik (CTO, Physicians EMR, LLC) amalik@pemr.com/

14 Dated: Oct 01, 2015 Drummond Group Inc North Hwy 183 Suite B Austin, TX Subject: Documentation for g.3.1 Physicians EMR, LLC Version V2.1 conforms with ISO for the following (a)(1) Computerized provider order entry (a)(2) Drug-drug, drug-allergy interaction checks (a)(6) Medication list (a)(7) Medication allergy list (a)(8) Clinical decision support (b)(3) Electronic prescribing (b)(4) Clinical information reconciliation (a)(18) Computerized provider order entry Medications (a)(19) Computerized provider order entry Laboratory (a)(20) Computerized provider order entry Diagnostic imaging (b)(9) Clinical information reconciliation and incorporation Sincerely, Signature: Signature Block of Authorized Senior Company Representative: Adnan Malik (CTO, Physicians EMR, LLC) Document Safety Enhanced Design Page 1

15 USABILITY REPORT DTR g.3-2 Document Safety Enhanced Design Page 2

16 Usability Report Agenda: Contents USABILITY REPORT... 2 DTR g EXECUTIVE SUMMARY... 6 INTRODUCTION... 7 METHOD... 8 DESCRIPTION OF ENVISIONED USERS... 8 PARTICIPANTS... 9 STUDY DESIGN TASKS PROCEDURES TEST LOCATION TEST ENVIRONMENT PARTICIPANT INSTRUCTIONS Usability Metrics DATA SCORING RESULTS DATA ANALYSIS AND REPORTING DATA REPORTING EFFECTIVENESS AND EFFICIENCY SATISFACTION AREAS FOR IMPROVEMENT APPENDICES APPENDIX 1: PARTICIPANT DEMOGRAPHICS APPENDIX 2: NON-DISCLOSURE AGREEMENT APPENDIX 3: INFORMED CONSENT FORM Document Safety Enhanced Design Page 3

17 APPENDIX 4: FINAL QUESTIONS APPENDIX 5: SYSTEM USABILITY SCALE QUESTIONNAIRE Document Safety Enhanced Design Page 4

18 EHR Usability Test Report: Physicians EMR Version V2.1 Report based on ISO/IEC 25062:2006 Common Industry Format for Usability Test Reports. Date of Usability Test: 10/01/2015 Date of Report: 10/01/2015 Report Organized By: Infinity Physicians EMR (ipemr) Adnan Malik, (CTO, Physicians EMR, LLC) 450 W.SR-434, #3010 Longwood FL Adnan Malik (CTO, Physicians EMR, LLC) Document Safety Enhanced Design Page 5

19 EXECUTIVE SUMMARY A usability test of Physicians EMR Version V2.1, Complete EHR was conducted on 10/01/2015. The purpose of this test was to test and validate the usability of the current user interface, and provide evidence of usability in the EHR Under Test (EHRUT). During the usability test, 5 healthcare providers (physicians and nurse practitioners) served as participants and used the EHRUT in simulated, but representative tasks. This study collected performance data on 11 tasks typically conducted on an EHR: (a)(1) Computerized provider order entry (a)(2) Drug-drug, drug-allergy interaction checks (a)(6) Medication list (a)(7) Medication allergy list (a)(8) Clinical decision support (b)(3) Electronic prescribing (b)(4) Clinical information reconciliation (a)(18) Computerized provider order entry Medications (a)(19) Computerized provider order entry Laboratory (a)(20) Computerized provider order entry Diagnostic imaging (b)(9) Clinical information reconciliation and incorporation Through usability test, the administrator introduced the test, and trained participants to complete a series of tasks (given one at a time) using the EHRUT. During the testing, the administrator timed the test and, along with the data logger(s) recorded user performance data on paper. The administrator did not give the participant any support regarding how to accomplish the job. Document Safety Enhanced Design Page 6

20 The participants completed a series of usability tests using local browser sessions via logins to the cloud based EHR. The administrator did not give the participant any support regarding how to accomplish the job. The following types of data were collected for each participant: Path deviations from the optimal workflow Number and types of errors encountered Participant s satisfaction ratings of the system Recommended necessary improvements Time to complete the tasks Participant data was de-identified, so that no correspondence could be made from the identity of the participants to the data collected. Following the conclusion of the testing, the participants were asked to complete a post-test questionnaire. The results from the test scored the overall subjective satisfaction with the EHRUT based on performance with these tasks to between Easy and Very Easy. INTRODUCTION The EHRUT tested for this study was Physician EHR, Version V2.1; Complete EHR which is designed to record and present medical information to healthcare providers in Ambulatory setting. The usability testing attempted to represent realistic exercises and conditions. The purpose of this study was to test and validate the usability of the current user interface, and provide evidence of usability in the EHR Under Test (EHRUT). To this end, measures of effectiveness, efficiency and user satisfaction, such as: Task success Task errors Task time Path deviation Task ratings Document Safety Enhanced Design Page 7

21 METHOD DESCRIPTION OF ENVISIONED USERS The envisioned real world users of software include practice physicians, nurse practitioners, medical assistance s, office managers and front desk staff. For this usability test study we envisioned to use healthcare providers typically physicians and/or nurse practitioners in the age of years from different specialties. There was no gender, race, preferred language or ethnicity restriction. Document Safety Enhanced Design Page 8

22 PARTICIPANTS We approve that the minimum number of participants is 5 for all tests except admin tasks of configuration of clinical decision support intervention and drug-drug severity settings for which minimum number of participants were two. We also approve that the participants contest the description of the intended users. Total of 5 participants were tested on the EHRUT. Participants in the test were physicians and nurse practitioners. Participants were recruited by Physicians EMR, LLC. Furthermore, participants had no direct link to the development of our organization producing the EHRUT. Participants were not from the testing or supplier organization. Participants were given the chance to have the same positioning and level of training as the definite end users would have acknowledged. Here is a table of participants by features, including professional experience, demographics, computing experience and user needs for assistive technology. Participant names were replaced with Participant IDs so that an individual s data cannot be tied back to individual characteristics. Part. Id Gender Age Education Occupation Professional Experience Computer Experience Product Experience Assistive Technology Needs 1 P01 M 38yr MD Doctor 7 yr Yes Yes No 2 P02 M 45yr ACP Nurse 13 yr Yes Yes No 3 P03 F 53yr MD Doctor 21 yr Yes Yes No 4 P04 M 33yr MD Doctor 7yr Yes Yes No 5 P05 F 50yr MD Doctor 15yr Yes Yes No Document Safety Enhanced Design Page 9

23 STUDY DESIGN Overall, the objective of this test was to uncover areas where the application performed well that is, effectively, efficiently, and with satisfaction and areas where the application failed to meet the needs of the participants. The data from this test may serve as a baseline for future tests with an updated version of the same EHR. During the usability test, participants interacted with one EHR. Each participant used the system in the same location, and was provided with the same instructions. The system was evaluated for effectiveness, efficiency and satisfaction as defined by measures collected and analyzed for each participant: Number of tasks successfully completed within the allotted time without assistance Time to complete the tasks Number and types of errors Path deviations Participant s satisfaction ratings of the system TASKS We confirm that the user tasks are prioritized in accordance with the risks associated with user errors. A number of tasks were constructed that would be realistic and representative of the kinds of activities a user might do with this EHR, including: (a)(1) Computerized provider order entry (a)(2) Drug-drug, drug-allergy interaction checks (a)(6) Medication list (a)(7) Medication allergy list (a)(8) Clinical decision support (b)(3) Electronic prescribing (b)(4) Clinical information reconciliation Document Safety Enhanced Design Page 10

24 (a)(18) Computerized provider order entry Medications (a)(19) Computerized provider order entry Laboratory (a)(20) Computerized provider order entry Diagnostic imaging The test scenarios for each participant were centered on common specified tasks. We provided the participants with descriptions of each task. However, we did not provide specific, step-by-step instructions on how to accomplish the designated task. These tasks were ranked based on the risks associated, frequency of use and those that may be troublesome for users. These tasks are listed in ascending risk ranking: Risk Ranking Task Description (a)(2)Drug-drug, drug-allergy interaction checks (a)(8) Clinical Decision Support (b)(4) Clinical information reconciliation (b)(3) Electronic Prescribing (a)(1) Computerized Provider Order Entry (a)(7) Medication Allergy List (a)(6) Medication List (a)(18) Computerized provider order entry Medications (a)(19) Computerized provider order entry Laboratory (a)(20) Computerized provider order entry Diagnostic imaging (b)(9) Clinical information reconciliation and incorporation Document Safety Enhanced Design Page 11

25 PROCEDURES A participant ID was assigned to the participants. For the purpose of test confirmation, two staff members contributed in this test, one is data logger and the other is usability administrator. The administrator toned-down the session including managing tasks and instructions. The administrator also examined task times, took notes on participant comments and acquired post-task rating data. Second person assisted as took notes on task success, path deviations, number, comments and type of errors. Participants were trained to perform the tasks (see specific instructions below): Without using a think aloud practice. Making as few errors and deviations as possible in an effectively less time. Administrators were permitted to give clarifications on tasks and immaterial guidance, without giving proper help and support, but not instructions on use. For each task, the participants were given a written copy of the task. As soon as administrator finished with reading the questions, ask timing began. Once participants shown that they had successfully finished the tasks, the task time was ended. The administrator gave the participant the post-test questionnaire proceeding the session and acknowledged each individual for their participation. Data such as participants' demographic information, time on task, task success rate, errors, deviations, and post-test questionnaire were documented into a spreadsheet. Document Safety Enhanced Design Page 12

26 TEST LOCATION The test was performed in a virtual setting, using online GoToMeeting sessions. The participants joined the sessions from their respective home or work locations via a personal computer and browser. They were given access to the EHRUT through a steadfast and secure cloud based EHR login. TEST ENVIRONMENT The EHRUT would be usually be used in a facility or healthcare office. In this case, the testing was directed in Physicians EMR Healthcare Development Centre. For testing, the system used was a Dell Desktop running Windows 7. The participants used a mouse and a keyboard when interacting with the EHRUT. Physicians EMR used 28 inch screen size display, screen resolution of 1366 X 768 and 64 -bit color settings. The application was set up by highly experienced Physicians EMR Healthcare IT professions. According to the vendor s documentation describing the system set-up and preparation. The application itself was running on a windows 7 desktop using a test database on a LAN connection. Technically, the system performance (i.e., response time) was illustrative to what definite users would experience in a field implementation. Moreover, participants were instructed not to change any of the default system settings. PARTICIPANT INSTRUCTIONS The administrator reads the following instructions aloud to the each participant: Thank you for participating in this study. Your input is very important. Our session today will last about 150 minutes. During that time you will use an instance of an e electronic health record. I will ask you to complete a few tasks using this system and answer some questions. You should complete the tasks as quickly as possible making as few errors as possible. Please try to complete the tasks on your own following the instructions very closely. Please note that we are not testing you, we are testing the system, therefore if you have difficulty all this means is that something needs to be improved in the system. I will be here in case you need specific help, but I am not able to instruct you or provide help in how to use the application. Document Safety Enhanced Design Page 13

27 Overall, we are interested in how easy (or how difficult) this system is to use, what in it would be useful to you, and how we could improve it. I did not have any involvement in its creation, so please be honest with your opinions. All of the information that you provide will be kept confidential and your name will not be associated with your comments at any time. Should you feel it necessary you are able to withdraw at any time during the testing. Following the procedural instructions, participants were shown the EHR and as their first task, were given time 10 minutes to explore the system and make comments. Once this task was complete, the administrator gave the following instructions: For each task, I will read the description to you and say Begin. At that point, please perform the task and say Done once you believe you have successfully completed the task. I would like to request that you don t talk aloud or verbalize while you are doing the task. I will ask you your impressions about the task once you are done. Participants were then given 7 tasks to complete. Usability Metrics According to the NIST Guide to the Processes Approach for Improving the Usability of Electronic health Records, EHRs should support a process that provides a high level of usability for all users. The goal is for users to interact with the system effectively, efficiently, and with an acceptable level of satisfaction. To this end, metrics for effectiveness, efficiency and user satisfaction were captured during the usability testing. The goals of the test were to assess: Satisfaction with EHRUT by measuring ease of use ratings Effectiveness of EHRUT by measuring participant errors and success rates Efficiency of EHRUT by measuring the path deviations and average task time Document Safety Enhanced Design Page 14

28 DATA SCORING The following table details how tasks were scored, errors evaluated, and the time data analyzed. Measure Effectiveness: Task Success Effectiveness: Task Errors Effectiveness: Task Deviations Effectiveness: Task Time Rationale and Scoring A task was counted as a Success if the participant was able to achieve the correct outcome, without assistance, within the time allotted on a per task basis. Total number of tasks and max allowed time are recorded for each individual task. Task times were recorded for successes. If the participant abandoned the task, did not reach the correct answer or performed it incorrectly, or reached the end of the allotted time before successful completion, the task was counted as an Error. The participant s path (i.e., steps) through the application was recorded. Deviations occur if the participant, for example, went to a wrong screen, clicked on an incorrect menu item, followed an incorrect link, or interacted incorrectly with an on-screen control. This path was compared to the optimal path. Each task was timed from when the administrator said Begin until the participant said, Done. If he or she failed to say Done, the time was stopped when the participant stopped performing the task. Only task times for tasks that were successfully completed were included in the average task time analysis. Document Safety Enhanced Design Page 15

29 Satisfaction: Task Rating Participant s subjective impression of the ease of use of the application was measured by administering both a simple post- task question as well as a post-session questionnaire. After each task, the participant was asked to rate Overall, this task was: on a scale of 1 (Very Difficult) to 5 (Very Easy). These data are averaged across participants. Common convention is that average ratings for systems judged easy to use should be 3.3 or above. To measure participants confidence in and likeability of Physicians EMR overall, the testing team administered the System Usability Scale (SUS) post-test questionnaire. Questions included, I think I would like to use this system frequently, I thought the system was easy to use, and I would imagine that most people would learn to use this system very quickly. Table : Details of how observed data were scored. Document Safety Enhanced Design Page 16

30 RESULTS DATA ANALYSIS AND REPORTING We confirm that the test results provided an analysis of the use, tested performance and error rates in order to identify risk prone errors. The results of the usability test were calculated according to the methods specified in the Usability Metrics section above. None of the participants or participant data was excluded from test data analysis and reporting. The usability testing results for the EHRUT are detailed below with individual test results in the beginning and a complete data analysis report in the end: DATA REPORTING The following tasks were performed as a realistic representation of the kinds of activities a user might do with the EHRUT, along with a task rating. Task Ratings 1=Very Easy, 2= Easy, 3= Neither Easy/Nor Difficult, 4= Difficult, 5= Very Difficult Risk Ratings Low, Medium, High Users U1= Physician, U2= MA, U3= Nurse, U4= Office Manager, U5= Front desk Task Risk Path Deviations Errors User(s) Suggestions Record Laboratory Low None None U1,U2,U3,U4,U5 None Order Change Laboratory Low None None U1,U2,U3,U4,U5 None Access Laboratory Low None None U1,U2,U3,U4,U5 None Order Record Medication Order High Users did not initially realize that orders needed to be select as erx or print None U1,U2,U3,U4,U5 Workflow requires a Little trial and error learning curve, but is effective once the Document Safety Enhanced Design Page 17

31 prescription process is learned. Change Medication High None None U1,U2,U3,U4,U5 None Access Medication Low None None U1,U2,U3,U4,U5 None Order Record Imaging Low None None U1,U2,U3,U4,U5 None Order Change Imaging Low None None U1,U2,U3,U4,U5 None Order Access Imaging Low None None U1,U2,U3,U4,U5 None Order Generate and High None None U1,U2,U3,U4,U5 None Electronically Indicate Drug-drug and Drug- allergy Interventions Adjust Severity Medium None None U1,U2,U3,U4,U5 None Level of Interventions Indicated for Drugdrug Interaction Checking Record Medication Medium None None U1,U2,U3,U4,U5 Unify user interface to List match the rest of the EHR. Change Medication Medium None None U1,U2,U3,U4,U5 None List Access Medication Low None None U1,U2,U3,U4,U5 None List Record Medication Low None None U1,U2,U3,U4,U5 None Change Medication Medium None None U1,U2,U3,U4,U5 None Access Medication Low None None U1,U2,U3,U4,U5 None Select/Activate Low None None U1,U2,U3,U4,U5 this part of physician Clinical Decision configuration Support capabilities instead of Interventions for it being a system Problem List administration function. Clinical Medium None None U1,U2,U3,U4,U5 None information reconciliation - CCDA received Clinical information Low None None U1,U2,U3,U4,U5 None reconciliation - Document Safety Enhanced Design Page 18

32 CCDA received & incorporate Task Risk Path Deviations Select/Activate Clinical Decision Support Interventions for Medication List Select/Activate Clinical Decision Support Interventions for Medication Allergy List Select/Activate Clinical Decision Support Interventions for Demographics Select/Activate Clinical Decision Support Interventions for Lab Test and Results Select/Activate Clinical Decision Support Interventions for Vital Signs Select/Activate Clinical Decision Support Errors User(s) Suggestions Low None None U1,U2,U3,U4,U5 Make this part of physician configuration capabilities instead of it being a system administration function. Low None None U1,U2,U3,U4,U5 Make this part of physician configuration capabilities instead of it being a system administration function. Low None None U1,U2,U3,U4,U5 Make this part of physician configuration capabilities instead of it being a system administration function. Low None None U1,U2,U3,U4,U5 Make this part of physician configuration capabilities instead of it being a system administration function. Low None None U1,U2,U3,U4,U5 Make this part of physician configuration capabilities instead of it being a system administration function. Low None None U1,U2,U3,U4,U5 Make this part of physician configuration Document Safety Enhanced Design Page 19

33 Interventions for Vital Signs and Demographics Trigger Clinical Decision Support Interventions for Problem List Trigger Clinical Decision Support Interventions for Medication List Trigger Clinical Decision Support Interventions for Medication Allergy List Trigger Clinical Decision Support Interventions for Demographics Trigger Clinical Decision Support Interventions for Lab Test and Results Trigger Clinical Decision Support Interventions for Vital Signs Trigger Clinical Decision Support Interventions for Vital Signs and capabilities instead of it being a system administration function. High None None U1,U2,U3,U4,U5 None High None None U1,U2,U3,U4,U5 None High None None U1,U2,U3,U4,U5 None Medium None None U1,U2,U3,U4,U5 None Medium None None U1,U2,U3,U4,U5 None High None None U1,U2,U3,U4,U5 None High None None U1,U2,U3,U4,U5 None Document Safety Enhanced Design Page 20

34 Demographics DISCUSSION OF THE FINDINGS EFFECTIVENESS AND EFFICIENCY Participants did not significantly deviate from optimal workflow paths. The user interface for electronic prescribing and recording of allergies took a little more effort to understand and to get a level of end user proficiency, but was found to be effective after completing the learning curve. SATISFACTION Given the subjective comments on effectiveness and efficiency recorded above, the participants recorded an overall score between Easy and Very Easy. AREAS FOR IMPROVEMENT The participants would prefer that the user interface styling of the electronic prescribing and allergy recording would more closely match the rest of the UI style of the EHRUT. Document Safety Enhanced Design Page 21

35 APPENDICES APPENDIX 1: PARTICIPANT DEMOGRAPHICS Following is a high-level overview of de-identified information about the participants in this study. Gender Men [3] Women [2] Total participants [5] Occupation/Role LPN [0] Optometrist [4] Admin Staff [4] Years of Experience All paper [0] Some paper, some electronic [7-21yrs] All electronic [] Document Safety Enhanced Design Page 22

36 APPENDIX 2: NON-DISCLOSURE AGREEMENT THIS AGREEMENT is entered into as of (date), between ( the Participant ) and the testing organization Physician EMR Software located at 450 W.SR-434, #3010 Longwood FL The Participant acknowledges his or her voluntary participation in today s usability study may bring the Participant into possession of Confidential Information. The term "Confidential Information" means all technical and commercial information of a proprietary or confidential nature which is disclosed by Test Company, or otherwise acquired by the Participant, in the course of today s study. By way of illustration, but not limitation, Confidential Information includes trade secrets, processes, formulae, data, know-how, products, designs, drawings, computer aided design files and other computer files, computer software, ideas, improvements, inventions, training methods and materials, marketing techniques, plans, strategies, budgets, financial information, or forecasts. Any information the Participant acquires relating to this product during this study is confidential and proprietary to Test Company and is being disclosed solely for the purposes of the Participant s participation in today s usability study. By signing this form the Participant acknowledges that s/he will receive monetary compensation for feedback and will not disclose this confidential information obtained today to anyone else or any other organizations. Participant s Printed Name: Signature: Date: Document Safety Enhanced Design Page 23

37 APPENDIX 3: INFORMED CONSENT FORM Physician EMR Software would like to thank you for participating in this study. The purpose of this study is to evaluate an electronic health records system. If you decide to participate, you will be asked to perform several tasks using the prototype and give your feedback. The study will last about 120 minutes. Agreement I understand and agree that as a voluntary participant in the present study conducted by Physician EMR Software. I am free to withdraw consent or discontinue participation at any time. I understand and agree to participate in the study conducted by Physician EMR Software. I understand and agree that the purpose of this study is to make software applications more useful and usable in the future. I understand and agree that the data collected from this study may be shared outside of Physician EMR Software. I understand and agree that data confidentiality is assured, because only de-identified data i.e., identification numbers not names will be used in analysis and reporting of the results. I agree to immediately raise any concerns or areas of discomfort with the study administrator. I understand that I can discontinue participation at any time. Please check one of the following: o YES, I have read the above statement and agree to be a participant. o NO, I choose not to participate in this study. Signature: Date: Document Safety Enhanced Design Page 24

38 APPENDIX 4: FINAL QUESTIONS The questionnaire below lists the responses of one of the study participants. Question: What was your overall impression of this system? Answer: Easy to use, follows a practical clinical workflow. Question: What aspects of the system did you like most? Answer: Clinical decision support and SOAP Note Question: What aspects of the system did you like least? Answer: Past Medication entry required a slight learning curve. Question: Were there any features that you were surprised to see? Answer: Ease of use of Clinical Decision Support was nice to see. Question: What features did you expect to encounter but did not see? That is, is there anything that is missing in this application? Answer: Not that I could determine. Question: Compare this system to other systems you have used. Would you recommend this system to your colleagues? Answer: Yes Document Safety Enhanced Design Page 25

39 APPENDIX 5: SYSTEM USABILITY SCALE QUESTIONNAIRE The questionnaire below lists the responses of one of the study participants. # Question Answer 1. I think that I would like to use this system frequently. Strongly agree 2. I found the system unnecessarily complex. Strongly disagree 3. I think that I would need the support of a technical person to be able to use this system. 4. I found the various functions in this system were well integrated. Agree Strongly disagree 5. I thought the system was easy to use Agree 6. I found the various functions in this system were well integrated. Agree 7. I found the system very cumbersome to use. Strongly disagree 8. I felt very confident using the system. Strongly disagree 9. I thought there was too much inconsistency in this system. Disagree 10. I would imagine that most people would learn to use this system very quickly. Agree Document Safety Enhanced Design Page 26

40 Subject: Documentation for g.4: Quality Management System (QMS) This letter is to attest that in order to improve the quality and management of the company and better satisfy the requirements and expectations of its customers Infiniti Physicians EMR (ipemr) uses its own developed QMS standard which is modeled after various ISO standards. We have a dedicated QA department to ensure that we deliver a quality EMR product for each and every release. To accomplish this, all products are carefully designed with collaboration between our own expert clinical team, subject matter experts and our customers. The ultimate goals are to improve patient care, reduce errors and simplify business processes so that physician s time spent with patients is quality time and that the health information can be properly documented, coded and accurately billed. Regards, Signature: Signature Block of Authorized Senior Company Representative: Adnan Malik (CTO, Physicians EMR, LLC) amalik@pemr.com/ Date signed: 10/5/2015 Document Quality Management System (QMS) Page 1

41 Quality Management System Project: Infiniti Physicians EMR (ipemr) Reference: MU-2014 Certification Procedure: (g)(4) Software Development Home-grown QMS is used. The Infiniti Physicians EMR (ipemr) is used to collect feedback from various stakeholders: customers, partners, subject-matter experts and colleagues Product Managers review proposed features, determine priorities, decide upon scope of future releases of the software. Project Managers are responsible for the planning Systems analysts describe requirements from users, subject-matter experts, and partners and transform these into Functional Design documents, describing Use Cases. Development leads review designs, resolve open questions, and discuss any alternatives to proposed solutions with the systems analysts. Software Developers are creating the software following the Coding Guidelines and Coding Principles. Software is being reviewed by peers. Document Quality Management System (QMS) Page 2

42 Criterion # (a)(1) (a)(2) (a)(3) (a)(4) (a)(5) (a)(6) (a)(7) (a)(8) (a)(9) (a)(10) (a)(11) (a)(12) (a)(13) (a)(14) (a)(15) (b)(1) (b)(2) (b)(3) (b)(4) (b)(5)(A) (b)(7) (c)(1,2,3) (d)(1) (d)(2) (d)(3) (d)(4) (d)(5) (d)(6) (d)(7) (d)(8) (e)(1) (e)(2) (e)(3) (f)(1) (f)(2) (f)(3) (g)(1) (g)(2) Name Computerized provider order entry Drug-drug, Drug-allergy Interaction Checks Ambulatory Demographics Vital signs, body mass index, and growth Charts Problem list Medication list Medication allergy list Clinical Decision Support Search in Electronic Notes Drug Formulary checks Smoking Status Image Results Family health history Patient list creation Patient-specific education resources Transitions of care receive, display and incorporate trans.. Trans. of care, create transmit transition of care/referral sum Electronic prescribing Clinical Information reconciliation Incorporate laboratory tests and values/results Data portability Clinical quality measures Authentication, access, control, and authorization Auditable events and tamper-resistance Audit report(s) Amendments Automatic log-off Emergency access End-user device encryption Integrity View, download, and transmit to 3rd party Clinical summary (ambulatory setting only) Secure messaging Immunization information Transmission to Immunization Registry Transmission to PHA Syndromic Surveillance Automated Numerator Recording Automated Measure Calculation Document Quality Management System (QMS) Page 3

43 Software Testing Home-grown QMS is used. Test criteria and Test instructions are described in the Functional Design documents. Test scripts are created prior to testing, including expected Test outcome. Share Point is used by the test team to record and track issues. The development team-lead assign issues to developers to update code or ask Systems analyst to clarify requirements. Software updates (bug fixes) are being retested by the test engineers. Software Implementation Home-grown QMS is used. The following environments are utilized throughout the implementation of the software: o o o o o o o o Developer: each developer has his own environment Development: developer code is combined into this environment Unit Test: used for developer testing Integration: test customer data deployed; integration testing Alpha: test customer data deployed; QA testing QA: used to test 3rd party connectivity; end-user acceptance Pre-Prod: used to test pre-go-live Production: final release of tested software Document Quality Management System (QMS) Page 4

44 Software Maintenance Home-grown QMS is used. The following types of releases are utilized throughout the maintenance of the software: o o Functional: planned and prioritized releases based on market analysis Emergency: resolution of high-priority/blocking issues Updates to the software are performed outside of customer hours of operation, based on time zone and day-of-the-week, when possible. Document Quality Management System (QMS) Page 5

45 For public release: [Physician EMR LLC.] Attests to the validity of the information below to satisfy the documentation requirements for testing and certification of the ONC 2014 Edition criteria: (d)2. [Include the following information based on the EHR System under Test s functionality] 1. Does the EHR SUT allow the following? Disabling the audit log User can disable the Audit log by going into user settings; by default Audit log will be enabled. Figure 1.0 Monitoring and recording of audit log status changes (if disabling is possible) Yes audit log status can be change from above figure. Monitoring and recording of status changes to encryption, if encryption is used to satisfy the end user device encryption (d)7 criteria [IN (d)(2)-2.02 / IN (d)(2)-2.09] Not Applicable

46 2. If the audit log can be disabled, is the default state for audit log and audit log status recording enabled by default? Yes. Briefly describe how the EHR SUT can demonstrate the default settings. [IN (d)(2) ] Audit log will be enabled by default for new users, however super-user can change the status depending on required user settings. Figure If applicable, and if the EHR also allows it to be disabled, is the encryption of electronic health information on end-user devices enabled by default? Yes. Briefly describe how the EHR SUT enables the default settings. Not Applicable OR No. Confirm that the EHR SUT does not store electronic health information on end-user devices. [IN (d)(2)-1.03] As our app is a web-based EHR we do not store data locally. Also there is no data available in our temporary files or in browser cache file location. As we are using nocache pragma in each web page html header to stop any data from being cached. 4. Does the EHR SUT permit any users to delete electronic health information? [IN (d)(2)-3.03] From Infiniti Physician EMR we can delete the clinical information, however all deleted data will be recorded in audit report as mentioned in figure 3.0

47 Figure Does the EHR SUT audit logging capability monitor each of the required actions for all instances of electronic health information utilized by the EHR SUT in accordance with the specified standard ASTM E ? [IN (d)(2)-3.04] Yes. Confirm via statement. OR No. Explain and justify for each. Addition: On adding every new record in our EHR, addition function will be performed; log will be maintained/displayed in Audit log.

48 Deletion: We can delete the clinical information, however all deleted data will be recorded in audit report. Changes: Any changes in Infiniti Physicians EHR will be recorded as update/modification.