ONC HIT Certification Program

Transcription

1 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 01-Nov-2014 ONC HIT Certification Program Test Results Summary for 2014 Edition EHR Certification Part 1: Product and Developer Information 1.1 Certified Product Information Product Name: InPracSys EHR Product Version: 8.0 Domain: Ambulatory Test Type: Complete EHR 1.2 Developer/Vendor Information Developer/Vendor Name: InPracSys Address: 2225 Lyndale Ave S. Minneapolis MN Website: Phone: Developer/Vendor Contact: Ashu Kataria Page 1 of 11

2 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 01-Nov-2014 Part 2: ONC-Authorized Certification Body Information 2.1 ONC-Authorized Certification Body Information ONC-ACB Name: Drummond Group Address: North Hwy 183, Ste B , Austin, TX Website: Phone: ONC-ACB Contact: Bill Smith This test results summary is approved for public release by the following ONC-Authorized Certification Body Representative: Bill Smith ONC-ACB Authorized Representative Certification Committee Chair Function/Title Signature and Date 2/2/ Gap Certification The following identifies criterion or criteria certified via gap certification x (a)(1) (a)(17) x (d)(5) (d)(9) x (a)(6) (b)(5)* x (d)(6) x (f)(1) x (a)(7) x (d)(1) x (d)(8) *Gap certification allowed for Inpatient setting only No gap certification Page 2 of 11

3 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 01-Nov Inherited Certification The following identifies criterion or criteria certified via inherited certification (a)(1) (a)(14) (c)(3) (f)(1) (a)(2) (a)(15) (d)(1) (f)(2) (a)(3) (a)(16) Inpt. only (d)(2) (f)(3) (a)(4) (a)(17) Inpt. only (d)(3) (f)(4) Inpt. only (a)(5) (b)(1) (d)(4) (f)(5) Optional & (a)(6) (b)(2) (d)(5) Amb. only (a)(7) (b)(3) (d)(6) (f)(6) Optional & (a)(8) (b)(4) (d)(7) Amb. only (a)(9) (b)(5) (d)(8) (g)(1) (a)(10) (b)(6) Inpt. only (d)(9) Optional (g)(2) (a)(11) (b)(7) (e)(1) (g)(3) (a)(12) (c)(1) (e)(2) Amb. only (g)(4) (a)(13) (c)(2) (e)(3) Amb. only x No inherited certification Page 3 of 11

4 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 01-Nov-2014 Part 3: NVLAP-Accredited Testing Laboratory Information Report Number: TEB Test Date(s): 6/4/2014, 6/5/2014, 8/5/2014, 9/2/ NVLAP-Accredited Testing Laboratory Information ATL Name: Drummond Group EHR Test Lab Accreditation Number: NVLAP Lab Code Address: North Hwy 183, Ste B , Austin, TX Website: Phone: ATL Contact: Beth Morrow For more information on scope of accreditation, please reference NVLAP Lab Code Part 3 of this test results summary is approved for public release by the following Accredited Testing Laboratory Representative: Timothy Bennett Test Proctor ATL Authorized Representative Function/Title Signature and Date 2/2/2015 Nashville, TN Location Where Test Conducted 3.2 Test Information Additional Software Relied Upon for Certification Additional Software Applicable Criteria Functionality provided by Additional Software EMR Direct b.1, b.2, e.1 Direct messaging services No additional software required Page 4 of 11

5 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 01-Nov Test Tools Test Tool Version x Cypress x eprescribing Validation Tool HL7 CDA Cancer Registry Reporting Validation Tool HL7 v2 Electronic Laboratory Reporting (ELR) Validation Tool 1.8 x HL7 v2 Immunization Information System (IIS) Reporting Validation Tool 1.8 x HL7 v2 Laboratory Results Interface (LRI) Validation Tool 1.7 x HL7 v2 Syndromic Surveillance Reporting Validation Tool 1.7 x Transport Testing Tool 179 x Direct Certificate Discovery Tool No test tools required Test Data Alteration (customization) to the test data was necessary and is described in Appendix [insert appendix letter] No alteration (customization) to the test data was necessary Standards Multiple Standards Permitted The following identifies the standard(s) that has been successfully tested where more than one standard is permitted Criterion # Standard Successfully Tested (a)(8)(ii)(a)(2) (a)(13) x (b)(1) HL7 Version 3 Implementation Guide: URL-Based Implementations of the Context-Aware Information Retrieval (Infobutton) Domain x (a)(3) IHTSDO SNOMED CT International Release July 2012 and US Extension to SNOMED CT March 2012 Release (b)(2) HL7 Version 3 Implementation Guide: Context-Aware Knowledge Retrieval (Infobutton) Service-Oriented Architecture Implementation Guide (j) HL7 Version 3 Standard: Clinical Genomics; Pedigree Page 5 of 11

6 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 01-Nov-2014 Criterion # (a)(15)(i) (a)(16)(ii) (b)(2)(i)(a) (b)(7)(i) (e)(1)(i) (e)(1)(ii)(a)(2) (e)(3)(ii) x (b)(1) HL7 Version 3 Implementation Guide: URL-Based Implementations of the Context-Aware Information Retrieval (Infobutton) Domain (g) Network Time Protocol Version 3 (RFC 1305) (i) The code set specified at 45 CFR (c)(2) (ICD-10- CM) for the indicated conditions (i) The code set specified at 45 CFR (c)(2) (ICD-10- CM) for the indicated conditions Standard Successfully Tested Annex A of the FIPS Publication [list encryption and hashing algorithms] AES-128 SHA (g) Network Time Protocol Version 3 (RFC 1305) Annex A of the FIPS Publication [list encryption and hashing algorithms] AES-128 SHA (b)(2) HL7 Version 3 Implementation Guide: Context-Aware Knowledge Retrieval (Infobutton) Service-Oriented Architecture Implementation Guide (g) Network Time Protocol Version 4 (RFC 5905) x (a)(3) IHTSDO SNOMED CT International Release July 2012 and US Extension to SNOMED CT March 2012 Release x (a)(3) IHTSDO SNOMED CT International Release July 2012 and US Extension to SNOMED CT March 2012 Release x (g) Network Time Protocol Version 4 (RFC 5905) Common MU Data Set (15) x (a)(3) IHTSDO SNOMED CT International Release July 2012 and US Extension to SNOMED CT March 2012 Release (b)(2) The code set specified at 45 CFR (a)(5) (HCPCS and CPT-4) None of the criteria and corresponding standards listed above are applicable Newer Versions of Standards Page 6 of 11

7 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 01-Nov-2014 The following identifies the newer version of a minimum standard(s) that has been successfully tested Newer Version Applicable Criteria No newer version of a minimum standard was tested Optional Functionality Criterion # (a)(4)(iii) (b)(1)(i)(b) (b)(1)(i)(c) (b)(2)(ii)(b) (b)(2)(ii)(c) (f)(3) Common MU Data Set (15) Common MU Data Set (15) Optional Functionality Successfully Tested Plot and display growth charts Receive summary care record using the standards specified at (a) and (b) (Direct and XDM Validation) Receive summary care record using the standards specified at (b) and (c) (SOAP Protocols) Transmit health information to a Third Party using the standards specified at (a) and (b) (Direct and XDM Validation) Transmit health information to a Third Party using the standards specified at (b) and (c) (SOAP Protocols) Ambulatory setting only Create syndrome-based public health surveillance information for transmission using the standard specified at (d)(3) (urgent care visit scenario) Express Procedures according to the standard specified at (b)(3) (45 CFR (a)(4): Code on Dental Procedures and Nomenclature) Express Procedures according to the standard specified at (b)(4) (45 CFR (c)(3): ICD-10-PCS) x No optional functionality tested Page 7 of 11

8 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 01-Nov Edition Certification Criteria* Successfully Tested Criteria # Version Version Criteria # TP** TD*** TP TD (a)(1) x (c)(3) x (a)(2) 1.2 (d)(1) 1.2 x (a)(3) x (d)(2) 1.5 x (a)(4) x (d)(3) 1.3 x (a)(5) x (d)(4) 1.3 (a)(6) (d)(5) 1.2 (a)(7) (d)(6) 1.2 x (a)(8) 1.2 x (d)(7) 1.2 x (a)(9) (d)(8) 1.2 x (a)(10) (d)(9) Optional 1.2 x (a)(11) 1.3 x (e)(1) x (a)(12) 1.3 x (e)(2) Amb. only x (a)(13) 1.2 x (e)(3) Amb. only 1.3 x (a)(14) 1.2 (f)(1) x (a)(15) 1.5 x (f)(2) (a)(16) Inpt. only x (f)(3) (a)(17) Inpt. only 1.2 (f)(4) Inpt. only x (b)(1) (f)(5) Optional & x (b)(2) Amb. only x (b)(3) (f)(6) Optional & x (b)(4) Amb. only x (b)(5) (g)(1) (b)(6) Inpt. only x (g)(2) x (b)(7) x (g)(3) 1.3 x (c)(1) x (g)(4) 1.2 x (c)(2) No criteria tested *For a list of the 2014 Edition Certification Criteria, please reference (navigation: 2014 Edition Test Method) **Indicates the version number for the Test Procedure (TP) ***Indicates the version number for the Test Data (TD) Page 8 of 11

9 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 01-Nov Clinical Quality Measures* Type of Clinical Quality Measures Successfully Tested: x Ambulatory Inpatient No CQMs tested *For a list of the 2014 Clinical Quality Measures, please reference (navigation: 2014 Clinical Quality Measures) Ambulatory CQMs CMS ID Version CMS ID Version CMS ID Version CMS ID Version x 155 v x 138 v2 x 157 v2 52 x 123 v x 127 v x 129 v3 145 x 165 v x 166 v3 x 69 v2 131 x 147 v x 134 v2 x 153 v Inpatient CQMs CMS ID Version CMS ID Version CMS ID Version CMS ID Version Page 9 of 11

10 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 01-Nov Automated Numerator Recording and Measure Calculation Automated Numerator Recording Automated Numerator Recording Successfully Tested (a)(1) (a)(9) (a)(16) (b)(6) (a)(3) (a)(11) (a)(17) (e)(1) (a)(4) (a)(12) (b)(2) (e)(2) (a)(5) (a)(13) (b)(3) (e)(3) (a)(6) (a)(14) (b)(4) (a)(7) (a)(15) (b)(5) x Automated Numerator Recording was not tested Automated Measure Calculation Automated Measure Calculation Successfully Tested x (a)(1) x (a)(9) (a)(16) (b)(6) x (a)(3) x (a)(11) (a)(17) x (e)(1) x (a)(4) x (a)(12) x (b)(2) x (e)(2) x (a)(5) x (a)(13) x (b)(3) x (e)(3) x (a)(6) x (a)(14) x (b)(4) x (a)(7) x (a)(15) x (b)(5) Automated Measure Calculation was not tested Attestation Attestation Forms (as applicable) x Safety-Enhanced Design* x Quality Management System** x Privacy and Security Appendix A B C 3.3 Appendices Attached below. *Required if any of the following were tested: (a)(1), (a)(2), (a)(6), (a)(7), (a)(8), (a)(16), (b)(3), (b)(4) **Required for every EHR product Page 10 of 11

11 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 01-Nov-2014 Test Results Summary Change History Test Report ID Description of Change Date 2014 Edition Test Report Summary Page 11 of 11

12 USER CENTER DESIGN REPORT TEST REPORT UPDATE This test report was updated in December 2015 to satisfy User Center Design Report specifications by ONC. The new Test Report ID is amended as follows: Part 3: NVLAP-Accredited Testing Laboratory Information: Report Number plus the suffix _Dec2015.

13 Ashu Kataria 2225 Lyndale Ave S. Minneapolis, MN Ph: Fax: November 19, 2015 InPracSys EHR, v8.0 For public release: Innovative Practice Systems, DBA InPracSys, used the following user centered design (UCD) process based on NISTIR 7741 NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records in developing and designing their HIT system, InPracSys. Sincerely, Ashu Kataria President

14 Usability Test Report of InPracSys EHR Version 8.0 Applications Tested: InPracSys EHR V8.0 Dates Usability Test: 05/12/2014 Report: 05/20/2014 Prepared by Ashu Kataria, President, InPracSys Contact Ashu Kataria InPracSys 2225 Lyndale Ave S Minneapolis, MN (612)

15 Contents CONTENTS 1 EXECUTIVE SUMMARY 2 INTRODUCTION 5 METHOD 5 RESULTS 9 APPENDICES 10

16 Usability Test Report of InPracSys EHR Report based on ISO/IEC 25062:2006 Common Industry Format for Usability Test Reports InPracSys EHR, v8.0 Date of usability testing: 05/12/2014 Date of report: 05/20/2014 Report prepared by: The InPracSys Implementation Team EXECUTIVE SUMMARY A usability test of the InPracSys EHR was conducted the week of 05/12/2014 via several Go-to- Meeting virtual testing sessions by the InPracSys Implementation Team. The purpose was to test and validate the updated user interface, and show the usability of the EHR Under Test (EHRUT). A test environment mirroring the production environment was set up with test data adequate to perform the tests as well as replicate the production environment. PHI was also addressed. No live PHI was used. During the usability test, 5 healthcare providers and/or other users matching the target demographic criteria (physicians, office workers, medical assistants, third party consultants, IT admin and implementation specialists) served as participants and used the EHRUT in simulated, but representative tasks. This study collected performance data on 7 tasks typically conducted on an EHR: (a) (1) Computerized provider order entry (a) (2) Drug-drug, drug-allergy interaction checks (a) (6) Medication list (a) (7) Medication allergy list (a) (8) Clinical decision support (b) (3) Electronic prescribing (b) (4) Clinical information reconciliation During the 30 minute one-on-one usability test, each participant was greeted by the test administrator and was asked to accept the InPracSys User Agreement. Upon login they were informed that they could withdraw at any time. Participants had varying prior experience with the EHR. The administrator introduced the test, and asked participants to complete a series of tasks (given one at a time) using the EHRUT. The administrator timed the test and, along with the data logger(s) recorded user performance data on paper and electronically. The administrator provided the standard training used to teach users to complete the task. InPracSys used Go-To-Meeting for the test. Each user was informed of usage and test requirements. All users were familiar with the EHRUT so no formal training was needed. The following types of data were collected for each participant: Number of tasks successfully completed within the allotted time Time to complete the tasks Errors Participants satisfaction ratings of the system

17 All participant data was de-identified no correspondence could be made from the identity of the participants to the data collected. When the testing ended, participants were thanked but not compensated for their participation, as mandated by state rules. Various recommended metrics, as described in the NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records, were used to evaluate the usability of the EHRUT. Following is a summary of the performance and rating data collected on the EHRUT g.3 Safety Enhanced Design g.4 Quality Management System The SUS scored a respectably high score of 4.82 based on performance of tasks assigned to the tester. A risk factor scale of 1-5 was used, where 5 represented the highest risk to patient safety In addition to the performance data, the following qualitative observations were made: Major Findings Provider Findings Providers were assigned tasks relevant to their roles: CPOE, clinical decisions, e-prescription drug interaction and contraindication checks (a) (1) Computerized provider order entry On a scale of 1 to 5 (5 being best) users found the system to be at 4.5 in terms of usability and user friendliness. Automation is built in to automatically incorporate data from labs and external sources, thereby enhancing patient safety and ease of data entry (a) (2) Drug-drug, drug-allergy interaction checks On a scale of 1 to 5 (5 being best) users found the system to be a 4.2 in terms of alerting the provider to existence of drug interactions, drug contraindications etc., using third party sources (a) (8) Clinical decision support On a scale of 1 to 5 (5 being best) users found the system to be a 4.9 in terms of alerting the physician with flags relevant to patient care (iodine allergy and the risk of CT with contrast). Such decision support features greatly enhance patient safety and product quality (b) (3) Electronic prescribing The system uses a third party e-prescription provider. Considerable automation has been put in place to achieve a quick 3-click prescription. Nurses and other Users Other users were assigned supporting tasks like reconciliation medications, allergy and other information (a) (6) Medication list All users found this feature extremely easy to use, as it helps maintain accurate data and greatly enhances patient safety (a) (7) Medication allergy list Users easily created allergies and medicines in patient records. The ease of use of the system was greatly appreciated.

18 (b) (4) Clinical information reconciliation Users were able to easily incorporate allergies, medications and other information from outside clinics with confidence and ease. Overall, providers found the system quite easy to use. Reconciling clinical data took longer than other tasks due to the complex nature of the process. There was a no significant correlation between assigned risk factors and error rates. For the highest risk tasks the error rates were low or null. The most significant correlation of risk to error was in creating imaging orders and in e-prescribing medications. Those errors did not result in any inaccurate data recording because the interface automatically warned of the error prior to the user saving their work. In the case of image order creation, the users forgot to designate the facility where the test was being performed. On the e- prescribing interface the errors were related to choosing the prescribing provider. This error is unlikely when the user is logged in as a provider and not a test participant. AREAS FOR IMPROVEMENT During the test the following areas of improvement were noted and plans have been made to achieve better performance: Reduction of clicks to achieve desired results More improvements in the UI to make it more user friendly

19 INTRODUCTION User Centered Design User-centered design (UCD) is a process in which the needs and limitations of the application s end users are given extensive attention at each stage of the design process. In addition UCD is a multi-stage problem solving process that not only requires application designers to analyze and foresee how users are likely to use a product, but also to "test" the validity of their assumptions with regard to user behavior in real world tests with actual users of the product or application under test. Such testing is necessary as it is often challenging for the designers of a product to understand intuitively what a user of their design experiences, and what each user's learning curve may look like. The main difference from other product design philosophies is that UCD optimizes the product around how users can, want, or need to use the product, rather than forcing the users to change their behavior to accommodate the product making the product easier to use while at the same time avoiding risk of errors and missed data elements.. InPracSys EHR and User Centered Design InPracSys EHR is designed with UCD at its core and is extensively reviewed and tested. The tests are also repeated when necessary. The purpose of this test was to evaluate the performance and usability of the InPracSys application by real users who are currently using an older, previously Drummond-certified version of the InPracSys software. The test represented real world scenarios commonly faced by our users, which mimicked the business flow of their clinic. The test covered all aspects of usability, patient safety, ease of data entry, accuracy and robustness. INTENDED USERS Physicians users that interact with patient data and provide and make health care decisions Nurses - users that interact with patient data and assist providers make health care decisions Medical Assistant - users that interact with limited patient clinical data and assist providers make health care decisions Clinic Office Staff (front and back office) - users that interact with scheduling and billing patient data and assist providers make health care decisions Lab technicians - users that interact with patient data and assist providers make health care decisions IT Adminstrators - - users that interact minimal patient data but provide authentications and user access controls to allow other staff to perform their functions Users were assigned tests based on their roles. These tests were designed to closely represent their daily work. METHOD PARTICIPANTS A total of 5 participants were tested on the EHRUT. T h e y were recruited by InPracSys and were not compensated in any way. T h e participants had no direct connection to the organization producing the EHRUT, and were not affiliated with the testing or supplier organization. The participants came from a mix of backgrounds and had various demographic characteristics.the following is a table of participants by characteristics, including demographics, professional experience,

20 computing experience and user needs for assistive technology. Participant names were replaced with Participant IDs so that individuals data cannot be tied back to their identities. Part ID Gender Education Role 1 A Male MD School Physician 2 B Male College Implementation Specialist 3 C Female College Nurse 4 D Female College Medical Assistant 5 E Male College IT/Admin 5 participants were recruited and all w e r e p r e s e n t for the study. Participants were scheduled for 30 minute sessions. STUDY DESIGN Overall, the objective of this test was to uncover areas where the application performed well that is, effectively, efficiently, and with satisfaction and areas where the application failed to meet the needs of the participants. The data from this test may serve as a baseline for future tests with an updated version of the same EHR and/or as a comparison with other EHRs (provided the same tasks are used). In short, this testing serves as both a means to evaluate current usability, but also to identify areas where improvements must be made. During the usability test, participants interacted solely with the InPracSys EHR. Each participant used the system in their practice location, and was provided with the same instructions. The system was evaluated for effectiveness, efficiency and satisfaction as defined by measures collected and analyzed for each participant: Tasks successfully completed within the allotted time: Task completion time Error count and type of error Tester comments T e s t e r satisfaction rating TASKS A number of tasks were constructed that would be realistic and representative of the kinds of tasks a user might perform with this EHR, including: CPOE entering, editing and accessing meds Medication orders - entering, editing and sending meds orders Lab orders - entering, editing and sending or scheduling orders to labs for tests Radiology orders- entering, editing and sending or scheduling imaging orders for investigations Drug allergy - entering, editing and displaying Drug allergies with user error trapping Drug contraindication entering, editing and displaying Drug- Drug interactions with user error trapping Access orders access and display with alerts for user error trapping Radiology results - receiving and action for follow-up with alerts Labs results - receiving and action for follow-up with alerts User privileges setting and adjusting Severity level of notification - setting and adjusting CDS - to set, change and view Medication list and reconciliation - view, reconciliation and alerts

21 Allergy list - view, reconciliation and alerts CDS o Problems o Medications o Allergies o Other interventions Diagnostic and therapeutic reference information setting-up, adjusting and display Electronic prescribing prescription process, alerts, sending of the Rx. o New erx o Refills o Error handling PROCEDURES Participants were greeted as they arrived to the session, a n d their identity was verified and matched with the name on the assigned participant ID. Each participant agreed to the InPracSys User Agreement, w h i c h was electronically recorded. The administrator moderated the session and provided instructions and tasks. The administrator also monitored task times, obtained post-task rating data, and recorded participant comments. Participants were instructed to perform the tasks listed above as quickly as possible, with as few errors and deviations as possible. For each task, the participants were given a written copy of the task. Task timing began once the administrator finished reading the question. The task time was stopped once the participant indicated they had successfully completed the task. Users work flow for his or her tasks is organized in the InPracSys EHR by user roles and permissions, and the application minimizes the risk associated with errors by utilizing alerts in that user s workflow, e.g. the a scheduling conflict is alerted in scheduler user role has a lower priority than an allergy alert in a provider s role. This work-flow and ranking of alerts to minimize risk was tested. Following the session, the administrator gave participants a post-test questionnaire and thanked each individual for their participation. Participants' demographic information, task success rate, time on task, and post-test questionnaire were recorded into a spreadsheet. TEST ENVIRONMENT The EHRUT would typically be used in a healthcare office or facility. In this instance, the testing was conducted at the practice facility. For testing, the computer was supplied by the user, running Windows. The participants used a mouse and keyboard when interacting with the EHRUT. Technically, the system performance was representative of what actual users would experience in a field implementation. PARTICIPANT INSTRUCTIONS The administrator read the following instructions aloud to the each participant: Thank you for participating in this study. Your input is very important. Our session today will last about no more than 30 minutes. During that time you will use an instance of the InPracSys electronic health record. This study is mandated by the ONC as part of Meaningful Use Stage 2 application testing. We may be testing users that have varying experience with the InPracSys system. I will ask you to complete a few tasks using this system and answer some questions. You should complete the tasks as quickly as possible making as few errors as possible. Please try to complete

22 the tasks following the instructions very closely. Please note that we are not testing you, we are testing the system, therefore if you have difficulty all this means is that something needs to be improved in the system. Overall, we are interested in how easy (or how difficult) this system is to use, what in it would be useful to you, and how we could improve it. I did not have any involvement in its creation, so please be honest with your opinions. All of the information that you provide will be kept confidential and your name will not be associated with your comments at any time. Should you feel it necessary you are able to withdraw at any time during the testing. Following the procedural instructions, participants were shown the EHR and as their first task, were given time to review the system and to make comments. Once this task was complete, the administrator gave the following instructions: For each task, I will read the description to you and say Begin. At that point, please perform the task and say Done once you believe you have successfully completed the task. I would like to request that you not talk aloud or verbalize while you are doing the tasks. I will ask you your impressions about the task once you are done. Participants were then given 6 tasks to complete, with subtasks to cover all as noted in the section Tasks.. USABILITY METRICS According to the NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records, EHRs should support a process that provides a high level of usability for all users. The goal is for users to interact with the system effectively, efficiently, and with an acceptable level of satisfaction. To this end, metrics for effectiveness, efficiency and user satisfaction were captured during the usability testing. The goals of the test were to assess: 1. Effectiveness of the InPracSys EHR by measuring participant success rates and errors 2. Efficiency of the InPracSys EHR by measuring the average task time 3. Satisfaction with the InPracSys EHR by measuring ease of use ratings 4. Compare assigned risk level to error rates 5. Range alerts according to roles and risk associated with missed data or user errors

23 RESULTS Safety Or Error Risk Average Task Time (sec) Average User Ranking # of Errors CPOE Medication order Record Edit Lab order Record Edit Radiology orders Record Edit Drug Allergy Drug-Drug and drug allergy interaction checks Adjust severity level alerts for Drug-Drug and drug allergy interaction Drug Contraindication Access Orders Radiology Medication Labs User Privileges Severity Level of Notification CDS - to set, change and view Medication List and Reconciliation Allergy List and Reconciliation Record Change - Access Record Change - Access CDS Problems Medications Allergies Demographics Labs and Results Other interventions (Vitals) Diagnostic and Therapeutic Reference Information Configuration Electronic Prescribing New erx Refills Error handling Scale Average 4.82

24 Analysis: Overall, providers found the system quite easy to use. Reconciling clinical data took longer than other tasks due to the complex nature of the process. Data entry was easy and InPracSys EHR script design allows for quicker flow thru the application thereby minimizing data entry errors. The results section performed especially well given the complexity of the process. This section was also very intuitive. Low error rates were attributed to through testing of the application s flow, usability oriented design and alerts that did not allow the user to make mistakes. Conclusion: We concluded that the application design from usability standpoint scored very high at 4.82 where 5 is the highest and best score. We also concluded that the access to data and action task times were well within our expectation and compared better when matched with published peer data, as well as with internal InPracSys EHR benchmarks.. APPENDICES The following appendices include supplemental data for this usability test report. Following is a list of the appendices provided: None.

25 Arjun Kataria 2225 Lyndale Ave S. Minneapolis, MN January 19, 2015 Melissa Martin EHR Registration Coordinator Drummond Group 4700 Bryant Irvin Court Fort Worth, TX Dear Ms. Martin: The following is our attestation of our Quality Management System: We have a home grown QMS system based on leading industry standards, but better suited to company and client needs. It begins with a ticket system that it generated by either the client, or our testers. This ticket is then moved through the approval process, as necessitated by the ticket. If needed, it ultimately moves to the development team. The development team performs the work and testing on development servers and moves their work to a staging server. The work is tested on the staging server, either by our team or by the client based on the nature of the ticket and the requirements. Where needed, outside consultants and usability experts are consulted and action taken, appropriate to the need. As needed, documents and applications are secure to ensure HIPAA compliance with PHI and other data. Backups of current applications and data are always done before changes and or upgrades are done to production applications to ensure that roll backs are possible when if an error should arise. Sincerely, Arjun Kataria Project Coordinator

26 Ashu Kataria President InPracSys - EHR 2225 Lyndale Ave S Minneapolis MN help@inpracsys.com For public release: InPracSys - EHR attests to the validity of the information below to satisfy the documentation requirements for testing and certification of the ONC 2014 Edition criteria: (d)2. 1. Does the EHR SUT allow the following? Disabling the audit log No the audit log cannot be disabled. monitoring and recording of audit log status changes (if disabling is possible) NA monitoring and recording of status changes to encryption, if encryption is used to satisfy the end user device encryption (d)7 criteria - NA - The InPracSys EHR does not store electronic health information on end-user devices. [IN (d)(2)-2.02 / IN (d)(2)-2.09] 2. If the audit log can be disabled, is the default state for audit log and audit log status recording enabled by default? The audit log cannot be disabled. [IN (d)(2) ] 3. If applicable, and if the EHR also allows it to be disabled, is the encryption of electronic health information on end-user devices enabled by default? The InPracSys EHR does not store electronic health information on end-user devices. [IN (d)(2)-1.03] 4. Does the EHR SUT permit any users to delete electronic health information? The InPracSys EHR does not permit deletes of electronic health information. [IN (d)(2)-3.03]

27 Drummond Group January 30, 2015 Page 2 5. Does the EHR SUT audit logging capability monitor each of the required actions for all instances of electronic health information utilized by the EHR SUT in accordance with the specified standard ASTM E ? [IN (d)(2)-3.04] Yes. Each of the following actions are logged: Addition, Deletion, Changes, Queries, Print, Copy 6. Describe the method(s) through which the audit logs are protected from being changed, overwritten, or deleted by the EHR technology itself. [IN (d)(2)-4.01] Audit log files are named based on a Microsoft naming convention with date/time stamps and are controlled by the operating system, so that there is no way a file can be over written SQL provides additional audit trail data. That data is stored within SQL, with appropriate time stamps and database primary keys. As a result, this data cannot be over written. SQL audit logs are also backed up as a part of our normal, regular, scheduled daily back up routine. 7. Describe the method(s) through which the EHR SUT is capable of detecting whether the audit logs have been altered. NOTE This type of alteration would be from outside the EHR (e.g. hacking, manual tampering, other software besides the EHR). [IN (d)(2)-5.01] InPracSys ensures that the audit cannot be altered by server OS lock down procedures that ensure that no one, inside or outside, the enterprise has access to the audit logs. Additionally, audit trail data is also added to SQL database with appropriate user information and time stamps. Further security comes from scheduled and regular backup of SQL databases including offsite backups that require admin approvals for access I hereby attest that all above statements are true, as an authorized signing authority on behalf of my organization. Ashu Kataria President InPracSys EHR 1/30/2015