ONC HIT Certification Program

Transcription

1 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 01-Jun-2015 ONC HIT Certification Program Test Results Summary for 2014 Edition EHR Certification Part 1: Product and Developer Information 1.1 Certified Product Information Product Name: ModuleMD WISE Product Version: 8.1 Domain: Ambulatory Test Type: Complete EHR 1.2 Developer/Vendor Information Developer/Vendor Name: ModuleMD Address: 8359 Office Park Drive Grand Blanc MI Website: Phone: x1004 Developer/Vendor Contact: Lloyd Williams Page 1 of 12

2 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 01-Jun-2015 Part 2: ONC-Authorized Certification Body Information 2.1 ONC-Authorized Certification Body Information ONC-ACB Name: Drummond Group Address: North Hwy 183, Ste B , Austin, TX Website: Phone: ONC-ACB Contact: Bill Smith This test results summary is approved for public release by the following ONC-Authorized Certification Body Representative: Bill Smith ONC-ACB Authorized Representative Certification Body Manager Function/Title Signature and Date 7/6/ Gap Certification The following identifies criterion or criteria certified via gap certification (a)(1) (a)(19) (d)(6) (h)(1) (a)(6) (a)(20) (d)(8) (h)(2) (a)(7) (b)(5)* (d)(9) (h)(3) (a)(17) (d)(1) (f)(1) (a)(18) (d)(5) (f)(7)** *Gap certification allowed for Inpatient setting only **Gap certification allowed for Ambulatory setting only x No gap certification Page 2 of 12

3 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 01-Jun Inherited Certification The following identifies criterion or criteria certified via inherited certification x (a)(1) (a)(16) Inpt. only x (c)(2) x (f)(2) x (a)(2) (a)(17) Inpt. only x (c)(3) x (f)(3) x (a)(3) (a)(18) x (d)(1) (f)(4) Inpt. only x (a)(4) (a)(19) x (d)(2) x (a)(5) (a)(20) x (d)(3) x (a)(6) x (b)(1) x (d)(4) x (a)(7) x (b)(2) x (d)(5) x (a)(8) x (b)(3) x (d)(6) (f)(7) x (a)(9) x (b)(4) x (d)(7) (g)(1) x (a)(10) x (b)(5) x (d)(8) x (g)(2) x (a)(11) (b)(6) Inpt. only (d)(9) Optional x (g)(3) x (a)(12) x (b)(7) x (e)(1) x (g)(4) x (a)(13) (b)(8) x (e)(2) Amb. only (h)(1) x (a)(14) (b)(9) x (e)(3) Amb. only (h)(2) x (a)(15) x (c)(1) x (f)(1) (h)(3) (f)(5) Amb. only (f)(6) Amb. only No inherited certification Page 3 of 12

4 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 01-Jun-2015 Part 3: NVLAP-Accredited Testing Laboratory Information Report Number: SG Test Date(s): N/A 3.1 NVLAP-Accredited Testing Laboratory Information ATL Name: Drummond Group EHR Test Lab Accreditation Number: NVLAP Lab Code Address: North Hwy 183, Ste B , Austin, TX Website: Phone: ATL Contact: Beth Morrow For more information on scope of accreditation, please reference NVLAP Lab Code Part 3 of this test results summary is approved for public release by the following Accredited Testing Laboratory Representative: Sonia Galvan Test Proctor ATL Authorized Representative Function/Title Signature and Date 7/6/2015 Houston, TX Location Where Test Conducted 3.2 Test Information Additional Software Relied Upon for Certification Additional Software Applicable Criteria Allscripts eprescribe a.2, 8, 10; b.3, 4; g.2 Functionality provided by Additional Software e-prescribing vendor EMR Direct phimail b.1, 2; e.1 HISP Adobe Reader a.12 Viewing narrative results No additional software required Page 4 of 12

5 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 01-Jun Test Tools Test Tool Version x Cypress 2.6 x eprescribing Validation Tool HL7 CDA Cancer Registry Reporting Validation Tool HL7 v2 Electronic Laboratory Reporting (ELR) Validation Tool x HL7 v2 Immunization Information System (IIS) Reporting Validation Tool x HL7 v2 Laboratory Results Interface (LRI) Validation Tool x HL7 v2 Syndromic Surveillance Reporting Validation Tool x Transport Testing Tool 181 x Direct Certificate Discovery Tool Edge Testing Tool No test tools required Test Data Alteration (customization) to the test data was necessary and is described in Appendix [insert appendix letter] x No alteration (customization) to the test data was necessary Standards Multiple Standards Permitted The following identifies the standard(s) that has been successfully tested where more than one standard is permitted Criterion # Standard Successfully Tested (a)(8)(ii)(a)(2) (a)(13) x (b)(1) HL7 Version 3 Implementation Guide: URL-Based Implementations of the Context-Aware Information Retrieval (Infobutton) Domain x (a)(3) IHTSDO SNOMED CT International Release July 2012 and US Extension to SNOMED CT March 2012 Release (b)(2) HL7 Version 3 Implementation Guide: Context-Aware Knowledge Retrieval (Infobutton) Service-Oriented Architecture Implementation Guide (j) HL7 Version 3 Standard: Clinical Genomics; Pedigree Page 5 of 12

6 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 01-Jun-2015 Criterion # (a)(15)(i) x (b)(1) HL7 Version 3 Implementation Guide: URL-Based Implementations of the Context-Aware Information Retrieval (Infobutton) Domain Standard Successfully Tested (b)(2) HL7 Version 3 Implementation Guide: Context-Aware Knowledge Retrieval (Infobutton) Service-Oriented Architecture Implementation Guide (a)(16)(ii) (b)(2)(i)(a) (b)(7)(i) (g) Network Time Protocol Version 3 (RFC 1305) (i) The code set specified at 45 CFR (c)(2) (ICD-10- CM) for the indicated conditions (i) The code set specified at 45 CFR (c)(2) (ICD-10- CM) for the indicated conditions (g) Network Time Protocol Version 4 (RFC 5905) x (a)(3) IHTSDO SNOMED CT International Release July 2012 and US Extension to SNOMED CT March 2012 Release x (a)(3) IHTSDO SNOMED CT International Release July 2012 and US Extension to SNOMED CT March 2012 Release (b)(8)(i) (e)(1)(i) (e)(1)(ii)(a)(2) (e)(3)(ii) (i) The code set specified at 45 CFR (c)(2) (ICD-10- CM) for the indicated conditions Annex A of the FIPS Publication [list encryption and hashing algorithms] 128 bit AES 1024 bit RSA x (g) Network Time Protocol Version 3 (RFC 1305) Annex A of the FIPS Publication [list encryption and hashing algorithms] 128 bit AES 1024 bit RSA (a)(3) IHTSDO SNOMED CT International Release July 2012 and US Extension to SNOMED CT March 2012 Release (g) Network Time Protocol Version 4 (RFC 5905) Common MU Data Set (15) x (a)(3) IHTSDO SNOMED CT International Release July 2012 and US Extension to SNOMED CT March 2012 Release (b)(2) The code set specified at 45 CFR (a)(5) (HCPCS and CPT-4) Page 6 of 12

7 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 01-Jun-2015 Criterion # Standard Successfully Tested None of the criteria and corresponding standards listed above are applicable Newer Versions of Standards The following identifies the newer version of a minimum standard(s) that has been successfully tested Newer Version Applicable Criteria No newer version of a minimum standard was tested Optional Functionality Criterion # x (a)(4)(iii) (b)(1)(i)(b) (b)(1)(i)(c) (b)(2)(ii)(b) (b)(2)(ii)(c) (e)(1) (f)(3) (f)(7) Common MU Data Set (15) Common MU Data Set (15) Optional Functionality Successfully Tested Plot and display growth charts Receive summary care record using the standards specified at (a) and (b) (Direct and XDM Validation) Receive summary care record using the standards specified at (b) and (c) (SOAP Protocols) Transmit health information to a Third Party using the standards specified at (a) and (b) (Direct and XDM Validation) Transmit health information to a Third Party using the standards specified at (b) and (c) (SOAP Protocols) View, download and transmit data to a third party utilizing the Edge Protocol IG version 1.1 Ambulatory setting only Create syndrome-based public health surveillance information for transmission using the standard specified at (d)(3) (urgent care visit scenario) Ambulatory setting only transmission to public health agencies syndromic surveillance - Create Data Elements Express Procedures according to the standard specified at (b)(3) (45 CFR (a)(4): Code on Dental Procedures and Nomenclature) Express Procedures according to the standard specified at (b)(4) (45 CFR (c)(3): ICD-10-PCS) No optional functionality tested Page 7 of 12

8 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 01-Jun Edition Certification Criteria* Successfully Tested Criteria # Version Version Criteria # TP** TD*** TP TD (a)(1) (c)(3) (a)(2) 1.2 (d)(1) 1.2 (a)(3) (d)(2) 1.6 (a)(4) (d)(3) 1.3 (a)(5) (d)(4) 1.3 (a)(6) (d)(5) 1.2 (a)(7) (d)(6) 1.2 (a)(8) 1.3 (d)(7) 1.2 (a)(9) (d)(8) 1.2 (a)(10) (d)(9) Optional 1.2 (a)(11) 1.3 (e)(1) (a)(12) 1.3 (e)(2) Amb. only (a)(13) 1.2 (e)(3) Amb. only 1.3 (a)(14) 1.2 (f)(1) (a)(15) 1.5 (f)(2) (a)(16) Inpt. only (f)(3) (a)(17) Inpt. only 1.2 (f)(4) Inpt. only (a)(18) (a)(19) (a)(20) (b)(1) (f)(5) Amb. only (f)(6) Amb. only (b)(2) (f)(7) Amb. only 1.1 (b)(3) (g)(1) (b)(4) (g)(2) (b)(5) (g)(3) 1.4 (b)(6) Inpt. only (g)(4) 1.2 (b)(7) (h)(1) 1.1 (b)(8) (h)(2) 1.1 (b)(9) (h)(3) 1.1 (c)(1) (c)(2) Page 8 of 12

9 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 01-Jun-2015 Criteria # x No criteria tested Version Version Criteria # TP** TD*** TP TD *For a list of the 2014 Edition Certification Criteria, please reference (navigation: 2014 Edition Test Method) **Indicates the version number for the Test Procedure (TP) ***Indicates the version number for the Test Data (TD) Page 9 of 12

10 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 01-Jun Clinical Quality Measures* Type of Clinical Quality Measures Successfully Tested: x Ambulatory Inpatient No CQMs tested *For a list of the 2014 Clinical Quality Measures, please the CMS ecqm Library (Navigation: June 2014 and April 2014 Updates) Ambulatory CQMs CMS ID Version CMS ID Version CMS ID Version CMS ID Version x 2 v x 155 v2 22 x 117 v2 137 x 156 v x 138 v x 123 v x 124 v x 125 v x 126 v x 127 v x 165 v x 146 v2 x 166 v3 x 69 v2 x 131 v x 153 v x 154 v2 182 Inpatient CQMs CMS ID Version CMS ID Version CMS ID Version CMS ID Version Page 10 of 12

11 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 01-Jun Automated Numerator Recording and Measure Calculation Automated Numerator Recording Automated Numerator Recording Successfully Tested (a)(1) (a)(11) (a)(18) (b)(6) (a)(3) (a)(12) (a)(19) (b)(8) (a)(4) (a)(13) (a)(20) (b)(9) (a)(5) (a)(14) (b)(2) (e)(1) (a)(6) (a)(15) (b)(3) (e)(2) (a)(7) (a)(16) (b)(4) (e)(3) (a)(9) (a)(17) (b)(5) x Automated Numerator Recording was not tested Automated Measure Calculation Automated Measure Calculation Successfully Tested x (a)(1) x (a)(11) (a)(18) (b)(6) x (a)(3) x (a)(12) (a)(19) (b)(8) x (a)(4) x (a)(13) (a)(20) (b)(9) x (a)(5) x (a)(14) x (b)(2) x (e)(1) x (a)(6) x (a)(15) x (b)(3) x (e)(2) x (a)(7) (a)(16) x (b)(4) x (e)(3) x (a)(9) (a)(17) x (b)(5) Automated Measure Calculation was not tested Attestation Attestation Forms (as applicable) x Safety-Enhanced Design* x Quality Management System** x Privacy and Security Appendix A B C 3.3 Appendices Attached below. *Required if any of the following were tested: (a)(1), (a)(2), (a)(6), (a)(7), (a)(8), (a)(16), (a)(18), (a)(19), (a)(20), (b)(3), (b)(4), (b)(9). **Required for every EHR product Page 11 of 12

12 Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 01-Jun-2015 Test Results Summary Change History Test Report ID Description of Change Date 2014 Edition Test Report Summary Page 12 of 12

13 Electronic Health Record Usability Test Report ModuleMD WISE v8.0 Complete EHR

14 EHR Usability Test Report of ModuleMD WISE v8.0 Report based on ISO/IEC 25062:2006 Common Industry Format for Usability Test Reports Module MD WISE v8.0 Date of Usability Test: Monday, May 20th, 2013 through Monday, July 29th, 2013 Date of Report: 08/16/2013 Date of Update: 06/01/2015 Report Prepared By: ModuleMD, LLC Jeff McKenzie, Client Services Manager Amardip Chandpuri, Quality Analyst Addendum Prepared By: Ellyn Serra, Client Services Manager ModuleMD, LLC 8359 Office Park Drive Grand Blanc, MI

15 TABLE OF CONTENTS 1 EXECUTIVE SUMMARY INTRODUCTION METHOD PARTICIPANTS STUDY DESIGN TASKS PROCEDURE TEST LOCATION TEST ENVIRONMENT USABILITY METRICS RESULTS DATA SCORING DATA ANALYSIS AND REPORTING APPENDICES APPENDIX 1: NON-DISCLOSURE AGREEMENT APPENDIX 2: TASKS TO BE PERFORMED APPENDIX 3: FINAL QUESTIONS APPENDIX 4: SYSTEM USABILITY SCALE QUESTIONNAIRE ADDENDUM

16 1. EXECUTIVE SUMMARY This report documents the findings of a usability test of ModuleMD WISE v8.0 Complete EHR by ModuleMD. A usability test of ModuleMD WISE v8.0 Complete EHR was conducted every other Monday after each release beginning from Monday, May 20th, 2013 through Monday, July 29th, 2013 at the headquarters of ModuleMD, LLC and various remote locations by ModuleMD employees and clients. The purpose of this testing to test and validate the usability of the new user interface, and provide evidence of usability in the EHR Under Test (EHRUT). During the usability test, one healthcare provider and four ModuleMD employees matching the target demographic criteria served as participants and used the EHRUT in simulated, but representative tasks. This study collected performance data on 37 tasks typically conducted on an EHR: Computerized Provider Order Entry (CPOE): 1. Record medication order 2. Change medication order 3. Access medication order 4. Record lab order 5. Change lab order 6. Access lab order 7. Record Rad/Image order 8. Change Rad/Image order 9. Access Rad/Image order Drug-drug, Drug-allergy Interaction Checks: 10. Trigger one drug-drug contraindication intervention 11. Trigger one drug-allergy contraindication intervention 12. Adjust severity level of interventions indicated for drug-drug interaction Medication List: 13. Record active medication list 14. Change active medication list 15. Access active medication list 4

17 Medication Allergy List: 16. Record active medication allergy list 17. Change active medication allergy list 18. Access active medication allergy list Clinical Decision Support: 19. Create clinical decision support intervention for a problem list 20. Create clinical decision support intervention for a medication list 21. Create clinical decision support intervention for a medication allergy list 22. Create clinical decision support intervention for demographics 23. Create clinical decision support intervention for laboratory tests and values/results 24. Create clinical decision support intervention for vital signs 25. Trigger clinical decision support intervention for a problem list 26. Trigger clinical decision support intervention for a medication list 27. Trigger clinical decision support intervention for a medication allergy list 28. Trigger clinical decision support intervention for demographics 29. Trigger clinical decision support intervention for laboratory tests and values/results 30. Trigger clinical decision support intervention for vital signs Electronic Prescribing: 31. Electronically prescribe single medication for test patient 32. Complete active medication for test patient 33. Discontinue active medication for test patient 34. Renew active medication for test patient Clinical Information Reconciliation: 35. Create a single reconciled active medication list 36. Create a single reconciled active problem list 37. Create a single reconciled active medication allergy list Throughout the development period, there were 12 milestone release dates. Prior to each release date, usability tests were conducted by each participant. Each participant was asked to track the following types of data: Time to complete the tasks The number of errors Path deviations Evaluate the tasks for ease of success 5

18 Evaluate the tasks for level of difficulty All participant data was de-identified. Various recommended metrics, in accordance with the examples set forth in the NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records, were used to evaluate the usability of the EHRUT. Following is a summary of the performance and rating data collected on the EHRUT. Measure N Task Success Path Deviation Task Time Errors Task Ratings Task # Mean (SD) Deviations (Observed / Optimal) Mean (SD) Mean (SD) Mean (SD) 1 Record medication order Change medication order Access medication order Record lab order Change change lab order Access lab order Record Rad/Image order Change Rad/Image order Access Rad/Image order Trigger one drug-drug contraindication intervention Trigger one drug-allergy contraindication intervention Adjust severity level of interventions indicated for drugdrug interaction Record active medication list Change active medication list Access active medication list Record active medication allergy list Change active medication allergy list Access active medication allergy list Create clinical decision support intervention for a problem list

19 Create clinical decision support intervention for a medication list Create clinical decision support intervention for a medication allergy list Create clinical decision support intervention for demographics Create clinical decision support intervention for laboratory tests and values/results Create clinical decision support intervention for vital signs Trigger clinical decision support intervention for a problem list Trigger clinical decision support intervention for a medication list Trigger clinical decision support intervention for a medication allergy list Trigger clinical decision support intervention for demographics Trigger clinical decision support intervention for laboratory tests and values/results Trigger clinical decision support intervention for vital signs Electronically prescribe single medication for test patient Complete active medication for test patient Discontinue active medication for test patient Renew active medication for test patient Create a single reconciled active medication list Create a single reconciled active problem list

20 37 Create a single reconciled active medication allergy list The results from the System Usability Scale scored the subjective satisfaction with the system based on performance with these tasks to be Satisfactory with satisfaction score of In addition to the performance data, the following qualitative observations were made: Major findings in Release 1: o Not able to create clinical decision support intervention for a medication allergy list as the system has no option for Medication Allergy. o Not able to create clinical decision support intervention for a vital signs as the system has no option for vital signs. Major findings in Release 2: o Patient Search working very slow. o Under scan documents the uploaded file not visible in the upload queue. o Clinical decision support intervention headings are still not collapsing. Major findings in Release 3: o Allscripts connectivity issue from EMR. o BP Systolic and BP Diastolic radio buttons not working. o Under Clinical Decision Support for Vital Signs, when BMI Within Range radio button is selected only one text box is opening instead of two text boxes. Major findings in Release 4: o Not able to trigger clinical decision support for Demographics when Race and Ethnicity is a criteria for a test patient. o User not able to change lab order for a patient. o User not able to upload image under scan documents. 8

21 Areas for improvement in Release 1: o Add an option to create and trigger clinical decision support intervention for a medication allergy list. o Add an option to create and trigger clinical decision support intervention for vital signs. Areas for improvement in Release 2: o Need to increase the speed for patient search o All the heading under clinical decision support interventions should get collapsed. Areas for improvement in Release 3: o The user should be able to select radio buttons for every value associated with the radio button. o User should be able to connect to Allscripts on clicking the eprescribe button in Current Medications clinical category. o Two text boxes should open when the Within Range radio button is selected for BMI under Vital Signs for clinical decision support. Areas for improvement in Release 4: o The user should be able to trigger the clinical decision support with each and at least one combination. o User should be able to change lab order for a Patient. o User should be able to upload an image in Scan Documents and also the uploaded image should be available under upload queue before the Save button is selected. 9

22 2. INTRODUCTION The EHRUT tested for this study was ModuleMD WISE, version 8.0. Designed to present medical information to healthcare providers in ambulatory care, the EHRUT consists of integrated health records, management and revenue management into one comprehensive, interoperable solution. ModuleMD WISE assists physicians in improving the quality and coordination of care, while obtaining peak efficiencies in practice management. The usability testing attempted to represent realistic exercises and conditions. The purpose of this study was to test and validate the usability of the current user interface, and provide evidence of usability in the EHR Under Test (EHRUT). To this end, measures of effectiveness, efficiency and user satisfaction, such as task time, and ease of completion were captured during the usability testing. 10

23 3. METHOD 3.1 PARTICIPANTS A total of five participants were tested on the EHRUT. Participants in the test were doctor, Quality analysts, and client service support staff. Participants had a mix of background and demographic characteristics. The following is a table of participants by characteristics, including demographics, professional experience and computing experience. Participant names were replaced with Participant IDs so that an individual's data cannot be tied back to individual identities. Participant Demographics Part ID Gender Age Education Occupation/role Professional Experience Computer Experience Product Experience 101 M 40 BS Manager 16 yrs 16 yrs < 4 months 102 M 40 BE Quality Analyst 15 yrs 15 yrs >2 yrs 103 F 29 BE Quality Analyst 5+ yrs 5+ yrs >2 yrs 104 M 67 MD Doctor 35 yrs 15 yrs 13 yrs 105 M 32 MA Client Services 5+ yrs 10 yrs >2 yrs 3.2 STUDY DESIGN Overall, the objective of this test was to uncover areas where the application performed well - that is, effectively, efficiently, and with satisfaction - and areas where the application failed to meet the needs of the participants. The data from this test may serve as a baseline for future tests with an updated version of the same EHR and/or comparison with other EHRs provided the same tasks are used. In short, this testing serves as both a means to record or benchmark current usability, but also to identify areas where improvements must be made. The system was evaluated for effectiveness, efficiency and satisfaction as defined by measures collected and analyzed for each participant: Time to complete the tasks The number of errors Path deviations Evaluate the tasks for ease of success Evaluate the tasks for level of difficulty Additional information about the various measures can be found in Section on Usability Metrics. 11

24 3.3 TASKS A number of tasks were constructed that would be realistic and representative of the kinds of activities a user might do with this EHR, including: Computerized Provider Order Entry (CPOE): 1. Record medication order 2. Change medication order 3. Access medication order 4. Record lab order 5. Change lab order 6. Access lab order 7. Record Rad/Image order 8. Change Rad/Image order 9. Access Rad/Image order Drug-drug, Drug-allergy Interaction Checks: 10. Trigger one drug-drug contraindication intervention 11. Trigger one drug-allergy contraindication intervention 12. Adjust severity level of interventions indicated for drug-drug interaction Medication List: 13. Record active medication list 14. Change active medication list 15. Access active medication list Medication allergy List: 16. Record active medication allergy list 17. Change active medication allergy list 18. Access active medication allergy list Clinical Decision Support: 19. Create clinical decision support intervention for a problem list 20. Create clinical decision support intervention for a medication list 21. Create clinical decision support intervention for a medication allergy list 22. Create clinical decision support intervention for demographics 23. Create clinical decision support intervention for laboratory tests and values/results 12

25 24. Create clinical decision support intervention for vital signs 25. Trigger clinical decision support intervention for a problem list 26. Trigger clinical decision support intervention for a medication list 27. Trigger clinical decision support intervention for a medication allergy list 28. Trigger clinical decision support intervention for demographics 29. Trigger clinical decision support intervention for laboratory tests and values/results 30. Trigger clinical decision support intervention for vital signs Electronic Prescribing: 31. Electronically prescribe single medication for test patient 32. Complete active medication for test patient 33. Discontinue active medication for test patient 34. Renew active medication for test patient Clinical information reconciliation: 35. Create a single reconciled active medication list 36. Create a single reconciled active problem list 37. Create a single reconciled active medication allergy list Tasks were selected based on their frequency of use, criticality of function, and those that may be most troublesome for users. Tasks should always be constructed in light of the study objectives. 3.4 PROCEDURE Each participant was assigned a Participant ID and they reviewed and signed the Non-Disclosure Agreement (See Appendix 1). Every participant ran the tests and logged there results in the usability testing log. All the errors encountered were logged by the participants in Redmine and were assigned to the development team for correction. Each participant demographics are mentioned earlier under heading participants. Each participant was given the usability testing log and were instructed to perform the tasks as quickly as possible making as few errors and deviations as possible. 3.5 TEST LOCATION Testing was done at headquarters of ModuleMD, LLC and various remote locations by ModuleMD employees and clients. 13

26 3.6 TEST ENVIRONMENT The following specifications computers were used by the participants: Computer Name Computer Model Processor Installed RAM Operating System Browser Computer 1 Dell Intel Core i5 6 GB Windows 8 Chrome Computer 2 Lenovo Intel Core i5 6 GB Windows 7 Professional IE9/Chrome Computer 3 HP Intel Core GB Windows XP Professional IE9/Chrome Computer 4 Lenovo Intel Core i5 4 GB Windows 7 Home Premium Chrome Computer 5 Compaq Intel T GB Windows 7 Professional IE/9Chrome 3.7 USABILITY METRICS The goals of the test were to assess: 1. Measuring participant success rates and errors 2. Measuring the average task time and path deviations 3. Satisfaction by measuring ease of use ratings 4. RESULTS 4.1 DATA SCORING Task Success: A task was counted as a Success if the participant was able to achieve the correct outcome, without assistance, within the time allotted on a per task basis. Task Failure: If the participant did not reach the correct answer or performed it incorrectly, the task was counted as a Failure. Task Deviations: Participant Deviated from the actual steps Task Time: Time spent by each participant to complete the task Task Rating: How easy or difficult the task was for each participant RESULTS 4.2 DATA ANALYSIS AND REPORTING The results of the usability test were calculated according to the methods specified in the Usability Metrics section above. 14

27 5. APPENDICES 5.1 Appendix 1: Non-Disclosure Agreement THIS AGREEMENT is entered into as of 2013, between ( the Participant ) and the testing organization ModuleMD LLC located at 8359 Office Park Drive, Grand Blanc, MI, The Participant acknowledges his or her participation in this usability study may bring the Participant into possession of Confidential Information. The term "Confidential Information" means all technical and commercial information of a proprietary or confidential nature which is disclosed by ModuleMD LLC, or otherwise acquired by the Participant, in the course of the study. By way of illustration, but not limitation, Confidential Information includes trade secrets, processes, formulae, data, know-how, products, designs, drawings, computer aided design files and other computer files, computer software, ideas, improvements, inventions, training methods and materials, marketing techniques, plans, strategies, budgets, financial information, or forecasts. Any information the Participant acquires relating to this product during this study is confidential and proprietary to ModuleMD LLC and is being disclosed solely for the purposes of the Participant s participation in the usability study. By signing this form the Participant acknowledges that she/he will not disclose this confidential information obtained to anyone else or any other organizations. Participant s Printed Name: Signature: Date: 15

28 5.2 Appendix 2: TASKS TO BE PERFORMED Task 1: Record Medication Order Note: Before testing the participant should review the test patient s Medication Order. Optimal Path: Select Test Patient >> Open Visit Slip >> Click Current Medications >> Click eprescribe >> Search Medication >> Select Medication >> Complete Sig >> Add to Script Queue Very 16

29 Task 2: Change Medication Order Note: Before testing the participant should review the test patient s medications. Optimal Path: Select Test Patient >> Open Visit Slip >> Click Current Medications >> Click eprescribe >> Click Script Queue >> Click Edit link >> Modify Sig >> Add to Script Queue Very 17

30 Task 3: Access Medication Order Note: Before testing the participant should review the test patient s medications. Optimal Path: Select Test Patient >> Open Visit Slip >> Click Current Medications >> Click eprescribe >> Click Review Script Queue >> Process Order >> Exit eprescribe >> Click Prescriptions Very 18

31 Task 4: Record Lab Order Note: Before testing the participant should review the test patient s lab orders. Optimal Path: Select Test Patient >> Open Visit Slip >> Click Labs >> Click New/Existing Orders >> Click Order New >> Complete Lab Order Form >> Submit Very 19

32 Task 5: Change Lab Order Note: Before testing the participant should review the test patient s lab test values/results. Optimal Path: Select Test Patient >> Open Visit Slip >> Click Labs >> Click New/Existing Orders >> Select Order ID >> Click Reorder >> Complete Lab Order Form >> Submit Very 20

33 Task 6: Access Lab Order Note: Before testing the participant should review the test patient s lab test values/results. Optimal Path: Select Test Patient >> Open Visit Slip >> Click Labs >> Click New/Existing Orders >> Select Order ID >> Click View Order Very 21

34 Task 7: Record Rad/Image Order Note: Before testing the participant should review the test patient s image order. Optimal Path: Select Test Patient >> Open Visit Slip >> Click Labs >> Click New/Existing Orders >> Click Order New >> Complete Radiology Order Form >> Submit Very 22

35 Task 8: Change Rad/Image Order Note: Before testing the participant should review the test patient s image order. Optimal Path: Select Test Patient >> Open Visit Slip >> Click Labs >> Click New/Existing Orders >> Select Order ID >> Click Reorder >> Complete Radiology Order Form >> Submit Very 23

36 Task 9: Access Rad/Image Order Note: Before testing the participant should review the test patient s image order. Optimal Path: Select Test Patient >> Open Visit Slip >> Click Labs >> Click New/Existing Orders >> Select Order ID >> Click View Order Very 24

37 Task 10: Trigger One Drug-Drug Contraindication Intervention Note: Before testing the participant should review the test patient s drug-drug contraindication intervention. Optimal Path: Select Test Patient >> Open Visit Slip >> Click Current Medications >> Click eprescribe >> Search Medication >> Select Medication >> Complete Sig >> Add to Script Queue Very 25

38 Task 11: Trigger One Drug-Allergy Contraindication Intervention Note: Before testing the participant should review the test patient s drug-allergy contraindication intervention. Optimal Path: Select Test Patient >> Open Visit Slip >> Click Current Medications >> Click eprescribe >> Search Medication >> Select Medication >> Complete Sig >> Add to Script Queue Very 26

39 Task 12: Adjust Severity Level of Interventions Indicated for Drug-Drug interaction Note: Before testing the participant should review the test patient s severity level for drug-drug interaction Optimal Path: Login as Administrator >> Select Test Patient >> Select Practice Setup >> Select eprescribe >> Click Settings Tab >> Click DUR Settings link >> Adjust Settings >> Save Very 27

40 Task 13: Record Active Medication List Note: Before testing the participant should review the test patient s active medication list. Optimal Path: Select Test Patient >> Open Visit Slip >> Click Current Medications Very 28

41 Task 14: Change Active Medication List Note: Before testing the participant should review the test patient s active medication list. Optimal Path: Select Test Patient >> Open Visit Slip >> Click Current Medications >> Click eprescribe >> Click Review History >> Select Medication to change >> Done Very 29

42 Task 15: Access Active Medication List Note: Before testing the participant should review the test patient s active medication list. Optimal Path: Select Test Patient >> Open Visit Slip >> Click Current Medications >> Active Medication List Displayed Very 30

43 Task 16: Record Active Medication Allergy List Note: Before testing the participant should review the test patient s active medication allergy list. Optimal Path: Select Test Patient >> Open Visit Slip >> Click Adverse Drug Reactions >> Click Update >> Click pencil icon next to Active Allergies >> Click Add >> Select Medication >> Save Very 31

44 Task 17: Change Active Medication Allergy List Note: Before testing the participant should review the test patient s active medication allergy list. Optimal Path: Select Test Patient >> Open Visit Slip >> Click Adverse Drug Reactions >> Click Update >> Click pencil icon next to Active Allergies >> Click Edit >> Select Inactive >> Save Very 32

45 Task 18: Access Active Medication Allergy List Note: Before testing the participant should review the test patient s active medication allergy list. Optimal Path: Select Test Patient >> Open Visit Slip >> Click Adverse Drug Reactions Very 33

46 Task 19: Create clinical decision support intervention for a problem list Note: Before testing the participant should review the test patient s problem list. Optimal Path: Login >> Administration >> Practice Setup >> General >> Config. Clinical Decision Support >> New >> Diagnosis >> Select ICD >> Save Very 34

47 Task 20: Create clinical decision support intervention for a medication list Note: Before testing the participant should review the test patient s medication list. Optimal Path: Login >> Administration >> Practice Setup >> General >> Config. Clinical Decision Support >> New >> Medication >> Select Medication >> Save Very 35

48 Task 21: Create clinical decision support intervention for a medication allergy list Note: Before testing the participant should review the test patient s medication allergy list. Optimal Path: Login >> Administration >> Practice Setup >> General >> Config. Clinical Decision Support >> New >> Medication Allergy >> Select Medication Allergy >> Save Very 36

49 Task 22: Create clinical decision support intervention for a demographics Note: Before testing the participant should review the test patient s demographics. Optimal Path: Login >> Administration >> Practice Setup >> General >> Config. Clinical Decision Support >> New >> Demographics >> Select Demographics >> Save Very 37

50 Task 23: Create clinical decision support intervention for a laboratory tests and values/results Note: Before testing the participant should review the test patient s laboratory tests and values/results. Optimal Path: Login >> Administration >> Practice Setup >> General >> Config. Clinical Decision Support >> New >> Lab Results >> Select Lab Test >> Save Very 38

51 Task 24: Create clinical decision support intervention for vital signs. Note: Before testing the participant should review the test patient s vital signs. Optimal Path: Login >> Administration >> Practice Setup >> General >> Config. Clinical Decision Support >> New >> Vital Signs >> Select Vital Signs >> Save Very 39

52 Task 25: Trigger clinical decision support intervention for a problem list Note: Before testing the participant should review the test patient s problem list. Optimal Path: Select Test Patient >> Open Visit Slip >> Enter Problem List >> Click Clinical Support to trigger Very 40

53 Task 26: Trigger clinical decision support intervention for a medication list Note: Before testing the participant should review the test patient s medication list. Optimal Path: Select Test Patient >> Open Visit Slip >> Current Medications >> Click Clinical Support to trigger Very 41

54 Task 27: Trigger clinical decision support intervention for a medication allergy list Note: Before testing the participant should review the test patient s medication allergy list. Optimal Path: Select Test Patient >> Open Visit Slip >> Medication Allergy List >> Click Clinical Support to trigger Very 42

55 Task 28: Trigger clinical decision support intervention for demographics Note: Before testing the participant should review the test patient s demographics. Optimal Path: Select Test Patient >> Open Visit Slip >> Demographics >> Click Clinical Support to trigger Very 43

56 Task 29: Trigger clinical decision support intervention for laboratory tests and values/results Note: Before testing the participant should review the test patient s laboratory tests and values/results. Optimal Path: Select Test Patient >> Open Visit Slip >> Select Lab Tests >> Click Clinical Support to trigger Very 44

57 Task 30: Trigger clinical decision support intervention for vital signs Note: Before testing the participant should review the test patient s vital signs. Optimal Path: Select Test Patient >> Open Visit Slip >> Vital Signs >> Click Clinical Support to trigger Very 45

58 Task 31: Electronically prescribe single medication for test patient Note: Before testing the participant should review the test patient s medication. Optimal Path: Select Test Patient >> Open Visit Slip >> Click Current Medications >> Click eprescribe >> Review History >> Click Add Patient Reported Med >> Search & Select Medication >> Save >>Exit Very 46

59 Task 32: Complete active medication for test patient Note: Before testing the participant should review the test patient s active medications. Optimal Path: Select Test Patient >> Open Visit Slip >> Click Current Medications >> Click eprescribe >> Review History >> Click Medication to Complete >> Click Complete >> Done Very 47

60 Task 33: Discontinue active medication for a test patient Note: Before testing the participant should review the test patient s active medications to be discontinued. Optimal Path: Select Test Patient >> Open Visit Slip >> Click Current Medications >> Click eprescribe >> Review History >> Click medication to discontinue >> Click Discontinue >> Done Very 48

61 Task 34: Renew active medication for a test patient Note: Before testing the participant should review the test patient s active medications to be renewed. Optimal Path: Select Test Patient >> Open Visit Slip >> Click Current Medications >> Click eprescribe >> Review History >> Add Patient Reported Medication >> Search & Select Medication >> Click Save >> Exit Very 49

62 Task 35: Create a single reconciled active medication list Note: Before testing the participant should review the test patient s active medication list. Optimal Path: Select Test Patient >> Open Visit Slip >> Click Current Medications >> Click eprescribe >> Review History >> Add Patient Reported Medications >> Search & Select Medication >> Save >> Exit >> Click Download erx under Current medications to View list Very 50

63 Task 36: Create a single reconciled active problem list Note: Before testing the participant should review the test patient s active problem list. Optimal Path: Select Test Patient >> Open Visit Slip >> Click Assessment >> Add All Active Problems >> Add New >> Select a problem >> Click Add New >> Exit Very 51

64 Task 37: Create a single reconciled active medication allergy list Note: Before testing the participant should review the test patient s active medication allergy list. Optimal Path: Select Test Patient >> Open Visit Slip >> Click Adverse Drug Reactions >> Click Update >> Click pencil icon next to active allergies >> Select allergy >> Save >> Done >> In EMR click Adverse Drug Reactions to view list. Very 52

65 Appendix 3: FINAL QUESTIONS: Q1. What was your overall impression of this system? Q2. What aspects of the system did you like most? Q3. What aspects of the system did you like least? Q4. Were there any features that you were surprised to see? Q5. What features did you expect to encounter but did not see? That is, is there anything that is missing in this application? Q6. Compare this system to other systems you have used. Q7. Would you recommend this system to your colleagues? 53

66 5.4 Appendix 4: SYSTEM USABILITY SCALE QUESTIONNAIRE: Strongly Disagree Strongly Agree 54

67 1. I think that I would like to use this system frequently 2. I found the system unnecessarily complex I thought the system was easy to use I think that I would need the support of a technical person to be able to use this system 5. I found the various functions in this system were well integrated 6. I thought there was too much inconsistency in this system 7. I would imagine that most people would learn to use this system very quickly 8. I found the system very cumbersome to use 9. I felt very confident using the system 10. I needed to learn a lot of things before I could get going with this system Total Score = Average Score = 32.9 (329/10) 55

68 Average SUS Score = (32.9 *2.5) 56

69 Electronic Health Record Usability Test Report ModuleMD WISE ADDENDUM 6/1/2015 This ADDENDUM is attached to and forms part of the Usability Test Report ModuleMD WISE v8.0 Complete EHR. ENHANCEMENT TO EHR: Addition of e-prescribing ability through DrFirst, creating version 8.1. WORKFLOW ENHANCEMENTS: The following are the workflow enhancements in Safety- Enhanced Design of ModuleMD WISE : a.1 CPOE Medication Drug-Drug, Drug-Allergy Interaction Checks Medication List Electronic Prescribing USABILITY TESTED: Usability of each test criteria was tested from May 1, 2015, through May 31, 2015, in the medical records for multiple test patients a.1 CPOE - Medication a. record medication order b. change medication order c. access medication order Drug-Drug, Drug-Allergy Interaction Checks a. trigger one drug-drug contraindication intervention b. trigger one drug-allergy contraindication intervention 57

70 c. adjust severity level of interventions indicated for drug-drug interaction Medication List a. record active medication list b. change active medication list c. access active medication list Electronic Prescribing a. electronically prescribe single medication for test patient b. complete active medication for test patient c. discontinue active medication for test patient PERFORMANCE: 1. Ease of use a. Quicker SSO launch from ModuleMD to DrFirst b. DrFirst dashboard allows for easy navigation c. Return to EMR button saves additional click to log out 2. Issues a. Help button results in Web Page error message 3. Areas for improvement a. No recommendations IMPACT: None. Users will have the choice between 2 e-prescribing applications which will have no impact on reporting the above criteria. 58

71 Providing solutions...not just software Quality Management System Attestation For reporting information related to testing of (g)(4) Vendor and Product Information Vendor Name Product Name Product Version Quality Management System ModuleMD, LLC ModuleMD Wise Type of Quality Management System (QMS) used in the development, testing, implementation, and maintenance of EHR product. X Based on Industry Standard (for example ISO9001, IEC 62304, ISO13485, etc). Standard: A modified or "home- grown" QMS. No QMS was used. Was one QMS used for all certification criteria or were multiple QMS applied? X One QMS was used. Multiple QMS used. Description or documentation of QMS applied to each criteria: Not Applicable. Statement of Compliance I, the undersigned, attest that the statements in this document are accurate. Vendor Signature by an Authorized Representative Date 1/22/2014 ModuleMD LLC. * 8359 Office Park Dr, Grand Blanc, MI )

72 Providing solutions...not just software Privacy and Security Attestation Vendor and Product Information Vendor Name Product Name Product Version Privacy and Security (d)(2) Auditable events and tamper- resistance ModuleMD, LLC ModuleMD Wise Audit Log: X Cannot be disabled by any user. Audit Log can be disabled Not Applicable (did not test to this criteria) The EHR enforces that the audit log is enabled by default when initially configured Audit Log Status Indicator: X Cannot be disabled by any user. Audit Log can be disabled X The EHR enforces a default audit log status. Identify the default setting (enabled or disabled): There is no Audit Log Status Indicator because the Audit Log cannot be disabled. Encryption Status Indicator (encryption of health information locally on end user device): X Cannot be disabled by any user. Audit Log can be disabled X The EHR enforces a default encryption status. Identify the default setting (enabled or disabled): There is no Encryption Status Indicator because the EHR does not allow health information to be stored locally on end user devices. Identify the submitted documentation that describes the inability of the EHR to allow users to disable the audit logs, the audit log status, and/or the encryption status: Local Data Storage.pdf ModuleMD LLC. * 8359 Office Park Dr, Grand Blanc, MI )

73 Providing solutions...not just software Identify the submitted documentation that describes the method(s) by which the EHR protects 1) recording of actions related to electronic health information, 2) recording of audit log status, and 3) recording of encryption status from being changed, overwritten, or deleted by the EHR technology: Auditable Events and_tamper- Resistance.pdf Local Data Storage.pdf (d)(7) End- user device encryption Storing electronic health information locally on end- user devices (i.e. temp files, cookies, or other types of cache approaches). Not Applicable (did not test to this criteria) Identify the submitted documentation that describes the method(s) by which the EHR technology detects whether the audit log has been altered: X The EHR does not allow health information to be stored locally on end- user devices. Identify the submitted documentation that describes the functionality used to prevent health information from being stored locally: Auditable Events and_tamper- Resistance.pdf Local Data Storage.pdf The EHR does not allow health information to be stored locally on end- user devices. Identify the FIPS approved algorithm used for encryption: Identify the submitted documentation that describes how health information is encrypted when stored locally on end- user devices: (d)(8) Integrity X Not Applicable (did not test to this criteria) X The EHR enforces default configuration settings that either enforces the encryption of locally stored health information or prevents health information from being stored locally. Identify the default setting: Identify the hashing algorithm used for integrity (SHA- 1 or higher): Statement of Compliance I, the undersigned, attest that the statements in this document are accurate. Vendor Signature by an Authorized Representative Date 1/22/2014 ModuleMD LLC. * 8359 Office Park Dr, Grand Blanc, MI )