ONC HIT Certification Program

Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 01-Jun-2015 ONC HIT Certification Program Test Results Summary for 2014 Edition EHR Certification Part 1: Product and Developer Information 1.1 Certified Product Information Product Name: ModuleMD WISE Product Version: 8.1 Domain: Ambulatory Test Type: Complete EHR 1.2 Developer/Vendor Information Developer/Vendor Name: ModuleMD Address: 8359 Office Park Drive Grand Blanc MI 48439 Website: www.modulemd.com Email: Lloyd.williams@modulemd.com Phone: 248-434-0444 x1004 Developer/Vendor Contact: Lloyd Williams Page 1 of 12

Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 01-Jun-2015 Part 2: ONC-Authorized Certification Body Information 2.1 ONC-Authorized Certification Body Information ONC-ACB Name: Drummond Group Address: 13359 North Hwy 183, Ste B-406-238, Austin, TX 78750 Website: Email: www.drummondgroup.com ehr@drummondgroup.com Phone: 817-294-7339 ONC-ACB Contact: Bill Smith This test results summary is approved for public release by the following ONC-Authorized Certification Body Representative: Bill Smith ONC-ACB Authorized Representative Certification Body Manager Function/Title Signature and Date 7/6/2015 2.2 Gap Certification The following identifies criterion or criteria certified via gap certification 170.314 (a)(1) (a)(19) (d)(6) (h)(1) (a)(6) (a)(20) (d)(8) (h)(2) (a)(7) (b)(5)* (d)(9) (h)(3) (a)(17) (d)(1) (f)(1) (a)(18) (d)(5) (f)(7)** *Gap certification allowed for Inpatient setting only **Gap certification allowed for Ambulatory setting only x No gap certification Page 2 of 12

Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 01-Jun-2015 2.3 Inherited Certification The following identifies criterion or criteria certified via inherited certification 170.314 x (a)(1) (a)(16) Inpt. only x (c)(2) x (f)(2) x (a)(2) (a)(17) Inpt. only x (c)(3) x (f)(3) x (a)(3) (a)(18) x (d)(1) (f)(4) Inpt. only x (a)(4) (a)(19) x (d)(2) x (a)(5) (a)(20) x (d)(3) x (a)(6) x (b)(1) x (d)(4) x (a)(7) x (b)(2) x (d)(5) x (a)(8) x (b)(3) x (d)(6) (f)(7) x (a)(9) x (b)(4) x (d)(7) (g)(1) x (a)(10) x (b)(5) x (d)(8) x (g)(2) x (a)(11) (b)(6) Inpt. only (d)(9) Optional x (g)(3) x (a)(12) x (b)(7) x (e)(1) x (g)(4) x (a)(13) (b)(8) x (e)(2) Amb. only (h)(1) x (a)(14) (b)(9) x (e)(3) Amb. only (h)(2) x (a)(15) x (c)(1) x (f)(1) (h)(3) (f)(5) Amb. only (f)(6) Amb. only No inherited certification Page 3 of 12

Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 01-Jun-2015 Part 3: NVLAP-Accredited Testing Laboratory Information Report Number: SG-06082015-2937-1 Test Date(s): N/A 3.1 NVLAP-Accredited Testing Laboratory Information ATL Name: Drummond Group EHR Test Lab Accreditation Number: NVLAP Lab Code 200979-0 Address: 13359 North Hwy 183, Ste B-406-238, Austin, TX 78750 Website: Email: www.drummondgroup.com ehr@drummondgroup.com Phone: 512-335-5606 ATL Contact: Beth Morrow For more information on scope of accreditation, please reference NVLAP Lab Code 200979-0. Part 3 of this test results summary is approved for public release by the following Accredited Testing Laboratory Representative: Sonia Galvan Test Proctor ATL Authorized Representative Function/Title Signature and Date 7/6/2015 Houston, TX Location Where Test Conducted 3.2 Test Information 3.2.1 Additional Software Relied Upon for Certification Additional Software Applicable Criteria Allscripts eprescribe 170.314.a.2, 8, 10; 170.314.b.3, 4; 170.314.g.2 Functionality provided by Additional Software e-prescribing vendor EMR Direct phimail 170.314.b.1, 2; 170.314.e.1 HISP Adobe Reader 170.314.a.12 Viewing narrative results No additional software required Page 4 of 12

Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 01-Jun-2015 3.2.2 Test Tools Test Tool Version x Cypress 2.6 x eprescribing Validation Tool 1.0.5 HL7 CDA Cancer Registry Reporting Validation Tool 1.0.3 HL7 v2 Electronic Laboratory Reporting (ELR) Validation Tool 1.8.2 x HL7 v2 Immunization Information System (IIS) Reporting Validation Tool 1.8.2 x HL7 v2 Laboratory Results Interface (LRI) Validation Tool 1.7.2 x HL7 v2 Syndromic Surveillance Reporting Validation Tool 1.7.2 x Transport Testing Tool 181 x Direct Certificate Discovery Tool 3.0.4 Edge Testing Tool 0.0.5 No test tools required 3.2.3 Test Data Alteration (customization) to the test data was necessary and is described in Appendix [insert appendix letter] x No alteration (customization) to the test data was necessary 3.2.4 Standards 3.2.4.1 Multiple Standards Permitted The following identifies the standard(s) that has been successfully tested where more than one standard is permitted Criterion # Standard Successfully Tested (a)(8)(ii)(a)(2) (a)(13) x 170.204(b)(1) HL7 Version 3 Implementation Guide: URL-Based Implementations of the Context-Aware Information Retrieval (Infobutton) Domain x 170.207(a)(3) IHTSDO SNOMED CT International Release July 2012 and US Extension to SNOMED CT March 2012 Release 170.204(b)(2) HL7 Version 3 Implementation Guide: Context-Aware Knowledge Retrieval (Infobutton) Service-Oriented Architecture Implementation Guide 170.207(j) HL7 Version 3 Standard: Clinical Genomics; Pedigree Page 5 of 12

Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 01-Jun-2015 Criterion # (a)(15)(i) x 170.204(b)(1) HL7 Version 3 Implementation Guide: URL-Based Implementations of the Context-Aware Information Retrieval (Infobutton) Domain Standard Successfully Tested 170.204(b)(2) HL7 Version 3 Implementation Guide: Context-Aware Knowledge Retrieval (Infobutton) Service-Oriented Architecture Implementation Guide (a)(16)(ii) (b)(2)(i)(a) (b)(7)(i) 170.210(g) Network Time Protocol Version 3 (RFC 1305) 170.207(i) The code set specified at 45 CFR 162.1002(c)(2) (ICD-10- CM) for the indicated conditions 170.207(i) The code set specified at 45 CFR 162.1002(c)(2) (ICD-10- CM) for the indicated conditions 170. 210(g) Network Time Protocol Version 4 (RFC 5905) x 170.207(a)(3) IHTSDO SNOMED CT International Release July 2012 and US Extension to SNOMED CT March 2012 Release x 170.207(a)(3) IHTSDO SNOMED CT International Release July 2012 and US Extension to SNOMED CT March 2012 Release (b)(8)(i) (e)(1)(i) (e)(1)(ii)(a)(2) (e)(3)(ii) 170.207(i) The code set specified at 45 CFR 162.1002(c)(2) (ICD-10- CM) for the indicated conditions Annex A of the FIPS Publication 140-2 [list encryption and hashing algorithms] 128 bit AES 1024 bit RSA x 170.210(g) Network Time Protocol Version 3 (RFC 1305) Annex A of the FIPS Publication 140-2 [list encryption and hashing algorithms] 128 bit AES 1024 bit RSA 170.207(a)(3) IHTSDO SNOMED CT International Release July 2012 and US Extension to SNOMED CT March 2012 Release 170. 210(g) Network Time Protocol Version 4 (RFC 5905) Common MU Data Set (15) x 170.207(a)(3) IHTSDO SNOMED CT International Release July 2012 and US Extension to SNOMED CT March 2012 Release 170.207(b)(2) The code set specified at 45 CFR 162.1002(a)(5) (HCPCS and CPT-4) Page 6 of 12

Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 01-Jun-2015 Criterion # Standard Successfully Tested None of the criteria and corresponding standards listed above are applicable 3.2.4.2 Newer Versions of Standards The following identifies the newer version of a minimum standard(s) that has been successfully tested Newer Version Applicable Criteria No newer version of a minimum standard was tested 3.2.5 Optional Functionality Criterion # x (a)(4)(iii) (b)(1)(i)(b) (b)(1)(i)(c) (b)(2)(ii)(b) (b)(2)(ii)(c) (e)(1) (f)(3) (f)(7) Common MU Data Set (15) Common MU Data Set (15) Optional Functionality Successfully Tested Plot and display growth charts Receive summary care record using the standards specified at 170.202(a) and (b) (Direct and XDM Validation) Receive summary care record using the standards specified at 170.202(b) and (c) (SOAP Protocols) Transmit health information to a Third Party using the standards specified at 170.202(a) and (b) (Direct and XDM Validation) Transmit health information to a Third Party using the standards specified at 170.202(b) and (c) (SOAP Protocols) View, download and transmit data to a third party utilizing the Edge Protocol IG version 1.1 Ambulatory setting only Create syndrome-based public health surveillance information for transmission using the standard specified at 170.205(d)(3) (urgent care visit scenario) Ambulatory setting only transmission to public health agencies syndromic surveillance - Create Data Elements Express Procedures according to the standard specified at 170.207(b)(3) (45 CFR162.1002(a)(4): Code on Dental Procedures and Nomenclature) Express Procedures according to the standard specified at 170.207(b)(4) (45 CFR162.1002(c)(3): ICD-10-PCS) No optional functionality tested Page 7 of 12

Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 01-Jun-2015 3.2.6 2014 Edition Certification Criteria* Successfully Tested Criteria # Version Version Criteria # TP** TD*** TP TD (a)(1) 1.3 1.5 (c)(3) 1.11 1.11 (a)(2) 1.2 (d)(1) 1.2 (a)(3) 1.2 1.4 (d)(2) 1.6 (a)(4) 1.4 1.3 (d)(3) 1.3 (a)(5) 1.4 1.3 (d)(4) 1.3 (a)(6) 1.3 1.4 (d)(5) 1.2 (a)(7) 1.3 1.3 (d)(6) 1.2 (a)(8) 1.3 (d)(7) 1.2 (a)(9) 1.3 1.3 (d)(8) 1.2 (a)(10) 1.2 1.4 (d)(9) Optional 1.2 (a)(11) 1.3 (e)(1) 1.11 1.5 (a)(12) 1.3 (e)(2) Amb. only 1.2 1.6 (a)(13) 1.2 (e)(3) Amb. only 1.3 (a)(14) 1.2 (f)(1) 1.2 1.2 (a)(15) 1.5 (f)(2) 1.3 1.3 (a)(16) Inpt. only 1.3 1.2 (f)(3) 1.3 1.3 (a)(17) Inpt. only 1.2 (f)(4) Inpt. only 1.3 1.3 (a)(18) 1.1 1.5 (a)(19) 1.1 1.5 (a)(20) 1.1 1.5 (b)(1) 1.7 1.4 (f)(5) Amb. only 1.2 1.2 (f)(6) Amb. only 1.4 1.4 (b)(2) 1.4 1.6 (f)(7) Amb. only 1.1 (b)(3) 1.4 1.4 (g)(1) 2.0 2.0 (b)(4) 1.3 1.4 (g)(2) 2.0 2.0 (b)(5) 1.4 1.2 (g)(3) 1.4 (b)(6) Inpt. only 1.3 1.3 (g)(4) 1.2 (b)(7) 1.4 1.7 (h)(1) 1.1 (b)(8) 1.2 1.2 (h)(2) 1.1 (b)(9) 1.1 1.1 (h)(3) 1.1 (c)(1) 1.11 1.11 (c)(2) 1.11 1.11 Page 8 of 12

Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 01-Jun-2015 Criteria # x No criteria tested Version Version Criteria # TP** TD*** TP TD *For a list of the 2014 Edition Certification Criteria, please reference http://www.healthit.gov/certification (navigation: 2014 Edition Test Method) **Indicates the version number for the Test Procedure (TP) ***Indicates the version number for the Test Data (TD) Page 9 of 12

Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 01-Jun-2015 3.2.7 2014 Clinical Quality Measures* Type of Clinical Quality Measures Successfully Tested: x Ambulatory Inpatient No CQMs tested *For a list of the 2014 Clinical Quality Measures, please the CMS ecqm Library (Navigation: June 2014 and April 2014 Updates) Ambulatory CQMs CMS ID Version CMS ID Version CMS ID Version CMS ID Version x 2 v3 90 136 x 155 v2 22 x 117 v2 137 x 156 v2 50 122 x 138 v2 157 52 x 123 v2 139 158 56 x 124 v2 140 159 61 x 125 v2 141 160 62 x 126 v2 142 161 64 x 127 v2 143 163 65 128 144 164 66 129 145 x 165 v2 68 130 x 146 v2 x 166 v3 x 69 v2 x 131 v2 147 167 74 132 148 169 75 133 149 177 77 134 x 153 v2 179 82 135 x 154 v2 182 Inpatient CQMs CMS ID Version CMS ID Version CMS ID Version CMS ID Version 9 71 107 172 26 72 108 178 30 73 109 185 31 91 110 188 32 100 111 190 53 102 113 55 104 114 60 105 171 Page 10 of 12

Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 01-Jun-2015 3.2.8 Automated Numerator Recording and Measure Calculation 3.2.8.1 Automated Numerator Recording Automated Numerator Recording Successfully Tested (a)(1) (a)(11) (a)(18) (b)(6) (a)(3) (a)(12) (a)(19) (b)(8) (a)(4) (a)(13) (a)(20) (b)(9) (a)(5) (a)(14) (b)(2) (e)(1) (a)(6) (a)(15) (b)(3) (e)(2) (a)(7) (a)(16) (b)(4) (e)(3) (a)(9) (a)(17) (b)(5) x Automated Numerator Recording was not tested 3.2.8.2 Automated Measure Calculation Automated Measure Calculation Successfully Tested x (a)(1) x (a)(11) (a)(18) (b)(6) x (a)(3) x (a)(12) (a)(19) (b)(8) x (a)(4) x (a)(13) (a)(20) (b)(9) x (a)(5) x (a)(14) x (b)(2) x (e)(1) x (a)(6) x (a)(15) x (b)(3) x (e)(2) x (a)(7) (a)(16) x (b)(4) x (e)(3) x (a)(9) (a)(17) x (b)(5) Automated Measure Calculation was not tested 3.2.9 Attestation Attestation Forms (as applicable) x Safety-Enhanced Design* x Quality Management System** x Privacy and Security Appendix A B C 3.3 Appendices Attached below. *Required if any of the following were tested: (a)(1), (a)(2), (a)(6), (a)(7), (a)(8), (a)(16), (a)(18), (a)(19), (a)(20), (b)(3), (b)(4), (b)(9). **Required for every EHR product Page 11 of 12

Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 01-Jun-2015 Test Results Summary Change History Test Report ID Description of Change Date 2014 Edition Test Report Summary Page 12 of 12

Electronic Health Record Usability Test Report ModuleMD WISE v8.0 Complete EHR

EHR Usability Test Report of ModuleMD WISE v8.0 Report based on ISO/IEC 25062:2006 Common Industry Format for Usability Test Reports Module MD WISE v8.0 Date of Usability Test: Monday, May 20th, 2013 through Monday, July 29th, 2013 Date of Report: 08/16/2013 Date of Update: 06/01/2015 Report Prepared By: ModuleMD, LLC Jeff McKenzie, Client Services Manager Amardip Chandpuri, Quality Analyst jeff.mckenzie@modulemd.com amardip.chandpuri@modulemd.com Addendum Prepared By: Ellyn Serra, Client Services Manager ellyn.serra@modulemd.com ModuleMD, LLC 8359 Office Park Drive Grand Blanc, MI 48439 2

TABLE OF CONTENTS 1 EXECUTIVE SUMMARY... 04 2 INTRODUCTION... 10 3 METHOD... 11 3.1 PARTICIPANTS... 11 3.2 STUDY DESIGN... 11 3.3 TASKS... 12 3.4 PROCEDURE... 13 3.5 TEST LOCATION... 13 3.6 TEST ENVIRONMENT... 14 3.7 USABILITY METRICS... 14 4 RESULTS... 14 4.1 DATA SCORING... 14 4.2 DATA ANALYSIS AND REPORTING... 14 5 APPENDICES... 15 5.1 APPENDIX 1: NON-DISCLOSURE AGREEMENT... 15 5.2 APPENDIX 2: TASKS TO BE PERFORMED... 16 5.3 APPENDIX 3: FINAL QUESTIONS... 54 5.4 APPENDIX 4: SYSTEM USABILITY SCALE QUESTIONNAIRE... 55 6 ADDENDUM... 56 3

1. EXECUTIVE SUMMARY This report documents the findings of a usability test of ModuleMD WISE v8.0 Complete EHR by ModuleMD. A usability test of ModuleMD WISE v8.0 Complete EHR was conducted every other Monday after each release beginning from Monday, May 20th, 2013 through Monday, July 29th, 2013 at the headquarters of ModuleMD, LLC and various remote locations by ModuleMD employees and clients. The purpose of this testing to test and validate the usability of the new user interface, and provide evidence of usability in the EHR Under Test (EHRUT). During the usability test, one healthcare provider and four ModuleMD employees matching the target demographic criteria served as participants and used the EHRUT in simulated, but representative tasks. This study collected performance data on 37 tasks typically conducted on an EHR: Computerized Provider Order Entry (CPOE): 1. Record medication order 2. Change medication order 3. Access medication order 4. Record lab order 5. Change lab order 6. Access lab order 7. Record Rad/Image order 8. Change Rad/Image order 9. Access Rad/Image order Drug-drug, Drug-allergy Interaction Checks: 10. Trigger one drug-drug contraindication intervention 11. Trigger one drug-allergy contraindication intervention 12. Adjust severity level of interventions indicated for drug-drug interaction Medication List: 13. Record active medication list 14. Change active medication list 15. Access active medication list 4

Medication Allergy List: 16. Record active medication allergy list 17. Change active medication allergy list 18. Access active medication allergy list Clinical Decision Support: 19. Create clinical decision support intervention for a problem list 20. Create clinical decision support intervention for a medication list 21. Create clinical decision support intervention for a medication allergy list 22. Create clinical decision support intervention for demographics 23. Create clinical decision support intervention for laboratory tests and values/results 24. Create clinical decision support intervention for vital signs 25. Trigger clinical decision support intervention for a problem list 26. Trigger clinical decision support intervention for a medication list 27. Trigger clinical decision support intervention for a medication allergy list 28. Trigger clinical decision support intervention for demographics 29. Trigger clinical decision support intervention for laboratory tests and values/results 30. Trigger clinical decision support intervention for vital signs Electronic Prescribing: 31. Electronically prescribe single medication for test patient 32. Complete active medication for test patient 33. Discontinue active medication for test patient 34. Renew active medication for test patient Clinical Information Reconciliation: 35. Create a single reconciled active medication list 36. Create a single reconciled active problem list 37. Create a single reconciled active medication allergy list Throughout the development period, there were 12 milestone release dates. Prior to each release date, usability tests were conducted by each participant. Each participant was asked to track the following types of data: Time to complete the tasks The number of errors Path deviations Evaluate the tasks for ease of success 5

Evaluate the tasks for level of difficulty All participant data was de-identified. Various recommended metrics, in accordance with the examples set forth in the NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records, were used to evaluate the usability of the EHRUT. Following is a summary of the performance and rating data collected on the EHRUT. Measure N Task Success Path Deviation Task Time Errors Task Ratings Task # Mean (SD) Deviations (Observed / Optimal) Mean (SD) Mean (SD) Mean (SD) 1 Record medication order 30 2.3 2.3 171.46 2.06 2.6 2 Change medication order 30 2.43 0.75 169.71 2.53 2.66 3 Access medication order 30 1.9 1.96 123.92 1.5 2.76 4 Record lab order 30 2.33 2.23 152.41 4.06 2.56 5 Change change lab order 30 2.03 1.11 140.61 2.7 2.93 6 Access lab order 30 1.86 1.93 103.69 2 3.13 7 Record Rad/Image order 30 2.3 1.13 185.15 3.56 2.6 8 Change Rad/Image order 30 2.33 2.2 182.43 2.66 2.7 9 Access Rad/Image order 30 2.23 0.77 137.08 2.16 2.96 10 Trigger one drug-drug contraindication intervention 30 2.13 2.4 196.83 3.96 2.56 11 12 Trigger one drug-allergy contraindication intervention Adjust severity level of interventions indicated for drugdrug interaction 30 2 1.2 187.73 3.1 2.63 30 1.93 2.13 158.84 2.36 3.06 13 Record active medication list 30 1.86 2.26 160.22 2.93 2.86 14 Change active medication list 30 1.9 2.2 142.36 2.23 2.76 15 Access active medication list 30 1.66 1.96 120.30 1.9 3 16 17 18 Record active medication allergy list Change active medication allergy list Access active medication allergy list 30 1.9 2.0 134.95 2.5 3.0 30 2.0 1.9 135.90 2.0 3.2 30 1.6 1.6 102.10 1.1 3.5 19 Create clinical decision support intervention for a problem list 30 1.96 2.1 107.79 2.5 3.1 6

20 21 22 Create clinical decision support intervention for a medication list Create clinical decision support intervention for a medication allergy list Create clinical decision support intervention for demographics 30 2.1 2.2 155.59 3.86 2.83 30 1.96 0.7 167.32 3.86 3.03 30 1.53 1.76 94.13 1.06 3.9 23 Create clinical decision support intervention for laboratory tests and values/results 30 2.16 2.23 144.39 4.03 3.03 24 Create clinical decision support intervention for vital signs 30 2.24 1.13 137.95 2.9 3.2 25 Trigger clinical decision support intervention for a problem list 30 1.96 1.9 127.08 2.53 3.2 26 27 28 Trigger clinical decision support intervention for a medication list Trigger clinical decision support intervention for a medication allergy list Trigger clinical decision support intervention for demographics 30 1.93 1.86 92.97 2.26 3.16 30 2.16 0.74 103.41 2.7 3.06 30 1.5 1.43 58.84 0.73 4.1 29 Trigger clinical decision support intervention for laboratory tests and values/results 30 2.36 2.1 131.09 3.03 2.93 30 Trigger clinical decision support intervention for vital signs 30 2.2 1.06 98.18 2.13 3.36 31 Electronically prescribe single medication for test patient 30 1.76 1.73 122.04 1.36 2.9 32 Complete active medication for test patient 30 2.06 1.13 136.63 2.63 2.83 33 Discontinue active medication for test patient 30 2.03 1.83 106.02 1.43 3 34 Renew active medication for test patient 30 2.33 2.03 146.48 3.03 2.56 35 Create a single reconciled active medication list 30 2.23 2.06 129.35 2.4 3 36 Create a single reconciled active problem list 30 2.33 2.03 118.25 1.8 2.8 7

37 Create a single reconciled active medication allergy list 30 2.06 1.08 107.10 2.4 2.8 The results from the System Usability Scale scored the subjective satisfaction with the system based on performance with these tasks to be Satisfactory with satisfaction score of 82.25 In addition to the performance data, the following qualitative observations were made: Major findings in Release 1: o Not able to create clinical decision support intervention for a medication allergy list as the system has no option for Medication Allergy. o Not able to create clinical decision support intervention for a vital signs as the system has no option for vital signs. Major findings in Release 2: o Patient Search working very slow. o Under scan documents the uploaded file not visible in the upload queue. o Clinical decision support intervention headings are still not collapsing. Major findings in Release 3: o Allscripts connectivity issue from EMR. o BP Systolic and BP Diastolic radio buttons not working. o Under Clinical Decision Support for Vital Signs, when BMI Within Range radio button is selected only one text box is opening instead of two text boxes. Major findings in Release 4: o Not able to trigger clinical decision support for Demographics when Race and Ethnicity is a criteria for a test patient. o User not able to change lab order for a patient. o User not able to upload image under scan documents. 8

Areas for improvement in Release 1: o Add an option to create and trigger clinical decision support intervention for a medication allergy list. o Add an option to create and trigger clinical decision support intervention for vital signs. Areas for improvement in Release 2: o Need to increase the speed for patient search o All the heading under clinical decision support interventions should get collapsed. Areas for improvement in Release 3: o The user should be able to select radio buttons for every value associated with the radio button. o User should be able to connect to Allscripts on clicking the eprescribe button in Current Medications clinical category. o Two text boxes should open when the Within Range radio button is selected for BMI under Vital Signs for clinical decision support. Areas for improvement in Release 4: o The user should be able to trigger the clinical decision support with each and at least one combination. o User should be able to change lab order for a Patient. o User should be able to upload an image in Scan Documents and also the uploaded image should be available under upload queue before the Save button is selected. 9

2. INTRODUCTION The EHRUT tested for this study was ModuleMD WISE, version 8.0. Designed to present medical information to healthcare providers in ambulatory care, the EHRUT consists of integrated health records, management and revenue management into one comprehensive, interoperable solution. ModuleMD WISE assists physicians in improving the quality and coordination of care, while obtaining peak efficiencies in practice management. The usability testing attempted to represent realistic exercises and conditions. The purpose of this study was to test and validate the usability of the current user interface, and provide evidence of usability in the EHR Under Test (EHRUT). To this end, measures of effectiveness, efficiency and user satisfaction, such as task time, and ease of completion were captured during the usability testing. 10

3. METHOD 3.1 PARTICIPANTS A total of five participants were tested on the EHRUT. Participants in the test were doctor, Quality analysts, and client service support staff. Participants had a mix of background and demographic characteristics. The following is a table of participants by characteristics, including demographics, professional experience and computing experience. Participant names were replaced with Participant IDs so that an individual's data cannot be tied back to individual identities. Participant Demographics Part ID Gender Age Education Occupation/role Professional Experience Computer Experience Product Experience 101 M 40 BS Manager 16 yrs 16 yrs < 4 months 102 M 40 BE Quality Analyst 15 yrs 15 yrs >2 yrs 103 F 29 BE Quality Analyst 5+ yrs 5+ yrs >2 yrs 104 M 67 MD Doctor 35 yrs 15 yrs 13 yrs 105 M 32 MA Client Services 5+ yrs 10 yrs >2 yrs 3.2 STUDY DESIGN Overall, the objective of this test was to uncover areas where the application performed well - that is, effectively, efficiently, and with satisfaction - and areas where the application failed to meet the needs of the participants. The data from this test may serve as a baseline for future tests with an updated version of the same EHR and/or comparison with other EHRs provided the same tasks are used. In short, this testing serves as both a means to record or benchmark current usability, but also to identify areas where improvements must be made. The system was evaluated for effectiveness, efficiency and satisfaction as defined by measures collected and analyzed for each participant: Time to complete the tasks The number of errors Path deviations Evaluate the tasks for ease of success Evaluate the tasks for level of difficulty Additional information about the various measures can be found in Section on Usability Metrics. 11

3.3 TASKS A number of tasks were constructed that would be realistic and representative of the kinds of activities a user might do with this EHR, including: Computerized Provider Order Entry (CPOE): 1. Record medication order 2. Change medication order 3. Access medication order 4. Record lab order 5. Change lab order 6. Access lab order 7. Record Rad/Image order 8. Change Rad/Image order 9. Access Rad/Image order Drug-drug, Drug-allergy Interaction Checks: 10. Trigger one drug-drug contraindication intervention 11. Trigger one drug-allergy contraindication intervention 12. Adjust severity level of interventions indicated for drug-drug interaction Medication List: 13. Record active medication list 14. Change active medication list 15. Access active medication list Medication allergy List: 16. Record active medication allergy list 17. Change active medication allergy list 18. Access active medication allergy list Clinical Decision Support: 19. Create clinical decision support intervention for a problem list 20. Create clinical decision support intervention for a medication list 21. Create clinical decision support intervention for a medication allergy list 22. Create clinical decision support intervention for demographics 23. Create clinical decision support intervention for laboratory tests and values/results 12

24. Create clinical decision support intervention for vital signs 25. Trigger clinical decision support intervention for a problem list 26. Trigger clinical decision support intervention for a medication list 27. Trigger clinical decision support intervention for a medication allergy list 28. Trigger clinical decision support intervention for demographics 29. Trigger clinical decision support intervention for laboratory tests and values/results 30. Trigger clinical decision support intervention for vital signs Electronic Prescribing: 31. Electronically prescribe single medication for test patient 32. Complete active medication for test patient 33. Discontinue active medication for test patient 34. Renew active medication for test patient Clinical information reconciliation: 35. Create a single reconciled active medication list 36. Create a single reconciled active problem list 37. Create a single reconciled active medication allergy list Tasks were selected based on their frequency of use, criticality of function, and those that may be most troublesome for users. Tasks should always be constructed in light of the study objectives. 3.4 PROCEDURE Each participant was assigned a Participant ID and they reviewed and signed the Non-Disclosure Agreement (See Appendix 1). Every participant ran the tests and logged there results in the usability testing log. All the errors encountered were logged by the participants in Redmine and were assigned to the development team for correction. Each participant demographics are mentioned earlier under heading participants. Each participant was given the usability testing log and were instructed to perform the tasks as quickly as possible making as few errors and deviations as possible. 3.5 TEST LOCATION Testing was done at headquarters of ModuleMD, LLC and various remote locations by ModuleMD employees and clients. 13

3.6 TEST ENVIRONMENT The following specifications computers were used by the participants: Computer Name Computer Model Processor Installed RAM Operating System Browser Computer 1 Dell Intel Core i5 6 GB Windows 8 Chrome Computer 2 Lenovo Intel Core i5 6 GB Windows 7 Professional IE9/Chrome Computer 3 HP Intel Core 2 6320 2 GB Windows XP Professional IE9/Chrome Computer 4 Lenovo Intel Core i5 4 GB Windows 7 Home Premium Chrome Computer 5 Compaq Intel T2600 2 GB Windows 7 Professional IE/9Chrome 3.7 USABILITY METRICS The goals of the test were to assess: 1. Measuring participant success rates and errors 2. Measuring the average task time and path deviations 3. Satisfaction by measuring ease of use ratings 4. RESULTS 4.1 DATA SCORING Task Success: A task was counted as a Success if the participant was able to achieve the correct outcome, without assistance, within the time allotted on a per task basis. Task Failure: If the participant did not reach the correct answer or performed it incorrectly, the task was counted as a Failure. Task Deviations: Participant Deviated from the actual steps Task Time: Time spent by each participant to complete the task Task Rating: How easy or difficult the task was for each participant RESULTS 4.2 DATA ANALYSIS AND REPORTING The results of the usability test were calculated according to the methods specified in the Usability Metrics section above. 14

5. APPENDICES 5.1 Appendix 1: Non-Disclosure Agreement THIS AGREEMENT is entered into as of 2013, between ( the Participant ) and the testing organization ModuleMD LLC located at 8359 Office Park Drive, Grand Blanc, MI, 48439 The Participant acknowledges his or her participation in this usability study may bring the Participant into possession of Confidential Information. The term "Confidential Information" means all technical and commercial information of a proprietary or confidential nature which is disclosed by ModuleMD LLC, or otherwise acquired by the Participant, in the course of the study. By way of illustration, but not limitation, Confidential Information includes trade secrets, processes, formulae, data, know-how, products, designs, drawings, computer aided design files and other computer files, computer software, ideas, improvements, inventions, training methods and materials, marketing techniques, plans, strategies, budgets, financial information, or forecasts. Any information the Participant acquires relating to this product during this study is confidential and proprietary to ModuleMD LLC and is being disclosed solely for the purposes of the Participant s participation in the usability study. By signing this form the Participant acknowledges that she/he will not disclose this confidential information obtained to anyone else or any other organizations. Participant s Printed Name: Signature: Date: 15

5.2 Appendix 2: TASKS TO BE PERFORMED Task 1: Record Medication Order Note: Before testing the participant should review the test patient s Medication Order. Optimal Path: Select Test Patient >> Open Visit Slip >> Click Current Medications >> Click eprescribe >> Search Medication >> Select Medication >> Complete Sig >> Add to Script Queue Very 16

Task 2: Change Medication Order Note: Before testing the participant should review the test patient s medications. Optimal Path: Select Test Patient >> Open Visit Slip >> Click Current Medications >> Click eprescribe >> Click Script Queue >> Click Edit link >> Modify Sig >> Add to Script Queue Very 17

Task 3: Access Medication Order Note: Before testing the participant should review the test patient s medications. Optimal Path: Select Test Patient >> Open Visit Slip >> Click Current Medications >> Click eprescribe >> Click Review Script Queue >> Process Order >> Exit eprescribe >> Click Prescriptions Very 18

Task 4: Record Lab Order Note: Before testing the participant should review the test patient s lab orders. Optimal Path: Select Test Patient >> Open Visit Slip >> Click Labs >> Click New/Existing Orders >> Click Order New >> Complete Lab Order Form >> Submit Very 19

Task 5: Change Lab Order Note: Before testing the participant should review the test patient s lab test values/results. Optimal Path: Select Test Patient >> Open Visit Slip >> Click Labs >> Click New/Existing Orders >> Select Order ID >> Click Reorder >> Complete Lab Order Form >> Submit Very 20

Task 6: Access Lab Order Note: Before testing the participant should review the test patient s lab test values/results. Optimal Path: Select Test Patient >> Open Visit Slip >> Click Labs >> Click New/Existing Orders >> Select Order ID >> Click View Order Very 21

Task 7: Record Rad/Image Order Note: Before testing the participant should review the test patient s image order. Optimal Path: Select Test Patient >> Open Visit Slip >> Click Labs >> Click New/Existing Orders >> Click Order New >> Complete Radiology Order Form >> Submit Very 22

Task 8: Change Rad/Image Order Note: Before testing the participant should review the test patient s image order. Optimal Path: Select Test Patient >> Open Visit Slip >> Click Labs >> Click New/Existing Orders >> Select Order ID >> Click Reorder >> Complete Radiology Order Form >> Submit Very 23

Task 9: Access Rad/Image Order Note: Before testing the participant should review the test patient s image order. Optimal Path: Select Test Patient >> Open Visit Slip >> Click Labs >> Click New/Existing Orders >> Select Order ID >> Click View Order Very 24

Task 10: Trigger One Drug-Drug Contraindication Intervention Note: Before testing the participant should review the test patient s drug-drug contraindication intervention. Optimal Path: Select Test Patient >> Open Visit Slip >> Click Current Medications >> Click eprescribe >> Search Medication >> Select Medication >> Complete Sig >> Add to Script Queue Very 25

Task 11: Trigger One Drug-Allergy Contraindication Intervention Note: Before testing the participant should review the test patient s drug-allergy contraindication intervention. Optimal Path: Select Test Patient >> Open Visit Slip >> Click Current Medications >> Click eprescribe >> Search Medication >> Select Medication >> Complete Sig >> Add to Script Queue Very 26

Task 12: Adjust Severity Level of Interventions Indicated for Drug-Drug interaction Note: Before testing the participant should review the test patient s severity level for drug-drug interaction Optimal Path: Login as Administrator >> Select Test Patient >> Select Practice Setup >> Select eprescribe >> Click Settings Tab >> Click DUR Settings link >> Adjust Settings >> Save Very 27

Task 13: Record Active Medication List Note: Before testing the participant should review the test patient s active medication list. Optimal Path: Select Test Patient >> Open Visit Slip >> Click Current Medications Very 28

Task 14: Change Active Medication List Note: Before testing the participant should review the test patient s active medication list. Optimal Path: Select Test Patient >> Open Visit Slip >> Click Current Medications >> Click eprescribe >> Click Review History >> Select Medication to change >> Done Very 29

Task 15: Access Active Medication List Note: Before testing the participant should review the test patient s active medication list. Optimal Path: Select Test Patient >> Open Visit Slip >> Click Current Medications >> Active Medication List Displayed Very 30

Task 16: Record Active Medication Allergy List Note: Before testing the participant should review the test patient s active medication allergy list. Optimal Path: Select Test Patient >> Open Visit Slip >> Click Adverse Drug Reactions >> Click Update >> Click pencil icon next to Active Allergies >> Click Add >> Select Medication >> Save Very 31

Task 17: Change Active Medication Allergy List Note: Before testing the participant should review the test patient s active medication allergy list. Optimal Path: Select Test Patient >> Open Visit Slip >> Click Adverse Drug Reactions >> Click Update >> Click pencil icon next to Active Allergies >> Click Edit >> Select Inactive >> Save Very 32

Task 18: Access Active Medication Allergy List Note: Before testing the participant should review the test patient s active medication allergy list. Optimal Path: Select Test Patient >> Open Visit Slip >> Click Adverse Drug Reactions Very 33

Task 19: Create clinical decision support intervention for a problem list Note: Before testing the participant should review the test patient s problem list. Optimal Path: Login >> Administration >> Practice Setup >> General >> Config. Clinical Decision Support >> New >> Diagnosis >> Select ICD >> Save Very 34

Task 20: Create clinical decision support intervention for a medication list Note: Before testing the participant should review the test patient s medication list. Optimal Path: Login >> Administration >> Practice Setup >> General >> Config. Clinical Decision Support >> New >> Medication >> Select Medication >> Save Very 35

Task 21: Create clinical decision support intervention for a medication allergy list Note: Before testing the participant should review the test patient s medication allergy list. Optimal Path: Login >> Administration >> Practice Setup >> General >> Config. Clinical Decision Support >> New >> Medication Allergy >> Select Medication Allergy >> Save Very 36

Task 22: Create clinical decision support intervention for a demographics Note: Before testing the participant should review the test patient s demographics. Optimal Path: Login >> Administration >> Practice Setup >> General >> Config. Clinical Decision Support >> New >> Demographics >> Select Demographics >> Save Very 37

Task 23: Create clinical decision support intervention for a laboratory tests and values/results Note: Before testing the participant should review the test patient s laboratory tests and values/results. Optimal Path: Login >> Administration >> Practice Setup >> General >> Config. Clinical Decision Support >> New >> Lab Results >> Select Lab Test >> Save Very 38

Task 24: Create clinical decision support intervention for vital signs. Note: Before testing the participant should review the test patient s vital signs. Optimal Path: Login >> Administration >> Practice Setup >> General >> Config. Clinical Decision Support >> New >> Vital Signs >> Select Vital Signs >> Save Very 39

Task 25: Trigger clinical decision support intervention for a problem list Note: Before testing the participant should review the test patient s problem list. Optimal Path: Select Test Patient >> Open Visit Slip >> Enter Problem List >> Click Clinical Support to trigger Very 40

Task 26: Trigger clinical decision support intervention for a medication list Note: Before testing the participant should review the test patient s medication list. Optimal Path: Select Test Patient >> Open Visit Slip >> Current Medications >> Click Clinical Support to trigger Very 41

Task 27: Trigger clinical decision support intervention for a medication allergy list Note: Before testing the participant should review the test patient s medication allergy list. Optimal Path: Select Test Patient >> Open Visit Slip >> Medication Allergy List >> Click Clinical Support to trigger Very 42

Task 28: Trigger clinical decision support intervention for demographics Note: Before testing the participant should review the test patient s demographics. Optimal Path: Select Test Patient >> Open Visit Slip >> Demographics >> Click Clinical Support to trigger Very 43

Task 29: Trigger clinical decision support intervention for laboratory tests and values/results Note: Before testing the participant should review the test patient s laboratory tests and values/results. Optimal Path: Select Test Patient >> Open Visit Slip >> Select Lab Tests >> Click Clinical Support to trigger Very 44

Task 30: Trigger clinical decision support intervention for vital signs Note: Before testing the participant should review the test patient s vital signs. Optimal Path: Select Test Patient >> Open Visit Slip >> Vital Signs >> Click Clinical Support to trigger Very 45

Task 31: Electronically prescribe single medication for test patient Note: Before testing the participant should review the test patient s medication. Optimal Path: Select Test Patient >> Open Visit Slip >> Click Current Medications >> Click eprescribe >> Review History >> Click Add Patient Reported Med >> Search & Select Medication >> Save >>Exit Very 46

Task 32: Complete active medication for test patient Note: Before testing the participant should review the test patient s active medications. Optimal Path: Select Test Patient >> Open Visit Slip >> Click Current Medications >> Click eprescribe >> Review History >> Click Medication to Complete >> Click Complete >> Done Very 47

Task 33: Discontinue active medication for a test patient Note: Before testing the participant should review the test patient s active medications to be discontinued. Optimal Path: Select Test Patient >> Open Visit Slip >> Click Current Medications >> Click eprescribe >> Review History >> Click medication to discontinue >> Click Discontinue >> Done Very 48

Task 34: Renew active medication for a test patient Note: Before testing the participant should review the test patient s active medications to be renewed. Optimal Path: Select Test Patient >> Open Visit Slip >> Click Current Medications >> Click eprescribe >> Review History >> Add Patient Reported Medication >> Search & Select Medication >> Click Save >> Exit Very 49

Task 35: Create a single reconciled active medication list Note: Before testing the participant should review the test patient s active medication list. Optimal Path: Select Test Patient >> Open Visit Slip >> Click Current Medications >> Click eprescribe >> Review History >> Add Patient Reported Medications >> Search & Select Medication >> Save >> Exit >> Click Download erx under Current medications to View list Very 50

Task 36: Create a single reconciled active problem list Note: Before testing the participant should review the test patient s active problem list. Optimal Path: Select Test Patient >> Open Visit Slip >> Click Assessment >> Add All Active Problems >> Add New >> Select a problem >> Click Add New >> Exit Very 51

Task 37: Create a single reconciled active medication allergy list Note: Before testing the participant should review the test patient s active medication allergy list. Optimal Path: Select Test Patient >> Open Visit Slip >> Click Adverse Drug Reactions >> Click Update >> Click pencil icon next to active allergies >> Select allergy >> Save >> Done >> In EMR click Adverse Drug Reactions to view list. Very 52

Appendix 3: FINAL QUESTIONS: Q1. What was your overall impression of this system? Q2. What aspects of the system did you like most? Q3. What aspects of the system did you like least? Q4. Were there any features that you were surprised to see? Q5. What features did you expect to encounter but did not see? That is, is there anything that is missing in this application? Q6. Compare this system to other systems you have used. Q7. Would you recommend this system to your colleagues? 53

5.4 Appendix 4: SYSTEM USABILITY SCALE QUESTIONNAIRE: Strongly Disagree Strongly Agree 54

1. I think that I would like to use this system frequently 2. I found the system unnecessarily complex 1 2 3 4 5 3. I thought the system was easy to use 1 2 3 4 5 4. I think that I would need the support of a technical person to be able to use this system 5. I found the various functions in this system were well integrated 6. I thought there was too much inconsistency in this system 7. I would imagine that most people would learn to use this system very quickly 8. I found the system very cumbersome to use 9. I felt very confident using the system 10. I needed to learn a lot of things before I could get going with this system Total Score = 329 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 Average Score = 32.9 (329/10) 55

Average SUS Score = 82.25 (32.9 *2.5) 56

Electronic Health Record Usability Test Report ModuleMD WISE ADDENDUM 6/1/2015 This ADDENDUM is attached to and forms part of the Usability Test Report ModuleMD WISE v8.0 Complete EHR. ENHANCEMENT TO EHR: Addition of e-prescribing ability through DrFirst, creating version 8.1. WORKFLOW ENHANCEMENTS: The following are the workflow enhancements in Safety- Enhanced Design of ModuleMD WISE : 1. 170.314.a.1 CPOE Medication 2. 170.314.0.2 Drug-Drug, Drug-Allergy Interaction Checks 3. 170.314.0.6 Medication List 4. 170.314.3 Electronic Prescribing USABILITY TESTED: Usability of each test criteria was tested from May 1, 2015, through May 31, 2015, in the medical records for multiple test patients. 1. 170.314.a.1 CPOE - Medication a. record medication order b. change medication order c. access medication order 2. 170.314.0.2 Drug-Drug, Drug-Allergy Interaction Checks a. trigger one drug-drug contraindication intervention b. trigger one drug-allergy contraindication intervention 57

c. adjust severity level of interventions indicated for drug-drug interaction 3. 170.314.0.6 Medication List a. record active medication list b. change active medication list c. access active medication list 4. 170.314.3 Electronic Prescribing a. electronically prescribe single medication for test patient b. complete active medication for test patient c. discontinue active medication for test patient PERFORMANCE: 1. Ease of use a. Quicker SSO launch from ModuleMD to DrFirst b. DrFirst dashboard allows for easy navigation c. Return to EMR button saves additional click to log out 2. Issues a. Help button results in Web Page error message 3. Areas for improvement a. No recommendations IMPACT: None. Users will have the choice between 2 e-prescribing applications which will have no impact on reporting the above criteria. 58

Providing solutions...not just software Quality Management System Attestation For reporting information related to testing of 170.314(g)(4) Vendor and Product Information Vendor Name Product Name Product Version Quality Management System ModuleMD, LLC ModuleMD Wise 8.0.0 Type of Quality Management System (QMS) used in the development, testing, implementation, and maintenance of EHR product. X Based on Industry Standard (for example ISO9001, IEC 62304, ISO13485, etc). Standard: A modified or "home- grown" QMS. No QMS was used. Was one QMS used for all certification criteria or were multiple QMS applied? X One QMS was used. Multiple QMS used. Description or documentation of QMS applied to each criteria: Not Applicable. Statement of Compliance I, the undersigned, attest that the statements in this document are accurate. Vendor Signature by an Authorized Representative Date 1/22/2014 ModuleMD LLC. * 8359 Office Park Dr, Grand Blanc, MI 48439 ) 248-434-0444 8 www.modulemd.com

Providing solutions...not just software Privacy and Security Attestation Vendor and Product Information Vendor Name Product Name Product Version Privacy and Security 170.314(d)(2) Auditable events and tamper- resistance ModuleMD, LLC ModuleMD Wise 8.0.0 Audit Log: X Cannot be disabled by any user. Audit Log can be disabled Not Applicable (did not test to this criteria) The EHR enforces that the audit log is enabled by default when initially configured Audit Log Status Indicator: X Cannot be disabled by any user. Audit Log can be disabled X The EHR enforces a default audit log status. Identify the default setting (enabled or disabled): There is no Audit Log Status Indicator because the Audit Log cannot be disabled. Encryption Status Indicator (encryption of health information locally on end user device): X Cannot be disabled by any user. Audit Log can be disabled X The EHR enforces a default encryption status. Identify the default setting (enabled or disabled): There is no Encryption Status Indicator because the EHR does not allow health information to be stored locally on end user devices. Identify the submitted documentation that describes the inability of the EHR to allow users to disable the audit logs, the audit log status, and/or the encryption status: Local Data Storage.pdf ModuleMD LLC. * 8359 Office Park Dr, Grand Blanc, MI 48439 ) 248-434-0444 8 www.modulemd.com

Providing solutions...not just software Identify the submitted documentation that describes the method(s) by which the EHR protects 1) recording of actions related to electronic health information, 2) recording of audit log status, and 3) recording of encryption status from being changed, overwritten, or deleted by the EHR technology: Auditable Events and_tamper- Resistance.pdf Local Data Storage.pdf 170.314(d)(7) End- user device encryption Storing electronic health information locally on end- user devices (i.e. temp files, cookies, or other types of cache approaches). Not Applicable (did not test to this criteria) Identify the submitted documentation that describes the method(s) by which the EHR technology detects whether the audit log has been altered: X The EHR does not allow health information to be stored locally on end- user devices. Identify the submitted documentation that describes the functionality used to prevent health information from being stored locally: Auditable Events and_tamper- Resistance.pdf Local Data Storage.pdf The EHR does not allow health information to be stored locally on end- user devices. Identify the FIPS 140-2 approved algorithm used for encryption: Identify the submitted documentation that describes how health information is encrypted when stored locally on end- user devices: 170.314(d)(8) Integrity X Not Applicable (did not test to this criteria) X The EHR enforces default configuration settings that either enforces the encryption of locally stored health information or prevents health information from being stored locally. Identify the default setting: Identify the hashing algorithm used for integrity (SHA- 1 or higher): Statement of Compliance I, the undersigned, attest that the statements in this document are accurate. Vendor Signature by an Authorized Representative Date 1/22/2014 ModuleMD LLC. * 8359 Office Park Dr, Grand Blanc, MI 48439 ) 248-434-0444 8 www.modulemd.com