The Usability Certification Quandary in Healthcare Presented to: 2012 Symposium on Human Factors and Ergonomics in Health Care: Bridging the Gap Presented by: Dick Horst UserWorks, Inc. March 13, 2012 0
Certification ala Usability Seal of approval Historical aversion But market keeps demanding it Some areas of current activity: Medical devices Voting systems Programmable thermostats Electronic health records My motivation 1
Current context for usability of EHRs Impetus provided by Recovery Act funding for Electronic Health Records Center for Medicare and Medicaid Services incentive program to reimburse healthcare organizations for adoption of EHRs Meaningful Use Usability discipline continuing to mature; at least in certain areas (web design, aviation, military systems) becoming part of the process Industry, government regulatory bodies considering whether and how usability should be part of the process of evaluating and certifying EHRs 2
3
Three stages of meaningful use Stage 1 (which began in 2011 and remains the starting point for all providers): meaningful use consists of transferring data to EHRs and being able to share information, including electronic copies and visit summaries for patients. Stage 2 (to be implemented in 2014 under the proposed rule): meaningful use includes standards such as online access for patients to their health information and electronic health information exchange between providers. Stage 3 (expected to be implemented in 2016): meaningful use includes demonstrating that the quality of health care has been improved. 4
Meaningful Use: 15 Core Objectives for Eligible Professionals 1. Computerized provider order entry (CPOE) 2. E-Prescribing (erx) 3. Report ambulatory clinical quality measures to CMS/States 4. Implement one clinical decision support rule 5. Provide patients with an electronic copy of their health information, upon request 6. Provide clinical summaries for patients for each office visit 7. Drug-drug and drug-allergy interaction checks 8. Record demographics 9. Maintain an up-to-date problem list of current and active diagnoses 10. Maintain active medication list 11. Maintain active medication allergy list 12. Record and chart changes in vital signs 13. Record smoking status for patients 13 years or older 14. Capability to exchange key clinical information among providers of care and patient-authorized entities electronically 15. Protect electronic health information 5
Meaningful Use: 14 Core Objectives for Hospitals 1. Computerized provider order entry (CPOE) 2. Drug-drug and drug-allergy interaction checks 3. Record demographics 4. Implement one clinical decision support rule 5. Maintain up-to-date problem list of current and active diagnoses 6. Maintain active medication list 7. Maintain active medication allergy list 8. Record and chart changes in vital signs 9. Record smoking status for patients 13 years or older 10. Report hospital clinical quality measures to CMS or States 11. Provide patients with an electronic copy of their health information, upon request 12. Provide patients with an electronic copy of their discharge instructions at time of discharge, upon request 13. Capability to exchange key clinical information among providers of care and patient-authorized entities electronically 14. Protect electronic health information 6
A role for usability? Stage 1 not yet Activity at ONC, NIST, ARHQ CCHIT pilot program Advocacy by HFES Stage 2 Meaningful Use Notice of Public Rule Making appeared in the Federal Register on March 7; comments due May 7, 2012 Stage 2 --? Stage 3 --??? 7
But not everyone is onboard Industry concerns: Cost of implementation, return on investment Stifling innovation Already doing usability engineering; don t need more regulations Reliability of usability measures Government concerns; Usability is in the eye of the beholder Is usability a science? Is usability practice mature enough to support a certification process? 8
EHR Usability Hearing 9
Challenges in evaluating usability of EHRs Variety of settings, user roles, tasks From sole practitioners to paperless hospitals From temporary help to attending physicians and health system managers Telemedicine Variety of products, functionality eprescriptions Medical histories, record keeping Billing Etc, etc, etc Variety of platforms, media Lab to field transition Work settings can be chaotic distractions, interruptions Team performance not just individuals Interoperability among systems, interfaces with devices Need to consider workflows, not just screen designs 10
What has been done thus far Agency for Healthcare Research and Quality (AHRQ) reports Electronic Health Record Usability: Evaluation and Use Case Framework Electronic Health Record Usability: Interface Design Considerations EHR Usability: Vendor Practices & Perspectives NIST reports (NISTIR 7769) Human Factors Guidance to Prevent Healthcare Disparities with the Adoption of EHRs (NISTIR 7741) NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records (NISTIR 7742) Customized Common Industry Format Template for Electronic Health Record Usability Testing (NISTIR 7743) Usability in Health IT: Technical Strategy, Research, and Implementation (NISTIR 7804) Technical Evaluation, Testing and Validation of the Usability of Electronic Health Records NIST Conferences on usability of EHRs Office of the National Coordinator (ONC) policy committee hearing Certification Commission for Health Information Technologies (CCHIT) usability certification of ambulatory medical systems 11
CCHIT Usability Rating Model Trial basis -- Usability is rated as part of the CCHIT Certified 2011 Ambulatory EHR Certification Program Utilizes the clinical juror observations during the regular inspection process (about 30-40 minutes) to gather data Jurors answer a series of questionnaires based on observations After Scenario Questionnaire (ASQ) jurors rate perceived efficiency (time and effort), learnability, and confidence after viewing scenarios 4 questions after each scenario 16 overall Strongly Disagree Strongly Agree 1-5 Perceived Usability Questionnaire (PERUSE) jurors rate screen-level design attributes based on reasonably observable characteristics 20 questions divided among each of the scenarios; Jurors are allowed to revisit answers to these questions Strongly Disagree Strongly Agree 1-4 (no mid-point) System Usability Survey (SUS) jurors rate the assessment of usability, and satisfaction with the application 10 questions after all four scenarios have been demonstrated Strongly Disagree Strongly Agree 1-5 12
Ongoing efforts that will push the envelope NIST Usability Framework project AHRQ usability toolkit project Strategic Health IT Advanced Research Projects ( SHARP ) R&D projects from the Office of the National Coordinator 13
Lessons from other usability certification efforts? NIST initiative on usability of voting systems Energy Star initiative on usability certification of programmable thermostats Human Factors International proprietary usability certification of products Others? 14
Usability certification for programmable thermostats 15
HFI certification of products 16
What needed in general re usability evaluation of EHRs Win hearts and minds! Convince industry and government regulatory bodies that usability measurement is scientifically valid and reliable Research to continue to refine usability measures Agree on procedures, protocols to increase reliability (a cookbook that is not a cookbook) Research on methods, tools to manage the user-centered design process Demonstrate (publicize) return on investment Usability Professionals Association initiative on ROI and making the business case Mike Wiklund book, Usability Testing of Medical Devices (but focused on devices, not EHRs) But much more needed Do we need: EHR-specific design guidelines? Accreditation of testing entities? Pre-market application; post-market surveillance? 17
What needed specifically re EHR usability evaluation Life cycle perspective; user-centered design approach, iterative testing At least late in development, test products where they live (in the clinical setting with representative users) Focus on user workflow as much as screen design Focus not only on task scenarios that pertain to a given workstation but on the flow of information between them Between user roles Between product modules Develop tools to facilitate the usability evaluation process Managing changes, implementation of recommendations Modeling workflow of a particular workplace Managing EHR customization for a given workplace without invalidating previous, generic certification 18