University of Michigan Health System Adult and Children s Emergency Departments

Transcription

1 University of Michigan Health System Adult and Children s Emergency Departments Efficiency and Effectiveness of Electronic Health Record (EHR) Use in the Emergency Department Final Report To: Jeffrey Desmond, M.D., Client - Associate Chair for Clinical Operations, UMHS Emergency Department Mary Duck, Coordinator - Senior Management Consultant, UMH Program and Operations Analysis Alex Lai, Coordinator - IOE 481 Graduate Student Instructor From: IOE 481 Project Team 9 Sarah Bach Samuel Pettinato Megan Taylor Date: April 23, 2013 i

2 Table of Contents Executive Summary 1 Goals and Objectives..1 Methods...1 Checklists 1 Surveys 1 Focus Groups..2 Findings..2 Checklists 2 Surveys 2 Focus Groups..2 Conclusions.3 Determining a Methodology...3 Assessing MiChart Usage...3 Improving MiChart Usage..3 Recommendations...3 Introduction.4 Background.4 Key Issues...4 Goals and Objectives..5 Project Scope..5 Methods...5 Shortcomings of Task Timing 5 ii

3 Checklists. 6 Surveys. 6 MiChart Self-Assessments...7 Focus Groups... 8 Findings. 8 Shortcomings of Task Timing...8 Checklists..8 Surveys..10 Nurse: MiChart User Levels.10 Nurse: Most and Least Used Features..11 Nurse: Correlations...12 Nurse: Training Preferences.13 Physician Assistant: MiChart User Levels...14 Physician Assistant: Most and Least Used Features.14 Physician Assistant: Correlations.15 Physician Assistant: Training Preferences 16 Physician Resident: MiChart User Levels 16 Physician Resident: Most and Least Used Features.17 Physician Resident: Correlations..18 Physician Resident: Training Preferences 19 Physician Faculty: MiChart User Levels..19 Physician Faculty: Most and Least Used Features...20 Physician Faculty: Correlations 21 iii

4 Physician Faculty: Training Preferences..22 Comparison between Roles...22 Focus Groups 23 Conclusions...24 Determining a Methodology.24 Assessing MiChart Usage.24 Top 3 Defining MiChart Features.24 Improving MiChart Usage 24 Recommendations.25 Expected Impact 25 Appendix A: Epic s Post-Live Checklist for Nurses 26 Appendix B: Epic s Post-Live Checklist for Providers 29 Appendix C: MiChart Usage Survey for Nurses..33 Appendix D: MiChart Usage Survey for Providers..39 Appendix E: Nurse Focus Group Summary and Transcript.46 Appendix F: Provider Focus Group Summary and Transcript.51 Appendix G: Nurse MiChart Self-Assessment Feature Usage Rankings.56 Appendix H: Physician Assistant MiChart Self-Assessment Feature Usage Rankings...60 Appendix I: Resident MiChart Self-Assessment Feature Usage Rankings..64 Appendix J: Faculty MiChart Self-Assessment Feature Usage Rankings 68 Appendix K: Recommended Training Focuses 73 Appendix L: Top 3 Defining Features for Each Role iv

5 List of Tables and Figures Figure 1. Most Used Advanced MiChart Usage Features among Nurses 9 Figure 2. Most Used Advanced MiChart Usage Features among Providers 9 Figure 3. Overall ED MiChart User Level Breakdown 10 Figure 4. Histogram of Nurse MiChart Self-Assessment Scores.11 Figure 5. Nurse MiChart User Level Breakdown.11 Figure 6. Nurse Training Method Preferences by Age Group..13 Figure 7. Histogram of Physician Assistant MiChart Self-Assessment Scores 14 Figure 8. Physician Assistant MiChart User Level Breakdown...14 Figure 9. Histogram of Physician Resident MiChart Self-Assessment Scores 16 Figure 10. Physician Resident MiChart User Level Breakdown..17 Figure 11. Resident Training Method Preferences by Age Group...19 Figure 12. Histogram of Faculty MiChart Self-Assessment Scores.19 Figure 13. Faculty MiChart User Level Breakdown 20 Figure 14. Faculty Training Method Preferences by Age Group.22 Figure 15. ED MiChart User Level Breakdown by Role..22 Figure 16. ED MiChart User Level Breakdown by Role..23 Table 1. MiChart User Classification Scale 7 Table 2. Survey and MiChart Self-Assessment Response Rates for all Roles.10 Table 3. Most and Least Used MiChart Features by Nurses 12 Table 4. Nurse User Levels with Previous EHR Experience 12 Table 5. Nurse Actual User Level vs. Perceived User Level Table 6. Nurse Perceived User Level vs. Actual User Level v

6 Table 7. Most and Least Used MiChart Features by Physicians Assistant...15 Table 8. Physician Assistant User Levels with Previous EHR Experience..15 Table 9. Physician Assistant Actual User Level vs. Perceived User Level...16 Table 10. Physician Assistant Perceived User Level vs. Actual User Level 16 Table 11. Most and Least Used MiChart Features by Residents..17 Table 12. Resident User Levels with Previous EHR Experience.18 Table 13. Resident Actual User Level vs. Perceived User Level. 18 Table 14. Resident Perceived User Level vs. Actual User Level. 18 Table 15. Most and Least Used MiChart Features by Faculty.20 Table 16. Faculty User Levels with Previous EHR Experience...21 Table 17. Faculty Actual User Level vs. Perceived User Level...21 Table 18. Faculty Perceived User Level vs. Actual User Level...21 vi

7 EXECUTIVE SUMMARY The University of Michigan Health System (UMHS) Emergency Department (ED), both Adult and Children s divisions, uses an electronic health record (EHR) system to record patient information. MiChart-ASAP (MiChart), UMHS s customized version of the healthcare software provider Epic s EHR, was implemented within the ED on June 15, The Associate Chair for Clinical Operations, project client, in the UMHS ED was interested in determining a methodology to assess and improve MiChart usage efficiency and effectiveness. Thus, this study focused on assessing the efficiency and effectiveness of MiChart use in the UMHS Adult and Children s ED. Yet, this study recommends a methodology for MiChart usage assessment that can be generalized to other departments throughout UMHS as MiChart continues to be implemented. Lastly, this study recommends additional training measures that can be implemented to improve MiChart usage. Goals and Objectives The primary goal of this project was to determine a methodology to assess and improve efficiency and effectiveness of MiChart usage in the UMHS ED. Secondary project goals were: Improve efficiency and effectiveness of MiChart use within the ED Standardize methodology used to measure MiChart efficiency and effectiveness Develop findings that can be applied to improve MiChart training Methods Various methods were tested to determine a methodology to assess and improve MiChart usage effectiveness and efficiency. Task timing was one of the methods tested, but it resulted in insignificant findings. A detailed report on the method of task timing can be found in the full report. All other methods used are summarized below. Checklists The team used Epic s Post-Live Checklist for Nurses and Post-Live Checklist for Providers to determine which advanced features of MiChart were utilized by nurses and providers. These checklists each list 70 to 80 advanced tasks that MiChart users should be able to implement and can be found in Appendix A and B, respectively. Using the checklists, the team shadowed 10 providers in the Main area of the ED for 2 to 3 hour time periods, resulting in a total of 28 hours of provider data collection. The team shadowed 18 different nurses in the Main area of the ED for 3 to 4 hour time periods, resulting in a total of 46 hours of nurse data collection.. The team used the checklist results to determine which advanced MiChart features were most used. The 15 to 20 most used MiChart features were included in the MiChart Self-Assessments created by the team and included in the surveys. Surveys The MiChart Self-Assessments were distributed to nurses, physician assistants, residents, and faculty throughout both the Adult and Children s ED in an online survey created through Qualtrics. The surveys contained 30 to 35 questions including questions focused on the user s 1

8 background, MiChart training preferences, general MiChart opinions, and the MiChart Self- Assessments. The main goal of the surveys was to quickly and objectively identify MiChart user levels. User levels were determined by the following scale: scores above 75% were identified as high functioning users, scores 50 to 75% were identified as medium functioning users, and scores less than 50% were identified as low functioning users. The survey results also were used to identify advanced MiChart features that are most and least commonly used. Least used advanced MiChart features were recommended as focuses in additional training. Additional training methods were recommended based on results from the training portion of the survey. Lastly, the survey results identified any correlations present between user level and perceived user level, age, or previous EHR experience. Focus Groups The team led two focus groups, one for nurses and one for providers, to discuss MiChart usage strategies. These sessions were used to identify usage root causes that affect efficiency and effectiveness and to understand why users use MiChart the way they do. The focus group discussion questions were created by the team with input from the Associate Chair for Clinical Operations. Findings Using the methods and analysis techniques described above, the following results were found. Checklists The most used advanced features observed from Epic s Post-Live Checklists for Nurses and Providers were calculated using Microsoft Excel. The 15 to 20 most used advanced features determined from this checklist observation were included in the MiChart Self-Assessments. Figures containing specific advanced feature observation data are included in the full report. Surveys Through online distribution of the surveys the following response rates were achieved: 31% of Nurses, 50% of Physician Assistants, 84% of Physician Residents, and 72% of Physician Faculty. The overall user level breakdown of the ED survey respondents was found to be: 14% low users, 74% medium users, and 12% high users. Each role within MiChart had a user level breakdown similar to the overall user level breakdown; however, findings by role can be seen in the full report. Further data regarding each role s most and least used features, correlations between user level and perceived user level, age, and previous EHR experience, and training preferences are also presented in the full report. Focus Groups The nurse and provider focus groups had 9 nurses and 8 providers in attendance, respectively. Generally, the team found that nurses were not familiar with customization and advanced 2

9 features, were eager for additional training opportunities, and still had difficulty with intake and output and blood documentation. Providers were quite familiar with customization and advanced features, were interested in additional optional training materials, and felt they could improve efficiency through real time dictation. Complete summaries and transcripts of the nurse and provider focus groups are provided in Appendix E and F, respectively. Conclusions The primary project goal was to determine a methodology to assess and improve MiChart usage efficiency and effectiveness. Thus, conclusions were developed for the following 3 main topics of determining a methodology, assessing MiChart usage, and improving MiChart usage. Determining a Methodology The team concluded that the method of task timing, which involved shadowing users and timing common tasks performed in MiChart, is not an appropriate method for this study. The MiChart Self-Assessments created, with influence from the Epic Post Live Checklists, did accurately identify MiChart user levels. Assessing MiChart Usage Across all MiChart user roles it can be concluded that a majority of MiChart users are medium functioning users. Additionally, all other users are nearly evenly distributed between high functioning and low functioning users. It was found that user level is not strongly correlated with perceived user level, age, or previous EHR experience, both Epic or otherwise. Improving MiChart Usage To improve MiChart usage different training methods and areas of focus are preferred and needed between different roles. However, overall it was concluded that customization features and MiChart Tip Sheets and Videos are not utilized to their fullest potential. Recommendations The team recommends that this methodology to assess MiChart usage be continued in order to document progress made with MiChart usage improvements. The team also recommends that further MiChart training be implemented for each role as outlined below: Nurses: Classroom sessions with instructors every 6 months to 1 year Physician Assistants and Residents: Presentations during scheduled meetings every 6 months to 1 year Physician Faculty: Mentoring session with colleagues every 6 months Features to be addressed during training should be determined by the combined metric of less than 50% observed usage and an average score of less than 3 on the MiChart Self-Assessment. Features that fit this metric can be found in Appendix K. 3

10 Introduction Use of Electronic Health Records (EHR) is becoming widespread throughout hospitals today. The University of Michigan Health System (UMHS) uses software developed by the healthcare software provider Epic as their form of an EHR. Epic currently holds about 40% of the EHR market share and has created an EHR software package that is highly customizable to the specific institution in which it is implemented. The UMHS Emergency Department (ED) implemented MiChart-ASAP (MiChart), a customized version of Epic s EHR, in June However, the MiChart roll out has not been completed for all of UMHS. Nurses and providers received MiChart training, but it was unknown whether they are using MiChart as trained and how effectively they are using MiChart. Therefore, the Associate Chair for Clinical Operations in the UMHS ED asked the IOE 481 student project team from the University of Michigan to determine a methodology to assess and improve MiChart usage efficiency and effectiveness in the UMHS ED. The developed methodology was created with the goal of being general enough to be implemented throughout UMHS once the MiChart roll out is completed. Additionally, the team was asked to identify current weaknesses of MiChart usage and to recommend improvements to training. The purpose of this final report is to present the student team s methods, findings, conclusions, and recommendations. Background The UMHS ED, both Adult and Children s divisions, at the University of Michigan uses an EHR system to record patient information. The UMHS chose to standardize their EHR system and selected Epic healthcare software in This comprehensive Epic system is used by both nurses and providers and collects demographic information and clinical information; it also sends and receives orders, tests, and results. MiChart, UMHS s customized version of Epic, was implemented within the ED on June 15, 2012; however, has yet to be implemented throughout all of UMHS. All staff received 8-14 hours of training on the MiChart system prior to its implementation; however, a learning curve still exists. While the team was informed that Deloitte consultants had partnered with UMHS to show the workload and process flow benefits of EHR implementation, no studies had specifically focused on effectiveness and efficiency of MiChart usage itself. Thus, this study sought to assess and improve MiChart usage efficiency and effectiveness in the UMHS ED, both Adult and Children s divisions. This study also recommends a methodology for MiChart usage assessment that can be generalized to other departments throughout UMHS as MiChart continues to be implemented. Lastly, this study recommends additional training that can be implemented to improve MiChart usage. Key Issues The issues that created the need for this project were as follows: Unknown how MiChart is being used within the ED, thus standardized work may or may not exist No methodology exists to measure the efficiency and effectiveness of MiChart usage Overall efficiency could be improved if the gaps between MiChart training and actual usage were measured, understood, and improved upon 4

11 Goals and Objectives The primary goal of this project was to determine a methodology to assess and improve efficiency and effectiveness of MiChart usage in the UMHS ED. Secondary project goals were: Improve efficiency and effectiveness of MiChart use within the ED Standardize methodology used to measure MiChart usage efficiency and effectiveness that can be applied to other departments as MiChart is implemented Develop findings that can be applied to improve MiChart training Project Scope This study focused solely on use of MiChart within the UMHS ED. The study consisted of observations and checklists completed in the Adult ED and surveys and focus groups regarding MiChart use performed in both the Adult and Children s ED. Any other department of the UMHS was not analyzed in this study; however, the methodology developed from this study is able to be generalized and implemented throughout all of UMHS. While the Children s ED staff was invited to participate in the survey and focus groups, no observations of the Children s ED were performed. Lastly, no analysis of how MiChart usage relates to quality of care was performed. Methods The methods used to complete the project goal of assessing and improving MiChart usage efficiency and effectiveness are as follows. The methodology used to complete this study affected the follow departments: UMHS Adult Emergency Department UMHS Children s Emergency Department Shortcomings of Task Timing The team shadowed 18 nurses in the Main area of the ED for 3 to 4 hour time periods, resulting in a total of 46 hours of data collection. These hours covered daytime shifts on Monday through Friday from February 13, March 1, The team observed all MiChart use by the shadowed nurse both at his or her desk and at the patient s bedside. The team observed all tasks in the MiChart system and recorded the time taken to complete each task. The main task categories observed included: Chart review for patient information Triage assessment and documentation Order placement Care documentation Medication administration Procedure documentation Results review 5

12 Originally, the team planned to compare these task times across MiChart users to classify users into high functioning, medium functioning, and low functioning user groups. These user groups would be analyzed to determine which functions and usage strategies led to increased effectiveness and efficiency between user groups. However, through task timing the team did not observe enough tasks to have statistically relevant data. To increase data collection, the team extended nurse shadowing and task timing from one to two weeks; however, the quantity of data needed was still not reached. At least 10 data points for each task per user would be needed to statistically compare differences across users. The team collected on average 19 data points per observed user when at least 80 data points per observed user would be needed for comparison. Therefore, the results of this data were not the main focus of the project, but did eliminate task timing as a potential method to measure MiChart usage efficiency and effectiveness. Thus, as the team s goal was to determine a methodology to assess and improve MiChart usage efficiency and effectiveness, different methods were implemented. Checklists The team used Epic s Post-Live Checklist for Nurses and Post-Live Checklist for Providers to determine which advanced features of MiChart are utilized. These checklists were created by Epic and therefore include advanced features of the Epic EHR system as determined by Epic. These checklists each list advanced features that MiChart users should be able to implement. Copies of both the Nurse and Provider Post-Live Checklists are provided in Appendix A and B, respectively. The appropriate checklist, nurse or provider, was completed by the team for each MiChart user observed. Using the checklists, the team shadowed 10 providers in the Main area of the ED for 2 to 3 hour time periods, resulting in a total of 28 hours of data collection from March 11-15, The team used data collected from the previous two weeks of nurse task timing observations to complete the checklist for nurses observed. As needed, the team followed up with previously observed nurses to complete missing checklist information. Using the checklist results, the team computed the percentage of observed users who implemented each advanced MiChart feature with Microsoft Excel to determine which features were most used. The 15 to 20 most used MiChart features found from Epic s Post-Live Checklists were included in the MiChart Self-Assessments created by the team. These MiChart Self-Assessments were distributed to all MiChart users to quickly and objectively identify high functioning, medium functioning, and low functioning user groups. Surveys The MiChart Self-Assessments were distributed to nurses and providers throughout the ED in an online survey created through Qualtrics. The surveys were electronically distributed to all nurses, physician assistants, residents, and faculty in the ED and were available from March 25, 2013 April 8, The surveys contained 30 to 35 questions including the MiChart Self- Assessments. Questions not related to the MiChart Self-Assessment were divided into the following categories: Background: Addressed user s role, age, and previous electronic health record usage 6

13 Training : Addressed both previous training received and preferences regarding future training methods and topics General MiChart Opinions: Allowed the user to express any likes, dislikes, or additional comments and questions regarding MiChart MiChart Self-Assessments MiChart Self- Assessments included the most used features as identified by the team s results, unused features which may provide opportunity for efficiency improvement, and unobserved features deemed essential by the Associate Chair for Clinical Operations. The MiChart Self- Assessments included 20 to 30 advanced usage questions. The survey responders reported their use of each feature on the following 0-5 scale: 0- I do not know how to use this feature, 1- Never, 2-Rarely, 3-Sometimes, 4-Frequently, 5-Always. All survey questions were selected through discussion between the team and the client. The entire survey was also reviewed by Barry DeCicco, Statistician for UMHS, who has extensive experience with survey creation and analysis. The Nurse and Provider Surveys are provided in Appendix C and D, respectively. The main goal of the surveys was to quickly and objectively identify MiChart user levels. Therefore, the MiChart Self-Assessment results were analyzed to determine the user level classification of survey respondents. Each survey respondent s total points on the MiChart Self- Assessment were calculated with Microsoft Excel. Using the total number of available points, based on the number of questions in the MiChart Self-Assessment, each survey respondent s percentage was calculated with Microsoft Excel. User group classification was determined by the scale shown in Table 1. Table 1. MiChart User Classification Scale MiChart User Classification Score on MiChart Self-Assessment (%) High Functioning User Over 75% Medium Functioning User 50-75% Low Functioning User Under 50% In addition to determining MiChart user group classifications, the most and least commonly used advanced MiChart features were identified from the survey results. Using Microsoft Excel, the mean response value for each advanced MiChart feature question was calculated. The most and least used advanced MiChart features were calculated for all MiChart users, each role within MiChart, and each MiChart user level within each role. Least used advanced MiChart features were recommended as focuses in additional training. Additional training methods were 7

14 recommended based on results from the training portion of the survey. For each role, the team determined the most popular training method and frequency of training option. Survey results were also analyzed with Microsoft Excel to determine any correlations present between MiChart user classification and perceived user level, age, or previous electronic health record experience. Any frequently expressed sentiment in the general opinions section was summarized and reported. Focus Groups The team led two focus groups, one for nurses and one for providers, to discuss MiChart usage strategies. These sessions were used to identify usage root causes that affect efficiency and effectiveness and to understand why users use MiChart the way they do. The focus group discussion questions were created by the team with input from the Associate Chair for Clinical Operations. The Provider Focus Group took place on Monday, March 25, 2013 from 3:30PM- 4:30PM and the Nurse Focus Group took place on Wednesday, March 27, 2013 from 9AM- 10AM. The team summarized the opinions found from the focus groups. Findings The following results were found after using the methods and analysis techniques described above to assess and improve MiChart usage efficiency and effectiveness. Shortcomings of Task Timing Task timing, as described above, did not result in enough data to statistically analyze. The times found were also found to be highly variable. Due to the many different ways to perform the same function within MiChart task times were not representative of MiChart functionality level. For example, when nurses conduct the initial assessment they can either perform the assessment and then return to their computer to enter the data into MiChart or enter the data into MiChart at the patient s bedside while performing the assessment. The times varied from 2.2 minutes to 9.6 minutes. In addition, these times were not indicative of MiChart efficiency because charting at the patient s bedside can lead to higher overall process efficiency as the nurse does not have to walk back to their desk to complete the chart. Therefore, task timing was found not to be an appropriate method for assessing MiChart usage efficiency and effectiveness. Checklists The team calculated the most used advanced features from Epic s Post-Live Checklists by determining what percentage of observed users implemented each feature. The most used advanced features as obtained from data of 18 nurses and 10 providers are shown in Figure 1 and 2, respectively. 8

15 Uses SmartPhrases Can search for reports in SnapShot Uses OrderSets Uses time and date shortcuts Searches for things by completion matching Can access her dashboard to view reports. Validates vitals Presses TAB instead of using the mouse. Searches for medications by first few letters Presses ENTER instead of clicking ACCEPT Reviews labs using flowsheets or graphs. Sorts by clicking column headers Adds comments to findings 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Figure 1. Most Used Advanced MiChart Usage Features among Nurses (Source: IOE 481 Team Observations, Sample Size: 18 Nurses, Dates: Feb. 13- March 1, 2013) Uses the Facility List and Database Lookup tabs. Understands standing and future orders. Knows naming conventions for SmartTools. Can search for reports in SnapShot Finds diagnoses by completion matching Can order a facility-administered medication. Can view patient images on the Media tab Uses SmartPhrases and SmartLinks in charting Adds comments to findings Reviews results using flowsheets or graphs. Searches by completion matching on partial words. Sorts by clicking column headers Presses ENTER instead of clicking ACCEPT Percentage of Nurses Observed 0% 10% 20% 30% 40% 50% 60% 70% 80% 90%100% Percentage of Providers Observed Figure 2. Most Used Advanced MiChart Usage Features among Providers (Source: IOE 481 Team Observations, Sample Size: 10, Dates: March 11-15, 2013) 9

16 Surveys Through distribution of the MiChart Usage Surveys developed by the team through Qualtrics the following response numbers and rates were obtained are shown in Table 2. Table 2. Survey and MiChart Self-Assessment Response Rates for all Roles (Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 147, Dates: March 25-April 8, 2013) Role Number of Responses Response Rate % of Total Nurse 75 31% 51% Physician Assistant 9 50% 6% Physician Resident 27 84% 18% Physician Faculty 36 72% 25% Total % All of the above response rates met or exceeded the Associate Chair for Clinical Operations expectations for response rates. Therefore, these response rates are high enough to ensure that the data provides an accurate representation of each group. Overall, the MiChart user level breakdown across all ED survey respondents is shown in Figure 3. High 12% Low 14% Medium 74% Figure 3. Overall ED MiChart User Level Breakdown (Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 147, Dates: March 25-April 8, 2013) The team analyzed the data further by each role (Nurse, Physician Assistant, Physician Resident, and Physician Faculty) as each role uses MiChart differently. Nurse: MiChart User Levels Findings from the Nurse MiChart Self-Assessment scores resulted in a mean of 62% with a standard deviation of 11%. The maximum and minimum scores were 85% and 35%, respectively. A histogram of the Nurse MiChart Self-Assessment scores, shown in Figure 4, displays the concentration of scores in the range of 50-70%. 10

17 30 25 Frequency MiChart Self-Assessment Score (%) Figure 4. Histogram of Nurse MiChart Self-Assessment Scores (Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 75, Dates: March 25-April 8, 2013) The Nurse MiChart user level breakdown was very similar to the overall ED MiChart User Level Breakdown and is shown in Figure 5. High: 13% Low: 12% Medium: 75% Figure 5. Nurse MiChart User Level Breakdown (Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 75, Dates: March 25-April 8, 2013) Nurse: Most and Least Used Features The average score received by each feature was calculated in Microsoft Excel to understand what features of MiChart are most and least used. The average score was calculated across all Nurse survey respondents with the following 0 to 5 scale: 0- I do not know how to use this feature, 1- Never, 2-Rarely, 3-Sometimes, 4-Frequently, 5-Always. The five most and least used features by Nurses, along with their average score, are shown in Table 3. 11

18 Table 3. Most and Least Used MiChart Features by Nurses (Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 75, Dates: March 25-April 8, 2013) Average Most Used: Score I add medications to the patient s home medication list when appropriate 4.5 I use OrderSets for NIPCO's. I select only the appropriate items and can add orders on the fly. 4 I edit/correct my documentation when needed before completing a patient s chart. 3.9 I delete medications from the patient s home medication list when appropriate. 3.8 I review my documentation before completing a patient s chart. 3.7 Least Used: I search for things like orders, reports, and recipients by completion matching on partial words. 1.6 I use the Trauma and Code Narrator solely to document trauma cases without taking notes on paper. 1.5 I use SmartPhrases and SmartLinks in charting. 1.3 I create personal SmartPhrases. (Also known as dot phrases.) 1.1 I access MiChart-ASAP Tip Sheets and Videos if I have questions. 0.9 In addition, the team ranked the entire feature checklist from the MiChart Self-Assessment both overall and by user level within the role of Nurses. These lists of most and least used features can be found in Appendix G. Nurse: Correlations The team investigated possible correlations within the survey data. Specifically, it was hypothesized that a correlation may exist between MiChart user level and age or previous EHR experience, both with Epic or other software. The correlation coefficient between age and MiChart user level was found to be The percentage of Nurses of each user level with previous EHR experience is shown in Table 4. Table 4. Nurse User Levels with Previous EHR Experience (Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 75, Dates: March 25-April 8, 2013) User Level EHR Experience (Epic) EHR Experience (Other) Low (9) 11% (1) 89% (8) Medium (56) 5% (3) 91% (51) High (10) 30% (3) 100% (10) The team also investigated whether a correlation existed between a users actual user level and what they perceived their user level to be. Survey responders were asked if they believed themselves to be a low, medium, or high functioning user. The resulting percentages of each user 12

19 type and what they perceived their user level to be are shown in Table 5. The percentages of each perceived user level group s actual user level is shown in Table 6. Table 5. Nurse Actual User Level vs. Perceived User Level (Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 75, Dates: March 25-April 8, 2013) User Level: Low (Perceived) Medium (Perceived) High (Perceived) Low (Actual) (9) 22% (2) 78% (7) 0% Medium (Actual) (56) 5% (3) 82% (46) 13% (7) High (Actual) (10) 0% 40% (4) 60% (6) Table 6. Nurse Perceived User Level vs. Actual User Level (Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 75, Dates: March 25-April 8, 2013) User Level: Low (Actual) Medium (Actual) High (Actual) Low (Perceived) (5) 40% (2) 60% (3) 0% Medium (Perceived) (57) 12% (7) 81% (46) 7% (4) High (Perceived) (13) 0% 54% (7) 46% (6) Nurse: Training Preferences In response to the training frequency questions, the team found that of Nurses surveyed 39% believed training should be offered annually and 31% believed training should occur every 6 months. It was also found that 27% of Nurses preferred the training method of a classroom session with a training instructor and 23% preferred elective, self-directed e-learning online resources. Training preferences did vary with age as seen in Figure 6. 50% 45% 40% 35% 30% 25% 20% 15% 10% 5% 0% Classroom session with training instructor Presentations during scheduled meetings Mentoring session with colleagues Elective, self-directed e-learning online resources Years Old Years Old Years Old Figure 6. Nurse Training Method Preferences by Age Group (Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 75, Dates: March 25-April 8, 2013) 13

20 Physician Assistant: MiChart User Levels The Physician Assistant MiChart Self-Assessments had a mean score of 60% and a standard deviation of 12%. The maximum score was 85% and the minimum score was 46%. A histogram of the Physician Assistant MiChart Self-Assessments scores is shown in Figure Frequency MiChart Self-Assessment Score (%) Figure 7. Histogram of Physician Assistant MiChart Self-Assessment Scores (Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 9, Dates: March 25-April 8, 2013) The Physician Assistant MiChart user level breakdown, shown in Figure 8, was very similar to the overall ED MiChart User Level Breakdown. High: 11% Low: 11% Medium: 78% Figure 8. Physician Assistant MiChart User Level Breakdown (Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 9, Dates: March 25-April 8, 2013) Physician Assistant: Most and Least Used Features The average score for each MiChart feature was calculated across all Physician Assistant survey respondents, using the following scale 0 to 5 scale: 0- I do not know how to use this feature, 1- Never, 2-Rarely, 3-Sometimes, 4-Frequently, 5-Always. The five most and least used features by Physician Assistants, along with their average score, are shown in Table 7. 14

21 Table 7. Most and Least Used MiChart Features by Physicians Assistant (Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 9, Dates: March 25-April 8, 2013) Average Most Used: Score I find and view patient images and scans on the Media tab in Chart Review. 4 When using NoteWriter, I use the Review of Systems function. 3.9 When using NoteWriter, I use the Physical Exam tab. 3.9 I use the Facility List and Database Lookup tabs to find orders that aren t on my preference list. 3.5 I trend a patient s previous lab results using tools such as flowsheets, graphs, or results. 3.4 Least Used: I use the history tab to enter and/or edit patients past family history. 1.7 I delete medications from the patient s home medication list when appropriate. 1.7 I use time and date shortcuts, such as n, t-1, m+2, and y I add orders to my preference list. I can edit orders display names so that they re easier to find. 1.6 I access MiChart-ASAP Tip Sheets and videos if I have questions. 1.2 In addition, the team ranked the entire feature checklist from the Physician Assistant MiChart Self-Assessment both overall and by user level. These lists of most and least used features can be found in Appendix H. Physician Assistant: Correlations The team investigated possible correlations within the survey data. Specifically, it was hypothesized that a correlation may exist between MiChart user level and age or previous EHR experience, both with Epic or other software. The correlation coefficient between age and MiChart user level was found to be The percentage of Physician Assistants of each user level with previous EHR experience is shown in Table 8. Table 8. Physician Assistant User Levels with Previous EHR Experience (Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 9, Dates: March 25-April 8, 2013) User Level EHR Experience (Epic) EHR Experience (Other) Low (1) 0% 0% Medium (7) 0% 71% (5) High (1) 100% (1) 0% The team also investigated whether a correlation existed between a users actual user level and what they perceived their user level to be. Survey responders were asked if they believed themselves to be a low, medium, or high functioning user. The resulting percentages of each user type and what they perceived their user level to be are shown in Table 9. The percentages of each perceived user level group s actual user level is shown in Table

22 Table 9. Physician Assistant Actual User Level vs. Perceived User Level (Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 9, Dates: March 25-April 8, 2013) User Level: Low (Perceived) Medium (Perceived) High (Perceived) Low (Actual) (1) 0% 100% (1) 0% Medium (Actual) (7) 14% (1) 86% (6) 0% High (Actual) (1) 0% 0% 100% (1) Table 10. Physician Assistant Perceived User Level vs. Actual User Level (Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 9, Dates: March 25-April 8, 2013) User Level: Low (Actual) Medium (Actual) High (Actual) Low (Perceived) (1) 0% 100% (1) 0% Medium (Perceived) (7) 14% (1) 86% (6) 0% High (Perceived) (1) 0% 0% 100% (1) Physician Assistant: Training Preferences In response to the training frequency questions, the team found that of 9 Physician Assistants surveyed 56% believed training should occur every year and 33% believed training should occur every 6 months. It was also found that a majority of 67% believed training should be offered during scheduled meetings. This majority training preference of preferring training during scheduled meetings was consistent among all age groups surveyed. Therefore, training preferences did not vary with age. Physician Residents: MiChart User Levels The Resident Assistant Self-Assessment had a mean score of 60% and a standard deviation of 9%. The maximum score was 79% and the minimum score was 45%. A histogram of the Physician Residents MiChart Self-Assessments scores is shown in Figure Frequency MiChart Self-Assessment Score (%) Figure 9. Histogram of Physician Resident MiChart Self-Assessment Scores (Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 27, Dates: March 25-April 8, 2013) 16

23 The Resident MiChart user level breakdown, shown in Figure 10, was very similar to the overall ED MiChart User Level Breakdown. High: 7% Low: 8% Medium: 85% Figure 10. Physician Resident MiChart User Level Breakdown (Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 27, Dates: March 25-April 8, 2013) Physician Resident: Most and Least Used Features The average score for each MiChart feature was calculated across all Physician Assistant survey respondents, using the following 0 to 5 scale: 0- I do not know how to use this feature, 1- Never, 2-Rarely, 3-Sometimes, 4-Frequently, 5-Always. The five most and least used features by Physician Residents along with their average score, are shown in Table 11. Table 11. Most and Least Used MiChart Features by Residents (Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 27, Dates: March 25-April 8, 2013) Average Most Used: Score When using NoteWriter, I add comments to specific findings or entire sections as needed. 3.7 I press ENTER after typing a search term instead of clicking Accept or Search. 3.7 I review patient information using CareWeb. (Excluding access through MiChart-ASAP) 3.6 I create personal SmartPhrases (Also known as dot phrases) 3.6 I create and share macros for typical exams. 3.5 Least Used: I add orders to my preference list. I can edit orders display names so that they re easier to find. 1.6 I use the history tab to enter and/or edit patients past family history. 1.5 I edit (add and/or delete) medications from the patient s home medication list when appropriate. 1.5 I use time and date shortcuts, such as n, t-1, m+2, and y I access MiChart-ASAP Tip Sheets and videos if I have questions

24 The feature above in Table 11 in italics corresponds to a low MiChart user level. Therefore, the fact that it is a most used feature should be viewed negatively. In addition, the team ranked the entire feature checklist from the Residents MiChart Self-Assessment both overall and by user level. These lists of most and least used features can be found in Appendix I. Physician Resident: Correlations The team investigated correlations within the survey data. Specifically, it was hypothesized that a correlation may exist between MiChart user level and age or previous EHR experience. The correlation coefficient between age and MiChart user level was found to be The percentage of Residents of each user level with previous EHR experience is shown in Table 12. Table 12. Resident User Levels with Previous EHR Experience (Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 27, Dates: March 25-April 8, 2013) User Level: EHR Experience (Epic) EHR Experience (Other) Low (2) 100% (2) 100% (2) Medium (23) 59% (14) 95% (22) High (2) 50% (1) 100% (2) The team also investigated whether a correlation existed between a users actual user level and what they perceived their user level to be. Survey responders were asked if they believed themselves to be a low, medium, or high functioning user. The resulting percentages of each user type and what they perceived their user level to be are shown in Table 13. The percentages of each perceived user level group s actual user level is shown in Table 14. Table 13. Resident Actual User Level vs. Perceived User Level (Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 27, Dates: March 25-April 8, 2013) User Level: Low (Perceived) Medium (Perceived) High (Perceived) Low (Actual) (2) 50% (1) 50% (1) 0% Medium (Actual) (23) 26% (6) 52% (12) 22% (5) High (Actual) (2) 0% 0% 100% (2) Table 14. Resident Perceived User Level vs. Actual User Level (Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 27, Dates: March 25-April 8, 2013) User Level: Low (Actual) Medium (Actual) High (Actual) Low (Perceived) (7) 14% (1) 86% (6) 0% Medium (Perceived) (13) 8% (1) 92% (12) 0% High (Perceived) (7) 0% 71% (5) 29% (2) 18

25 Physician Resident: Training Preferences In response to the training frequency questions, the team found that of 27 Physician Residents surveyed 52% believed training should occur every year and 26% believed training should occur every 6 months. It was also found that 37% believed training should be elective, self-directed e- learning online resources and 33% preferred presentations during scheduled meetings. Training preferences did vary slightly with age as seen in Figure % Presentations during 40% scheduled meetings 35% 30% 25% 20% 15% 10% 5% 0% Years Old Years Old Elective, self-directed e- learning online resources Other (One on One, Come to resident conference, MI chart person available to ask questions during shift, none) Figure 11. Resident Training Method Preferences by Age Group (Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 9, Dates: March 25-April 8, 2013) Physician Faculty: MiChart User Levels The mean score on the Faculty MiChart Self-Assessment was 60%, with a standard deviation of 13%. The maximum and minimum scores were 87% and 36%, respectively. The histogram shown in Figure 12 displays the range of scores. Frequency MiChart Self Assessment Score (%) Figure 12. Histogram of Faculty MiChart Self-Assessment Scores (Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 36, Dates: March 25-April 8, 2013) 19

26 The Faculty MiChart user level breakdown, shown in Figure 13, was very similar to the overall ED MiChart User Level Breakdown. High: 14% Low: 22% Medium: 64% Figure 13. Faculty MiChart User Level Breakdown (Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 36, Dates: March 25-April 8, 2013) Physician Faculty: Most and Least Used Features The average score for each MiChart feature was calculated across all Faculty survey respondents, using the following 0-5 scale: 0- I do not know how to perform this function, 1- Never, 2- Rarely, 3- Sometimes, 4- Frequently, 5- Always. The five most and least used features by Faculty, along with their average score, are shown in Table 15. Table 15. Most and Least Used MiChart Features by Faculty (Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 36, Dates: March 25-April 8, 2013) Average Most Used: Score While using the In-Basket function, I know which attestation tab to use for different circumstances. 4.5 When using NoteWriter, I use the Physical Exam tab. 4 When using NoteWriter, I add comments to specific findings or entire sections as needed. 3.9 While using the In-Basket function, I use protocol notes for patients on observation protocols. 3.8 I use the Facility List and Database Lookup tabs to find orders that aren t on my preference list. 3.7 Least Used: I use the history tab to enter and/or edit patient s surgical history. 1.6 I use the history tab to enter and/or edit patient s family history. 1.6 I use time and date / shortcuts, such as n, t-1, m+2, and y I add orders to my preference list. I can edit orders display names so that they re easier to find. 1.4 While using the In-Basket function, I know what to do with deficiencies that are not mine

27 In addition, the team ranked the entire feature checklist from the Faculty MiChart Self- Assessment both overall and by user level. These lists of most and least used features can be found in Appendix J. Physician Faculty: Correlations The correlation coefficient between age and MiChart user level was found to be The percentage of Faculty of each user level with previous EHR experience is shown in Table 16. Table 16. Faculty User Levels with Previous EHR Experience (Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 36, Dates: March 25-April 8, 2013) User Level: EHR Experience (Epic) EHR Experience (Other) Low (8) 0% 100% (8) Medium (23) 26% (6) 87% (20) High (5) 20% (1) 60% (3) Faculty were asked to rate themselves as a low, medium, or high functioning user. This perceived user level was compared to their actual user level based on their MiChart Self- Assessment score. The relationship between Faculty actual user level and Faculty perceived user level is shown in Table 17. The percentages of the each perceived user level group s actual user level is shown in Table 18. Table 17. Faculty Actual User Level vs. Perceived User Level (Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 36, Dates: March 25-April 8, 2013) User Level: Low (Perceived) Medium (Perceived) High (Perceived) Low (Actual) (8) 25% (2) 63% (5) 12% (1) Medium (Actual) (23) 13% (3) 70% (16) 17% (4) High (Actual) (5) 0% 20% (1) 80% (4) Table 18. Faculty Perceived User Level vs. Actual User Level (Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 36, Dates: March 25-April 8, 2013) User Level: Low (Actual) Medium (Actual) High (Actual) Low (Perceived) (5) 40% (2) 60% (3) 0% Medium (Perceived) (22) 23% (5) 73% (16) 4% (1) High (Perceived) (9) 12% (1) 44% (4) 44% (4) 21

28 Physician Faculty: Training Preferences The team found that out of 36 Faculty surveyed, 36% believe training should occur every 6 months, 19% believe training should occur every year, and 17% believe that additional training should never be offered. The survey showed that 31% of Faculty prefers the training method of a mentoring session with colleagues, 19% prefer elective, self-directed e-learning online resources, and 17% prefer a classroom session with a training instructor. Faculty training preferences were found to somewhat vary by age group as shown in Figure % 40% 35% 30% 25% 20% 15% 10% 5% 0% Years Old Years Old Years Old Classroom session with training instructor Presentations during scheduled meetings Mentoring session with colleagues Elective, self-directed e-learning online resources Other Figure 14. Faculty Training Method Preferences by Age Group (Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 36, Dates: March 25-April 8, 2013) Comparison between Roles Overall the team found that across all roles the user level breakdown was consistent with a majority being medium functioning users. The user level breakdowns within each role were similar and are shown below for comparison in Figure % 80% 60% 40% 20% 0% Nurse PA Resident Faculty Low Medium High Figure 15. ED MiChart User Level Breakdown by Role (Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 147, Dates: March 25-April 8, 2013) 22