Evaluation of an Electronic Charting System in the BCIT Nursing Simulation Lab

Evaluation of an Electronic Charting System in the BCIT Nursing Simulation Lab Prepared by: Johanne Mattie Research Associate, Technology and Product Evaluation Group As part of the Stage 3 ehealth Go To Market Accelerator British Columbia Institute of Technology July 12, 2010 With assistance from: Angie Wong, Connie Evans, Christine Flegal 1

Table of Contents Executive Summary... 3 Introduction... 4 Creation and Implementation of the ECS... 4 Evaluation Methodology... 4 Results... 5 Technology Experience / Ownership... 5 Ease of use of Adept/MDIT ECS:... 6 Attitudes towards Technology:... 6 Likes/ Dislikes:... 8 Discussion... 10 Conclusions... 11 Recommendations... 12 2

Executive Summary This project involved the creation, implementation, and evaluation of an Electronic Charting System (ECS) for the British Columbia Institute of Technology (BCIT) Nursing Simulation Lab. The project aimed to introduce nursing students to an ECS early in their careers, as well as investigate usability issues related to electronic charting from a nursing perspective- i.e. to better understand nurses needs, priorities, and barriers to usage. The ECS was piloted with both Term 1 and Term 4 nursing students during the January to May 2010 term. During the term, Term 1 and Term 4 students used the ECS to chart patient vitals as well as review patient histories, blood work, nurses notes (flow sheets), and doctors orders. Students completed baseline questionnaires prior to the pilot, as well as final questionnaires at the end of the pilot to provide feedback on the ECS. In addition, teaching staff from the Simulation Lab shared comments from students and noted areas for improvement with the ECS. Several issues relating to software usability, training, and limited exposure to the ECS were identified. In general, Term 1 students were more receptive to the ECS which may highlight the importance of getting nursing students to use this technology early in their careers, prior to them establishing workflow and charting practices using paper based documentation. If done successfully, it is anticipated that building electronic charting competency with today s students will facilitate the deployment and adoption of the technology within the healthcare system. It is anticipated that a positive introduction to these technologies will make nursing students more accepting of e-health technology in general and will make them more likely to accept other e-health technologies throughout their nursing careers. In addition, it is anticipated that having nurses use ECS will help increase the adoption of these technologies in other sectors (eg. amongst physicians and other health care providers). 3

Introduction The BCIT Nursing Simulation Lab is a learning environment for nursing students that houses eight human patient simulators that breathe and imitate 'real' patients. The lab aims to simulate an acute care hospital setting and uses real charts and equipment to create a realistic environment. Through the use of scenarios that build on course learning outcomes, nursing students have the opportunity to practice their roles as health care professionals by communicating, assessing and providing interventions to the human patient simulators. Prior to this initiative, patient charting and document access in the Simulation Lab were done using a paper based system. This project involved the creation, implementation and evaluation of an Electronic Charting System (ECS) for the Nursing Simulation Lab. The project aimed to introduce nursing students to an ECS early in their careers, as well as investigate usability issues related to electronic charting from a nursing perspective- i.e. to better understand nurses needs, priorities, and barriers to usage. Creation and Implementation of the ECS To better define requirements for the Nursing Simulation Lab ECS, BCIT worked with Adept/MDIT to discuss workflow and charting requirements. Vancouver Coastal Health (VCH) provided BCIT with their paper-based charting documents and shared results from a Patient Care Information System evaluation conducted with nurses at the University of British Columbia Hospital. Using all of this information, Adept/MDIT adapted their Clinical Information system to create an ECS specifically aimed at meeting the needs of the BCIT Nursing Simulation lab. Once the system was finalized, two laptops and two desktop workstations were set up in the Simulation Lab, and simulated patient data was entered into the ECS. Nursing teaching staff familiarized themselves with the ECS and outlined how the system would be integrated into the Simulation Lab training for nursing students. Evaluation Methodology The ECS was piloted with both Term 1 and Term 4 nursing students during the January to May 2010 term. Term 1 students were included in the evaluation as they were new to nursing and had no prior charting experience. Term 4 nursing students were included as they had more complex patients to chart, and already had experience using paper-based charting to provide comparative feedback. Prior to commencing the pilot, an introductory meeting was held with all Term 1 and Term 4 students to explain the objectives of the ECS initiative, describe the evaluation process, answer questions, and have the students sign informed consent forms. Additionally, students were asked to complete a Baseline Questionnaire, probing them on their level of comfort / experience with technology, as well as some of their thoughts about electronic health solutions for nursing. While 4

all Term 1 and Term 4 students were expected to use the ECS as part of the nursing curriculum, participation in the evaluation was voluntary. ECS training for the students consisted of a 5-10 min orientation by the Nursing Simulation Lab instructors during the first Lab session of the term. Students were provided with an overview of the system and shown how to conduct basic tasks such as reviewing doctors orders and entering vital signs. Instructors were available to answer questions, but aside from that, students were left to explore the ECS on their own as they used it. (Training on the ECS was limited due to scheduling and course constraints). During the term, Term 1 and Term 4 students used the ECS to chart patient vitals as well as review patient histories, blood work, nurses notes (flow sheets), and doctors orders. Term 1 students had access to the ECS approximately 3 times each; Term 4 students had access to it approximately 4 times each. (Note: Some students may not have used the ECS every time they had access to it as documenters were designated in each student group). Introductory sessions were observed and video recorded. Upon term completion, students completed a final questionnaire to provide feedback on the ECS. In addition, teaching staff from the Simulation Lab shared comments they heard from students throughout the term and noted areas for improvements/ additional features they would like to see on the ECS. Results A total of 143 students (n= 83 Level 1; n= 60 Level 4) completed the Baseline Questionnaires and a total of 132 students (n= 83 Level 1; n= 49 Level 4) completed the End of Term Questionnaires. Highlights from the questionnaires are summarized below. Please note: Participants were asked to respond to various statements using a five point Likert Scale of strongly agree to strongly disagree. For reporting purposes, we combined the strongly agree, agree responses and the strongly disagree, disagree responses. We then divided these totals by the total number of participants that completed a particular question. Rational for this is that several Term 4 students did not notice the questionnaire was double-sided and only filled out the first page of the questionnaire; therefore these responses are not included in the percentage calculations reported below. Technology Experience / Ownership As there were no notable differences in the results relating to Technology Experience/ Ownership from Term 1 and Term 4 students, results of these two groups are presented together in the section below. All students reported using computers regularly, and most considered themselves either experienced (81.7%) or expert users (5.6%). Almost all students owned computers (97.2%) and felt comfortable using them (96.5%). Students used computers for a variety of different reasons including school (98.6%), communicating (95.1%), internet (93.0%), entertainment (87.4%), and nursing related work (67.8%). 5

The majority of respondents owned a cell phone (98.6%), and felt comfortable using cell phone technology (95.8%). Only 11.9% of respondents owned a PDA, and less than half (44.1%) reported feeling comfortable using PDA technology. Prior to using the ECS in the BCIT Simulation Lab, very few students had had experience using an electronic charting system (only 2.8% claimed they had lots of experience; 16.9 % claimed they had a little experience). While 14.1% of students had never heard of electronic charting systems, most had heard of them but had never used them (66.2%). For those that had used ECS, there were 9 different systems that students had experience with; the most commonly mentioned were Synapse and Paris. Ease of use of Adept/MDIT ECS: Most Term 1 students felt the ECS they used in the Nursing Simulation Lab was easy to use (76.3%) and easy to navigate (62.0%). They also felt it was easy to enter data (85.0%) and view information (64.6%), and found the screen layout logical and easy to understand (68.4%). Most Term 1 students found electronic charting to be a fast and efficient way to enter patient data (80.0%), an effective way to access patient data (83.8%), and felt the ECS would fit well with the way nurses work with patients (82.5%). Most felt the experience with the ECS had been positive (75.0%). It should be noted that many students responded with no opinion, (depending on the question, between 7.5% - 21.5%). Contrarily, only 38.8% of Term 4 students found the ECS easy to use, and only 34.7% found it easy to navigate. Forty nine percent (49.0%) felt it was easy to enter data, 53.1% felt it was easy to view information, and 38.8% found the screen layout logical and easy to understand. Only 49.0% of Term 4 students felt that electronic charting is a fast and efficient way to enter data, however 67.3% felt that electronic charting is an effective way to access patient data. Fifty one percent (51%) felt the ECS would fit well with the way nurses work with patients. Only 38.8% of Term 4 students felt the experience had been positive. Again, it should be noted that many students responded with no opinion (between 4.1%- 18.4%). Attitudes towards Technology: Tables 1 and 2 below compare students responses to a number of questions relating to attitudes towards technology (asked both pre and post ECS pilot). Sections highlighted in yellow represent a positive change in attitude towards technology after the pilot; responses highlighted in red represent a negative change. In general, Term 1 students felt more positive about the technology after the ECS pilot compared to Term 4 students. Possible reasons for this will be proposed in the Discussion section below. 6

Table 1. Comparison of questions related to attitudes towards technology both pre and post ECS pilot- Term 1 responses QUESTION Rating Term 1 Term 1 - Baseline End of Term % change (n=83) (n=72) I like to keep myself informed about the latest agree 43.4% N/A 1 electronic technologies and applications no opinion 36.1% N/A disagree 20.5% N/A I would like to use computer and mobile device technologies in my nursing studies I would like to use computer and mobile device technologies in my career as a nurse. I think computer and mobile device technologies will play an important role in my career as a nurse. I think it is important that nursing students learn about new computer and mobile device technologies that could be used in nursing. I am interested in learning about the latest computer and mobile device technologies that could be used in nursing agree 80.7% 87.5 +6.8% no opinion 16.9% 6.3% -10.6% disagree 2.4% 6.3% +3.9% agree 83.1% 90.4% +7.3% no opinion 15.7% 6.0% -9.7% disagree 1.2% 3.6% +2.4% agree 83.1% 91.6% +8.5% no opinion 14.5% 7.2-7.3% disagree 2.4% 1.2% -1.2% agree 91.5% 92.8% +1.3% no opinion 8.5% 4.8% -3.7% disagree 0% 2.4% +2.4% agree 85.5% 81.9% -3.6% no opinion 12.1% 15.7% +3.6 disagree 2.4% 2.4% 0% 1 Please note, this question was only asked in the Baseline questionnaire. There was no follow up in the End of Term questionnaire. 7

Table 2. Comparison of questions related to attitudes towards technology both pre and post ECS pilot- Term 4 responses QUESTION Rating Term 4 Term 4 - Baseline End of Term % change (n=60) (n=49) I like to keep myself informed about the latest agree 48.3% N/A 2 electronic technologies and applications no opinion 28.3% disagree 23.3% I would like to use computer and mobile device technologies in my nursing studies I would like to use computer and mobile device technologies in my career as a nurse. I think computer and mobile device technologies will play an important role in my career as a nurse. I think it is important that nursing students learn about new computer and mobile device technologies that could be used in nursing. I am interested in learning about the latest computer and mobile device technologies that could be used in nursing agree 81.7% 71.4% -10.3% no opinion 13.3% 14.3% +1.0 disagree 5.0% 14.3% +9.3 agree 81.4% 76.3% -5.1% no opinion 15.3% 15.8% +0.5 disagree 3.4% 7.9% +4.5 agree 85.0% 84.2% -0.8% no opinion 6.7% 7.9% +1.2 disagree 8.3% 7.9% -0.4 agree 95.0% 86.8% -8.2% no opinion 3.3% 10.5% +7.2 disagree 1.7% 2.6% +.9 agree 88.3% 81.6% -6.7% no opinion 8.3% 13.2% +4.9 disagree 3.3% 5.3% +2.0 Likes/ Dislikes: A summary of the ECS likes and dislikes reported in the questionnaires is presented below. Detailed comments made by students during the term, as well as observations from teaching staff about potential improvements to the ECS were provided to the vendor. Most liked features of the ECS included: Easy to read / legibility (e.g. compared to handwriting, doctors orders) 2 Please note, this question was only asked in the Baseline questionnaire. There was no follow up in the End of Term questionnaire. 8

Easy to access / retrieve information on patient (accessible at multiple locations/sites) Fast, efficient, organized Easy to navigate / enter information Logical layout, easy to view Least liked features of the ECS included: Not enough training/ exposure to the system Challenges navigating and searching the system Time it took to enter information Usability issues with the screen layout Patient information is too far from patient - system was not mobile Limited number of computers Technology challenges (e.g. potential of crashes, what to do if there is a power outage; potential for errors with data entry) Desired features in an ECS included: User friendly (e.g. faster data, less screens to go through; more tick/ text boxes where additional information can be added) Simple layout; clarity/legibility: pictures, colour, bigger fonts, clearer buttons for each function, easy to interpret headings, tabs instead of icons, ability to use different windows at the same time. Quick to enter data Reliable; error proof; ability to quickly identify errors Easy to find patient info or to see cautions or warnings for patient Easy access to terminals and/or portability to enter/view data anywhere Potential to link hospitals/clinics together Additional features which should be included in an ECS included: Vital signs / flow sheets with quick access and entry Notes (for additional information) Doctor s orders, nurses notes Privacy/ security: password protected logins, automatic logouts, Nurse s initials or ID number Diagrams, visual aids Patient history & status updates MARs (Medication Administration Records) Charting options Help button, pause button. Messenger system for communication in pharmacy/doctors Lab test values & results Different assessment types/ assessment reminders Instructions, training/ take home tutorial/ handbook to clarify any FAQ s or common errors made. 9

Referrals/consults Touch screen Tabs on top browser to navigate Discussion Several students noted they were grateful for the opportunity to use and familiarize themselves with the ECS. The Nursing Simulation Lab ECS pilot informed them of how relevant this technology could be in the workplace. They also noted that using it in an educational setting will help ease the switchover from paper to electronic records. In fact, several students noted the system is better than relying on traditional handwriting I can never read anything during clinical How are we supposed to be accurate in our patients care?! Despite the interest in moving towards electronic charting, results from the evaluation demonstrate there are some issues related to nurses using the ECS that must be resolved if this technology is to be accepted and used readily. Wonderful job to start incorporating a system that will be valuable. [It] just needs improvement. In particular, usability issues relating to navigating the system, entering data, and screen layout were brought up. An efficient means of entering and viewing data were noted as critical, so optimizing the system is essential. Some specific recommendations of how the usability of the system can be improved were provided to the vendor. In general, Term 4 students were more critical of the usability of the ECS than their Term 1 counterparts. Several possible explanations for this discrepancy are proposed: Term 4 students had prior charting experience so already had pre-conceived ideas, workflow patterns, routines associated with charting Term 4 students used the ECS for more complex tasks so would have had more exposure to different aspects of the system Term 4 students may have been feeling more overwhelmed and/or burned out with course work and therefore had less energy to devote to dealing with the ECS Term 4 students may have felt that inputting data into the ECS took more time than paper charting. Term 1 had no previous experience with charting so may have been more open to utilizing this method Term 1 students are new to nursing, and may have been less critical and/or eager to please by responding positively In general, Term 4 students had less positive attitudes towards technology after the pilot. Again, it is difficult to say definitively why this is the case, however it is proposed that these attitudes may have been influenced by their less positive experiences with the ECS. Challenges with the using the ECS may have left them feeling less optimistic about the role technology might play in nursing in general. 10

As previously mentioned, the lack of adequate training was a problem for the students, thus this may have contributed to the usability issues discussed above. Unfortunately, due to time constraints in the lab course, there was little time available that could be devoted to training and orientation. In hindsight, more time should have been found to familiarize students with the ECS as this would likely have resulted in a much more positive experience for all. It is also likely that the limited exposure students had to the ECS contributed to their perceptions. In most cases, students only used the system once per month. The long wait period between uses, together with the fact that students only had a short window of time to actually use the system, resulted in a lack of proficiency. Consequently, it is likely that more positive evaluations would have been the outcome had they been more fluent with the system. Other issues that were raised related to the physical set up of computers. Students felt that there should be a point of data access for each patient, and ideally this should be mobile. During the evaluation, 2 laptop and 2 desktop systems were used. There had been concerns about keeping computers sanitized when working directly with the patients, consequently it had been decided that for the pilot, data would not be entered at the patient bedside. This resulted in the students writing notes on paper at the patient bedside, then transferring the data into the ECS at the workstation. This not only resulted in extra work for the nursing students, but it also increased the risk of error with data entry. (Note- After discussion with VCH, it was determined that VCH had addressed this issue and had created sanitizing procedures that they kindly shared with BCIT. This procedure is on file for use in subsequent terms with the ECS.) Students also suggested using mobile devices to access the ECS. While there are usability issues associated with the small form factor of mobile devices, the option is worthy of further investigation. Of interest was the fact that there was still a considerable amount of documentation that was done on paper, including the bedside charting mentioned above, as well as some procedural tasks where no easy electronic solution was available (e.g. MARS and patient consent forms). Additionally, teaching staff felt it was important that students still learn to chart using paper documents so students were required to do both forms of charting. In order for the ECS to be completely integrated into nursing, efforts will need to be made to find solutions that eliminate or at least minimize the need for paper documentation. Several companies are working on solutions (e.g. RFID or scanner based MARS systems, electronic signatures on consent forms) that should be investigated further. Conclusions Through the pilot, several issues relating to software usability, training, and exposure to the ECS were identified. In general, Term 1 students were more receptive to the ECS which may highlight the importance of engaging nursing students to use this technology early in their careers, prior to them establishing workflow and charting practices using paper based documentation. If done successfully, it is anticipated that building electronic charting competency with today s students will facilitate the deployment and adoption of the technology within the healthcare 11

system. It is anticipated that a positive introduction to these technologies will make nursing students more accepting of e-health technology in general and will make them more likely to accept other e-health technologies throughout their nursing careers. In addition, it is anticipated that having nurses use ECS will help increase the adoption of these technologies in other sectors (e.g. amongst physicians and other health care providers). Recommendations Consider implementing the specific ECS modifications provided to the vendor to improve usability and efficiency of the system Formalize the training and orientation of ECSs e.g. have a Simulation Lab session that is focused purely on the ECS, or create an on-line training/ review module that could be done by students at home Continue to have nursing students use the ECS, preferably by increasing frequency of usage so that competency is built up quickly Continue to work with the Health Authorities to determine their long term visions for nursing charting systems Explore the use of mobile workstations and/or handheld devices for access to the ECS Adopt standardized sanitizing procedures for ECS workstations (based on VCH procedures) Eliminate or minimize paper based documentation practices in the Simulation Lab by charting at the patient bedside, and consider solutions for MARS, consent forms etc. 12