The Effect of EHR Implementation on Provider Perspectives of Developmental Surveillance, Screening, and Referrals for Early Intervention Services

Transcription

1 The Effect of EHR Implementation on Provider Perspectives of Developmental Surveillance, Screening, and Referrals for Early Intervention Services By Blair Alicia Brown, Master of Public Health May 2012

2 ii ACKNOWLEDGEMENTS I would like to thank my advisor, Dr. Jennifer Breaux and my preceptor Dr. Benjamin Sanders for their expertise and guidance throughout the course of my research. I would also like to thank Dr. Bruce Bernstein and Dr. Ed Gracely for their knowledge and assistance with the collection of survey data and analysis using SPSS.

3 iii TABLE OF CONTENTS Abstract...vi Introduction...1 Background...1 Developmental Delays.1 Electronic Health Records...7 Specific Aims...12 Research Design & Methods...13 Participant Recruitment.13 Data Collection..13 Data Analysis.15 Data Storage/Protection.15 Human Subjects.16 Results Quantitative 16 Qualitative..21 Discussion Limitations.27 Future Research.28 Research Challenges..30 Conclusions...32 References.33 LIST OF APPENDICES Appendix A Paper-based Survey Appendix B.Qualtrics Survey

4 iv LIST OF TABLES Table 1: Median Values for Provider Responses Pre and Post-EHR Implementation.19

5 v LIST OF ILLUSTRATIONS Figure 1: Survey Questions 8 & 10 Differences in Pre and Post Responses 20

6 vi ABSTRACT The Effect of EHR Implementation on Developmental Surveillance, Screening, and Referrals for Early Intervention Services Blair Alicia Brown MPH 1 Benjamin W Sanders MD MSPH 2,Jennifer Breaux DrPH MPH 1 1 Drexel University School of Public Health 2 Drexel University College of Medicine Objectives: Electronic health records (EHR) are becoming more widespread in the healthcare community. Research suggests that EHR is more efficient, cost-effective, and provides higher quality patient care than paper-based records. There is limited data available on how the transition to EHR affects developmental care in pediatrics, specifically from the provider s perspective. Methods:A mixed-methods survey for providers was designed asking about their perceptions on the ease of completing developmental surveillance, use of validated screening tools, and the overall effectiveness of the referral process for early intervention services. Providers were given surveys one week prior to EHR implementation and three months post-implementation. Surveys were done on paper and using Qualtrics online survey software. The data were collected and pre and post responses were analyzed in SPSS for significant differences using the Wilcoxon Signed Rank Test. Qualitative responses were summarized into themes and compared pre and post. Results: All 16 providers completed both the pre and post-survey. Overall both pre and postimplementation, more than 60% of providers regularly assess child development. Prior to EHR providers were somewhat more likely to use validated screening tools and believed EHR would improve their assessments. Post-implementation surveys show that providers feel their assessments of development are worse. Based upon qualitative responses, providers believe access to screening tools and time are the largest barriers to completing effective developmental assessments. Conclusions: Long term, EHR has the potential to improve the quality of care, specifically pediatric developmental care. After only three months of EHR providers are most likely still becoming familiar with the system. The lack of availability of screening tools and referral forms stream-lined into EHR may be a limiting factor in the current efficiency of EHR. As providers gain confidence using the system and the necessary screening tools and forms are added, EHR has the potential to improve developmental surveillance, screening, and referrals.

7

8 1 Introduction This study provides insight on the role of electronic health records (EHR) in improving the developmental care pediatricians deliver. Specifically, this study evaluates provider perceptions about developmental screening, surveillance, and referrals and how those perceptions change after the implementation of EHR. The problem addressed by this study is the nationally low rate of surveillance, screening, and referral for developmental delay by pediatricians. National rates of surveillance, screening, and referral for developmental delay suggest that less than 40% of children with delays are referred for Early Intervention (American Academy of Pediatrics (AAP), 2003). Previous studies show that physicians feel they are not knowledgeable enough about developmental delays and the use of screening tools to test for delays. In addition, many physicians mention time and the burdensome referral process as barriers to screening for delays (Sices, Feudtner, Mclaughlin, Drotar, &Williams, 2003). Children with delays that go undetected and untreated ultimately place an undue strain on the child, the child s family, the community, and the health system as delays may lead to life-long social, emotional, and academic difficulties (Aylward, 1997). Background Developmental Delays A developmental delay is a significant lag, according to the expected time frames, in one or more areas of development (AAP, 2006; Tervo, 2003). The four domains of development include speech and language, gross motor, fine motor, and personal and social (Tervo, 2003). Children are at risk for or develop delays for various reasons. Some children are born with risk factors for developing delays because of prematurity or other factors such as fetal alcohol

9 2 syndrome. Environmental factors also put children at risk because of their social or physical environment, which may adversly affect their growth and development. Finally, there are children who are diagnosed with a medical condition that is known to adversly affect development such as Down syndrome or Autism. Early identification of a child with a developmental delay or a child at risk for delay is essential because the child can then be further evaluated for possible Early Intervention services. These services, depending upon the delay may include a combination of speech therapy, physical or occupational therapy, and programs that emphasize cognitive and social skills. Depending upon whether the child is at risk for delay or has a delay, intervention services can prevent the development of delays or minimize the effects of delays. In some children who may have developmental delays that cannot be minimized through current interventions, there are still services and techniques such as aids and adaptations as well as behavioral techniques that can be used to help maximize a child s functioning. Research has shown that intervention programs are most beneficial for children during the first few years of life as this is the critical period in child development. Some children do receive Early Intervention past the age of three if it is still needed but most often the first three years of life are when interventions are most successful. Children who require and receive Early Intervention at a young age are able to improve their development significantly as compared to children who do not receive services (Majnemer, 1998). Delays that are left undetected and untreated often result in a lifetime of difficulties for the child and the family (AAP, 2006). Developmental surveillance is the ongoing process of monitoring the status of a child by gathering information about the child s development and behavior from multiple sources, including skillful direct observation of the child s behavior and elicitation of concerns from

10 3 parents, caregivers, and relevant professionals. The primary care pediatrician generally undertakes this role, as she is the health professional with the most frequent and direct access to the child s health and underlying social and familial situation from birth to adolescence (Dworkin & Glascoe, 1997; Gilbride, 1995; Rydz, Shevell, Majnemer, & Oskoui, 2005; Squires, Nickel & Eisert, 1996). Surveillance involves interviews at every well-child visit, recording longitudinal progress, and administration of validated screening tools. Developmental monitoring for many pediatricians often becomes limited to informal interviews because of time but effective developmental surveillance requires much more. The success of surveillance depends on the ongoing assessment of development and there are often structural barriers that prevent many children from receiving frequent care resulting in poor developmental surveillance. For these reasons, the use of standardized tools as part of surveillance may assist in the accurate detection of delays (Dutton (1979); Rydz, Shevell, Majnemer, & Oskoui, 2005, Sices, 2007). Screening is a part of surveillance that involves widespread detection of risks for developmental delay in children. Screening tests for development are given to populations of children generally in clinics, schools, day care centers, or recreation centers to identify early deviations from normal, leading to referral for further assessment (Aylward, 1997). Since the tools are used on a larger scale, there is a strong need to assure the external, internal, and cultural validity of screening questionnaires. A validated tool is one that has been tested and shown to actually measure what the tool is intended to measure. These tools have also been tested for reliability, meaning they are able to produce consistent results under identical conditions regardless of who is using the tool (Rydz, Shevell, Majnemer, & Oskoui, 2005). Standardized screening tools increase detection rates and should be given to populations of children as part of regular surveillance following the American Academy of Pediatrics and Bright Futures

11 4 guidelines at 9, 18, and 30 months (AAP, 2006). There are a number of validated questionnaires used to assess developmental milestones at various ages. At St. Christopher s Hospital for Children in Philadelphia, pediatricians use two different validated screening tools to assess children for developmental delays; the PEDS and the M-CHAT. The PEDS (Parents Evaluation of Developmental Status) is a developmental screening tool for children birth-8 years of age. It is one of several developmental screening tools that obtain and address parent s concerns. The test itself takes approximately two minutes to complete and asks parents 10 questions, eight of which begin by asking, Do you have any concerns about how your child Questions touch on all four domains of development and ask about how the child interacts with others and how the child uses his/her body. The tool has been effective in detecting developmental delays that may not otherwise be mentioned during a wellchild visit and also allows the pediatrician to provide parents with developmental advice for specific concerns that do not require a referral to Early Intervention (PEDS, 2012; Rydz, Shevell, Majnemer, & Oskoui, 2005). The M-CHAT (Modified-Checklist for Autism in Toddlers) is a validated screening tool completed by parents. It is designed to identify children ages 16 to 30 months who should be referred for further evaluation for a possible autism spectrum disorder diagnosis. This screening tool asks 23 yes/no questions about a child s behaviors, such as, Does your child take an interest in other children? Does your childsmile in response to your face or your smile? and Does your child walk? (M-CHAT, 2012). Both the PEDS and the M-CHAT are screening tools that are completed by parents. Research shows that parents are able to give accurate information when answering questionnaires about their child s development, regardless of socio-economic background

12 5 (Dobos, Dworkin, & Bernstein, 1994; Rydz, Shevell, Majnemer, & Oskoui, 2005). These tools provide parents with the opportunity to discuss any developmental concerns they may have about their child with the child s provider. Research has shown that 70% of mothers have concerns about their child s development but only 28% actually ask questions of their child s healthcare provider. By filling out the parent questionnaire, these issues are more often discussed during well-child exams (Hickson, Alteimer, & O Connor, 1983). Despite the usefullness of screening tools in detecting delays, there are some qualities that may be associated with their underuse. According to the national survey How do primary care physicians identify young children with developmental delays, as few as 51% of pediatricians use validated screening tools (Sices, Feudtner, Mclaughlin, Drotar, & Williams, 2003). The biggest problem is time; administering, scoring, and discussing a standardized screening test can take up valuable time during short well-child exams. Since screening tools are used to assess development in young children, screening depends on a child s behavior and often the short time in which the child is seeing the provider is not representative of how the child behaves on a day-to-day basis. In addition, there are many screening tests available and physicians often find the process of choosing which test is appropriate to be burdensome and, sometimes the results of one test will contradict the results of another. Finally, administering screening tools on a regular basis can be very costly and many physicians feel they are not well compensated for this process (AAP, 2001; Glascoe & Dworkin, 1993; Sices, Feudtner, Mclaughlin, Drotar, & Williams, 2003). Once a child is identified as exhibiting a delay in any area, s/he should be referred for further evaluation and treatment through Early Intervention. Children in the United States who are identified as having a developmental delay are eligible to receive services to address their

13 6 developmental delay or disability, free of charge. This policy is part of Part C of the Individuals with Disabilities Education Act (IDEA), passed in 1990 and amended in 2004 (United States Department of Education, 2004). Although Early Intervention is part of a federal legislation, it is administered at the state level (Bailey, Hebbeler, Spiker, Scarborough, Mallik, & Nelson, 2005; United States Department of Education, 2012). Early Intervention, or ChildLink in Philadelphia, includes developmental assessment, support, and services designed to meet a child s developmental needs as well as the family s needs in relation to enhancing their child s development. After ChildLink evaluates a child, s/he may be deemed eligible for specialized services such as speech therapy, physical therapy, and/or occupational therapy. Often these services are delivered through a home visit program so that families are not burdened with transporting their child to and from various service programs (ChildLink, 2006). The rates of surveillance, screening and referral for developmental delays both nationally and at St. Christopher s Hospital for children are concerning and suggest room for improvement. One survey, the Early Childhood Longitudinal Study Birth Cohort, estimated that the rate of developmental delays that would qualify a child for Early Intervention was around 13% for most states. Only ten percent of the children from the study who were identified as delayed (eligible for Early Intervention), actually received Early Intervention. Of that 10 percent, African Americans and the uninsured were disproportionately underrepresented (Rosenberg, Zhang, & Robinson, 2008). In a 2002 survey of pediatricians, providers estimated that they referred 39% of their patients that had possible developmental problems to Early Intervention. Yet, those patients with global delays, delayed speech, loss of milestones, sensory impairment, abnormal muscle tone, and delayed motor development were referred 75 to 90% of the time (AAP, 2003). This suggests providers are more likely to refer a child for Early Intervention when they are

14 7 confident that the child has an obvious diagnosable delay versus risk factors for delay or a possible delay. The current screening system in place to identify, diagnose, and treat children with developmental delays, especially in vulnerable populations, does not appear to be effective. Many children who do have a developmental delay go undiagnosed and therefore, untreated. Children who are diagnosed often get lost in the system during the referral process and do not receive eligible services. Various factors appear to contribute to children going undiagnosed and untreated. Many pediatricians attribute their limited ability to perform developmental screening during regular well-child visits to time constraints. Both residents and pediatricians feel they are not educated enough to perform the screenings and/or recommend proper resources to families. Finally, there is a lack of communication and coordination of care between a child s healthcare professionals and therapists. Electronic Health Records (EHR) In recent years interest in using electronic health records among healthcare professionals has drastically increased (Miller & Sim, 2004). Healthcare providers have high expectations that EHR will prove to be a successful tool in improving the quality of patient care, reducing costs, and improving access to important patient information. While the concept of EHR appears promising, the number of practices currently using EHR is limited with recent estimates of EHR adoption in ambulatory care being slightly over 10% (Adams, Mann, & Bauchner, 2003; Miller & Sims, 2004; Morton & Wiedenbeck, 2009). In many cases the costs of transitioning between a paper-based and electronic records system is high and the financial benefits can take years to have an effect. With the implementation of EHR, practices must train staff to use the new system

15 8 (Miller & Sims, 2004). The American Recovery and Reinvestment Act of 2009 encourages EHR adoption by Beginning in 2011, any Medicare physician that implements and reports use of EHR will be eligible for an incentive payment up to $18,000. Starting in 2015, as a penalty, physicians that do not use EHR will receive reduced Medicare payments (starting at 1%) (American Medical Association (AMA), 2009). Current research on EHR and the transition between paper-based records and EHR is relatively small most likely because EHR is such a new concept. Specifically research that examines the transition to EHR and patient, provider, resident and educator perceptions is not extensive (Gadd & Penrod, 2001). In addition, little is known about the effect that the transition to EHR has on ambulatory pediatric quality of care, including developmental surveillance and screening (Adams, Mann, & Bauchner, 2003). A variety of studies looked at patient perceptions of EHR implementation with most of the data collected through surveys or interviews. One review looked specifically at patient satisfaction with physician use of EHR throughout the literature. Both qualitative and quantitative data from previous studies were examined. Many of the studies had methodological concerns such as non-randomization, and difficulty measuring certain variables such as patient satisfaction. Overall, the literature suggests that patient satisfaction remained the same or improved once EHR was implemented in a practice (Irani et al., 2009, Johnson et al., 2005). Specifically, a study by Gadd et al., (2000) that was included in the review by Irani et al., (2009) involved conducting pre and post implementation surveys and interviews with patients and physicians. The interviews and surveys looked at whether patients and/or physicians thought the use of EHR affected patient care. The results suggested that while patients did not feel the switch to EHR affected the quality of their care they were concerned about the privacy of their

16 9 medical records once they became electronic. Physicians agreed with patients that they were concerned for the safety of patient records but physicians also expressed concern that moving to EHR prevented them from providing quality care. This is similar to the research we conducted by surveying physicians pre and post implementation to look at how EHR affected their assessments of development. A second study by Gadd et al., (2001) specifically looked at physician attitudes towards implementation of EHR in an outpatient academic medical center. The article uses a validated survey instrument developed by Cork et al., (1998). The results showed that after 6 months, physicians were dissatisfied with EHR and many were less optimistic about the benefits of EHR than when they first started using EHR. The biggest issues were time and patient privacy and satisfaction. Although, the study did find that physicians still saw EHR as having potential for improving access, organization, and quality of medical records. This report only looks at the first 6 months after implementing EHR. It is possible that this is still the break-in period as other studies have shown there to be an adjustment period after first introducing EHR (Gamm et al., 1998). In addition to physicians and patients, EHR also has the potential to be helpful to medical educators and residents. Another study investigated the impacts of EHR implementation on all staff in a resident clinic by interviewing staff members. The results found that many residents and physicians were not happy with the amount of time needed to document information in the EHR system. Many faculty and residents did not feel the use of EHR led to better patient care and felt that using EHR was too time-consuming. Many were also disenchanted by the lack of communication between staff and the negative effects EHR had on patient care. Ultimately at this practice, physicians have started to shy away from documentation during a patient visit

17 10 instead saving it for after in hopes that this will improve patient quality of care. The disadvantage to this is increased workload for physicians that are already pressed for time (Bloom & Huntington, 2010). In another study by Aaronson et al., (2001) resident perspectives on the use of EHR were also studied. Residents were both interviewed and given a survey about their perceptions and experiences in using EHR. Overall, this study found many residents were frustrated with using EHR for various reasons. Given the demanding schedule of a resident, the use of EHR tends to increase workload especially initially, therefore many residents are resistant to using EHR. In addition, many feel this takes away from their patient relationships. This study also looked at how much influence EHR training has on a resident s perspective on using EHR. It suggests that residents who received better training were more likely to find the use of EHR to be beneficial. One study did find EHR to be considered a successful tool for use by residents and medical educators. Keenan et al., (2006) surveyed residents and found them to prefer EHR. Although they felt there were some drawbacks to EHR similar to other literature such as decreased patient interaction and increased workloads, overall residents felt the use of EHR led to better quality patient care. This study also discusses the use of EHR in medical education, an area with limited literature the use of EHR as a teaching tool in educating physicians. Results suggest that through features such as computerized clinical decision support systems EHR can be a useful tool in medical education. Computerized clinical decision support systems provide physicians with notices during a documented visit about certain tests that should be done based upon the patients history or any new symptoms the physician has entered into the system. In addition, EHR is a great tool for educators and residents because it is an easily accessible

18 11 database allowing for easy access to information to teach residents about quality improvement or for educators to evaluate resident competency. There is relatively limited research on the transition to EHR in an urban pediatric ambulatory setting. The only study available was by Adams et al., (2003) and looked specifically at the implementation of EHR in an urban pediatric ambulatory setting. This transition was evaluated by analyzing charts pre and post-implementation. In addition, clinicians were surveyed on their views on ease of use of EHR. Overall in this specific practice the transition to EHR was found to be successful and well received by physicians. This specific study is different than most because it specifically looked at how EHR can be used to prompt physicians to do specific tests based upon information that is entered by the user. Many physicians found this feature to be incredibly helpful and this may contribute to the overall happiness with EHR expressed in this study. In addition, the health care system where the research was conducted had their own self-made EHR system versus one of the mainstream widely distributed EHR systems used throughout the country. St. Christopher s Hospital for Children uses one of many mainstream EHR systems available, NextGen. The current literature on how the transition to EHR is perceived by patients, physicians, educators, and residents is limited especially in an ambulatory pediatric setting. Future research should focus on specifically how these various stakeholders perceive EHR and why those perceptions are present. In addition, the effects of the transition to EHR on the overall quality of patient care, specifically in a pediatric setting need to be considered. It may also be interesting to see how perspectives of EHR in a pediatric setting may be different when EHR systems that use prompting through computerized clinical decision support systems are implemented. Finally, research on how the transition from paper-based to EHR affects provider perceptions and actual

19 12 rates of screening, surveillance, and referrals is limited. This study mimics some of the studies discussed above by conducting pre and post-ehr implementation surveys with providers to get their perspectives on EHR and looking at how EHR affects developmental surveillance, screening and referral. This study is different from previous studies in that the focus is specifically on how provider perspectives on developmental assessment may change due to the transition to a mainstream EHR system in a pediatric ambulatory setting. Specific Aims Objective 1: Determine provider perceptions on the surveillance, screening and referral process for developmental delays Survey all providers gathering quantitative and qualitative data about their perceptions and experiences Objective 2: Determine whether provider perceptions of screening, surveillance, and referrals for delays change after the implementation of EHR. Provider perceptions of screening, surveillance, and referrals will be analyzed pre and post EHR implementation through a survey using quantitative questions on a Likert scale and qualitative free-response questions. Surveys will be given immediately prior to EHR implementation and again three months post EHR implementation. Surveys are attached in Appendix I and Appendix II.

20 13 Research Design and Methods This study took advantage of a natural experimental design, which means regardless of the study, the switch to EHR was going to happen at St. Christopher s providing an optimal opportunity to gather data in a before and after situation. The study measures subjective provider perspectives on the developmental screening and referral process for children ages 0 to 36 months. With samples done at EHR transition and three months afterwards. Participant Recruitment Surveys were given to General Pediatrics providers at the Center for Child and Adolescent Health at St. Christopher s Hospital for Children in Philadelphia, PA, before the implementation of EHR and three months after implementation. Principal Investigator, Dr. Benjamin Sanders, approached all possible eligible providers to participate in the study. All providers had the option to opt-out and not be a study participant. If providers agreed to participate in the study they completed both the consent form and the paper pre-survey and both were returned to an honest broker, an individual who is not involved in the research process in any way. Data Collection Providers were given a paper-based survey during the first week of December, prior to December 6 th, 2011, when General Pediatrics went live with EHR. The honest broker removed the first page of each survey containing the participant s name and assigned each participant a study ID number. The honest broker is maintaining the list of providers and study ID numbers in a confidential location for the duration of the study. Post-implementation surveys were

21 14 completed online during the second week of March 2012 utilizing the online survey software, Qualtrics. Qualitrics was chosen because Drexel University provides free access to the survey software for all students and faculty. Qualtrics is a survey software program that allows users to create online surveys and conduct statistical analysis on the results. While Qualtrics is similar to other survey software programs such as Survey Monkey, Qualtrics is more robust as it allows surveys to be sent via , provides ongoing tabulation of responses to the surveyor, and directly converts data for downloading into SPSS for analysis (Qualtrics, 2011). The surveys included 21 questions that asked providers about their previous experience with EHR, and their perceived efficiency, efficacy, and comfort with developmental surveillance, screening, and referrals. Providers were asked the same questions pre and post-implementation except for minor changes. See Appendix I for the paper based survey and Appendix II for a sample of the online Qualtrics survey. After de-identification of the pre-surveys by the honest broker, the data were transcribed and stored using Microsoft Access. Microsoft Access was used because it is effective at handling large data sets and is easily compatible with SPSS, the statistical analysis program used to analyze our data. Data from the post-survey in Qualtrics was de-identified, collected and converted to an SPSS file automatically within Qualtrics. Responses to the qualitative, free response questions were analyzed by three people, each developing a list of major themes. All three lists of themes were compared and a final determination of the main themes was agreed upon.

22 15 Data Analysis The data were analyzed using SPSS statistical software. The median response to each quantitative question, pre and post-implementation was found. The median was used instead of the mean because the variables being considered are ordinal variables on a Likert scale meaning they represent a rank and not numerical values that are equal distance apart. The Likert scale consists of ordered response options that represent an individual s level of agreement or disagreement with the question being asked (Likert, 1932). For example, one scale used in the survey allowed providers to choose one of the following options: much worse, somewhat worse, same, somewhat better, or much better. All questions were further analyzed using the Wilcoxon Signed Ranks Test to test for a significant difference between the pre and post-survey responses. This test was chosen because the Wilcoxon is a non-parametric test used for finding whether there is a significant difference when comparing repeated measurements on a single group. More specifically, the Wilcoxon Signed Rank Test shows the magnitude and the direction of change in a group (Wilcoxon, 1945). Chi-Square tests of independence were used to look at the associations of provider s reported experience with EHR and their predictions of how their developmental assessments would change with EHR implementation. This test was chosen because it allows one to test if observations on two different variables are independent of each other (Dawson & Trapp, 2001). In this case, does a provider s previous experience using EHR affect their response to how they believe EHR will affect their developmental assessment? Data Storage/Protection All surveys were collected and maintained by the honest broker that assigned each provider with a study ID number for the duration of the research project. The honest broker will

23 16 destroy the study ID number key at the conclusion of the research project, which Dr. Sanders intends to continue until February To protect the identity of providers, researchers only had access to the study ID numbers. Data collected was maintained on the computer in Dr. Sanders office, which is locked. The computer is password protected and the documents containing the data were also password protected. Only Dr. Sanders, our statistical advisor Dr. Bernstein who is Director of Research in General Pediatrics at St. Christopher s, and I had access to the document containing the data. All data were backed up on an additional password protected hard drive. Human Subjects This study was approved by the St. Christopher s Hospital for Children Institutional Review Board in partnership with the Drexel University College of Medicine Institutional Review Board as an expedited review. All researchers did complete the necessary IRB training. All providers remained anonymous throughout the research process. All provider identifying information will be destroyed by the honest broker at the conclusion of the research project in February Results Quantitative Surveys were given to 16 of the 21 providers in General Pediatrics. Two providers were excluded because of their involvement with the study, one provider was leaving the practice, and two other providers do not see their own patients in General Pediatrics. Survey data pre and post- EHR implementation were collected and the median value was found for each question. The

24 17 results show that 12 of the 16 providers had no prior experience using EHR before it was implemented at St. Christopher s. The remaining four providers had anywhere from months of previous experience using EHR. Providers were asked pre and post implementation about whether they routinely conducted developmental surveillance during regular well-child exams. Both pre and post implementation over 60% of providers said they regularly assessed child development in all four developmental domains. Prior to EHR implementation seven of 16 providers said they rarely use the PEDS questionnaire to assess developmental progress with the remaining nine providers using the PEDS at least one or more times. In the post-survey the number of providers rarely using the PEDS to assess development increased to ten providers, although this difference between pre and post was not significant when tested using the Wilcoxon Signed Ranks Test. Providers were fairly varied in their response to questions about relying upon their registration staff to insert or prepare the PEDS or M-CHAT questionnaire with responses spanning from strongly disagree to strongly agree. Providers were also varied in their response to the question, It is easy for me to complete and send a referral for evaluation of developmental concerns for my patients. There was no significant change between provider s answers pre and post-implementation but responses varied from strongly disagree to strongly agree. Most providers say they rarely or only some of the time receive information about the results and often this is information from parents. Few of the survey questions actually showed a significant difference as determined by the Wilcoxon Signed Ranks Test. Two questions, specifically those about how often providers use the M-CHAT to assess development and how well providers expect their developmental assessment to function post-ehr implementation did show a significant difference in responses

25 18 pre and post (Questions 8 and 10). The medians for each quantitative question pre and post are shown in Table 1.

26 19 Table 1: Median Values for Provider Responses Pre and Post-EHR Implementation. Pre Post Question 2 1 N/A Question Question Question Question Question 8* Question Question 10* Question Question Question Question Question Question Question See Appendix I for full survey questions. *denotes significant difference p<0.05, Wilcoxon Signed Ranks Sums Test

27 20 Of the questions tested for a significant difference, two showed a significant difference between the pre and post surveys (p<0.05), questions eight and ten (Figure 1). Survey questions with significant differences between the pre and post responses Median During the first 36 months of life, I typically assess developmental progress using the M-CHAT questionnaire: After the transition to using EHR, I expect my assessments of developmental progress to function: Pre Post Figure 1. Survey questions 8 and 10 both had significant differences in the median from pre to post implementation. This was further explored using the Wilcoxon Signed Rank Test and was found to be significant for both questions at a p-value < Question 10 was further explored to test for an association between prior EHR experience and a change of opinion in how provider s thought EHR affects their developmental assessments. Using a Chi-Square Test, there was no significant correlation (p<0.05) between a provider s

28 21 answer to the question about previous experience using EHR (Question 2), and how much their answer to question 10 changed. Qualitative Qualitative results showed the following major themes amongst providers both pre and post-ehr implementation. The most common theme across all survey questions was the perceived lack of time for developmental monitoring as illustrated by the provider quotes below. After the transition to EHR, I expect my assessments of developmental progress to function? Ability to navigate the system [EHR] will place me at a disadvantage thus slow up my visit even more. The following is the biggest barrier that keeps me from doing good developmental monitoring? time would be the biggest factor as the patients are usually scheduled 15 minutes apart and have multiple issues to address. Less available time as EHR continues to be less time efficient than the paper chart Providers also reported that availability of screening tools and issues with parental reporting negatively affected their ability to accurately assess development. The following is the biggest barrier that keeps me from doing good developmental monitoring? Variability of parent reports different depending on who is there at the visit and how they understand the questions. After the transition to EHR, I expect my assessments of developmental progress to function?

29 22 EHR will remind me to assess development and will have the questionnaires at my fingertips. PEDS is not included in EHR and finding the MCHAT is cumbersome. Common themes among providers regarding the Early Intervention referral process suggest poor communication between the provider and Early Intervention, burdensome paperwork, and issues with faxing as the major reasons providers felt the referral and follow-up process was inefficient. Once St. Chris patients are referred by our providers, the Early Intervention referral process currently works? Don t always receive updates from ChildLink. Within St. Christopher s General Pediatrics clinics, the Early Intervention referral process (up to and including sending the referral form) currently works? Too many steps required in referral process print form, fill out, fax. Since the transition to EHR, St. Chris referrals to Early Intervention function? There have been no changes [to EI referrals] since starting EHR Harder to refer because lack of resources available on EHR. Discussion The use of EHR is becoming more widespread in health care organizations because research suggests it is cost effective, efficient, and has been shown to improve the quality of care (Miller & Sim, 2004). In this study, three months after the implementation of EHR providers do

30 23 not appear to find EHR to be more efficient, specifically for completing developmental assessments. Based upon provider s responses it appears that developmental assessments and referrals function either the same or worse since the transition to EHR. Prior to EHR implementation most providers stated in their free-response answers that time was the biggest barrier to doing effective developmental assessments. In particular providers said overbooked schedules and other more pressing healthcare issues to discuss during the well-child visit were the biggest reason they had no time to conduct developmental assessments. We hypothesized that after the implementation of EHR, time would be less of an issue because providers would be able to document everything on EHR and not have the burden of paper charts and forms. After three months of using EHR providers are still citing time as one of the biggest barriers in their ability to conduct developmental assessments. This may be because it is still early in the transition process between paper-based charting and EHR at St. Christopher s. Time may still be a significant barrier because providers are getting used to using EHR. It is possible that over time, as providers become more comfortable with using EHR, we may see an improvement in the efficiency of EHR and providers may be less likely to suggest time as a significant barrier. A review of previous studies has shown that there is a transition period before providers become comfortable with EHR and begin to see the effects of its use (Gamm et al., 1998). Time was also a barrier that providers listed when specifically asked about barriers to doing developmental assessment. Numerous providers stated in their free-response answers that their assessments were worse after the transition because their documentation in EHR is far more robust and they are still trying to learn the system and therefore have less time for developmental assessments. We initially hypothesized that the transition would result in providers doing

31 24 developmental assessments more often and more thoroughly. When comparing the pre and post quantitative data there was a significant difference between provider s answers regarding how well their developmental assessments function. Overall, providers felt their assessments function worse now that EHR is in use. Based upon answers in the free-response section that followed this survey question, providers suggested a couple of reasons for this trend. Currently, the PEDS questionnaire is not available in EHR at St. Christopher s so if providers wish to assess development using the PEDS they would need to do the assessment on paper and then it would have to be scanned into the child s chart. The other screening tool used at St. Christopher s, the M-CHAT, is currently available in EHR but it is difficult to administer because it needs to be completed by the parent. In EHR parents fill out such questionnaires by using a feature called patient mode, a setting on EHR that allows the patient to fill in information without the ability to edit any other parts of the record. At the time of the three-month survey, the patient mode option was not available in EHR at St. Christopher s. The provider needs to either go through the M-CHAT questionnaire with the parent or, needs to have the parent fill it out on paper and then it has to be scanned into the child s chart. Once EHR has been in use for a longer time period providers will most likely become proficient at using the system and will be able to more easily navigate around EHR and document their assessments. In addition, once patient mode becomes available and the PEDS is mainstreamed into EHR, providers may go back to using these screening tools as they did prior to EHR. Despite the overall trend where providers felt their assessment of development was worse, some providers stated in their free-responses answers that they thought the new developmental milestones checklists in EHR were easy to use and effective. These checklists are part of developmental surveillance and contain developmental milestones from all four

32 25 developmental domains. One provider even stated they felt the new checklists were more complete than similar checklists on the paper-based charts. For this reason, some providers felt their developmental assessment had improved with EHR. Given the major differences between how providers thought EHR would affect their developmental assessments and how their perceptions changed after implementation, it may be interesting to compare the quantitative data on their responses and how they changed pre and post with how much previous experience they have with EHR. One could suspect that providers with previous EHR experience might have more accurately predicted how their developmental assessment would function once EHR was in use. While the Chi-Square Test did not show a significant correlation between previous EHR experience and how well provider s predicted the functioning of their developmental assessments in EHR, there was a general trend showing that providers who had previous experience were more likely to have consistent results about their developmental assessments before and after EHR implementation. Given the small sample size, especially with only four providers who have had previous EHR experience, it is hard to know its significance. It is possible that these results will not hold with a larger sample size or we may see that providers with previous experience in using EHR did not change their opinions about their developmental assessment pre and post. This may be because providers with previous EHR experience know what to expect during the transition from using paper-based records to electronic records. This trend is important because it suggests that the transition to EHR at St. Christopher s is proceeding similarly to transitions at other hospitals where providers previously used EHR. This could suggest that the differences in provider s perceptions about their developmental assessments at St. Christopher s may be due to the difficulties of getting accustomed to using EHR.

33 26 In addition to developmental assessment, some of the survey questions focused on the referral process with Early Intervention. Although an area where we expected improvement, there was little change after implementation. Providers expressed in their pre-survey freeresponse answers that they thought EHR would improve the referral process. In general, providers felt the referral process was not effective even before the use of EHR because of poor communication between providers and Early Intervention staff. Some providers stated that they found the referral process to be burdensome because they need to fill out a referral form and then fax the form; often they are left unsure as to whether the form even reaches ChildLink. At the same time, some providers thought the paper referral form process was streamlined and easy to use. It is possible that there was no observable change between pre and post-implementation and referrals because EHR has only been in use for three months. Often three months is not enough time for patients to receive a follow-up call from ChildLink, schedule an evaluation, and develop a treatment plan therefore, we would not expect the reporting of evaluation results to change within only three months of EHR use. This is an area with potential for improvement possibly after a few more months of EHR use. In addition, at the time of the post-survey the Early Intervention referral form was not available in EHR so providers have to go to another website to print out the form and then fax it. One week after the 3-month post-survey, the referral form became available in EHR so this may improve the process for some providers. Finally, if providers someday have the ability to fill out and electronically send the form to ChildLink from within EHR, this may improve the overall referral process.

34 27 Limitations The results of this study are limited for several reasons. First, the sample size of providers is relatively small which does not give this study much power. Including other providers such as residents may have increased the overall power of the study although that was difficult because residents started using EHR at a later date than full-time providers. In addition, the current data is only from three months after EHR implementation. One study showed that it took up to one year for providers to be comfortable with using EHR (Gamm et al., 1998). Therefore data collected before one year may not accurately represent EHR at its fullest potential Another limitation is that this study is based primarily on provider perceptions and no actual rates of developmental assessment. This is potentially problematic because providers may feel embarrassed about reporting an accurate evaluation of their developmental assessments for fear that they would be criticized for performing poorly. Including actual rates of developmental surveillance, screening and referrals from patient charts pre and post-implementation would give this study more validity. It may also be an issue that providers were already aware that St. Christopher s would be making the transition to EHR and had received basic training in using EHR prior to receiving the pre-survey. These factors may have influenced how providers answered questions in the pre-survey because they knew what to expect based upon what they had seen during their EHR training sessions. In addition, knowing that we were conducting a study looking at EHR and specifically developmental assessments may have motivated providers who normally do not conduct developmental assessments as often, to be more vigilant about doing assessments. Finally, the overall design of the study did not allow for a randomized experiment with a control group because we took advantage of a natural experimental design. This prevented the