Continuing Medical Education Program: a credit system evaluation

Original Article Continuing Medical Education Program: a credit system evaluation Programa de Educação Médica Continuada: avaliação de um sistema de créditos Claudio Schvartsman 1, Milton Glezer 2, Renato Melli Carrera 3, Ângela Tavares Paes 4, Augusto Paranhos Junior 5, Cláudio Luiz Lottenberg 6 ABSTRACT Objectives: To analyze the Continuing Medical Education (CME) program developed at the Hospital Israelita Albert Einstein, formally initiated in 2002, carried out in six successive 12-month cycles and focused on participating physicians, as to compliance with the program over time and as to educational production measured by a credit system. Methods: This is a descriptive (population) study that used a scoring model based on credit measured by time (one credit = one class hour) with formal and informal educational tools to analyze the compliance of the clinical staff with the program and the credit distribution profile regarding events promoted by the institution and elsewhere in the participants, in those who attain the goal established, and in a group that participated in six cycles and reached the institutional goal. For comparisons between the cycles, generalized estimation equations were used with normal (for means of credits) and binomial (for rates of physicians who attained the goals) distributions. Results: Comparing the six cycles, compliance of the clinical staff grew gradually and there was an increase in scoring of credits over time (growing credit mean). Distribution of the credits according to the type of activity was similar to that presented in literature, with greater representation in participation in congresses, courses, scientific meetings, and publications, among other educational modalities. Conclusions: The analysis of this credit system showed consolidation of the CME program over time with a mature and consistent attitude of the participants. Keywords: Education, medical; Education, medical, continuing; Hospitals, general; Evaluation/methods RESUMO Objetivos: Analisar o programa de Educação Médica Continuada (EMC) desenvolvido no Hospital Israelita Albert Einstein, iniciado formalmente em 2002, através de seis ciclos sucessivos de 12 meses, com foco de atenção voltado para médicos participantes, quanto à adesão ao programa no decorrer do tempo e quanto à produção educacional medida através de um sistema de créditos. Métodos: Estudo descritivo (populacional) que utilizou um modelo de pontuação através de crédito medido por tempo (um crédito = uma hora-aula), baseado em ferramentas educacionais formais e informais, analisandose a adesão do corpo clínico ao programa e o perfil da distribuição de créditos em eventos promovidos pela instituição e fora desta para os participantes, para os que cumpriram a meta estabelecida e para um grupo que participou dos seis ciclos com meta institucional atingida. Para as comparações entre os ciclos, foram utilizadas equações de estimação generalizadas com distribuição Normal (para as médias de créditos) e Binomial (para as freqüências de médicos com meta cumprida). Resultados: Comparando-se os seis ciclos, a adesão do corpo clínico cresceu gradativamente e houve aumento da pontuação de créditos no decorrer do tempo (média de créditos crescente). A distribuição dos créditos segundo o tipo de atividade foi similar ao apresentado na literatura, com maior representação a participação em congressos, cursos, reuniões científicas e publicações, entre outras modalidades educacionais. Conclusões: A análise desse sistema de créditos demonstrou consolidação do programa de EMC no decorrer do tempo com atitude madura e consistente dos participantes. Descritores: Educação médica; Educação médica continuada; Hospitais gerais; Avaliação/métodos Study carried ou at Hospital Israelita Albert Einstein HIAE, São Paulo (SP), Brazil. 1 PhD in Medicine, Faculdade de Medicina da Universidade de São Paulo - USP; Vice President of Sociedade Beneficente Israelita Brasileira Albert Einstein - SBIBAE, São Paulo (SP), Brazil. 2 Clinical director of Hospital Israelita Albert Einstein HIAE, São Paulo (SP), Brazil. 3 PhD; Hospital Israelita Albert Einstein HIAE, São Paulo (SP), Brazil. 4 Biostatistician of Hospital Israelita Albert Einstein HIAE, São Paulo (SP), Brazil. 5 PhD; Manager of Clinical Research of Instituto Israelita de Ensino e Pesquisa Albert Einstein - IIEP, São Paulo (SP), Brazil. 6 PhD; President of Sociedade Beneficente Israelita Brasileira Albert Einstein SBIBAE, São Paulo (SP), Brazil. Corresponding author: Claudio Schvartsman Avenida Albert Einstein, 627, 11º andar, sala 1.116 Morumbi CEP 05651-901 São Paulo (SP), Brasil Tel.:11 3747-3116/3661-7610 e-mail: schvartsman@einstein.br Received on Mar 19, 2008 Accepted on Oct 24, 2008

474 Schvartsman C, Glezer M, Carrera RM, Paes AT, Paranhos Junior A, Lottenberg CL INTRODUCTION The growing demand for new information in modern medicine generates the assertion that Continuing Medical Education (CME) programs are a part of current medical activity (1). The medical education process is a progressive state that should be adapted to changes in the current healthcare system, to which it should directly respond. The ideal program should induce changes based on the best evidence for action in welfare, enabling access to formal and informal sources of information within the institution (2) and also objectively assessing which educational tools are being utilized by the physicians of the institution. This concept allows for two principles: the scoring process through credits with the purpose of revalidation of clinical activity and the development of educational strategies based on the problem (3). Several medical specialty societies in other countries have carried out and participated in CME programs since the 1980s, and in this way qualified professional activity is linked to obtaining credits within a previously established period. Therefore, most recertification programs determine the type of knowledge that their members should acquire and how many credits should be awarded for the new and appropriate information (4). In 2002, the Hospital Israelita Albert Einstein (HIAE) established a credit collection system for their clinical staff as a basis for the CME program that has been maintained since then. In parallel, over time it has made available different formal and informal educational tools (5) for their clinical staff. Considering the characteristics of a private and open hospital, periodical evaluation of the updating process and behavioral change of the medical staff, that reproduces the progress of medical practice in this new millennium, has been necessary and is consistent with the various CME programs. OBJECTIVE To evaluate the participation of the clinical staff of HIAE in the CME program in its six successive 12- month cycles, initiated on August 1, 2002, up to July 31, 2008; to assess the progression of average credits obtained through formal and informal educational activities for the physicians who complete the established goal of 40 credits per cycle, and for all who participated in the program in some way; to assess the distribution of the credits obtained according to the various educational activities carried out at the institution and elsewhere. METHODS Hospital Israelita Albert Einstein (HIAE) is a highly complex private general hospital with about 500 active hospital beds and a clinical staff of approximately five thousand registered physicians; about 500 of these professionals are hired. Individuals evaluated were those who performed clinical and/or educational activities during the period in which they participated in the program. From August 2002 to July 2008, six successive 12-month cycles were analyzed. Each cycle was initiated on August 1 st and ended on July 31 st of the following year. A credit system was established for the educational activities in the medical area offered at the institution and elsewhere, according to the fundamental criterion of one credit for every class hour, prepared to be used at the institution, available at http://medicalsuite.einstein.br/ (6). The educational activities include activities in congresses (regional, national, and international), in courses and seminars (regional, national and international), publications (books, chapters, national and international journals indexed in different databases), routine teaching activities (undergraduate and graduate), distance CME, institutional norms and protocols, official scientific meetings, among others, with different levels of participation in the various activities described. In order to strengthen the internal connection with the program, the activities developed at the hospital received special scores. For an institutional goal to be achieved, a minimal margin of 40 credits per 12-month cycle was established, which is in proportion to five-year programs in which the minimal margin is 200 credits (7). The goal established of 40 credits was analyzed per 12-month cycle. In this way, besides the total number of credits, the attendance of professionals with attained goals (with 40 credits or more) and with non-attained goals (less than 40 credits) were also analyzed. From the joint evaluation of the six completed cycles, a group of physicians that participated in all cycles merited a separate analysis. The total number of credits per cycle was summarized as means ± standard errors. In order to describe the development over the cycles, bar and line graphs were used. For the comparison among the cycles, generalized estimate equation models were used considering Normal (for means of credits) and Binomial (for attendance of professionals with attained goals) distributions. A significance level of 0.05 (α = 5%) was adopted. The abovementioned tests were calculated with the help of SAS version 9.1 software.

Continuing Medical Education Program: a credit system evaluation 475 RESULTS In the period from the first cycle (August 1 st, 2002, to July 31 st, 2003) to the sixth cycle (August 1 st, 2007, to July 31 st, 2008), the absolute number of active physicians varied annually, according to the criteria adopted for the development of medical activities at the Hospital. Nevertheless, participation in the CME program grew at each cycle (Figure 1). Comparing the proportions of professionals who reached the goal, a statistically significant growth was noted from the 1 st to the 2 nd cycle (57% versus 64%, p < 0.001) and from the 2 nd to the 3 rd cycle (64% versus 68%, p = 0.0270). This percentage stabilized until the 5 th cycle and began to rise again in the 6 th cycle (69% versus 74%, p < 0.001). In absolute numbers, 821 physicians reached the goal in the 1 st cycle, with an increasing progression reaching the number of 1,757 in the 6 th cycle. The number of physicians with non-attained goals remained relatively stable over time (1 st cycle: 625; 2 nd cycle: 660; 3 rd cycle: 637; 4 th cycle: 650; 5 th cycle: 669, and 6 th cycle: 622) (Figure 1). However, it is worth pointing out that in relative numbers, the total number of non-attained goal individuals dropped over time. Participating physicians 2,500 2,000 1,500 1,000 500 0 821 1,167 1,333 1,455 1,475 1,757 625 660 637 650 669 622 1 st cycle 2 nd cycle 3 rd cycle 4 th cycle 5 th cycle 6 th cycle Not attained goal Attained goal Figure 1. Number of physicians per participation in the CME program and goal attainment, in six cycles The mean number of credits also showed expressive growth from the 1 st to the 2 nd cycle (60.0 ± 1.8 versus 79.3 ± 2.0, p < 0.001) and from the 2 nd to the 3 rd cycle (79.3 ± 2.0 versus 88.2 ± 2.1, p = 0.0001). From then on, there was stabilization on a considerable plateau, corresponding to more than the double of the institutional goal set at 40 credits (91.03 ± 2.06 in the 4 th cycle; 94.69 ± 2.13 in the 5 th cycle, and 95.22 ± 1.97 in the 6 th cycle). Among those who reached the goals in the cycles, this plateau was even more consistent, and reached three times the institutional goal (1 st cycle: 98.58 ± 2.40; 2 nd cycle: 117.64 ± 2.47; 3 rd cycle: 125.37 ± 2.51; 4 th cycle: 126.94 ± 2.45; 5 th cycle: 133.39 ± 2.51; 6 th cycle: 125.63 ± 2.26) (Figure 2). As to the evaluation of credit distribution by activity, the congresses showed the highest representation in the composition of credit means per activity, followed by courses, scientific meetings, and publications. This distribution was similar in all cycles. With the exception of the 1 st cycle, in which the credit mean in congresses and courses was a little lower, there were no significant differences among the means of credits per type of activity (Figure 3). Of the 3,910 physicians who participated in the program, 2,379 (60.8%) participated in all six cycles. Of these, 522 physicians (21.9%) reached the goal established of six cycles, with averages higher than those found for the groups analyzed previously (1 st cycle: 111.16 ± 3.27; 2 nd cycle: 135.84 ± 4.11; 3 rd Mean credits *significant 140.00 120.00 100.00 80.00 60.00 40.0 20.00 0.00 98.51 60.02 1 st cycle 117.54 79.29 Participants: 2 nd cycle < 3 rd cycle; p = 0.0001* 3 rd cycle = 4 th cycle; p = 0.4355 4 th cycle < 5 th cycle; p = 0.0292* 5 th cycle > 6 th cycle; p = 0.0101* 91.03 88.24 125.28 126.93 94.69 133.23 Goal attained: 2 nd cycle < 3 rd cycle; p < 0.0006* 3 rd cycle = 4 th cycle; p = 0.8265 4 th cycle = 5 th cycle; p = 0.1208 5 th cycle = 6 th cycle; p = 0.6294 Figure 2. Total number of credits (mean ± standard error) for total number of participants and for physicians who attained the goal ( 40 credits) in six cycles of the program Créditos 50 45 40 35 30 25 20 15 10 5 0 Congress Course Scientific meeting 95.22 125.60 2 nd cycle 3 rd cycle 4 th cycle 5 th cycle 6 th cycle Participants Goal attained Public 1 st cycle 2 nd cycle 3 rd cycle 4 th cycle 5 th cycle 6 th cycle DID ACT PREP STAN PROF LECT PROT HIAE GRAD ST SS MEC Remote CME PUBLIC = Publications; DID ACT PROF LECT = Didactic activities as professors, lecturers or speakers; PREP STAN PROT HIAE = Preparation of standards and protocols at the HIAE; GRAD ST SS MEC = Graduate studies sensu strictu credentialed by the MEC. Figure 3. Means of credits for total number of participating physicians in six cycles of the program, per activities carried out

476 Schvartsman C, Glezer M, Carrera RM, Paes AT, Paranhos Junior A, Lottenberg CL cycle: 147.76 ± 4.52; 4 th cycle: 152.79 ± 4.67; 5 th cycle: 161.84 ± 4.80; 6 th cycle: 157.26 ± 4.38). The growth of the credit mean also showed a behavior similar to the general pattern, with an increase of the credit mean between the 1 st and 2 nd cycles (p < 0.001), and between the 2 nd and 3 rd cycles (p = 0.0004), a stabilization between the 3 rd and 4 th cycles (p = 0.1725), a return to growth between the 4 th and 5 th cycles (p = 0.0288), and a new stabilization between the 5 th and 6 th cycles (p = 0.2023) (Figure 4). Credits 160,00 140,00 120,00 100,00 80,00 60,00 40,00 20,00 0,00 111,16 135,84 147,76 152,79 161,84 157,26 1 st cycle 2 nd cycle 3 rd cycle 4 th cycle 5 th cycle 6 th cycle Goal attained in 6 cycles 2 nd cycle < 3 rd cycle; p = 0.0004* 3 rd cycle = 4 th cycle; p = 0.1725 4 th cycle < 5 th cycle; p = 0.0288* 5 th cycle > 6 th cycle; p = NS *significant; NS = non significant Figure 4. Total number of credits (mean ± standard error) for physicians who attained the goal ( 40 credits) in six cycles of the program DISCUSSION The need for implantation of CME programs is well established. The Federal Council of Medicine (CFM, acronym in Portuguese), in its Resolution N o 1.755 of December 14, 2004, considered the importance of continuing medical education and determined the need for revalidation for maintenance of titles of specialist through periodical five-year cycles with programs that were to be prepared by each medical specialty society (8). The CME program of the HIAE was officially established on August 1 st, 2002, with the purpose of affording a preliminary diagnosis of the educational behavior of the clinical staff registered at this hospital. For this, the credit system composed of specific values for specific activities was the ideal tool created for this purpose. Despite the difficulty in establishing a clear cause and effect relation between the time of study and learning, we used the system based on hours of study (one hour of study = one credit) as an evaluation criterion, since it established an objective tool for carrying out educational activities. This system is defended and utilized in various worldwide reference centers (7). Similar to other CME programs, a minimal margin to be attained per period was established. Credits were distributed by categories composed by formal and informal products in which the weight of each activity presented a specific value. There are different program formats in the various centers that offer the CME, with total number of cycles varying between three and five years, requiring variable minimal scores for recertification. Since this is a hospital with an open clinical staff, the program is non-compulsory, and samples vary over time. Nonetheless, compliance with the program grew when cycles were compared over time. The credit means of the total number of participants and of those who attained the established goal became gradually greater with stabilization on an average plateau that corresponds to two or three times the established goal, showing growing and consistent interest of the clinical staff involved in the program. These two aspects are easily exemplified when we analyze the group of physicians that participated in the six cycles, contributing significantly to compliance with the program developed. The profile of this group shows the degree of fidelity and excellence of the search for constant updating. Credit distribution by activity showed the same pattern in each group, similar to what is described in the literature (9-10). Formal events with seasonal characteristics, such as national or international congresses, courses, and scientific meetings within each specialty were considered the main components in the composition of the credits obtained in all cycles. The score generated by publications merits mention, and was considered the fourth most significant score in the program, with credit means very close to those of scientific meetings. Besides the quantitative assessment developed by the system of credits, this program was also formatted to direct the possibility of educational events offered by the institution through interdisciplinary forums, national and international videoconferences, courses, symposiums, and institutional congresses. This offer, with the attendance and dimension reached, places the institution among those that actively participate in the development of continuing medical education programs (11). Extra-institutional events also exert a significant role in continuing professional development and were considered for the compilation of credits (6). Continuing professional development is the process by which the physician keeps updated, focused on the needs of his patient, on the healthcare service in which he works, and on his own development. By

Continuing Medical Education Program: a credit system evaluation 477 qualifying the process of professional development, the continuous acquisition of new knowledge, abilities, and attitudes generates competent practice, with no frank division between CME and continuing professional development (12). However, the offer and measurement of educational indicators correspond to only one component of the educational process. The translation of this knowledge into high-quality healthcare is fundamental for obtaining the desired result (12-13). CONCLUSIONS Participation of the clinical staff increased over time, as did the means of credits. Evaluation of the CME program adopted showed maturity and consistency of the clinical staff of this institution, besides displaying the importance given to constant learning in the medical field. With the consolidation of the program, besides maintenance of the system of credits developed, the promotion of strategies of educational products based on the problem promoting the development of daily practice constitutes the main focus of our future. REFERENCES 1. Matos-Ferreira A. Continuing medical education: a quality control system. Br J Urol. 1998;82(4):467-75. 2. Chan KKW. Medical education: from continuing medical education to continuing professional development. Asia Pacific Fam Med. 2002;1:88-90. 3. Barnes B. Integrating Quality Improvement, Physician Performance Improvement and CME - 2006 Spring Meeting Program. Society for Academic Continuing Medical Education, Key West, Fl-USA, 5-9 April, 2006. 4. Berube B. Royal College s CME initiative focuses on lifelong, practice-integrated learning. Can Med Assoc J. 1995;152:965-8. 5. Davidoff F. Continuing medical education resources. J Gen Intern Med. 1997;12:S15-S19. 6. Tabela de créditos da Educação Médica Continuada [Internet]. São Paulo: Hospital Israelita Albert Einstein. [cited 2006 Jul 21]. Disponível em: http:// medicalsuite.einstein.br 7. Peck C, McCall M, McLaren B, Rotem T. Continuing medical education and continuing professional development: international comparisons. BMJ. 2000;320(7232):432-5. 8. Conselho Federal de Medicina. Resolução CFM no 1.755 de 14 de dezembro de 2004 [Internet]. Brasília (DF): CFM; c 2005. [cited 2005 Jan 22]. Disponível em: http://www.portalmedico.org.br/php/pesquisa_resolucoes 9. Davis DA, O Brien MAT, Freemantle N, Wolf FM, Mazmanian P, Taylor-Vaisey A. Impact of formal continuing medical education: do conferences, workshops, rounds, and other traditional continuing education activities change physician behavior or health care outcomes? JAMA. 1999;282(9):867-74. 10. O Brien T, Fremantle N, Oxman AD, Wolf F, Davis DA, Herrin J. Continuing education meetings and workshops: effects on professional practice and health care outcomes. Cochrane review. In: The Cochrane Library, Issue 4, 2004. Oxford: update software. 11. Society for Academic Continuing Medical Education. Survey for 2004: descriptive results. [cited 2006 Jul 21]. Available from: www.sacme.org/ publications /SACME 12. Aparicio A, Willis CE. The continued evolution of the credit system. J Contin Educ Health Prof. 2005;25(3):190-7. 13. Weinberger SE, Duffy FD. Practice makes perfect or does it? Ann Intern Med. 2005;142(4):302-3.