How masters of medical education program could be evaluated in their working setting

Basic Research Journal of Social and Political Sciences Vol. 1(3) pp. 65-70 November 2012 Available online http//www.basicresearchjournals.org Copyright 2012 Basic Research Journal Full Length Research Paper How masters of medical education program could be evaluated in their working setting 1 Saeideh Ghaffarifar*, 2 Abolghasem Amini, 3 Fathieh Mortazavi, 4 Sedigheh Sadat Tavafian, 5 Fatemeh Sadeghi-Ghyassi 1 Medical education Department, Student of Health Education. Research Center of medical Education, EDC, Tabriz University of Medical Sciences. 2 Department of Medical Education, Research Center of medical education, EDC, Tabriz University of Medical Sciences. 3 EDC, Shahid Beheshti University of Medical Sciences. 4 Department of Health Education, Medical School, University of Tabiat Modarres 5 Iranian Center for Evidence-Based Medicine, Tabriz University of Medical Sciences. *Corresponding author email: sa.ghafarifar@yahoo.com Accepted 20 November, 2012 Purpose: Master course of medical education in Iran has been launched since 2002 to improve the quality of health services. Reports received from Iranian medical universities show most of graduates in this field are not aware of their professional role. In this study, we explain how a valid questionnaire for professional role evaluation of medical education masters could be designed and validated. Method: The primary draft of a researcher-maid questionnaire was designed in 2011 based on a literature review. To do validation process, the view points of 20 expert were obtained to determine content and face validity of the questionnaire through Content Validity Ratio (CVR) and Content Validity Index (CVI) at the levels of >0.51 and >0.79 respectively. We applied the index of average expressed variance to report convergence validity of items being acceptable at the level of at least 0.5 doing an exploratory factor analysis with the cut point of 0.5 to determine dimensions of the questionnaire. Results: According to 14 experts comments on the quantity and quality of the questionnaire, 9 items which did not meet preassumed criteria were eliminated, so 16 items remained in the questionnaire. Bartlett s test of Sphericity (Sig: 0.000) approved appropriateness of the data to do exploratory factor analysis that led to extraction of 8 factors with Eigen value more than 0.941. Conclusions: The findings of the study showed the questionnaire could assess the professional role of masters graduated from medical education program in Iran. However, further researches are needed to improve the questionnaire in future. Keywords: Medical education; professional role; graduate; evaluation tool; master INTRODUCTION Master program of Medical Education in Iran was launched in 2002, and despite being young, very soon, many health policy makers hoped to basically solve the health problems by the help of scholars and graduates of this field. So-called trusts led politicians of health care system to setting objectives like all Iranians should know all the different aspects of health including physical, emotional, psychological, social and spiritual, and achieve optimal level of health, and they should obtain a positive attitude toward health promotion in order to receive the most improved health services necessary to the prevention, treatment and rehabilitation with the best quality, easiest way and the fastest time when needed. It seems health policy makers have considered the weight of the objective of getting better quality of services higher than other above-mentioned goals. American Association of Medical Colleges (AAMC) believes that a community-centered health system provides rich and meaningful learning experiences for students of medical sciences. Such health-education systems provide simultaneously integrated ambulatory and hospital cares across the community. 1 With a deep professional look from medical educationists points of view on the objectives of launching the master

Ghaffarifar et al. 65 program of medical education, a major requirement reveals itself that medical students should be taught in the line of the real needs of the community in order to learn the three levels of prevention (Primary: concerned with health promotion activities that prevent the actual occurrence of a specific illness or disease, Secondary :promotes early detection or screening and treatment of disease and limitation of disability. This level of prevention is also called HEALTH MAINTENANCE. Tertiary: -directed towards recovery or rehabilitation of a disease or condition after the disease has been developed) and serve them at the community centers. 1 To fulfill this requirement, first, curriculums and course plans in all fields of medical sciences should necessarily be revised. By such doing, highly motivated selected candidates will make significant changes after graduation whether in the learning context, input, output and learning-teaching processes. Second, students during this program should be able to obtain competencies of effective, efficient and targeted learning, and easily achieve learning facilities which they need. In so-called education system, medical students' motivation and intention to acquire the necessary competencies is expected to reinforce even further than which has been anticipated in their educational curriculum, and the teachers issue the license to graduating students through valid, reliable and fair evaluations whether formative or summative. It means, the certificates will be issued to the capable graduates who will solve non-previously faced problems at unfamiliar educational settings. In other words, medical education masters graduated from revised curriculum are expected to make the best decisions with the help of reordering their previous experiences and making relationship with new ones. The question that arises here is that who will be responsible for these logical changes? Or in order to achieve a community- centered health education system, who(s) will seriously reflect positive applied and scientific experiences of universities of medical sciences and share them with others? If the answer accurately points to the graduates of medical education, the next question is, whether graduates in this field acquire the competencies needed for performing this great responsibility during their studying? More importantly, are they well aware of their professional role? Data hiding in the shadows of previous inductive research which reviewed medical education master degree curriculum in Iran in 2010, and reports from other universities in Iran (Dadghostar et al., 2011; Javdani, 2010; Fasihi et al., 2004; Dehghani et al., 2007; Bakhshi et al., 2002; Javadi et al., 2002) created this hypothesis in our mind, that most master graduates of medical education are not aware of their professional roles. Therefore, it is necessary to test this hypothesis, to investigate if master program of medical education have addressed, established and improved the competencies of medical education graduates to fulfill their professional roles or not. This study describes the process to design and validate a questionnaire in order to evaluate competencies of masters of medical education on their own points of view as a part of a master-thesis. METHOD In this study, the operational definition of professional role was determined reviewing the literature (Kachur, 2012; Wojtczak, 2012; Whitcomb, 2005, 2007; Coulehan, 2005; Khadem et al., 2008; Farzinpour et al., 2010; Ferguson et al., 2002; Lurie et al., 2009) and a draft of the items of the researcher- maid questionnaire were made available. A panel of 20 experts in the field of medical education judged the content validity of the questionnaire. In this way, those 20 experts were asked to specify the domain that the questionnaire measures.they were also asked to determine the area each item of the questionnaire evaluates. Then, they gave us feedback how the questionnaire could assess the professional role of masters graduated from medical education course, in addition to their comments to revise some items, even to omit a few of them. Qualitative evaluation of the questionnaire was also done based on their feedback receiving necessary corrections regarding grammar, wording, item allocation, and scaling of the items. Content Validity Ratio (CVR) and the Content Validity Index (CVI) were used to evaluate quantitative content validity of the questionnaire. Content validity of the items was rated based on a threepart scale and minimum validity of 0.51. The CVI score related to simplicity, relevance and clarity of each item was calculated on a four-part Likert scale, and items with CVI score higher than 0.79 were considered acceptable for the study. (Rimm and Jerusalem, 1999; Scholz et al., 2002; Thurstone, 1947; Hajizade and Asghari, 2011). The questionnaire was e-mailed to 112 masters of medical education who we had their e-mail address. They were asked to participate in this study and send us the completed one if they were willing to anonymously cooperation. The results of the study were obtained using SPSS.16 (Statistical Package for the Social Sciences, version 16). The convergence validity of the construct of professional role in this research was determined by Average Variance Explained (AVE), and its minimal acceptable level was assumed to be at least 0.5. Exploratory factor analysis (EFA) was used to determine dimensions of the construct of professional role. To assess the suitability of present data to perform an exploratory factor analysis Bartlett s Test of Sphericity was done. In order to data reduction, the number of factors for extraction was considered eight factors, Varimax rotation method was used, and absolute values less than 0.5 were suppressed. Maximum Iteration for

66. Basic Res. J. Soc. Polit. Sci. Table 1. Statistics presents content and face validity of the questionnaire to evaluate professional role of 94 graduates of medical education according to a panel of 14 experts Item Studying in this program has CVR CVI CVI CVI CVI Impact Number increased: My. (Simplicity) (Relevance) (Clarity) (Scale) Score 1 Skills respecting the honesty in 0.57 0.785 0.85 0.92 0.85 4.19 medicine. 2 Skills and capabilities regarding altruism. 1 0.92 0.92 0.71 0.85 4.44 3 Skills and capabilities regarding 1 0.92 0.92 0.92 0.92 4.67 respectfulness to others. 4 Interest to co-operate with university and 0.71 0.85 0.64 0.92 0.80 3.30 community service providers. 5 Motivation for learning to respond to the 0.86 0.85 0.71 0.92 0.83 3.76 needs of the community. 6 Motivation for proposing new methods of 0.86 0.85 0.92 0.71 0.83 3.30 teaching and learning to faculty members. 7 Awareness of my roles and 1 0.92 0.64 0.92 0.82 4.02 responsibilities in health system. 8 Knowledge about morals of practicing in 0.86 0.92 0.57 0.92 0.80 2.80 this profession 9 Active participation in curriculum 0.71 0.785 0.92 0.85 0.85 2.25 planning, revisions when necessary in college or university 10 Contribution on making decisions in 0.71 0.92 0.64 0.85 0.80 3.30 educational departments. 11 Commitment to make proper changes in 0.86 0.785 0.92 0.85 0.85 4.30 the learning environment. 12 Contribution on production of articles in 0.71 0.92 0.785 0.71 0.80 3.31 medical education. 13 Contribution on provision of research 0.57 0.85 0.785 0.92 0.85 3.30 projects in medical education. 14 Readiness to accept my new 0.71 0.92 0.92 0.64 0.83 4.19 professional role. 15 Knowledge about interdisciplinary cooperations. 0.57 0.92 0.785 0.71 0.80 2.98 16 Professional status and popularity. 0.71 0.785 0.64 0.85 0.758 2.25 convergence was considered 25. (Rimm and Jerusalem, 1999; Scholz et al., 2002; Thurstone, 1947; Hajizade and Asghari, 2011). RESULTS The draft of our researcher-made questionnaire with 25 items was designed presenting professional characteristics of masters of medical education. Those characteristics were extracted from existing literature in the field of medical education, and were included: altruism, accountability, dutifulness, loyalty, trustworthiness, honesty, respectfulness for others and providing the most efficient and effective service in the health care system based on what she/he has been taught. Equal and more similar items were excluded from the initial questionnaire based on the results of 20 experts opinions. Because of the lack of consensus on the judgment of experts, the opinions of 14 experts were used in the rest of study in order to qualitative and quantitative validation of the questionnaire. In this way, 9 Items of the 25 original items did not meet pre-assumed criteria and were omitted due to lack of a minimum content validity based on judgment of 14 experts. It remained 16 items, and content validity index (CVI) was calculated for each item based on mean CVI for relevance, clarity and simplicity of that item, according to the experts' comments and recommendations. (Table 1) 94 masters of medical education completed the questionnaire in a volunteer and anonymous type of participation. The results of the study were obtained using SPSS.16 based on the Maximum Iteration for convergence equal to12. Statistics of Kaiser-Meyer-Olkin (KMO) in this study was 0.474.The level of Bartlett's test of sphericity was significant (Sig: 0.000) which showed the suitability of the data of the study to perform an exploratory factor analysis. With regard to the extraction of eight factors, eight factors with Eigen values of 0.941 were extracted. Scree plot (Figure1) shows the adequacy of 8 items to evaluate the construct of professional role.

Ghaffarifar et al. 67 Figure 1. Eigen values of 16 items of the questionnaire to evaluate the professional role of masters of medical education Table 2. Rotated Component Matrix of the construct of professional role for its evaluation in masters of medical education Item q2.763 q5.745 q16 -.608 q15 -.786 Component 1 2 3 4 5 6 7 8 q6.643 q9.857 q7.585 q1.807 q11 -.611 q8.773 q3.575 q4 q14.843 q12.863 q13.887 q10 Extraction Method: Principal Component Analysis. Rotation Method: Varimax with Kaiser Normalization. Rotation converged in 12 iterations. Item 13 had more communality (0.813) with the construct of professional role, and the lowest communality belonged to item 16 ( 0.546).The Total Variance Explained (TVE) was equal to 69.286%. Table 2 shows component matrix after Varimax rotation.14 experts of medical education in our study rearranged component matrix, as much as it was possible for them. Table 3 provides re-structured component matrix regarding their points of view. It led to initial designing of the questionnaire of our research to assess the professional role of masters graduated from medical education program in Iran.

68. Basic Res. J. Soc. Polit. Sci. Table 3. Eight factors which measure the professional role of medical education graduates Factor Item Studying in this program has increased: My. Number 1 2 Skills and capabilities regarding altruism. 5 Motivation for learning to respond to the needs of the community. 16 Professional status and popularity. 2 15 Knowledge about interdisciplinary co-operations. 6 Motivation for proposing new methods of teaching and learning to faculty members. 3 9 Active participation in curriculum planning, revisions when necessary in college or university. 7 Awareness of my roles and responsibilities in health system. 4 1 Skills respecting the honesty in medicine. 11 Commitment to make proper changes in the learning environment. 5 8 Knowledge about morals of practicing in this profession. 3 Skills and capabilities regarding respectfulness to others. 6 14 Readiness to accept my new professional role. 4 Interest to co-operate with university and community service providers. 7 12 Contribution on production of articles in medical education. 8 13 Contribution on provision of research projects in medical education. 10 Contribution on making decisions in educational departments. CONCLUSIONS On one hand, the findings of this study create a mentality that the items of the designed questionnaire could reflect appropriateness of the tool to evaluate the professional role of masters of medical education. On our points of view, taking some scientific steps supports this idea, while keeping their all strengths and weaknesses in our mind. First, according to evidences, fixing a theoretical (conceptual) and functional (operational) definition of a construct, specifying the domain that a questionnaire should measure in addition to determining the area each item of a questionnaire should evaluate is the first step in designing of a questionnaire. (Rubio et al., 2003). It was the same as we did in this study. Based on a literature review, we determined how each graduate in the field of medical education should behave in a real working setting. In other words, a professional master of medical education in our researcher-maid questionnaire was introduced with traits of altruism, accountability, dutifulness, loyalty, trustworthiness, honesty, respectfulness for others and providing the most efficient and effective service in the health care system based on what she/he has been taught. Second, as evidences express judgment of a minimum of 2 and maximum of 20 professional experts are needed to content validation of a questionnaire. (Rubio et al., 2003; Grant and Davis, 1997). So, we asked a panel of 20 experts to judge on our researcher-made questionnaire draft of 25 items. Third, studies show content validation of a questionnaire requires diagnosis of irrational, nonerelated and none- quality items. (Rubio et al., 2003; Grant and Davis, 1997). Hence, 9 out of the 25 items in this study were eliminated due to lack of content validity based on judgment of a minimum of 14 experts. On the other hand, Results of construct validation of the questionnaire witness the fact that after the removal of nine inappropriate items, 16 remaining items are able to make a tool to measure the construct of professional role in the behavior of masters of medical education. The total variance explained (TVE) equal to 69.286% confirms convergence validity of this construct. It means the eight extracted factors could express 69% of the changes of the construct of professional role. In all, our research, by using Google and Yahoo search engines in Persian and English language databases, did not end to finding similar questionnaire to evaluate professional role of masters of medical education. It also gives us encouragement to think that the questionnaire was designed in this study could help health policy makers to evaluate masters professional role to assess how much medical education curriculum planning and its implementation has met the pre-determined goals and objectives in training capable masters to fulfill community demands. So, when it comes to policy makers expectations that all Iranians should know all the different aspects of health, this questionnaire could help them if medical education graduates are able to help the people to manage their health problems or not. Besides, this questionnaire would help us to test our previously mentioned hypothesis of that most of masters graduated from medical education program are not aware of their professional role. Recommendations and limitations Although this study had some good positive points, and led to the development of a valid tool for evaluating the professional role of medical education graduates, but like

Ghaffarifar et al. 69 all other studies was not without any limitation. As noted in the test results statistics of Kaiser-Meyer- Olkin (KMO) in this study was low (<0.5). The researchers believe that it is because of regular annual conventions of students and experts in the field of medical education. Such regular congresses, as hidden focus groups have created closely similar thoughts among the members of medical education community. So, the normal distribution of responses to most of the items has been strongly affected, and that is why we had the left or right skewness in response to the vast majority of the items. It is suggested to use the questionnaire in the other fields of medical education which they could provide more possibilities and opportunities for researchers to do a better factor and component analysis with KMO higher than this study which verifies sampling adequacy. It also suggested repeating factor analysis on the questionnaire data, using a 7-part Likert scale or more, in another study and comparing its findings with the results of this study. Because, we think such doing would probably let researchers to work on more normallydistributed data and achieve more acceptable factor analysis with higher KMOs. On the other hand, in this study when it came to content validation of the questionnaire, exactly the same as tradition and old habits, the items with the greatest overlap were excluded according to experts points of view to avoid duplication. It seems to be a confounding and intervening factor in this study which led to low KMO statistic. Hence, it is highly recommended to reach a consensus on the criteria of item-overlapping in future studies, before obtaining opinions of an expert panel. Because, removing some of the items that are in fact complement to each other rather than being the same could reduce accuracy of the predictive and explanatory power of the desired construct. It is also highly recommended to compare the predictive power of the construct of professional role of this tool with other similar tools if other researchers had access to them. It will not only help us to have further discussion on the designed questionnaire, but also lead to its development for conducting some robust studies in future. With regard to the concern of professional role being new among the masters of medical education in addition to the great challenges which have been reported by researchers all around the world facing its teaching and assessment in all fields of medical education, it is strongly suggested doing further evaluation of the reliability and internal consistency of the questionnaire by conducting other studies on different populations. Low number of participants in the study who completed the questionnaires was among other limitations which can be noted, and was resulted from low number of masters graduated from medical education program in Iran until the time this study was conducted and implemented. We hope to do future studies with larger sample sizes in order to overcome this limitation, and develop this questionnaire which was basically designed to evaluate professional role of masters of medical education on their points of view longing for meeting health policy makers objectives when it comes to committing the supervision of delivering heath services with the best quality to the masters of medical education. ACKNOWLEDGEMENT The authors wish to thank all scholars and graduates of medical education who participated and cooperated in this study. They will also gratefully appreciate precious advices of the professors of Biostatistics, Dr Ebrahim Hajizadeh and Dr Soghrat Faghihzadeh. The results of this study strongly indebted to the cooperation of officials and experts from the Education and Development Center of Shahid Beheshti University of Medical Sciences REFERENCES Bakhshi H, Esmaeelzadeh M, Taleghani F, Rafiee G, Mahdizadeh K (2002). Employment status of Rafsanjan Medical Sciences University graduates. J. Qazvin Univ. Med. Sci. 22:16-23. [In Persian] Coulehan J (2005). Viewpoint: today's professionalism: engaging the mind but not the heart. Acad. Med. 80(10):892-898. Dadghostar M, Changiz T, Vafamehr V (2011). Designing a tool for evaluation of a clinical course (a guide for implementation). Iran J Med Sci. 10(5):755-766. [In Persian] Dehghani MR, Kamyabi A, Haghdoost AA, Ruholamini A, Hosseini Nejad Z (2007). Occupational status of graduates of medical education sciences between years 2001 to 2007, Kerman University of Medical Sciences. Strides Dev. Med. Educ. 4(2):67-76. [In Persian] Farzinpour F, Emami AH, Davari Tanha F, Hosseini S, Farzanehnezhad AR (2010). Educational programs quality assessment based on graduates comments. Iran Red Crescent Med. J. 12(3):302-307. Fasihi Harandi T, Soltani Arabshahi SK, Tahamy SA, Mohammad Alizadeh S (2004). Quality of clinical education of the points of medical students view. J. Qazvin Univ. Med. Sci. 3:4-9. [In Persian] Ferguson E, James D, Madeley L (2002). Factors associated with success in medical School: systematic review of the literature. BMJ. 324(7343):952-7. Grant JS, Davis LL (1997). Selection and Use of Content Experts for Instrument Development. Res Nurs Health. 20(3):269 274. Hajizade E, Asghari M (2011). Statistical Methods and Analyses In health and biosciences. A research methodological approach. Iran, Jahad Daneshghahi. Javadi HR, Asefzadeh S, Mashatan M (2002). Qazvin Medical Sciences University graduates' professional situation. J. Qazvin Univ. Med. Sci. 22:24-32. [In Persian] Javdani H (2010). Globalization and higher education, the gap between theory and practice: new strategies for the development of higher education. J. Res. Planning in Higher Educ. 53:107-131. [In Persian] Kachur EK (2012). US medical education of glossary of terms and acronyms. ( http://mededdevelopment.com/glossary.htm). Accessed September 02. Khadem N, Khodadadi N, Ebrahim-Zadeh S (2008). Opinions of Mashhad Medical School graduates about coordination of educational goals of their program with medical services provided to the community. Med. J. Mashad. Univ. Med. Sci. 4:233-238. [In Persian] Lurie SJ, Mooney CJ, Lyness JM (2009). Measurement of the General

70. Basic Res. J. Soc. Polit. Sci. Competencies of the Accreditation Council for Graduate Medical Education: A Systematic Review. Acad. Med. 84(3):301-309. Rimm H, Jerusalem M (1999). Adaptation and validation of an Estonian version of the General Self-Efficacy Scale (ESES). Anxiety Stress Coping. 12(3):329-345. Rubio DMG, Berg-Weger M, Tebb SS, Lee ES, Rauch SH (2003). Objectifying content validity: Conducting a content validity study in social Work Research. Soc. Work Res. 27(2):94-104. Scholz U, Gutiérrez-Doña B, Sud S, Schwarzer R (2002). Is perceived self-efficacy a universal construct? Psychometric findings from 25 countries. Eur. J. Psychol. Assess. 18(3):242-251. Thurstone LL (1947). Multiple - factor analysis. Chicago: University of Chicago Press. Whitcomb ME (2005). Medical professionalism: can it be taught?. Acad. Med. 80(10):883-884. Whitcomb ME (2007). Professionalism in medicine. Acad. Med. 82(11):1009. Wojtczak A ( 2012). Glossary of medical education terms. (http://www.iime.org/glossary.htm). Revised February 2002. Accessed September 02.