FACULTY MENTORING PROGRAM EVALUATION

Transcription

1 1 FACULTY MENTORING PROGRAM EVALUATION Description and Goals The Faculty Mentoring Program for Women (FMPW) was created after critical assessment of the experiences of other institutions of higher education within a framework of professional development to provide field-specific and institution-specific career guidance to junior female faculty. 1 The pilot FMPW headed by the Center for Teaching and Learning (CETaL) director Dr. Christine Reimers started in While addressing the participating faculty in one of the focus groups meetings, the former evaluator described the program s goals: We want to increase your effectiveness and visibility through improved access to information and resources on campus. We see those resources and information supporting your academic activities and networking across the campus and community. We want to facilitate the attainment of your individual strategic [and] career objectives, particularly providing you an environment where you can discuss conflicting demands between teaching, service and research and also family obligations We really want to facilitate your understanding of how to set priorities in the context at UTEP. Formally stated, the program goals are: Access. Increase women faculty s effectiveness and visibility through improved access to information and resources that support academic activities and to networking opportunities for new ventures and relationships. 1 For more information on FMPW, see:

2 2 Balancing Work Loads. Facilitate the attainment of individual strategic career objectives by providing an environment where women faculty can discuss the conflicting demands of work and family and prioritizing the diverse demands of teaching, research, and service. Acculturation. Orient new women faculty to UTEP, including helping them avoid mere trial and error approaches to managing relationships with students, faculty, staff, and administrators. Mentors among senior faculty were selected to assist and advise junior female faculty regarding collaboration in research and publications, promotion and tenure expectations, department and university service, teaching, etc. One particularity of UTEP s program is that it invites all new women faculty into the program and then matches them with their mentors. Many other mentoring programs provide mentoring for only a subset of new faculty and have mechanisms in place for the mentees to choose their own mentors. The mentors in the FMPW specialize in providing performancerelated and career-related advice. Their role is critical in assisting mentees in managing their career, and specifying their goals, plans, and actions. Mentors are encouraged by program administration to develop strong collegial ties with mentees and provide informal social support to mentees with regard to new opportunities, liaisons, and resources. In the first year of the program it held monthly meetings on a variety of topics to which both mentors and mentees were invited. After the first year, additional monthly brownbag meetings for mentees only were instituted. The informal monthly meetings serve to discuss mentees general academic and organizational experience, to re-define their relationship with their mentors, to share acquired communicative skills and to work

3 3 on improvement of the program. They also provide new perspectives about critical issues, generate solutions for problems being experienced by the new faculty, and encourage mentors and mentees to apply their decisions to their particular situations. Past Evaluations The FMPW has maintained a high level of formative evaluation through such means as focus group reports, transcriptions of activities, and mentor s and mentees questionnaires. It is noteworthy that the first wave of evaluations was performed by external evaluators in , which is the pre-advance year. Originally, the first wave of evaluation was monthly. Later, an annual cycle of evaluation was adopted. Wave I evaluations were conducted using the methodology of Women Faculty Mentoring Program, University of Wisconsin-Madison. In later years, the program adopted different methodology of internal evaluation. Because of the incompatibility of questionnaires used Wave I with questionnaires used in later years, the cross-wave comparison of findings, starting at Wave I, is not feasible. The brief summary of the findings of Wave I is provided in the next section. The results of later waves were compared and analyzed across different FMPW areas and components. The list of the evaluation waves and surveys that correspond to them is presented below. 2 Wave I (Pre-ADVANCE, ): First year was End-of-the Year Survey of Mentors and Mentees. This survey was also complemented by twelve monthly surveys. Wave II (Pre-ADVANCE, ): First year was For parsimony, the surveys will be henceforth referred by their numbers in the list presented below. For example, the survey of veteran mentees conducted in will be referred as Survey 2.2.

4 Survey of Mentees (N=5) Survey of Mentees in Focus Groups (N=9) Survey of Mentors (N=11). These surveys were complemented by the follow-up qualitative interviews with mentees. Wave III (Pre-ADVANCE, ): 3.1. Survey of Mentees in Focus Groups (N=8) Survey of Veteran Mentees, Cohort of (N=14) Survey of Mentors (N=8). These surveys were complemented by the transcriptions of focus groups meetings. Wave IV (ADVANCE, ): 4.1. Survey of Mentees (N=6) Survey of Mentees in Focus Groups (N=6) Survey of Mentors in Focus Groups (N=6). Wave V (ADVANCE, ): 5.1. Survey of Mentees in Focus Groups, 6-Months Assessment (N=3) Survey of Mentees in Focus Groups, 18-Months Assessment (N=6) Survey of Mentees, 18-Month Assessment (N=2) Survey of Mentees, 6-Month Assessment (N=1). Brief Summary of Wave I Findings At the end of the first year of the program, attendees were asked to complete the questionnaire elaborated by Women s Faculty Mentoring Program, University of Wisconsin-Madison. A total of twenty-three questionnaires were completed and

5 5 returned. Participants, both mentors and mentees, were asked to evaluate success of the following planned activities: o October Partnering Agreement; o November Luncheon featuring successful mentor/mentee relationship; o January Focus Group; o February Workshop on personality and communication styles; o March Luncheon on Issues of Balance. In general, participants, who completed the questionnaire, found the planned activities moderately useful to very useful. Almost 90% of the respondents reported that all but one of the activities were moderately to very useful. Next, mentees and mentors were asked to rate their level of satisfaction with the support for academic activities and networking opportunities. In the case of support for academic activities, 83% of participants reported they were moderately to very satisfied with both kinds of support, while in the case of networking opportunities, the share of those who were satisfied was even higher 89%. The next series of questions deal with the FMPW assistance in a mentee s acculturation to the university that included relationships with administrators, other faculty and staff, and the public. Above eighty percent of the mentees and mentors indicated that they were moderately to very satisfied with how participation in the program assisted in the mentee s acculturation into UTEP. Tthe participants were asked about their approval of FMPW activites aimed at developing environments in which new women faculty could discuss the issue of conflicting demands of family and work and could discuss the prioritizing of research,

6 6 teaching and service. Over 70% of the respondents reported that they were moderately satisfied or very satisfied with the development of safe environments in which the issue of conflicting demands could be discusses. Eighty-five percent of the respondents reported that they were able to discuss prioritizing demands that included advice on the allocation of time for research, teaching, service and activities outside the university. Summary of Findings from Waves I, II, III, IV Although these evaluation surveys were conducted at different times, they encompass a series of questions that remained largely unchanged during the years. Below is an account of the closed-ended questions that FMPW participants were asked repeatedly in the evaluation surveys listed above. The closed-ended questions were assessed on a Likert scale with not useful, somewhat useful and very useful being options. Since Wave V questionnaires uses 6-item Likert scale, with options ranging from 1= not useful to 6= very useful, separate analyses have been conducted for this wave. The analyses of the findings from Wave V are included later in this report. The summary of the responses to the closed-ended questions in Waves I-IV is provided in Table 1. The columns present the percentages of respondents who considered their experiences corresponding to the questions as very useful or useful. Table 2 shows the frequency of occurrence recorded in the survey of veteran mentees in This is the only survey that provides information on how often interactions or discussions regarding a specific subject related to the close-ended questions occurred.

7 7 The first closed-ended question that appears virtually unchanged on all questionnaires concerns the satisfaction of mentees (or mentors) with the interactions that they had during the program with their respective mentors (or mentees). It is quite remarkable that in most evaluation surveys all respondents (100%) indicated that they considered these interactions productive. The only survey that stands alone because not all mentees considered meetings with their mentors to be useful is the survey of mentees conducted during focus group meetings in Even in this survey only 1 mentee did not find the productivity of interactions with her mentor satisfactory. Moreover, three of out of six (50%) respondents in this survey answered very useful in response to the question of utility of the encounters with their mentors. Similarly, interactions with mentoring team and individual mentees were rated as very useful by the majority of respondents in this survey. 3 With respect to the frequency of interactions reported by veteran mentees in 2003, 50% reported having at least 1 interaction with their mentors and 25% reported these interactions being continued even after leaving the program (see Table 2). Similarly, 62% of mentees reported having at least one interaction with other mentees, mentors and program staff during program meetings and workshops, and half of the answers to the question of interactions with the mentee peers fell into the category at least one. Observe that 42% of mentees reported that they still interact with their peers after having finished the program. The next groups of questions (2, 3 and 4) relate to the value of discussions about research, teaching, and service, respectively. Although the responses to these questions differ across the evaluation surveys, on average, the vast majority of respondents (both 3 Only the last two surveys conducted in 2005 included asked mentees to rate the utility of interactions with their peers.

8 8 mentors and mentees) were satisfied with the discussions about research, teaching, and service. Specifically, in 4 out of 7 evaluation surveys in Table 1 all respondents were satisfied with the discussions about teaching they had with their mentor (mentees). Only in one survey, the survey of mentees in 2005, half of the respondents (3 out of 6) reported medium to high levels of satisfaction with these discussions. The frequency of discussions about teaching was about the same as that of interactions along three program components, with 56, 70 and 55% of respondents reporting having at least one occurrence of such a discussion with their mentors, in program meetings and with their peers, respectively. Discussions about research were among the most highly rated program activities, as in 8 out of 9 evaluation surveys 100% of respondents deemed these discussions to be useful or very useful. Even in the survey of mentees in the focus groups of 2005, the only survey where not all respondents unanimously acclaimed satisfaction with this activity, still three valid answers two fell into category somewhat useful and one into very useful. Additionally, the findings in Table 1 indicate that discussions about research, from the standpoint of both mentees and mentors, were more productive than similar discussions about teaching and service. Table 2 indicates that 56% of veteran mentees reported having discussed teaching at least once with their mentors, 70% with their peers during workshops, and 55% with their peers in groups. The discussions about service received high, if not overwhelming, appraisal of mentees in evaluation surveys. Discussions concerning service to the University were rated as useful or very useful by 100% respondents in 5 out of 7 evaluation surveys during the years Eighty-six percent of respondents in survey of mentors,

9 9 2003, and 89% in survey of mentees in focus groups, 2005, gave high marks in their evaluation of this activity. Additionally, Table 2 demonstrates that 38% mentees reported having discussed service to the University with their mentors, 30% in program meetings and workshops and 58%with their mentee peers. Consequently, the evidence presented above suggests that discussions about service among fellow mentees were more frequent than similar group discussion in program workshops and with mentors. The next two questions deal with support for academic career and networking. Although the level of satisfaction of respondents with these activities is, generally, lower than the level of satisfaction with discussions concerning teaching, research and service, the majority of respondents in every evaluation survey taken during attach high value to FMPW in providing support for their academic career and connecting them with people across campus. Generally, support for the academic career received higher satisfaction marks by FMPW participants (96%, averaging across the surveys) than networking (83%). From the following it can be inferred that, through their communication with both mentors and peers, the majority of mentees are satisfied with the FMPW assistance towards their academic career and networking. Furthermore, veteran mentees in 2003 indicated they were less likely to talk with their mentors about support for academic career than about networking, as 36% mentioned talking at least once with their mentors about obtaining resources for their academic career compared to 50% of those who reported having at least one discussion about networking. On the other hand, the topic of resources and support for academic success was more frequently discussed in mentee-to-mentee interactions than the topic of networking. Compare seventy percent of veteran mentees reporting the topic of resources being mentioned with

10 10 their peers at least once to 40% of those who reported networking being mentioned at least once. The respondents in evaluation surveys were asked a question which referred to assistance in connecting with people in the El Paso/Ciudad Juarez community. In 4 out of 7 evaluation surveys, participants unanimously believed that they were satisfied with the assistance in connecting to community that they obtained (in the case of mentees) or that they provided (in the case of mentors). However, the strength of belief about the help that the program provided with respect to networking across the El Paso/Ciudad Juarez community has been eroded with time among mentees. In 2005, a lower percentage of mentees (averaged 50% across Surveys 4.1, 4.2 and 4.3) found the assistance in community networking to be satisfactory. With respect to the frequency of occurrence, community networking, according to veteran mentee accounts in 2003, was ever discussed in interactions with their mentor in only 21% of cases, while similar discussions had taken place at least once in program workshops (58%). Generally, both mentees and mentors positively assessed the FMPW contribution to balancing the demands of work, family, teaching, research and service. The question which prompts respondents to evaluate their experiences working with mentors and other mentees on balancing these demands was answered as useful or very useful by 82% of respondents, the average across the evaluation surveys in With respect to range, the positive responses from participants about the assistance provided by FMPW in balancing the demands of work, family, teaching, research and service vary across surveys without a distinct time pattern from 50% to 100%. According to the veteran mentees survey in 2003, this topic was not frequently discussed either in mentor-mentee

11 11 pairs, or in program-sponsored workshops. Approximately 30% of respondents noted that they discussed the balance issue with their mentors and peers in program workshops. However, the frequency of discussions concerning balancing the conflicting demands was high among fellow mentees, as evidenced by the fact that those who reported never talking about this problem with peers amount only to 25%, while the remaining 75% is divided equally between those who had already conversed with peers and those who were currently having discussion with peers about this issue. This implies that the FMPW, through the regular meetings and brownbags, facilitated mentee-to-mentee discussions of issues involving balancing work and family, though the mentors were less likely to take part in these discussions. The answers to the next question regarding assistance/advice in prioritizing responsibilities (semantically similar to the previous question) were largely positive in evaluation surveys. On average, 80% of program participants, both mentors and mentees, assessed as useful or very useful the assistance or advice that they gave (mentors) or received (mentees) in prioritizing responsibilities. It should be noted that, although the distribution of answers to this question in evaluation surveys was more or less even, the only survey where respondents were unanimous in their positive assessment of this topic was the survey of mentors, According to the veteran mentee survey of 2003, mentee-to-mentee interactions and program seminars more frequently addressed this issue, than interactions of mentor-mentee pairs (see Table 2). Acculturation to the University which has been stated as one of the primary goals of FMPW is the subject of the next question in the evaluation surveys, Respondents were asked to assess the overall utility of their communication with mentors

12 12 and fellow mentees in assisting their socialization with the culture of UTEP. The vast majority of responses in all surveys fell into useful or very useful category, giving an average of 90%. Similarly to the preceding question, Table 2 shows that the incidence of communication about adjusting to UTEP in mentor-mentee pairs was lower than in mentee-mentee pairs and during program workshops. It should also be mentioned that many veteran mentees continued to help each other with the adjustment to UTEP, as 38% of them in 2003 gave this activity an ongoing status. Hence, it can be inferred with certainty that FMPW helped significantly in implanting the UTEP culture in junior female faculty. In sum, the analyses presented in Tables 1 and 2 indicate that, regardless of the area in which FMPW made an effort to assist junior female faculty, the overwhelming majority of mentees and mentors alike across the evaluation survey undertaken in find their experiences satisfactory or better than satisfactory. However, several important caveats should be made. Meetings and interactions in mentee-mentor pairs or otherwise among mentees and with the program staff were, according to mentor and mentee accounts the most successful activity. These are closely followed by discussions about research and service to the University. More than 90% of participants find these activities to be useful or very useful. The least successful area of support that mentees received during their participation in FMPW was connecting with people in the El Paso/Ciudad Juarez community. However, even in this area, 76% (more than three fourths) of participants praised the effort that has been made by FMPW staff and mentors. It should also be mentioned that community networking was not one of the

13 13 primary goals of FMPW. Noteworthy also is that mentees and mentors did not differ significantly in their responses to all aforementioned questions. With respect to the question of how often these activities appeared on the agenda, there are substantial differences between and among the aforementioned areas. The discussions about research and teaching (results are not shown for parsimony) were the mostly frequent activities mentioned by veteran mentees in These findings are limited to the only survey that recorded frequency of occurrence with respect to a particular area, which was the survey of veteran mentees in With respect to the number of respondents, this was the largest survey among the evaluation surveys conducted in While comparing the incidence of communication across the areas of assistance provided by FMPW, the interactions among mentees were more likely to occur than interactions between mentors and mentees and interactions with the FMPW team during workshops. Although the design of the questionnaire does not allow us to relate the reported frequency of meetings to the overall satisfaction of FMPW by participants, the occurrence of specific activities is important because it is likely that those mentees who frequently meet with their mentors and peers benefit more from participation in the program than those who meet less often. Table 3 shows the results of satisfaction of FMPW participants with three FMPW components, specifically with mentors, mentees and program workshops. Although the question about the degree of benefit/support that each aforementioned component provided was asked only in 4 surveys, the results suggest that, on average, the majority of respondents were satisfied with all three components. However, the program s workshops/ meetings received higher appraisal from participants than other two

14 14 components. Observe that the differences between mentors and mentees in responses to this question were minimal. The question requires evaluation of program components by participants is an open-ended questions consisting of two parts. While the first part is quantitative in nature because it asks respondents to assign numeric ranks, the second part asks respondents to explain their ranking. The most common explanations for rankings were: My mentor has great experience in academic service. He always explains to me how the system works and how I can better work with it. My mentee peers are not from the same program and we tend to have different issues in our work. The program s workshops are informative. I also get to know other people. Both my mentor and the program workshop I attended are very helpful for me to learn about UTEP. However, I didn t get much opportunity to interact with other mentees. I feel getting together for the organized brown bag was very informative for me. The brown bags worked out better than non-brown bag interactions because we re all so busy, (mentors & mentees) brown bags were the best way to get us all together. The next table, Table 4, reveals results of overall satisfaction with FMPW that was recorded in 4 surveys, one focusing on mentors (2003) and three focusing on mentees (one from 2003 and two from 2005). Note that the differences in responses were found not among mentors and mentees but the years in which the surveys were taken. Thus, for example, the average percentage across two surveys of those who were satisfied (medium level of satisfaction) in 2003 was 53.65, while those who indicates the same level of satisfaction in 2005 were only 22.9 of participants. At the same time, the percentage of those who indicated the highest level of satisfaction rose from 25.9% in 2003 to 50% in Given that the percentage of those who indicated low level of satisfaction increased only slightly from 20.6 in 2003 to 27.1 in 2005, these findings suggest that overall satisfaction with FMPW increased significantly from 2003 to 2005.

15 15 To illustrate this point, several quotations from the program participants are cited below: I enjoy being a part of the mentoring program. I find it very useful for my career. The program has been very beneficial and useful to me I wish more people took advantage of all the opportunities. My mentoring group has been great we meet for dinner about once a month, and I feel comfortable looking to them for support. I very much appreciate the support from the FMP program. It is extremely valuable to new faculty members. I am happy I am part of this mentoring program. I hope to one day be as supportive as a mentor myself. I have learned much from collegiality and the administrative support ADVANCE has given me in my research! Thank you! Thank you! I ve gotten funding for my grant. I am very grateful! Brief Summary of Wave V Findings The questionnaires used in Wave V are parallel to those used in Waves II-IV. The only difference is that Wave V uses 6-item Likert scale instead of 3-item scale which was used in previous evaluation waves. The options range from 1= not useful to 6= very useful. Table 6 presents the answers of FMPW participants to the same 10 questions that appear on questionnaires of surveys in previous waves of evaluation. It should mentioned that some Wave V surveys have a very small number of respondents. Survey of Mentees, 6-Month Assessment, is by far the smallest, with only 1 respondent. Therefore, from the statistical point of view, the comparisons across the four surveys of Wave V are not

16 16 justified. It is more effective, though, to investigate the means of the participant responses across the surveys. On average, all questions, except the one that deals with community networking, were answered positively, with the average scores at 3 (medium satisfaction) or higher. The most highly appraised FMPW activities (with average scores of 4.3) were discussions concerning research and assistance in obtaining resources and/or support for academic success. Note that, while the discussions about research remained highly praised by the FMPW participants, assistance in obtaining resources was not among the FMPW activities that were highly appreciated in (see Table for comparison). This implies that the FMPW assistance in obtaining resources becomes more important to participants in Additionally, the assistance provided to mentees by the FMPW in connecting with people across the campus also remained highly significant (average score of 4.0), according to the participant accounts in Further, Table 6 reveals average satisfaction of FMPW participants with three components of the program with mentors, mentees and program workshops. Unlike the findings from earlier waves presented in Table 3, in Wave V FMPW participant believed that they received most help from their mentee peers (average score is 5.0) than from program workshops (4.8), although the difference between the two components is marginal. As before, the participants tended to praise the mentors assistance less than assistance that that received from mentee peers and program workshops. However, it should be pointed out that even in the case of mentors, the average satisfaction was higher that average, suggesting that all three FMPW were successful.

17 17 Table 1. Percentage of Useful and Very Useful in Answers to Questions of Evaluation Surveys of FMPW. Survey Questions 2.1 Surveys Total Meeting(s) and/or interaction(s) Discussion(s) concerning teaching Discussion(s) concerning research Discussion(s) concerning service to the University Assistance in obtaining resources and/or support for academic success Assistance/guidance about connecting with people across campus Assistance in connecting with people in the El Paso/Ciudad Juarez community Discussions concerning balancing the demands of work, family, teaching, etc Assistance/ Advice in prioritizing responsibilities Assistance/Advice in adjusting to UTEP The Surveys listed are: 2.1 Survey of Mentees, 2002 (N=5). 2.2 Survey of Mentees in Focus Groups, 2002 (N=9). 2.3 Survey of Mentors, 2002 (N=11). 3.1 Survey of Mentees in Focus Groups, 2003 (N=8). 3.3 Survey of Mentors, 2003 (N=8). 4.1 Survey of Mentors, 2005 (N=6). 4.2 Survey of Mentees in Focus Groups, 2005 (N=6). 4.3 Survey of Mentors in Focus Groups, 2005 (N=6).

18 18 Table 2. Percentage of Mentees Indicating Specific Frequency of Interactions with Mentors, Peers and Program Staff at Program Meetings/Workshops, Survey of Veteran Mentees, Mentor Program Meetings/Workshops Current or Past Peers 1 or more 1 or more 1 or more None times Ongoing None times Ongoing None times Ongoing Meeting(s) and/or interaction(s) Discussion(s) concerning my teaching Discussion(s) concerning my research Discussion(s) concerning service to the University Assistance in obtaining resources and/or support for academic success Assistance/guidance about connecting with people across campus Assistance in connecting with people in the El Paso/Ciudad Juarez community Discussions concerning balancing the demands of work, family, teaching, etc Assistance/ Advice in prioritizing responsibilities Assistance/Advice in adjusting to UTEP Total

19 19 Table 3. Average Satisfaction of Mentees with Mentors, Other Mentees and with Program Meetings/Workshops, Mentor Focus Group 2003 Veteran Mentee 2003 Individual Mentees 2005 Mentee Focus Group Mentor Focus Group 2005 My mentor My mentee peers The program s workshops/ meetings Note: 1= unsatisfied ; 2= satisfied ; 3= very satisfied. Total Table 4. Overall Satisfaction with FMPW (%). Mentor Focus Group 2003 Mentee Focus Group 2003 Individual Mentees 2005 Mentee Focus Group 2005 Mentor Focus Group 2005 Unsatisfied Satisfied Very satisfied Unsatisfied Satisfied Very satisfied

20 20 Table 5. Summary of Answers to Questions of Evaluation Surveys of FMPW, Wave V ( ). Survey Questions 5.1 Surveys Average 2 Meeting(s) and/or interaction(s) Discussion(s) concerning teaching Discussion(s) concerning research Discussion(s) concerning service to the University Assistance in obtaining resources and/or support for academic success Assistance/guidance about connecting with people across campus Assistance in connecting with people in the El Paso/Ciudad Juarez community. N/A Discussions concerning balancing the demands of work, family, teaching, etc Assistance/ Advice in prioritizing responsibilities Assistance/Advice in adjusting to UTEP The Surveys listed are: 5.1 Survey of Mentees in Focus Groups, 6-Months Assessment (N=3). 5.2 Survey of Mentees in Focus Groups, 18-Months Assessment (N=6). 5.3 Survey of Mentees, 18-Month Assessment (N=2). 5.4 Survey of Mentees, 6-Month Assessment (N=1). 2 The answers corresponds to the degree of appraisal by participants of a certain FMPW activity, ranging from 1= not useful to 6 very useful.

21 21 Table 6. Average Satisfaction of Mentees with Mentors, Other Mentees, and with Program Meetings/Workshops, Wave V ( ). 1 Mentees Focus Group, 18- Month Mentees Focus Group, 18-Month Mentees, 18-Month Mentees, 6-Month My mentor My mentee peers The program s workshops/ meetings Total 1 Note: 1= unsatisfied ; 6= very satisfied.