Program Evaluation. Graduate School of Human Resource Management and Industrial Relations. David F. Bush, Ph.D., SPHR. Villanova University

Transcription

1 Program Evaluation Graduate School of Human Resource Management and Industrial Relations David F. Bush, Ph.D., SPHR Villanova University Introduction The following report is based on my review of relevant materials and my observations of the Masters of Arts Program in Human Resource Management and Industrial Relations (HRM/IR) at St. Francis College of Loretto Pennsylvania during my site visit on April 6 through April 8, My comments are based on a number of data sources from the program, experience with other similar programs, my work with professional societies, such as SHRM (The Society for Human Resource Management), and my experience as a HR professional and as a professor in the field. My data sources from the program include catalogs and brochures describing the program, enrollment data, student evaluations, a meeting with 4 students from the Loretto program, telephone interviews with 6 adjunct faculty members (2 from each of the three program sites), and interviews and meetings with various administrators at the Loretto campus. Any statements that are made concerning student comments are based on only 4 of over 150 current students in the program during a meeting of approximately one hour duration. Their remarks must be considered with caution since this sample was neither random nor representative, given that 50% of the interviewed group is currently employed in the field, while about 90% of currently enrolled students are employed in HR. As a final preface to my evaluative comments, I should note that everyone associated with St. Francis was thoroughly cooperative, generous with their time, and helpful in sharing both data and their views on the HRM/IR program and related aspects of the College. I further acknowledge the critical assistance of Philip Benham, Ph.D., Director of the Graduate School of HRM/IR, Kathleen Owens, Ph.D., Vice President for Academic Affairs, and Judy A. McMullen, Assistant to the Director. The consensus of all individuals interviewed was that the HRM/IR Masters Program has good quality students, is well managed, is valued by the college administration, has a good faculty, has a 36 year history of success, and a curriculum which has recently undergone timely improvements. The adjunct faculty members interviewed over the telephone consisted of the following individuals: Dr. Ferdinand Molz, Samuel Pasquarelli, Esq., Ms. Jennifer Cooney, Mr. Charles Frame, David Andrews, Esq., and Dr. James Lakso. The four students interviewed in a group setting were: Ms. Margaret Hancheck, Ms. Christina Long, Ms. Joyce Page, and Mr. Michael Romano.

2 2 The following discussion is organized in to three sections: strengths, weaknesses, and recommendations. I will begin with the largest section, strengths. Strengths The current strengths of the HRM/IR Masters Program will be presented in a category by category description. A major source of data for these conclusions comes from a survey conducted by Dr. Benham and Ms. McMullen concerning the major issues of the program: administration, teaching, location and schedules, the library, curriculum, and marketing and advertising. While the alumni response rate was reasonable, the response rate for current students was quite good, averaging 69% across the three sites. 1. Program Administration. There is a clear consensus among the faculty, fellow administrators and the students, that the program is well run. Student ratings for administration produced an average rating across the three sites of 4 on a scale of 1 to 5, where 5 is excellent. The director and the assistant to the director both receive high marks from all groups interviewed. 2. Teaching and faculty quality. With the exception of one Loretto based instructor, teaching appears to be eminently satisfactory. Student ratings for Loretto and Pittsburgh rated 4.2, while Harrisburg ratings were alarmingly lower, scoring at a 3.4, approaching the scale midpoint of 3. However, student comments from Harrisburg did not reflect any great concern about teaching quality. The problem instructor for the Loretto students was identified by all four students as Dr. Heidi Sedwick. They asserted that she was "mediocre" and simply "regurgitated the textbook". Furthermore, she displayed discomfort and defensiveness when asked questions, especially if the questioner was male. In addition, she had assigned a student team to present in each class, which they deemed excessive. They expressed concern that they could not respond honestly to class evaluations because they believed that she would read them and identify those who protested from their handwriting. In reviewing Dr. Sedwick's credentials in her vita, I did not see evidence that she had an appropriate background for teaching a generalist course in Human Resource Management. Given the large number of instructors in three sites, it strikes me as exceptional that nearly all are highly regarded and only one is cause for concern. It is also noteworthy that some of the instructors are graduates of the program who have done well professionally and wish to continue their involvement with the program. 3. Facilities, Locations and Schedules. Location and scheduling received high ratings from both the interviewed faculty and the student survey. The mean rating across locations was approximately 3.9 on a scale from 1 to 5, with a range from 3.7 to 4.0. It is interesting that the lowest rating for this dimension comes from students attending at Loretto. While this is not a large deviation, it merits watching in future program reviews. The quality of

3 3 the facilities, including room characteristics and access, have improved in recent years. It would appear that the Harrisburg and Pittsburgh sites are more than satisfactory. 4. Curriculum. The curriculum receives good ratings from both current students and alumni. Ratings from the student survey were the highest for any evaluated dimension, with an average of 4.1 and a range from 4.0 to 4.2. However, the field of human resource management has been changing and new material must be added based on models developed by major professional organizations, such as SHRM. Consequently, the recommended changes are all be needed and appropriate. The students I interviewed also enjoyed the curriculum, but went on to make two suggestions regarding curriculum change. One was that Labor History should become an elective. The other is that the research course, which they referred to as their thesis, should be optional, with the alternative being two extra courses, i.e., 36 hours rather than the current 30 hours. Such an option may be an administrative complication, but it may have merit. While the latter is not currently under consideration, the former has been included in the curriculum revision. These students expressed general delight with the law classes. 5. Program Marketing. The material that has been prepared for marketing the program is generally good. The copy to be submitted to other publications looks appealing. While one older brochure looks dated, the new brochure is exceptionally good. In fact, it should probably become the standard for all St. Francis College graduate Programs. The cover looks very well done with the background of course names. The copy is very good, particularly the quotes from former students and professors. The visuals are very competitive. The final important ingredient is the tear off mailer. However, it should be noted that the current students do not have a positive view of marketing and advertising. In the survey they gave this area a low rating. It would be useful to conduct focus groups with current students to determine why they selected St. Francis and what their ideas are for more effective marketing. Be sure to enlist them in the process of attracting more new students to "theirprogram". Finally, be sure to use these marketing materials with area employers and targeted colleges and universities. 6. Enrollment. Enrollment appears to have been reasonably stabile across the three sites. Given that the goal if marketing is to increase the number of applicants, strategic decisions must be reached regarding program capacity constraints. With the current faculty and facilities, what is the maximum number of students that can be served? How much growth is desirable? 7. Student Quality. When the quality of students entering the program is assessed in terms of their undergraduate GPA, the recent trend is characterized by a gradual increase, with the mean for those entering in 1996 being approximately This ongoing progress combined with key characteristics of the targeted student market (adults employed in the HR field in one of the three regions), lead me to conclude that this is an acceptable level of student quality. While it would be helpful to have standardized test scores as an additional

4 4 measure of entering student ability, I do not advocate requiring entrance tests, such as the GRE, because their deterrent value would outweigh any useful contribution. 8. Outcomes for Graduates. Since the majority of students in the program are already functioning as practitioners in the field of Human Resource Management, placement data is not a very useful measure of the program's success. However, for those students who are enrolled full-time and have not yet entered the field, assistance from the Career Development Center and related placement data do take on greater importance. In fact, two of the students I interviewed, who were not employed in HR, expressed concern that the Career Development Center might not provide adequate exposure for their career aspirations. It may be helpful to enlist current student and alumni in creating a clearing house ofhr openings in the three market areas where the St. Francis program is well established. A job bank could be operated by a graduate assistant in the HRM/IR office utilizing the program's web page. 9. Program Reputation. The reputation of the program and its successful 36 year history is an important strength that produces many side effects. Many alumni are now in influential positions, allowing them to refer employees to the program and to hire recent graduates. Some are also in a position to make monetary contributions to the program, including personal and corporate donations. 10. Ongoing Review Process. An important strength of the program is its commitment and the commitment of the College to conducting program evaluations at a five year interval. These regular evaluations aid in the continuous improvement of the program. Weaknesses The program has few weaknesses, and since many of those which were noted in the last evaluation have been corrected, I am optimistic that this practice of continuous improvement will continue. 1. The library. The student ratings for the library are strongly indicative of a problem in the Harrisburg (2.2) and Pittsburgh (1.6) programs. Furthermore, the ratings from Loretto (3.4) students, while over 3, are low enough to be cause for concern. I suggest that a series of focus groups be conducted to clearly establish the basis for student dissatisfaction with the library in all locations. A negative comment about the library fee was received from a Pittsburgh respondent. I was informed that some students complain about the parttime library fee, even though this fee is the equivalent to one interlibrary loan and the students may make many interlibrary loan requests during a term. It appears that many adjunct professors avoid making library assignments and instead, hand out a large number of photocopied articles. While the HRM/IR collection in Loretto may be adequate, the program faces the challenge of establishing suitable library access in the Harrisburg and

5 Pittsburgh areas. It is important that the HRM/IR collection in Loretto have an adequate budget for ongoing growth. It is also important that the budget for this collection have annual increments commensurate with the inflation characteristic of the books and periodicals involved. A final point derived from the librarian, is the need for better communication of HR program needs to the library. It may be appropriate for Ms. McMullen to compile library requests from all instructors on a regular basis, so that the library orders can be made in a timely manner The bookstore. The bookstore did receive some negative comments from the students in the Loretto location. The exact nature of the complaint was unclear. The students who were interviewed complained about the bookstore hours. As in the case of the library, a series of focus groups should be conducted to clearly establish the basis for student dissatisfaction. The bookstore might consider surveying all of its customers to determine a schedule that would maximize customer satisfaction. Such a survey should of course include the graduate students. 3. Course/Instructor Evaluations. The Course/Instructor Evaluations should be revised so that they include a mixture of the open ended items that are currently in use and numerical rating scales. The scales will permit the conversion of ratings into statistical information. They will also act as reminders of some of the issues that the respondent may wish to discuss at greater length in an open ended response. Given the comments of the students that I interviewed, care must be taken to insure that proper procedures are followed in conducting in class evaluations. To insure a candid response, the evaluations must be administered by a student volunteer and placed into an envelope which is either hand carried to the office of the director or sent by mail. There was a clear concern with respect to Dr. Sedwick, that the professor would determine which students submitted adverse comments and punish them in some way. Recommendations Based on my review of the evaluation data and my direct observations at the college, I believe that the program is fundamentally sound, under excellent leadership and competently administered. I'd like to summarize a few recommendations that are needed for continuous improvement of the HRM/IR Masters Program at this time. 1. Recommendation for improving marketing. You should conduct focus groups with current students to determine why they selected St. Francis and what their ideas are for more effective marketing. Be sure to enlist them in the process of attracting more new students to "their program". Finally, be sure to use the new and improved marketing materials with area employers and targeted colleges and universities. 2. Set realistic enrollment goals. Enrollment goals should be based on system constraints imposed by classroom and library facilities, number of potential students, and available

6 faculty. Studies should be conducted to attempt to understand the nature of sustainable growth for the program Use SHRM certificate courses to attract students. By establishing SHRM based preparatory courses for the PHR/SPHR Standardized Tests under college sponsorship, you will develop an additional revenue source and a "feeder program" to lead accomplished candidates to your program. 4. Create a HR Job Bank. Enlist current student and alumni in creating a clearing house of HR openings in the three market areas where the St. Francis program is well established (Loretto, Pittsburgh, and Harrisburg). A job bank could be operated by a graduate assistant in the HRM/IR office utilizing the program's web page in cooperation with local SHRM chapters and the College's Career Development Center. In this way St. Francis also offers additional value added to the three HR communities. 5. Improve HR Library Facilities. The HRM/IR collection in Loretto should be strengthened annually and the program must establish suitable library access in the Harrisburg and Pittsburgh areas. It is important that the HRM/IR collection in Loretto have an adequate budget for ongoing growth. It is also important that the budget for this collection have annual increments commensurate with the inflation characteristic of the books and periodicals involved. The HRM/IR program needs to compile library requests from all instructors on a regular basis, so that the library orders can be made in a timely manner. 6. Improve Bookstore Services. It would be helpful to the Loretto students if the bookstore provided better access during the first two weeks of the semester. The exact nature of the needed change is unclear to me based on the available data. Communication between bookstore and evening students may be part of the issue. The purchase of books seems to be managed adequately in the other locations. However, since students mentioned a concern with books that were purchased not being used in class, professors should be reminded of this "customer service issue". 7. Improve Course/Faculty Evaluations. The Course/Instructor Evaluations should be revised so that they include a mixture of the open ended items that are currently in use and numerical rating scales. To insure a candid response, the evaluations must be administered by a student volunteer and placed into an envelope which is either hand carried to the office of the director or sent by mail. In conclusion, I confess that I am impressed with your HRM/IR program. It has an impressive history and an even more promising future. It works very well as a part-time, evening program. However, I am not persuaded that it works as well as a program for the full-time student who has just graduated from a baccalaureate program. Such students need financial support, like assistantships or fellowships, as well as internship experiences that may not be readily available near the Loretto campus. But, as an evening-based, part-time program

7 for employed professionals in the three markets (Loretto, Pittsburgh, and Harrisburg) it may be one of the most cost-effective HR masters programs in the country. 7 One of the great strengths that Dr. Benham brings to this program is his orientation to ongoing improvement. Continuous improvement is a paradigm that eliminates the option of complacency. My impression from this brief visit is that this orientation of continuous improvement is led from the top, with Dr. Owens providing much of the inspiration. Even though St. Francis does not have a formal program for continuous quality improvement, there seems to be a natural inclination to this approach among many of the people that I met on campus. A critical component of continuous quality improvement is an orientation to "customer satisfaction". In order to create additional HRM/IR program improvements, additional student data is needed. This is the reason that I have advocated the use of student focus groups. It was a pleasure to learn more about the HRM/IR Masters Program at St. Francis College. I hope that my comments are helpful.