Outcomes Assessment Research with Adult Education Alumni: What is the Impact of the Degree on our Graduates?

Transcription

1 1 Outcomes Assessment Research with Adult Education Alumni: What is the Impact of the Degree on our Graduates? Henry S. Merrill, Ed. D. This paper presents the results of a survey focused on the topic: What is the impact of the M. S. in Adult Education degree program on a graduate s life? We wanted to go beyond the metrics of placement and types of job. This research is important to the practice of adult and continuing education because outcomes assessment has become a major focus in higher education during the last three decades. Frequently discussed learning outcomes measures include where graduates are employed, whether the degree has been of assistance in their career, and graduates reflections on the value of the degree, what they liked about the program and suggested improvement for the graduate program. Learning outcomes may be assessed at many levels of the institution. This process is especially important at the degree program level. Introduction The call for accountability in post-secondary education created a form of institutional research that became known as outcomes assessment. The focus shifted from the input side of institutional resources to the output or outcomes side. The research focused on what happens as a result of the expenditure of the resources during educational process. The undergraduate education mission is the largest sector for most institutions, so the focus on outcomes assessment is primarily focused on this sector. Volkein provides this useful description: Simply put, the institutional goal should be the improvement of student learning and growth. Campuses need to carry out assessment and self-evaluation not for external accountability but for internal enhancement. Outcomes assessment does not judge undergraduate education but improves it. Faculty, especially, identify with this emphasis. (1999, p. 16). This describes a framework for continuous improvement cycles. Institutional programs and resources are reviewed in the context of effective learning outcomes at every level: school/program, department, degree(s) and even individual courses. One of the most comprehensive resources for outcomes assessment research and practice is the National Institute for Learning Outcomes Assessment. This organization maintains many examples of electronic surveys available to capture snapshots of undergraduate student perspectives on academic achievement and engagement. There are also alternative forms of assessment proposed for more in-depth perspective on learning outcomes. Some examples of these alternatives include e-portfolios of work produced by students throughout their degree program, a system of shared rubrics used at multiple institutions by faculty to focus assessment on student writing and depth of learning in multiple courses, and finally, online assessment communities of practice within institutions that may develop over time into a network across the institution and link together disciplines (and eventually disciplines and assessment projects across multiple institutions) (Banta, Griffin, Flateby, and Kahn, 2009). The outcomes assessment literature is less focused at the graduate level, except for medical and clinical education. The principles of effective outcomes assessment for undergraduate programs, however, can be adapted to graduate programs.

2 2 Alumni Surveys in Outcomes Assessment The literature on alumni surveys of graduate programs suggest there are many reasons for gathering information about the impact of degree completion: career development and quality of life; effectiveness of the curriculum and teaching on developing knowledge, skills, abilities and values; accountability for effective use of institutional resources; impact with employers in the field; and gauging alumni satisfaction and interest in continuing engagement with the institution (Borden, 2005; Cabrera, Weerts & Zulick, 2005, Volkwein, 2010). Alumni surveys are often recommended as a method of collecting data for outcomes assessment. Borden (2005) describes the limited purposes of alumni surveys in the decentralized and segmented efforts of a large university. Indiana University, with its eight campuses and wide variety of traditional arts and sciences schools and professional schools is a good example. Borden describes the development of an alumni survey for undergraduates since 1994 at the IUPUI campus. Despite this example of a long-term undergraduate alumni research project at IUPUI, Borden notes that there has been little success in establishing nationally benchmarked studies. Alumni research seems to be done primarily at the school, program or department, or degree level. There is only one document that focuses on assessment of graduate education for use at IUPUI. It is titled Questions for Departmental Self-Assessment of Graduate Programs (Queener, 2000). The questions provided in this self-assessment process are divided into these categories: recruitment; academic credentials of matriculated students; placement; advising and mentoring; financial resources; curriculum; and national reputation of the graduate program. These questions in these categories focus on a mix of inputs (recruitment, academic credentials, and financial resources) and the processes that lead to educational outcomes (advising and mentoring; curriculum) and evidence of student learning as judged by placement. Since the rankings and national reputation of a graduate program are influenced by so many factors (inputs, educational processes, a range of faculty accomplishments and student achievement) it is difficult to know how to characterize that category (Volkwein, 2010). The questions posed about placement category are (Queener, p. 2): List graduates of your program by current position, title and employing institution; also identify mentor for graduate work. How many of your graduates worked full time while pursuing their graduate degree? Part time? Did these students receive tuition assistance from their employer? How are students advised for placement? How typical of your field is your placement record? How could your placement be improved? This is a relatively limited list of questions that seems to identify evidence of learning only by factors related to placement. We know from research on the Capstone Portfolio completed by each IU adult education graduate in the last seven years that most of our students are employed full-time while they are working on the degree. The few that are working part-time or not employed at the time are usually in the process of making a career or life transition. The completion of the M. S. in adult education is usually an intentional part of that transition. The workforce sectors where the SCS students are generally employed are: post-secondary education (40%); business-hr/training (25%); government/military (10%) ; and self-employed/consulting

3 3 (10%). The remaining students (15%) are employed in health/clinical education, positions in non-profit organizations and retail sales or other transitional jobs. IU Adult Education Alumni Survey The IU School of Continuing Studies has used a survey developed by this research team to gather data on the Bachelor of General Studies (BGS) alumni. The survey used for the BGS alumni research project was adapted for this Adult Education Alumni Survey project. In May 2010 we mailed a two-page survey to 655 adult education graduates. The mailing list, provided by the IU Alumni Association, included graduates of the program from when the program was offered by the School of Education (approximately ) as well as from the years when the School of Continuing Studies has been the location of the program ( graduates). The letter/survey to graduates had a link for those who wished to complete the survey online. The IU Adult Education program has three distinct groups of graduates. Those in the first two groups graduated when the program was located in the IU School of Education. They include: 1) graduates of a doctoral degree program active from the 1950s through the 1980s (program closed in 1983); and 2) graduates of a M. S. degree program delivered through 1995 in typical evening class meeting format with some weekend classes. The third group contains graduates of the online M. S. in adult education degree located in the School of Continuing Studies from 1996 to the present. The IU Alumni Association provided valid addresses and surveys were sent to 656 graduates in all three categories. The response rate was 117 graduates (18%) responding to the survey. The distribution of respondents in each category was: School of Education Doctorates 37 32% School of Education M. S % School of Continuing Studies M. S % Current Involvement in Adult and Continuing Education Graduates were asked to list ways they were currently involved in adult and continuing education. Eighty-seven respondents (74%) reported 101 types of involvement. Thirty respondents (26%) left the question blank or indicated no involvement. We were able to get a detailed look at the types of involvement reported by the 74% who answered the question positively. Their open-ended responses were categorized as follows: Teaching/Training: 47 48% Counseling/Advising 5 6% Online learning technology/course design 5 6% Volunteering 26 31% Part of daily life 17 20% Taking courses 5 6% The respondents from the early years of the program in the school of education include many graduates who are now retired. We were able to take a closer look at the responses of retirees. There were 34 retired respondents (29% of all respondents), and 17% indicated ways they were involved in adult and continuing education. Ten percent of those mentioned some sort of volunteer work. Thirteen percent did not respond to the question or indicated they were not involved. We noted that two respondents reported disabling illness, and it is fair to assume that some other retirees may be limited by diminished energy or poor health.

4 4 The teaching/training involvement was one area of continuity. This 48% was distributed across all three groups of alumni respondents, ranging from 43% of the SoE masters alumni to 55% of the SCS M. S. alumni. Volunteerism was another area of continuity, ranging from a low of 24% for SCS alumni to a high of 38% of the doctoral alumni. Some of this variation might be due to the fact that the more recent SCS alumni are younger, employed and have less time for volunteer activities than the older alumni who earned doctorates and many are now retired. There are three areas of discontinuity between the groups in this category of engagement. Only graduates from the SCS alumni group are employed in counseling/advising roles and engaged in online learning technology and course design. The latter is not surprising since they are graduates of an online degree program and this technology is more prevalent in their careers. A puzzling discontinuity is that only 9% of SCS alumni group responded with open-ended comments that were included in the category of being engaged with adult education activities in other types of work or as part of their daily life. The two SoE groups ranged from 23-29%. Participation in Further Education The next section of the survey inquired about taking additional courses after completing their degree, about the types of courses completed and/or enrolling in another degree. Sixty-eight of the respondents (58%) reported pursuing further education after earning the adult education degree. Of these, 27% took credit courses, 21% took noncredit courses, and 9% took both credit and noncredit courses. Overall this is a group of people who appear to value further learning. The areas selected for further study were education (43%) and a variety others, including business, management, religion/ministry, nursing and counseling. It is interesting to note that the SCS alumni group participated less (45%) in further learning. However, this group is the only one with respondents who indicated they are pursuing other degrees (21% of this group). This again is not surprising since they are the younger group, more likely to be working full-time and pursuing career development strategies. Reasons for Selecting IU Adult Education Degree When asked why they selected this adult education program, 112 people offered 150 reasons. The largest number of respondents (61%) indicated that learning adult education theory and practice was relevant to current job responsibilities and career development was most important. The reasons were categorized as follows: To learn and acquire skills, relevant to current employment 61% To advance or change careers 21% Reputation of the program or a recommendation by a friend or advisor 12% Online course offerings (31% from online M. S. in SCS) 11% Program's flexibility as to time and place (15% from online M. S. in SCS) 11% To obtain a degree or credentials 5% Faculty and staff 5% Program structure and curriculum 3% Various other 5% Impact of Earning the IU Adult Education Degree The survey asked for information about the impact of the degree on employment or career plans and the impact of adult education on their life outside the workplace. When asked

5 5 about the impact of the adult education degree on their employment or career plans, 113 people responded with 140 ways. These impacts were categorized and tabulated as follows: Provided career enhancement and advancement opportunities 59% Imparted valuable knowledge and skills 39% No impact 7% Increased credibility and professional respect 6% Led to a salary increase 5% Various other 10% When asked about the impact of the AE degree on their life outside the workplace, 81 people responded with 88 ways. These statements were categorized and tabulated as follows: Providing intellectual growth and skills 48% Fostering personal growth and increasing self-confidence and esteem 47% Increasing their commitment to adult education 5% Generating new friendships and expanding their network 2% Bringing credibility and respect 4% Various other responses 2% There are two interesting areas of discontinuity among the groups in two of these categories. The SCS alumni responding indicated less impact in terms of intellectual growth and skills (36% vs. 58% of the SoE M. S. alumni and 52% of the doctoral graduates). In the area of personal growth and self-esteem, 54% of SCS alumni and 52% of doctoral alumni responding indicated impact. In contrast, 35% of the SoE M. S. indicated impact on their personal growth and self-esteem. Satisfaction with the IU Adult Education Degree The next area to report is about responses when asked what aspects of the degree program they liked or were most satisfied with. In this section, 107 people responded with 149 aspects. It is important to note that some researchers warn of a halo error when considering the relationship between alumni reported perceptions of the educational experience and actual outcomes (Pike, 1999). These aspects were categorized and tabulated as follows: Faculty and staff Peer interaction and learning from fellow students 32% Application of adult education principles and practicality of the program 17% Flexibility and convenience in regard to time and place 14% Certain specific courses and course content described 12% Online offerings 12% Intellectual challenge 8% Flexibility and adaptability of the curriculum 6% Being respected and treated as adults 6% Small class size 3% Various other 10% Suggestions for Program Improvement In the final question, alumni were invited to offer suggestions for improvement to the degree program. Many responded that they had been out of the program long enough that they

6 6 did not feel qualified to comment. The question yielded 48 suggestions. When a suggestion was made more than once, we established a category, as show below. Ideas mentioned only once are included in the Other category. The primary suggestions include: provide career and placement support; specific course content (e. g. research methodology, philosophy); increase alumni connections and involvement; incorporate internships; offer the doctorate again; increase online offerings/internet learning (from the SoE group who may not know the program is now online). Discussion This research was completed in 2010 and the results are being reviewed for ideas to strengthen the M. S. in Adult Education so it better prepares graduates for the profession. It is very interesting to see the areas of continuity that connect the graduates in these three groups with different types of course delivery over the last 50+ years. There are some notable areas of discontinuity across the groups of graduates such as perceptions of impact on intellectual and skills development as well as on personal growth and self-esteem. There are also areas where there appears to be continuity that links program focus, curriculum and culture across the years between these groups even though faculty and delivery systems have changed over time. Additional research using in-depth interviews would probably yield more useful suggestions for program improvements from the SCS alumni group. References Banta, Griffin, Flateby & Kahn (2009). Three promising alternatives for assessing college students knowledge and skills; Occasional paper # 2. Retrieved from Borden, V. M. H. (2005), Using alumni research to align program improvement with institutional accountability. New Directions for Institutional Research, 2005: Cabrera, A. F., Weerts, D. J. & Zulick, B. J. (2005), Making an impact with alumni surveys. New Directions for Institutional Research, 2005: Pike, G. R. (1999). The constant error of the halo in educational outcomes research. Research in Higher Education, Vol 40(1), p Queener, S. (2000). Questions for Departmental Self-Assessment of Graduate Programs. Retrieved from Volkwein, J. F. (1999). The four faces of institutional research. New Directions for Institutional Research, Spring, Vol. 1999; p9-19. Volkwein, J. (2010). Assessing alumni outcomes. New Directions for Institutional Research, Spring, Vol. 2010, p Henry S. Merrill, Ed. D., Associate Professor Emeritus, IU School of Continuing Studies, 1377 Red Oak Dr, Avon, IN hmerrill@iupui.edu Acknowledgments: Dr. Frank DiSilvestro and Joann Alexander for their work with this survey. Presented at the Midwest Research-to Practice Conference in Adult, Continuing, and Community Education, Lindenwood University, St. Louis, MO, September 21-23, 2011.