Using CAS Standards to Measure Learning Outcomes of Student Affairs Preparation Programs

Transcription

1 Using CAS Standards to Measure Learning Outcomes of Student Affairs Preparation Programs Dallin George Young Steven M. Janosik A majority of recent graduates from master s level student affairs preparation programs report having a clear understanding of 58 of 60 foundational learning outcomes derived from the Council for the Advancement of Standards in Higher Education (CAS) guidelines. While there were only two significant differences between selfreported learning of graduates from CAS compliant and non-compliant programs on 60 items, data suggest that recent graduates from CAS compliant programs report greater confidence in their preparation in 48 of the 60 outcomes presented in the questionnaire. Professional preparation is important for individuals in any profession. Unless individuals can complete successfully the professional training course for their chosen field, they will likely not achieve full status as a member of that profession (Stuit, Dickson, Jordan, & Schloerb, 1949). Professional preparation is also important for the profession itself. Hesburgh, Miller, and Wharton (1973) describe professional education as the introduction and updating of the theory, Dallin George Young is the director of housing at Dixie State College of Utah in St. George, UT. Steven M. Janosik is an associate professor of educational leadership and policy studies at Virginia Tech in Blacksburg, VA. 341

2 skills, and perspective for practitioners in the field. It is one of the hallmarks of any profession (Young, 1994). In 1986, a set of standards was developed for preparation programs at the master s level for student affairs professionals by the Council for the Advancement of Standards in Higher Education (CAS) (McEwen & Talbot, 1998). CAS is a consortium of 35 professional organizations concerned with the promotion of quality in programs and services related to higher education (CAS, 2003). The standards for graduate students in professional preparation programs in student affairs were developed as a guide for teaching faculty and to equip aspiring professionals with the minimum level of education for student affairs practice (McEwen & Talbot, 1998). Research using the CAS standards for professional preparation at the master s level came soon after the original publication of those standards. Von Destinon (1986) compared the curriculum of the master s program in student services at the University of Arizona with the newly-formed CAS standards. The importance of values education in preparation programs, as recommended in the CAS standards, also has been examined (Young & Elfrink, 1991). Learning outcomes of graduate students in student affairs preparation programs have been based on course evaluations (Harrow & Mann, 1996) and employer evaluations (Hyman, 1985). Other researchers have used instruments specifically designed to assess certain aspects of the learning experience such as the effects of intercultural interaction (McEwen & Roper, 1994). There is limited information available on graduate students in student affairs preparation programs and their learning outcomes based on the CAS standards, however. This study addressed this gap in the literature and seems especially appropriate given the purpose of this special issue of the NASPA Journal on professionalism and professional standards in the field of student affairs. The primary purpose of this study was to measure the degree to which recent graduates of student affairs preparation programs reported learning about the important foundational elements of the profession. Secondarily, we wanted to know if there were differences in learning between those who graduated from those program that met CAS standards and those that did not. 342

3 Instrument Development Method A questionnaire, using the Preparation Program Standards for Student Affairs in Higher Education Standards and Guidelines Self-Assessment Guide (SAG) published by CAS, was specially developed for this study. The Foundational Studies area of the curriculum, one of the 11 components recommended by CAS, served as the basis for the survey instrument. It contained six content areas: (a) historical foundations, (b) philosophical foundations, (c) psychological foundations, (d) cultural foundations, (e) sociological foundations, and (f) research foundations. The instrument s content validity was enhanced through its development. Focus groups were used in the initial step of instrument development. These focus groups were comprised of current master s degree students in a CAS compliant program. Group members identified learning outcomes in each of the six components in the Foundation Studies section of the CAS standards. After generating 94 items, the group members reviewed and revised the proposed outcomes. The number of items in each area ranged from 12 in Sociological Foundations to 20 in Psychological Foundations. Then, a panel of experts reviewed the learning outcomes generated by the focus groups and provided feedback on how to improve them. The experts were all current faculty, full-time and adjunct, in a CAS compliant student affairs preparation program. The experts identified the 10 items that best represented each of the criteria areas, based on which items they deemed most important as foundational learning outcomes in a student affairs preparation program. Also, the experts refined learning outcomes and suggested new ones if they were unable to identify 10 from the original list. The researchers recorded the items the experts identified as the most important and then, using this information, reduced the number of items in each component to 10. A sample of the final items, and their assigned criteria, included: (a) I have a clear understanding of the origin of higher education in the U.S. Historical Foundations, (b) I have a clear understanding of the holistic approach to student development Philosophical Foundations, (c) I have a clear understanding of typology theory Psychological Foundations, (d) I have a clear understanding of how 343

4 the historical role of access contributes to the culture of an institution Cultural Foundations, (e) I have a clear understanding of group theory Sociological Foundations, and (f) I have a clear understanding of the differences between qualitative and quantitative methods in research Research Foundations. In its final form, the instrument contained two sections. The first was a demographic section containing seven items that gathered information about the participants and the programs they attended. The second section of the instrument contained 60 items based on the Foundation Studies area of the CAS standards. Throughout this second section of the inventory, participants were asked to rate items on a Likert-type scale with responses ranging from 1 to 4 where 1 indicated strong disagreement and 4 indicated strong agreement. For each of the items, the participants assessed the degree to which their graduate preparation program had influenced their understanding of the learning outcome addressed in the item. Institutional and Graduate Student Samples A convenience sample was used in collecting the data. First, the American College Personnel Association s (ACPA) Directory of Graduate Programs Preparing Student Affairs Professionals (ACPA, 2005) was consulted to obtain a list of preparation programs in student affairs. Using the information in this list, we were able to sort each of the 98 schools into two groups: CAS compliant v. non-compliant. The selection criteria for inclusion in the CAS compliant or non-compliant groups came from the Preparation Program Standards for Student Affairs Professionals (CAS, 1998). According to CAS standards, student affairs preparation programs must have one full-time faculty member dedicating 100% of his or her time to the program, and the equivalent of one other full-time faculty member. Programs also must require two separate practical experiences totaling a minimum of 300 hours. Other criteria were not included since these two criteria were deemed to be the most objective and rigorous. This information came from the ACPA directory or the preparation program s Web site. 344

5 Program coordinators of all 98 programs listed in the ACPA directory were contacted and invited to participate. In many cases, program coordinators did not have a list of alumni s and could not participate. In other instances, preparation programs were very recently developed and had yet to graduate alumni. Some program coordinators were administering other assessments and did not want to make additional requests of their alumni. Still, other program coordinators agreed to participate in the data collection but failed to forward the introductory along to the program s alumni. Finally, some programs did not participate due to incorrect contact information contained either on the ACPA directory or on the program s Web site. In the end, of the 98 student affairs preparation programs listed in the ACPA directory, 13 agreed to participate in the study. Four were classified as non-compliant programs, and nine were classified as CAS compliant programs. The 13 program coordinators agreed to contact their recent graduates by and ask that they complete the questionnaire. Coordinators identified 527 graduates to whom the s were sent. Data Analysis Procedures Responses were sorted into two groups: CAS compliant and non-compliant. Next, we collapsed the data set by combining all responses that represented agreement (Strongly Agree and Agree) and all responses that represented disagreement (Strongly Disagree and Disagree). Chisquare tests were conducted to determine if there were any significant differences in reported learning between the two groups. Results A total of 191 respondents who graduated within the past two years completed the questionnaire. This represented 36.2% of the 527 graduates to whom the s were sent. Of those respondents, 92 (60.1%) were from CAS compliant schools and 61 (39.9%) were graduates from non-compliant schools. Reliability for this instrument was established by calculating an alpha coefficient on the data set after it was collected. The Chronbach s Alpha for the instrument was

6 Results Regarding Reported Learning Results are reported by foundational area. Frequencies and percentages, along with the results of the chi-square tests are shown in corresponding tables. In 18 of the 60 items, respondents were almost unanimous in their response. In these cases, the chi-square tests could not be conducted because frequencies in one or more cells in the chisquare matrix were less than five. Historical Foundations More than 90% of both groups reported a clear understanding of 5 of 10 items in the Historical Foundations area. When an 80% agreement level was used, three more items were added to the list. Respondents from CAS compliant programs reported having a clear understanding of the historical role of accountability in American higher education at much higher levels than respondents from non-compliant programs and about 56% of both groups reported having a clear understanding of the Carnegie classification system. None of the chi-square tests in this section of the questionnaire proved to be statistically significant. These results are reported in Table 1. Philosophical Foundations More than 90% of both groups reported a clear understanding of 4 of 10 items in the Philosophical Foundations area. Two additional items could be added to this list when an 80% agreement level was used. About 50% of respondents in both groups indicated they had a clear understanding of how pragmatism, existentialism, and realism influenced American higher education. Less than 50% of respondents in both groups indicated they had a clear understanding of how humanism influenced American higher education. None of the chi-square tests in this section of the questionnaire proved to be statistically significant. These results are reported in Table

7 Table 1 Results of Chi-Square Analysis on Historical Foundations Section of the CAS Foundation Studies Instrument 347

8 Table 1, continued 348

9 Table 2 Results of Chi-Square Analysis on Philosophical Foundations Section of the CAS Foundation Studies Instrument 349

10 Table 2, continued 350

11 Psychological Foundations More than 90% of both groups reported a clear understanding of 4 of 10 items in the Psychological Foundations area. Three additional items could be added to this list when an 80% agreement level was used. About 3/4 of respondents from CAS compliant programs reported clear understandings of typology theory and counseling theory. Few respondents at non-compliant programs reported such confidence, although these differences were not significant. With respect to involvement theory, respondents at non-compliant programs were significantly more likely to report a clear understanding. These results are reported in Table 3. Cultural Foundations More than 90% of both groups reported a clear understanding of 7 of 10 items in the Cultural Foundations area. The remaining three items could be added to this list when an 80% agreement level was used. None of the chi-square tests in this section of the questionnaire proved to be statistically significant. These results are reported in Table 4. Sociological Foundations More than 90% of both groups reported a clear understanding of 3 of 10 items in the Sociological Foundations area. Four additional items could be added to this list when an 80% agreement level was used. Respondents from CAS compliant programs reported a clearer understanding of group theory when compared to their peers at non-compliant programs. On the other hand, respondents from non-compliant programs reported a greater understanding of sociological and interaction theory when compared to their CAS compliant colleagues. None of the chi-square tests in this section of the questionnaire proved to be statistically significant. These results are reported in Table

12 Table 3 Results of Chi-Square Analysis on Psychological Foundations Section of the CAS Foundation Studies Instrument 352

13 Table 3, continued 353

14 Table 4 Results of Chi-Square Analysis on Cultural Foundations Section of the CAS Foundation Studies Instrument 354

15 Table 4, continued 355

16 Table 5 Results of Chi-Square Analysis on Sociological Foundations Section of the CAS Foundation Studies Instrument 356

17 Table 5, continued 357

18 Research Foundations There was no item in this section where more than 90% of both groups reported a clear understanding. Three items could be added when an 80% agreement level was used and an additional four items could be added when a 70% agreement level was considered. Respondents from CAS compliant programs reported a clearer understanding of the qualitative and quantitative research paradigms, the difference in research methods, and how to match methods to appropriate levels of analysis when compared to their peers at non-compliant programs. On the other hand, respondents from non-compliant programs reported a greater understanding of how to write in a scholarly style, and the difference between process goals and outcome goals when compared to their CAS compliant colleagues. None of these differences were statistically significant. The remaining item in this section of the questionnaire was significant, however. Respondents from non-compliant programs reported a much clearer understanding of the level of data represented by a variable in quantitative analysis. These results are reported in Table 6. Discussion Overall, the majority of respondents in CAS compliant programs in this study indicated having a clear understanding of all but one of the items in the CAS Foundation Studies questionnaire (humanism 48.9% agreement). The majority of respondents from non-compliant programs indicated having a clear understanding of all but three items (existentialism 47.5% agreement, realism 45.9% agreement, and humanism 36.1% agreement). All of these items were found in the philosophical foundations component. The understanding of a philosophical role in the development of higher education is highly conceptual. Forney (1994) suggested that master s students in student affairs programs are least likely to exhibit the learning style of assimilator, which is characterized by abstract conceptualization. This may help explain this result. When examining other items in the questionnaire by total group response, both groups of graduates were less likely to report high levels of understanding about the Carnegie classification system for insti- 358

19 Table 6 Results of Chi-Square Analysis on Research Foundations Section of the CAS Foundation Studies Instrument 359

20 Table 6, continued 360

21 tutional types (56.2%). This item was assigned to the Historical Foundations section. As indicated by the total affirmative responses by groups of items, recent graduates reported the greatest amount of understanding in Cultural Foundations and the least amount of understanding in Research Foundations. Items in this latter foundational area over which respondents were less likely to report highly levels of understanding included: (a) selecting an appropriate statistical method that matches different types of data 53.9%, (b) distinguishing between levels of data represented by a variable in a quantitative analysis 56.9%, and (c) choosing a research method to match the unit of analysis 68.0%. Such a result is not surprising. Many professional preparation programs, whether counseling-based or not, devote a great amount of class time to issues of tolerance, diversity, and multiculturalism; and while many programs incorporate assessment into the curriculum, most programs require no more than one introductory research class. It may also be that other in-class and field experiences are more likely to re-enforce or complement learning components found in Cultural Foundations rather than those identified in the research area. Sandeen and Barr (2006) suggest that assessment is very important to the successful practice of student affairs (p. 132). At least based on the responses in this study, the curricula in Master s level preparation programs may not provide enough preparation in assessment and research to help graduates play a meaningful role in this arena. Indeed, a review of the latest version of the ACPA directory of preparation programs revealed that only 12 out of the 90 programs listed require a thesis or a major research project (ACPA, 2005). We also found that recent graduates from CAS compliant programs reported greater understanding than their non-compliant counterparts in 48 of 60 items on the survey instrument. Statistically significant differences between respondents in CAS compliant and non-compliant programs were revealed on only two of the 60 items, however. Interestingly, graduates from non-compliant programs were more likely to report higher levels of understanding about involvement theory and understanding the level of data represented by a variable in quantitative analysis. Given the number of items in the questionnaire, one 361

22 would expect three items to be significant on chance alone using a probability level of.05. Most likely these results have little practical significance. While we cannot determine if the curriculum completed by these graduates was shaped by the CAS standards, these findings suggest that recent graduates have been exposed to curriculum issues that are defined as foundation studies by CAS. While the number of full-time faculty and the requirement of two field experiences that total at least 300 hours are convenient measures that help define CAS compliant programs from those that are not, these two characteristics alone were not able to account for any substantive difference in self-reported learning in the Foundational Studies curriculum component as recommended by CAS. If other measures of compliance were developed and used as distinguishing criteria (e.g., qualifications of the faculty, an analysis of course offerings and their content, other cocurricular program requirements, or the nature of the student-faculty advising relationship) the findings may have been different. When interpreting these findings, the limitations of the study should be kept in mind. The 60 items developed for our questionnaire may not accurately assess learning in the curriculum area referred to as Foundational Studies. The sample was a small one and was not randomly selected. Respondents may not have represented the population of professionals recently graduating from professional preparation programs. Several chi-square tests could not be conducted because small cell sizes violated the underlying assumptions of the statistic. Constructing a different kind of questionnaire that would allow the use of other types of inferential statistics may overcome this difficulty. Finally, we relied on self-report data and graduates may have overestimated their knowledge or understanding. Recent graduates often report high levels of satisfaction with their graduate school experience and this halo effect may have influenced their responses. The results of the study are positive, however, and they should not be dismissed given the simplicity of the methodology used or its limitations. Large numbers of program graduates from participating institutions report high levels of understanding about many of the foundational principles of the student affairs profession. 362

23 Implications for Preparation Programs and the Profession This is one of very few studies that attempts to measure learning outcomes based on the work by CAS that can be used to identify competencies of those who complete professional preparation programs. Much more needs to be done, however. While writers in the field have reinforced the importance of such learning as crucial to the well being of the profession (Sandeen & Barr, 2006; McEwen & Talbot, 1998), attempts to assess what has been learned in professional preparation program has been spotty at best. National rankings of preparation programs, student satisfaction evaluations, and even employer satisfaction evaluations are not sufficient measures. Assessing learning on a national basis, building on the methods similar to those found in this study, on the other hand, could help substantiate the real value of our preparation programs. Assessments completed on individual programs could help faculty members adjust their course syllabi and curriculum emphasis. Conclusion Leaders in both primary national associations in the student affairs profession have indicated an interest in certifying the knowledge base of their members. The phenomenon known as high stakes testing so common in our public schools may be coming to colleges and universities shortly. Indeed, the members of the Secretary of Education s Commission on the Future of Higher Education have called for the increased use of testing to ensure learning and accountability (Miller & Malandra, 2006). Elected officials in state legislatures have already requested that higher education oversight agencies develop processes to measure core competencies of undergraduates (SCHEV, 2004). The National Research Council is preparing to collect a wide variety of information that will attempt to demonstrate the value of a graduate degree and the learning outcomes that have been achieved (Ostriker & Kuh, 2003). While many in the student affairs profession and professional preparation programs have resisted the idea of testing and certification or even measuring learning outcomes in a systematic way, 363

24 trends in all segments of the education enterprise suggest that we may not have the luxury of taking such a position in the future. A steering committee of ACPA s Task Force on Certification has identified the knowledge needed to be successful student affairs practitioners from a review of the literature (Schoper, Stimpson, & Segar, 2006). In a report to the NASPA board of directors, Janosik (2002) presented a six-cell matrix that outlined a student affairs curriculum as a guide for practitioners. The CAS standards specifically developed for Professional Preparation Programs have existed for quite some time and have been revised regularly. Others have commented on the competencies and skill sets necessary for success in the student affairs field as well (e.g., Komives & Woodard, 2003; Pope, Reynolds, & Mueller, 2004; CAS, 2003). There is significant commonality among these models and lists (Segawa, in press). In the chapter titled The Foundation of Student Affairs, Sandeen and Barr (2006) suggest that student affairs professionals stop the arguments and test the theories (p. 27). With respect to measuring the learning outcomes of our preparation programs, it might also be time to stop discussing what we should know and instead, start measuring what we already expect our graduates to know. References American College Personnel Association. (2005). ACPA commission XII Graduate Preparation Program Directory. Retrieved April 10, 2005, from Council for the Advancement of Standards in Higher Education. (2003). The book of professional standards for higher education. Washington, DC: Author. Forney, D. S. (1994). A profile of student affairs master s students: Characteristics, attitudes, and learning styles. Journal of College Student Development, 35, Harrow, A. J., & Mann, B. A. (1996). Ethics and inquiry: A graduate course in higher education leadership. College Student Affairs Journal, 15, Hesburg, T., Miller, P., & Wharton, C. (1973). Patterns for lifelong learning. San Francisco: Jossey-Bass. 364

25 Hyman, R. E. (1985). Do graduate preparation programs address competencies important to student affairs practice? (Report No. CG ). East Lansing, MI: National Center for Research on Teacher Learning. (ERIC Document Reproduction Service No. ED ). Janosik, S. M. (March 22, 2002). The development and implementation of a National registry for student affairs administrators. Washington, DC: NASPA National Board. Komives, S. R., Woodard, Jr., D. B., & Associates. (Eds.). (2003). Student services: A handbook for the profession (4th ed.). San Francisco: Jossey-Bass. McEwen, M. K., & Roper, L. D. (1994). Interracial experiences, knowledge, and skills of master s degree students in graduate programs in student affairs. Journal of College Student Development, 35, McEwen, M. K., & Talbot, D. M. (1998). Designing the student affairs curriculum. In N. J. Evans & C. E. Phelps Tobin (Eds.), State of the art of preparation and practice in student affairs: Another look (pp ). Lanham, MD: University Press of America. Miller C., & Malandra G. (2006). Accountability and assessment. Retrieved September 15, 2006, from bdscomm/list/hiedfuture/reports/miller-malandra.pdf Ostriker, J. P., & Kuh, C. V. (2003). Assessing research-doctorate programs: A methodology study. Washington, DC: National Academies Press. Pope, R. L., Reynolds, A. L., & Mueller, J.A. (2004). Multicultural competence in student affairs. San Francisco: Jossey-Bass. Sandeen, A., & Barr, M. (2006). Critical issues in student affairs: Challenges and opportunities. San Francisco: Jossey-Bass. SCHEV. (2004). Overview of SCHEV s reports of institutional effectiveness. Retrieved September 15, 2006, from schev.edu Schoper, S., Stimpson, R., & Segar, T. (January 29, 2006). ACPA Certification Taskforce synthesis: A professional development curriculum. Washington, DC: American College Personnel Association. Segawa, M. (in press). Conferences, associations, and home-based: Professional development opportunities. In R. L. Ackerman & E. A. Langdon (Eds.), Mid-Level Managers. Washington, DC: National Association of Student Personnel Administrators. 365

26 Stuit, D. B., Dickson, G. S., Jordan, T. F., & Schloerb, L. (1949). Predicting success in professional schools. Menasha, WI: George Banta. Von Destinon, M. (1986). Educating student personnel administrators: Recommended standards and the University of Arizona master s program (Report No. HE ). East Lansing, MI: National Center for Research on Teacher Learning. (ERIC Document Reproduction Service No. ED ). Young, R. B. (1994). Student affairs professionals perceptions of barriers to participation in development activities. NASPA Journal, 31, Young, R. B., & Elfrink, V. L. (1991). Values education in student affairs graduate programs. Journal of College Student Development, 32,