Utilizing the Capstone Experience to Assess a MBA Program: Case Analyses, Groups, and Rubrics

Utilizing the Capstone Experience to Assess a MBA Program: Case Analyses, Groups, and Rubrics Floyd G. Willoughby, Oakland University James Suhay, Oakland University Assessment is fast becoming a requirement of both broad certification agencies such as AAHE (American Association for Higher Education) (AAHE, 2004) and of professional associations such as the AACSB (American Assembly of Collegiate Schools of Business) (AACSB, 1995,p. 21). This research details the assessment methodology of a MBA program by evaluating overall student performance and group processes. Written, group case analyses are utilized as direct measures of student outcomes. A two -reviewer model is used to evaluate the written assignments using weighted rubrics as guidelines. Group processes are also investigated by surveying the students on the workings of their groups. The rubrics and survey are discussed. INTRODUCTION Assessment is becoming a major activity for most educators. Emphasis on accountability for academic programs is the driving force for rigorous, regular measurement of student learning outcomes (NCA, AACSB). Assessment forces academic units to formulate goals and translate those goals into measurements of student learning outcomes. Instruments are then devised and implemented to measure student learning. The academic unit faculty is then required to implement changes to improve student learning. This research describes in detail how a group case analysis written assignment can be used to assessment an accredited MBA program. The case analysis exercise is an excellent method to measure students abilities to apply the business principles learned in the variety of business courses in the program, to identify and analyze business problems, and to synthesize realistic solutions to those problems. METHODOLOGY The SBA Graduate Curriculum Committee determined that the MBA Program was to be assessed by measuring the performance of student groups on an integrative business case in the Program capstone course, MGT 535. Two sections of MGT 535 were to be assessed. These sections combined totaled 13 groups. The primary instructor of MGT 535 was assigned the tasks of selecting the case, preparing instructions, administering the survey, collecting the completed group case analyses, evaluating the case analyses, analyzing the results of the assessment, reporting those findings to the Graduate Curriculum Committee, and writing the final report to the University Assessment Committee. 1

The case chosen for the assessment was Kacey Fine Furniture: Human Resource Management in the Face of Change (Winn, 1999). The case was chosen because it was an integrative case presenting issues in finance/accounting, strategic management, marketing, and management information. An Instruction Sheet accompanying the case was given to the students with an approximate two-week deadline. The Instruction Sheet (See Appendices) was intentionally vague. The hidden agenda was to not only assessing the students application of learned concepts, but also to measure their abilities to identify problems and determine the content of the report, i.e., the issues to be addressed. Since one of the major outcomes is the ability to work well in groups, the students were also informed that upon completion of the case analysis, they would also be asked to complete a Group Processes and Peer Evaluation Form (See Appendices). This form was intended to measure the degree of effectiveness of the groups and the extent to which all members did their share of the work. The two judges were the primary instructor for MGT 535 and an adjunct instructor who teaches MGT 535 on a regular basis. The Score Sheet (See Appendices) asks about the extent to which the written case analyses dealt with issues in various categories and the overall writing quality of the papers. The Rubric (See Appendices) is an evaluation aid intended to more accurately anchor an evaluator s rating of the paper in the categories. The Rubric is a descriptive rubric (Suskie, 2005). A descriptive rubric characterizes each level within the various assessment categories. The intent of the descriptive rubric is to reduce the variance among judges. The student groups submitted their case analyses for grading on the due date. The primary instructor administered the Group Processes and Peer Evaluation Survey. The case analyses were distributed to the second judge and the responses on the Group Survey was encoded and analyzed by the primary instructor. The results of the questionnaire to assess group dynamics were recorded in an Excel spreadsheet. The mean scores were compared across groups. The scores of the two judges were compared (See Appendices, Judges Scores of Case Studies for the Assessment of the MBA Program.). RESULTS There were 13 groups in the two MGT 535 course sections in Winter 2003. Each group submitted a group case analysis. One group paper had missing information so it was eliminated by the judges from the analysis. The scores of the two judges on the case analyses were fairly consistent. Judge F seemed to be a more severe rater than Judge J, but a general pattern of high and low scores is apparent. Specifically, the coverage of the environment, industry and competition were very weak. The discussions lack depth and detail. This is evident by the low scores that most of the papers received in the SWOT, Opportunities and Threats. The external environment is an area emphasized in the textbook and in lecture. The papers were also weak in the coverage of the internal functions of the firm. Hardly any papers covered the internal analysis in an organized and thorough manner. This resulted in incomplete SWOTS, Strengths and Weaknesses. Most of the papers did not clearly identify specific problems, but seemed to imply problem areas before presenting recommendations. 2

Recommendations were, with a couple of exceptions, brief and superficial. There was very little explanation of how recommendations would solve problems. Only one or two of the groups did acceptable outside research for the case. It was expected that information about the furniture industry as, competitors, and competition in the Denver area would be presented to enhance the understanding of the firm s situation. In general, the quality of the writing was below expectations. Most papers were poorly organized and seemed to jump from topic to topic without transitions. It appears that the written group case analyses were broken down into tasks and the tasks assigned to group members. The various completed assignments were then assembled without any editing. Some papers had misspellings and grammatical mistakes. The group activities assessment revealed no surprises. Inspection of the Questionnaire will reveal that a Likert type format is used. Every-other statement is a negative statement. Comparison of the statements and the means on the tally sheet shows that the high means (4.66, 4.8, 4.52, 4.66, and 4.52) are means for the positively stated statements and vice versa for the negatively stated statements. The group activities questionnaire means seemed to indicate that most of the groups worked well together, had meetings, etc. to get the job done on time. The peer evaluations would seem to indicate that almost all group members felt that the other members did their part. CONCLUSIONS, COMMENTS AND RECOMMENDATIONS There can be differing interpretations of the assessment results. The group assessment methodology was formulated for a couple of reasons. First, with groups doing the papers, there would be fewer papers to evaluate. Second, the group processes measurement could be included in the assessment. The researcher feels that the assessment sounded good in formulation, but did not implement well. Even though this was an embedded assessment, students knew that this assignment was different from what had been done in previous sections. It is feared that this created resentment toward the assignment. It might be concluded that the students decided to do the least work to complete the assignment. The assignments were not well done and displayed an overall lack of effort and skill on the part of the students. The students were graduating at the end of the semester and this may have made them less than enthusiastic about completing an assignment that they viewed as extra work. Anecdotal evidence (overheard conversations and direct comments to the professor) would suggest that groups were not as harmonious as the results would indicate. Experience has shown that students are very cautious about rating other students or about revealing that not all group members did their share. Thus, the group processes results may not be valid. If the results are valid, faculty has seriously misjudged the ability of the MBA Program to provide opportunities for the students to learn the expected skills. For example, aside from the fact that there may have been a lack of student effort to do an in-depth analysis of the business case, it could be expected that students would have much better writing skills than appears to be the case. Also, it could be expected that students would at the very least analyze all parts of the firm and its environment. It appears that the students, without an exact outline specifying the topics to be covered, cannot do an acceptable analysis of a business case. 3

The results of the assessment demonstrate that some major changes need to be made in the capstone pedagogy. First, it is obvious that students need a refresher on how to do business cases. Doing business cases should be a regular part of the course. It is clear that the groups did not follow any organized, systematic procedure in analyzing the case or in writing the paper. The lack of organization, misspellings and poor grammar may indicate that the graduating MBA s cannot write as well as was first thought. The writing component of the course should be increased. Lastly, the assessment could have been done much better. There were incorrect assumptions made about the students and groups There also administrative delays as well as. However, the results are useful in that some problem areas are revealed. REFEENCES AACSB, (1995). Achieving Quality and Continuous Improvement Through Self-evaluation and Peer Review, Standards for Accreditation, Business Administration and Accounting, Guidance for Self-evaluation. AAHE, Initiatives, Assessment Forum, 9 Principles of Good Practice for Assessing Student Learning, www.aahe.org/principl.htm, February 16, 2004. Suskie, L.(2005). Workshop on Embedded Assessment. Oakland University, April 12, 2005. Winn, J. (1999). Kacey Fine Furniture: Human Resources Management in the Face of Change APPENDICES 4

JUDGES SCORES OF CASE STUDIES FOR THE ASSESSMENT OF THE MBA PROGRAM Group/Judge Problem ID Ind & Comp Ana SWOT O & T SWOT S & W Rec s Research Writing Quality Total Score 3 Stooges F 1 1 3 2 1 4 3 15 J 3.5 5 4 4 3.5 3.5 5 28.5 KEER F 3 3 3 3 4 2 3 21 J 4.5 4 4.5 4.5 4 4 3.5 29 OverWork/UP F 2 5 4 4 4 2 3 24 J 3 5 4.5 4.5 4.5 4.5 4 30 TeamTecumseh F 2 2 4 3 5 5 4 25 J 4.5 4 5 5 4 3.5 4 30 Sh#theads F 1 2 3 3 2 2 2 15 J 4 5 5 5 4.5 4.5 4.5 32.5 Bushmasters F 3 2 2 3 5 5 5 25 J 3.5 5 4.5 4.5 5+ 5 5 32.5 DeerinHeadlights F 2 2 2 2 3 1 1 13 J 4.5 4.5 4 4 4 2 2.5 25.5 Big Three F 1 1 1 3 1 1 2 10 J 3 3 3 4 3.5 2 4 22.5 Gearheads F 2 1 2 2 2 2 2 13 J 3 3 4.5 4.5 4 3 3.5 25.5 Hackel F 2 1 2 2 2 1 2 12 J 3.5 4 4.5 4.5 4 2 3.5 26 Pedagogy F 1 1 1 1 2 1 2 9 J 3 2 2 2 3.5 2.5 2.5 20 CCSS F 2 1 4 3 2 1 4 17 J 5 4 4 4 5 3 4 29

MGT 535 Group Written Case Analysis Guidelines Learning Objectives The overall objective of this group written case analysis is to assess the student learning of graduating MBA students. Students will apply business and economic principles and concepts correctly and appropriately. Students will analyze and synthesize information in formulating realistic solutions to business problems that they identify. Students will demonstrate the ability to utilize correct grammar, punctuation and spelling in presenting the analysis and recommendations in a logical, persuasive, sustentative manner. Students will demonstrate their ability to work as a team in small groups, sharing work and responsibilities to complete the assigned task. Case Analysis Specifications 1. The paper is to be double-spaced with 1 margins. 2. The suggested maximum length of the paper is 20 pages not including References and Appendices. 3. All exhibits, graphs, figures and financial statements should be placed in the Appendices. Each of the above must be titled and numbered. 4. The suggested reference style for citations and references is the Publication Manual of the American Psychological Association, 5 th Edition. All research information, other than that obtained from the Case and Concepts texts must be properly cited and referenced. 5. Each group will submit one copy of their completed paper for grading. The paper will be unbound with a coversheet identifying the students only by group name. Policies 1. The written case analysis is a group project and all group members must participate. There can be assignment of responsibilities for lead investigation, analysis and formulation of solutions, however the paper must be a group effort with all group members contributing to the final paper. 2. Each group member will be asked to complete a short team work survey that will include the evaluation of group members contribution and the task and social workings of the group. 3. The aggregated, group peer evaluations will account for 30 points out of the 200 total points possible for the assignment. 4. The effective date of the case is the latest date that can be surmised from the information in the case. For example, if the case material includes a year-end financial statement for 1998, then any information source dated 1999 can be used.

FGW. Notes 5. All questions should be directed to Floyd, via email, telephone or in-person. 6. The completed paper and the team surveys are due on the specified date, at 6:30 PM for the Thursday class or 9:00 AM for the Saturday class NO EXCEPTIONS. 7. The completed papers will be evaluated by Professors Floyd G. Willoughby and one other professor to be named. Only Floyd s score will be used to determine the group score.

MGT 535 ASSESSMENT QUESTIONNAIRE WINTER 2003 Team Name: Instructions: The questions below ask about your experience as a group member in completing the Kacey Fine Furniture case analysis. You can be assured that strict confidentiality will be maintained. At no time will any of your responses be associated with your identity or be revealed to your group members. Data will be reported in the aggregate only, i.e. the unit of analysis is the group. Please answer each of the questions to the best of your ability. Please respond to the following statements by circling the number representing your answer. Strongly Agree Can t Disagree Strongly Agree Decide Disagree 1. Our group worked as a team. 5 4 3 2 1 2. We did not discuss group norms. 5 4 3 2 1. 3. We had an initial group meeting to 5 4 3 2 1 get organized. 4. We did not assign individual responsibilities. 5 4 3 2 1 5. Every group member did their share of the 5 4 3 2 1 work. 6. Group members fulfilled their responsibilities 5 4 3 2 1 as assigned. 7. Group members were not prepared for group 5 4 3 2 1 meetings. 8. Contributions of group members met my 5 4 3 2 1 expectations. 9. The final paper did not meet my expectations. 5 4 3 2 1 10. I could have done a better job if this were an 5 4 3 2 1 individual assignment. Give each of your group members a score for their performance on the case analysis. 100% = Great job! to 0% = They sucked! Team Member 1 2 3 4

MGT 535 Case Analysis Evaluation Rubric Score 1 2 3 4 5 Major issues not identified Some major issues identified, Some major issues identified Most major issues identified and nor discussed. but not discussed. but only marginally discussed. adequately discussed. Problem Identification Industry & Competitor Analyses SWOT Opportunities & Threats SWOT Strengths &Weaknesses Recommendations Research Writing Quality Major industry characteristics and competitors mentioned but not discussed. Important trends and events not mentioned. Major organization strengths and weaknesses not identified nor discussed. Recommendations do not appear to address or solve major problems. Research is minimal with less than 10 websites cited and has little effect on the overall quality of the paper. Paper is not well organized, there are writing style differences among sections, and many uncorrected technical errors. Industry characteristics and competitive rivalry not meaningfully discussed. Trends and events not covered in a thorough or meaningful manner. Some major issues identified, but not discussed. Recommendations address some major issues but are only marginally explained. Research is weak on major environmental and industry segments and is not well applied in the paper. Paper is not well organized and issues are not logically discussed. Uncorrected grammatical and spelling errors. Industry characteristics only marginally discussed, but implications for the firm not discussed. Some important trends missed and implications for the organization vague. Some major issues identified but only marginally discussed. Recommendations address most major issues, but are not elaborated upon not adequately explained. Research is adequate, but it is only marginally applied in the paper. Paper is logically organized, but grammar is awkward and there are some uncorrected technical errors. Industry characteristics discussed very well, but implications for the firm are not clear. Most important trends and events identified, but implications are not clearly stated. Most major issues identified and adequately discussed. Recommendations address all major issues and appear to be appropriate for solving organizational problems, but are not explained in-depth. Many reference sources and well applied in the paper. Paper is adequately organized, and grammar is acceptable but there are some uncorrected technical errors. Major problems identified and thoroughly discussed. Industry characteristics and organization s competitive position within the industry meaningfully discussed. Important trends identified and implications for organization are stated. Major organization strengths and weaknesses identified and appropriately discussed. Recommendations address all major issues, appear to be appropriate for solving organizational problems and explained in depth. Research is in-depth and thorough and is evident throughout the paper. Issues logically developed and paper is well organized and well written. No typographical or grammatical errors.