A SYSTEMATIC APPROACH FOR DEFINING AND ASSESSING PROGRAM EDUCATIONAL OBJECTIVES AND OUTCOMES Nikos J. Mourtos 1 Abstract The USA Accreditation Board for Engineering and Technology adopted recently a new set of criteria for evaluating engineering programs. The paper describes the design and implementation of a systematic process for defining and assessing Program Educational Objectives (criterion 2) and Program Outcomes (criterion 3). The process begins with definition of detailed course learning objectives to address specific Program Outcomes. New, innovative assignments are introduced in several courses to prepare students in the skills specified by these learning objectives. Student performance is assessed in regards to specific outcomes and changes are implemented to increase student achievement in each outcome. The result was significant curriculum. Program Educational Objectives are defined and evaluated using interviews with graduating seniors, alumni and employer surveys, and input from advisory boards. Index Terms accreditation, course design, program assessment, program educational objectives, program outcomes. INTRODUCTION The USA Accreditation Board for Engineering and Technology (ABET) adopted recently two new criteria for evaluating engineering programs: Criterion 2, Program Educational Objectives (PEOs,) and Criterion 3, Program Outcomes (POs) [1]. PEOs are defined with input from all program constituents and describe the expected accomplishments of graduates during the first several years following graduation. POs, on the other hand, describe what students are expected to know or be able to do by the time of graduation from the program. A systematic process must be in place to assess the achievement of both the POs before students graduate and the PEOs after graduates leave the program. This process needs to be ongoing to ensure the continuous improvement of each program. The paper describes the design and implementation of such a systematic process in the Aerospace (AE) and Mechanical (ME) Engineering Programs at San Jose State University. The relationship between PEOs, POs, and Course Learning Objectives (CLOs) is illustrated in Figure 1. Following the process of engineering design, one may view PEOs as a set of mission requirements (specification). Program Educational Objectives Program Outcomes Course Learning Objectives FIGURE 1 RELATIONSHIP BETWEEN CLOS, POS, AND PEOS Alumni will be able to meet the expecations set for them in the PEOs, if students have the skills described in the POs at the time they graduate. These skills are acquired mostly through the curriculum of each program. Hence, learning objectives in each course must represent a subset of the skills described in the POs. PROGRAM EDUCATIONAL OBJECTIVES PEOs reflect the career and professional accomplishments of graduates during the first several years after graduation. The process of definition and assessment of the PEOs is illustrated in Figure 2. from students, faculty, alumni, employers, and the two advisory boards (one for each program) is used to validate the definition of our PEOs, as well as to assess their achievement. PEOs are revisited periodically to ensure that they continue to reflect current industrial trends. Both programs are designed to fulfill the University [2], College [3], and Department [4] mission. They provide students with a broad understanding of basic concepts, as well as the contemporary skills required by industry. The coursework includes extensive laboratory experiences and many opportunities for students to work on hands-on, design projects. 1 Professor, Mechanical & Aerospace Engineering, San Jose State University, One Washington Square, San Jose, California 95192-0087 njmourtos@sjsu.edu 1
The foundation courses provide a basis for professional competence and the required knowledge to focus on a particular specialization upon graduation 2, in the work environment or in graduate school. Faculty Student Design and implement curriculum changes Define Program Educational Objectives Assess Program Educational Objectives PEOs met? Program is satisfactory Alumni Employer FIGURE 2 DEFINITION AND ASSESSMENT PROCESS FOR PEOS. Advisory Board The PEOs reflect our constituents expectations that our graduates should have: 1. A strong foundation in mathematics, basic science and engineering fundamentals, to successfully compete for entry-level positions or pursue graduate studies in AE / ME or related fields. 2. Contemporary professional and lifelong learning skills including hands-on laboratory experience, familiarity with computers, modern software, and information technology, to successfully compete in the local, national and global engineering market. 3. Strong communication and interpersonal skills, broad knowledge, and an understanding of multicultural and global perspectives to work effectively in multidisciplinary teams, both as team members and as leaders. 4. An understanding of the ethical choices inherent in the engineering profession to deal with issues such as public safety, honest product marketing, and respect for intellectual property. PROGRAM OUTCOMES ABET Criterion 3 requires engineering programs seeking accreditation to demonstrate that their graduates have: a. an ability to apply knowledge of mathematics, science, and engineering 2 Three options are offered in each Program: Aerodynamics and Propulsion, Dynamics and Control, Structures and Materials for AE, Mechanical Design, Mechatronics, and Thermal / Fluids for ME. b. an ability to design and conduct experiments, as well as to analyze and interpret data c. an ability to design a system, component, or process to meet desired needs d. an ability to function on multi-disciplinary teams e. an ability to identify, formulate, and solve engineering problems f. an understanding of professional and ethical responsibility g. an ability to communicate effectively h. the broad education necessary to understand the impact of engineering solutions in a global and societal context i. a recognition of the need for, and an ability to engage in life-long learning j. a knowledge of contemporary issues k. an ability to use the techniques, skills, and modern engineering tools necessary for engineering practice. Figure 3 shows the process for assessing outcomes. A course coordinator must show evidence that his / her course includes the necessary elements to satisfy a particular outcome and collect / analyze data to show that performance targets are met. Moreover, for each outcome there is a designated outcome champion. Champions validate the evidence presented by course coordinators for individual courses and have the final word on whether the performance of a program is satisfactory with regards to their outcome. They meet with course coordinators and instructors, discuss their findings and make recommendations for course. Outcome champions provide an additional level of accountability and ensure consistency in the process. Because outcomes are rather comprehensive and difficult to assess as stated, Felder and Brent [5, p.19] suggest that each outcome be analyzed into elements different abilities specified in the outcome and that a set of attributes be defined for each element actions that explicitly demonstrate mastery of the abilities specified. These attributes have been defined at one of the 6 levels of Bloom s taxonomy in the cognitive domain [6] or 5 levels in the affective domain [7]. Two outcome indicators are used to assess student attainment of program outcomes: (a) course performance ratings based on graded student work and (b) student surveys. To satisfy Criterion 3, performance targets were defined as follows: (a) The scores earned by all students in the assignments and test questions, which pertain to a particular outcome, in each course where this outcome is measured, must be at least 60% 3. (b) The ratings pertaining to this outcome, given by at least 70% of the students in each class surveyed, must be I agree on a 3-point Likert scale. If these targets are met in the courses chosen for assessment of an outcome, the 3 Corresponds to a grade of C-, the lowest passing grade in core courses. 2
outcome is achieved and no further action is needed in this course. When performance targets are met, courses are assessed on a 3-year cycle. When performance targets are not met in a course, are implemented and the course is assessed on a yearly basis until the targets are met. DESIGN AND ASSESSMENT The importance of course design becomes obvious from Figure 1, as CLOs form the foundation for the POs and the PEOs. Our first step was to define detailed and measurable CLOs for all courses, that describe what students should be able to do upon completion of the course. Typically, a course has 30 45 learning objectives. Although content specific, each CLO 4 addresses a specific PO, as shown in Table 1. TABLE 1 EXAMPLES OF CLOS FROM AE162 (AERODYNAMICS). THE RIGHT-HAND COLUMN SHOWS THE POS ADDRESSED BY EACH CLO [8]. Course Learning Objectives 27. Design and perform 5 an experiment to study the performance of an airfoil, analyze and interpret the results from this experiment, compare with analytical / computational predictions and other published experimental data 6, and explain any discrepancies 7. 36. Use the method of images to discuss and calculate aerodynamic interference for (a) wings flying in the vicinity of each other (i.e., wing/tail/canard combinations, biplanes, formation flying, etc.), (b) wind-tunnel boundaries, and (c) ground effects. 44. List several examples of regional, national, and / or global contemporary problems related to aerodynamics (ex. environmental issues, natural resources and energy conservation, etc.) articulate a problem / position statement for each, and explain what makes these issues particularly relevant to the present time. PO 3b 3g 3i 3k 3a 3e The next step is to design lectures, assignments, and other course activities that prepare students in the skills described by the CLOs. Some of the new, innovative assignments, which were introduced in several courses for this purpose, are shown in Table 2. Lastly, Figure 3 shows the process of course assessment. CONCLUSION The paper described the design and implementation of a systematic process for defining and assessing Program Educational Objectives and Outcomes. The AE and ME Programs at San Jose State University were evaluated in 2005. ABET evaluators found the approach described here 4 Only a few, selected CLOs are shown in Table 1. 5 Outcome is met by the requirement that students work in teams of 3-4 to design and perform their experiment, as well as to write their report. 6 Outcome 3i is met as students research the literature for published data and other resources. 7 Outcome 3g is met as students submit a full laboratory report for each experiment. 3g 3i 3j most comprehensive and expressed their satisfaction that it can indeed be used to ensure the continuous improvement of the programs. TABLE 2 NEW ASSIGNMENTS DESIGNED SPECIFICALLY TO SATISFY CRITICAL AREAS OF THE POS. Course assignment Courses in which assignment was introduced Students design the experiments they will perform in the various laboratories [9]. Students discuss economic, environmental, social, political, ethical, safety, liability, and manufacturability constraints in their design of aircraft / spacecraft. Students are taught team skills and required to assess formally the performance of their teammates using specific criteria. Students identify, formulate, and solve open-ended problems. Some of these problems involve integration of material from two or more courses [10]. Students research, present, and discuss in class safety, ethics, and liability issues in AE. Students research, present, and discuss in class contemporary engineering applications and their impact in a global and societal context [11]. ME120-Experimental Methods ME120-Experimental Methods ME195A&B-Senior Design Project ME111 Fluid Mechanics AE165 Flight Mechanics AE167-Aerospace Propulsion ME111 Fluid Mechanics AE165 Flight Mechanics AE167-Aerospace Propulsion REFERENCES [1] Criteria for Accrediting Engineering Programs, Effective for Evaluations During the 2005-2006 Accreditation Cycle, Engineering Accreditation Commission, Accreditation Board for Engineering and Technology, <http://www.abet.org/forms.shtml>. [2] San Jose State University mission statement, <http://www.sjsu.edu/about_sjsu/mission/>. [3] SJSU College of Engineering mission statement, <http://www.engr.sjsu.edu/overview.php>. [4] SJSU Department of Mechanical & Aerospace Engineering mission statement, <http://www.engr.sjsu.edu/mae/mae%20mission.php>. [5] Felder, R.M., Brent, R., Designing and Teaching Courses to Satisfy the ABET Engineering Criteria, ASEE Journal of Engineering Education, vol.92, no.1, January 2003, pp.7-25. PO 3b 3c 3a 3e 3f [6] Bloom, B.S., Taxonomy of Educational Objectives, Handbook 1, Cognitive Domain. New York: Addison Wesley, (1984). 3j 3
[7] Bloom, B.S., Karthwohl, D.R., Massia, B.B., Taxonomy of Educational Objectives, Handbook 2, Affective Domain. New York: Addison Wesley, (1984). [8] An example of course learning objectives / program outcomes matrix from the SJSU AE162 Aerodynamics course, <http://www.engr.sjsu.edu/nikos/courses/ae162/ae162lo.htm>. [9] Du, W.Y., Furman, B.J., Mourtos, N.J., On the ability to design engineering experiments, lead paper, proceedings, 8 th UICEE Annual Conference on Engineering Education, February 2005. [10] Mourtos, N.J., DeJong-Okamoto, N., Rhee, J., Open-Ended Problem-Solving Skills in Thermal-Fluids Engineering, Invited Paper, Global Journal of Engineering Education, vol.8, no.2, 2004. [11] DeJong-Okamoto, N., Rhee, J., Mourtos, N.J., Incorporating the Impact of Engineering Solutions on Society into Technical Engineering Courses, Invited Paper, Global Journal of Engineering Education, vol.9, no. 1, 2005. OUTCOME CHAMPIONS + Recommend / implement curriculum as needed. ABET + Define Program Outcomes. OUTCOME CHAMPIONS: Break down each outcome into elements. Define outcome attributes for each element. Define outcome indicators and performance targets. Identify courses that address this outcome. Select courses to be assessed for this outcome. Collect / organize course material from each of the selected courses (syllabus, student work, assignment / test scores for each student). Analyze data Performance targets met? OUTCOME CHAMPIONS: Write student survey questions for each outcome based on attributes Generate student survey including questions for each of the outcomes addressed in their course. Administer student surveys. Outcome satisfied. FIGURE 3 OUTCOME ASSESSMENT FLOWCHART 4
Define CLOs. List outcomes addressed in the course. Course Design List course activities / assignments / tests that address each outcome. At the end of the course, when student work has been graded, add for each student the points for all assignments / tests that pertain to a particular outcome (some assignments may address more than one outcome). Repeat the process for all outcomes. Implement course in the next course offering. Define performance targets (ex. 70% of the students perform at the 70% level in each outcome addressed in the course). Recommend course in content / delivery as needed. Performance target met for each outcome? Create student surveys with questions from all the outcomes addressed in the course. Define targets for survey responses (ex. 70% of respondents "agree" in each question). Build higher student confidence. Target met for each question of the survey? CLOs and associated POs are met; course is satisfactory. FIGURE 4 ASSESSMENT FLOW CHART 5