ABET Preparation - Helpful Ideas

Transcription

1 ABET Preparation - Helpful Ideas Prepared and Edited By: Jeffrey M. Forbes * and Dennis K. McLaughlin + with contributions from faculty at: University of California, Irvine University of California, San Diego University of Colorado Embry Riddle University University of Florida Iowa State University University of Maryland University of Minnesota North Carolina State University University of Notre Dame University of Oklahoma Penn State University Purdue University Saint Louis University Syracuse University U.S. Air Force Academy Virginia Tech August, 2006 * Glenn Murphy Professor of Aerospace Engineering Sciences, University of Colorado. Boulder, CO Professor of Aerospace Engineering, Penn State University, University Park, PA Chair, AIAA Academic Affairs Committee

2 Forward This document is a result of discussions within the Academic Affairs Committee of the American Institute of Aeronautics and Astronautics. Faculty in Aerospace Engineering Departments have expressed concerns for a number of years regarding the process of preparing for ABET 2000 accreditation visits. Such concerns centered on confusion as to what is an appropriate level of preparation for ABET visits and what actions are most beneficial. In an effort to make a positive contribution, the Academic Affairs Committee of AIAA undertook a project that involved asking members of the AAC and of ADCA, the Aerospace Department Chairs Association of the AIAA, to provide descriptions of problems their department was having and particularly useful actions they undertook to more effectively prepare for ABET reviews. Responders were asked to comment specifically on the resource commitment that was required for various levels of preparation. Faculty from seventeen universities responded with descriptions of their experiences with ABET preparation including both positive and negative aspects and suggestions for activities that programs could undertake to improve the results of their ABET reviews. These descriptions were compiled and edited to reduce the redundancy. The result is this document. It is intended to update this document on approximately a yearly basis as programs transition through from their first ABET 2000 visit to an equilibrium. The authors welcome any comments and suggestions for future revisions. I. Comments grouped according to topic General Organization of Effort 1. Form a standing committee whose responsibility it is to make sure the assessment feedback loop is active during each academic year. Results of assessments and the curriculum improvements that result from them should be documented. Committee members should rotate so that a significant fraction of the faculty are exposed to the process. This committee can ramp-up to produce the self-study report as the ABET visit approaches. 2. Information about ABET objectives, assessment, etc. should be a normal part of the information provided by the Department in course catalogs, web pages, newsletters, etc. Update this information periodically. Keep the ABET culture and level of consciousness alive between ABET visits. 2

3 3. Make sure that ABET materials such as Program Objectives, Outcomes, assessment methods, are documented identically in different places, i.e., ABET self-study, Department web site, Course Catalog, etc. Developing Objectives 1. Read the ABET definition of Program Objectives carefully before articulating your own. As of 2006, ABET defined Objectives to mean broad statements that describe the career and professional accomplishments that the program is preparing graduates to achieve. The Key word here is accomplishments. Program Objectives should not be changed very often. 2. Program Objectives can be used to distinguish how graduates from your particular program differ from graduates from other programs. 3. You should demonstrate how your Program Objectives meet the needs of your constituencies, and are consistent with your Institution s mission, and your strategic plan. 4. Many programs are stating the objectives of their undergraduate programs on their web sites. A word of caution: Not all examples available over the web conform well to the stated ABET definition (see 1. above). 5. Programs should make an effort to connect open-ended objectives with quantitative assessment of outcomes. Developing Outcomes 1. ABET defines program outcomes to be statements that describe what students are expected to know and be able to do by the time of graduation. These relate to the skills, knowledge, and behaviors that students acquire in their matriculation through the program. 2. Outcomes do not need to differ from the (a)-(k) outcomes of ABET, but if you do develop your own outcomes they must, at a minimum, meet ABET s (a)-(k), and you need to develop a clear mapping between your outcomes, and ABET s (a)-(k). Some evaluators are favorably impressed when programs develop their own variations of (a)-(k), but there is no advantage in effectively requiring more than (a)-(k). 3. If it is determined that Program Objectives are not being met, then outcomes should be revised accordingly. 3

4 Assessment Methods 1. Assessments of outcome achievement should be based on quantitative data, i.e. faculty assessments of actual student work. This needs to be documented, and if there are noted deficiencies, then changes in the curriculum should be made to address these deficiencies. These, too, must be documented. This is the feedback process that the evaluators are looking for. 2. Documentation that the processes are working and producing the desired results can take the following forms: meeting minutes, memos, tabular data, grading forms, etc. 3. Provide evidence that results are being used to improve the effectiveness of the program CLOSE THE LOOP! Assessment specific to objectives 1. Use your external advisory board or equivalent to periodically review Program Objectives this provides industry perspective. 2. Save minutes of external advisory board meetings. 3. The following tools are useful in evaluation of program objectives: a. Alumni Surveys (about behaviors and accomplishments rather than opinions) b. Industrial/Professional Advisory Boards c. Employers surveys and placement data d. Graduate programs at other universities 4. Alumni Surveys: a. Use FACT-based questions in alumni surveys to determine if Objectives are being met. (Using factual-based questions makes this direct assessment.) b. Have a survey designer look over the Alumni Survey to better bring out facts relevant to the Objectives, and to include definitions of the terms used in the survey. c. Surveys can be web-based, paper, or some combination. 5. Save anecdotal s from alumni and from employers. 6. Exiting students ask them to be receptive to surveys in the future. 4

5 7. If you have both AE and ME programs, ABET wants separate surveys sent to AE and ME alumni, industrial contacts, etc. This means you should keep separate lists of these AE and ME contacts. 8. Assessment of objectives should involve quantifiable data. Assessment specific to outcomes 1. A successful assessment methodology meets ABET requirements, provides needed data to assess and improve the program, is efficient and minimally burdensome to the faculty. 2. Some Departments use their two-semester capstone design course as the primary vehicle to provide direct assessment of "outcomes", in conjunction with grading/evaluating PDRs, CDRs, etc. a. Forms are used to grade students against outcomes. Be sure to express the outcomes on these forms identically to the way that you have defined them in the self-study and other materials. b. Strong recommendation from ABET Evaluator: Get an industry person to sit in on Senior Projects presentations. c. It was noted that reviewing the output of the capstone design courses is a very time efficient, integrated means for providing the students an opportunity to learn, practice, and demonstrate many elements listed in ABET criterion 3 (a)-(k). d. Written student reports with comments from the instructors serve as convincing evidence of the achievement of the program outcomes. 3. If your primary direct assessment method is focused on Senior Projects, consider devoting the Sunday afternoon of your ABET visit to a display of Senior Projects, with Senior Projects faculty and students present. 4. Some Departments use multiple direct measures of achievement for each outcome. This is where faculty gauge the degree of achievement based on review of student work. These measures can include but are not limited to: a. test questions b. specific homework problems c. projects d. design reports e. lab reports f. logbooks g. design competitions h. outside (external) reviewers (e.g. Advisory Board members) 5

6 5. Additional useful data can be obtained from a. Alumni b. senior surveys c. employer surveys d. exit interviews e. Advisory Board reviews 6. Although evaluation sheets for co-op students (from industry mentors) can be of value, it is of considerably less value if 100% of the students are not engaged in this activity. Outcomes assessments should be performed on work required of all students in the program. 7. Student Input: a. Senior exit surveys, or exit surveys for specific courses, are sometimes used. (For instance one can plot a-k results laterally over years to watch trends that need attention.) Some programs are attempting to develop web-based assessment tools for students. However, this self-assessment by students is not direct assessment, and hence does not carry nearly as much weight as direct assessment methods. b. Student Input can be utilized in a variety of ways: i) Class Evaluations: a-k, class objectives ii) Town hall meeting with students, once a year. Minutes kept on file. iii) Student curriculum improvement team (CIT) involved in the assessment and improvement process. iv) Student Advisory Board; what is going well, what is wrong. Feedback 1. Some Departments hold 1-day curricular retreats or rap sessions at the end of each academic year to discuss what changes needed to be made in the curriculum to address deficiencies noted in the Senior Projects assessments for that year. 2. It is important that any deficiencies noted as part of the assessment process be dealt with quickly, and that these course or curriculum modifications are documented. 3. One ABET evaluator recognized that they had a continual assessment and feedback process leading to program improvement, and that feedback and improvement were performed on a short time scale. 4. In one Department, faculty fill out a brief assessment sheet for each course which is passed on to the next instructor. 6

7 Frequently asked questions 1. How few examples of coursework are needed in order to make an outcomes assessment? 2. How can the process be made less burdensome to the faculty? 3. Does the program need to demonstrate that every student meets every expected outcome, in the same manner that every student must meet every minimum requirement for conferral of their degree? General Suggestions (things that worked well), as reported to the authors 1. We prepared a "draft" of a self study document that described our evolving process and methods of assessment. We then invited a recent ABET site visitor for a short mock-visit to discuss our process, data collection and goals for documentation. This proved useful and no significant "midcourse" corrections were recommended or implemented. 2. Prior to the visit we interacted with the Program Evaluators during a two month period and submitted written responses to them on a variety of issues. 3. A problem arose regarding ABET s constraint of a maximum of 2 pages for course outlines. In several cases, our course outlines exceed 2 pages because of descriptive text that states the relationship of the course s outcomes to the program outcomes. The solution we pursued was to ensure the first 2 pages contain the core information, with any additional information included after these 2 pages. General comments on the process Evaluators typically focus on the following things: 1. That documentation exists to show that outcomes have been adequately assessed. 2. That deficiencies noted in the assessment have been used to produce program improvement, and that these improvements have been documented. 3. The above process is ongoing, and a natural part of the Department s activities. 7

8 Less attention is devoted to: 1. How Outcomes and Objectives are defined and modified. Problems in assessment (as reported to the authors) 1. If too many elements of coursework (exams, homework, lab reports, etc.) are used to evaluate the same outcomes, then too much work is imposed on the faculty. It is doubtful that this approach can be sustained. 2. Mapping exam questions into (a)-(k) has been found to require too much resource commitment. Again, this is not likely to be sustainable. 3. While the spirit of EC2000 is embraced by most as a good means to improve program quality, ABET needs to provide more concise and effective documentation for their expectations and make it available in a fashion that recognizes the ongoing evolution of the assessment processes in the individual programs. 4. If ABET believes that a few homework assignments, test questions and projects are the best manner in which one "demonstrates that students have achieved each and every item in (a) through (k)" then the concern exists that this is just another case of "teaching to the test" and the ultimate goal of improved, internationally competitive academic programs will be lost. 5. A problem arose regarding direct assessment of the AE senior design project (MAE 159: Aircraft Design). We have a scoring rubric that, in the past, the AE ABET lead faculty (one of the MAE faculty) has completed by evaluating the students design reports. We feel that the problem with this is that a colleague essentially evaluates another colleague s teaching performance. We feel we are constrained by ABET s policy that instructors of design project courses are not allowed to complete the rubrics. This aspect of ABET s policy appears to be contradictory to the other aspect of ABET s policy that instructors of non-design courses are allowed to complete their own direct assessment forms. One possible solution is to have a panel of instructors and/or industrial professionals evaluate the students design reports but, again, it would seem this would amount to a peer evaluating another peer s teaching performance. 6. There is a rather general problem of ABET s policy of not considering, at all, instructors total course grades as a means of direct assessment. Our solution to this is to identify data on the components of these grades that relate the course outcomes to the program outcomes. 8

9 Needs requests to ABET / AIAA 1. It appears that the ABET Board and the Commissioners spend considerable time each year making sure that the wording in the statements of requirements for assessment of Objectives and Outcomes are very carefully worded. It seems however, that the descriptions that are included in the training information do not receive the same level of attention. A considerable number of problems (for the individual programs) result from inconsistent and ambiguous wording in these training documents. 2. Regarding the requirement of direct assessment methods. One option is the use of rubrics and specific performance criteria to evaluate achievement of each program outcome. It seems AIAA could develop some standard rubrics that could be used - at least as a base/starting point - since we all need to evaluate the same outcomes. 3. Provide examples of Matrices that map courses to outcomes, etc. 4. Having access to a set of examples to use as guidelines by the departments would have been extremely useful. The examples could be modified to fit each department as needed. Summary The readers should bear in mind that these comments have been assembled from input from 17 different sources. The authors attempted to represent the ideas from these contributors as faithfully as possible. As noted in the forward, the authors welcome comments and suggestions on this document so that improved versions can be produced in the future. File: AAC ABET Ideas-v 8_06.doc 8/21/06 9