Preparing for an ABET Accreditation Visit: Writing the Self-Study

Transcription

1 Preparing for an ABET Accreditation Visit: Writing the Self-Study Ronald E. Barr Mechanical Engineering Department University of Texas at Austin Abstract Engineering faculty must prepare for an ABET accreditation visit every six years. Since the ABET process involves demonstration of continuous assessment and improvement, one must have a plan that begins the process several years before the visit. While the current ABET process (EC2000) has been in place for almost two decades, faculty still struggle to determine the best way to prepare for ABET. There are two overriding aspects to this preparation: 1. Developing and maintaining the processes for assessing program objectives and student outcomes, and 2. Writing the self-study in anticipation of the site visit. This paper assumes that the program has the necessary assessment processes in place, and thus will focus on writing the self-study. The strategy proposed for writing the self-study is to look at it from the viewpoint of the ABET Program Evaluator (PEV). This viewpoint is based mainly on what the ABET evaluator needs to check-off when reading the self-study and the subsequent site visit. ABET has no specific guidelines for the self-study other than the published criterion. However ABET PEVs are provided several forms and documents prior to the site visit that helps them in their preparation. One such form (E062) is called the Program Evaluator Worksheet, which can provide insight for preparing the self-study. By looking at the ABET process from PEV s viewpoint, faculty can first see the desirable results they need to achieve during the site visit, and then start planning a self-study that will result in a successful ABET accreditation outcome. Introduction The key to understanding a successful ABET visit is to understand what the PEV must report in the evaluation forms. The main goal of the PEV is to determine if all nine ABET criteria have been met to some level of satisfaction, and to report any shortcomings on the ABET PEV forms. The Program Evaluator Worksheet (form E062) is used to check-off whether there are any concerns (C), weaknesses (W), or deficiencies (D) in the program, based on the expectations of each of the nine ABET criteria. These shortcomings () are defined by ABET as shown in Figure 1. The goal of the program faculty is to receive no weaknesses (W) or deficiencies (D) at the exit interview. This historically will result in a Next General Review (NGR) grade, which means re-accreditation for another six years. In the past, a concern (C) did not require a response, but recently ABET has also expected some response to concerns as well. It should be noted the form E062 has multiple columns, and it is first completed by the PEV while reading the self-study prior to the site visit. Thus, the self-study is important in that it gives the first impression of the program to the PEV, and could likely bias the final impression, even after a two-day visit to the program.

2 Concern: A concern indicates that a program currently satisfies a criterion, policy, or procedure; however, the potential exists for the situation to change such that the criterion, policy, or procedure may not be satisfied. Weakness: A weakness indicates that a program lacks the strength of compliance with a criterion, policy, or procedure to ensure that the quality of the program will not be compromised. Therefore, remedial action is required to strengthen compliance with the criterion, policy, or procedure prior to the next evaluation. Deficiency: A deficiency indicates that a criterion, policy, or procedure is not satisfied. Therefore, the program is not in compliance with the criterion, policy, or procedure. Figure 1: ABET Definitions of the Three Shortcomings. ABET annually publishes a document called Criteria for Accrediting Engineering Programs, which can be downloaded from the ABET website: This document lists the wordings for all nine criteria, and also includes a few definitions where appropriate. Meeting the nine criteria is what is expected for NGR accreditation. However, interpretation of the criteria is somewhat open to the reader. ABET also publishes a template for the self-study, which can also be downloaded from the ABET website. It should be noted that all the ABET documents are updated for the next review cycle, so the current set of documents are labeled for the review cycle. Accessing and reading these two documents is the first place to start when preparing the self-study. Writing the Self-Study The self-study template provided by ABET is organized in chapters that correspond to the nine ABET criteria. So it is advisable to write the self-study using the template without any departure from it. Background Information The first chapter of the self-study contains some background information (see Table 1). One must list contact information, including names, telephone numbers, and addresses of the program head and any other person that is part of the ABET preparation team. State the history of the program and any recent major changes to the program, particularly within the last six years. List and describe any options, tracks, or concentrations included in the program. Describe the educational delivery modes, such as day, night, or weekend Table 1: Background Information Contact Information Program History Options Program Delivery Modes Program Locations Public Disclosure Deficiencies, Weaknesses or Concerns from Previous Evaluation(s) and the Actions Taken to Address Them classes, as well as any web-based delivery modes. Include the program locations, as well as dual degree and international classes. Provide information concerning all the places where the

3 Program Education Objectives (PEOs), Student Outcomes (SOs), annual student enrollment and graduation data, are posted or made accessible to the public. If this information is posted to the Web, please provide the URLs. Finally, you must summarize any Deficiencies, Weaknesses, or Concerns remaining from the most recent ABET Final Statement. Describe the actions taken to address them, including effective dates of actions, if applicable. Criterion 1: Students The first ABET criterion 1 focuses on students. As shown in Table 2 the PEV will be looking for evidence about faculty evaluation of student performance, advising, transfer credits, and checks for students fulfilling all graduation requirements. What are the admission standards? What is the faculty advising protocol in the program for both academic and career matters? Do all students meet the same graduation standard that is enforced for both regular and transfer students? One could also link the transcript evaluation requirement with this criterion 1. Also, the PEV will want to talk to some undergraduate students during the site visit, and some of these questions in Table 2 might be addressed during these on-site interviews. Criterion 2: Program Educational Objectives Table 2: PEV list for Criterion 1 1. STUDENTS Evaluate student performance Monitor student progress Advise students regarding curricular and career matters Have and enforce policies for accepting both new and transfer students Have and enforce policies for awarding academic credit for courses taken at other institutions Have and enforce policies for awarding academic credit for work in lieu of courses taken at the institution The second ABET criterion 2 focuses on the Program Educational Objectives (PEOs). ABET defines PEOs as broad statements that describe Table 3: PEV list for Criterion 2 what graduates are expected to attain within a few 2. PROGRAM years of graduation. As shown in Table 3, the PEV EDUCATIONAL will be focusing on whether the PEOs are OBJECTIVES published, for example in the university catalog. Consequently, it is important to write acceptable Published and consistent with PEOs that state what the graduates will actually do institution s mission, the needs of the constituencies, and these after graduation (see Table 4). They also should be criteria consistent with university, college and department mission statements, as shown in Table 5. Hence it Documented, systematically is imperative to review mission statements as part utilized and effective process, involving program constituencies, of criterion 2 and to include them in this part of the for the periodic review of the self-study. The PEV will also be looking for program educational objectives. evidence that the PEOs have periodic review that involves both faculty and constituents. So some form of alumni input is needed for criterion 2. Most programs have an External Advisory Committee (EAC), and this group is a good source for input concerning the PEOs.

4 Institution Core Values Table 4: Writing Program Educational Objective (PEO) Statements Poor Graduates are prepared to work in the engineering fields of manufacturing and design Graduates have the educational background to go to graduate school and do research Graduates have leadership and teamwork skills Graduates are aware of ethics and professional responsibility in the workplace Table 5: Mapping PEOs to Institution Mission PEO 1 (Practice Engineering) PEO 2 (Advanced Education and Research) 1. Learning 2. Discovery PEO 3 (Leadership, Communication, and Teamwork) 3. Freedom 4. Leadership PEO 4 (Professional and Ethical) 5. Individual Opportunity 6. Responsibility Criterion 3: Student Outcomes Table 6: PEV list for Criterion 3 Good Graduates practice engineering in the fields of manufacturing and design in industry Graduates pursue advanced education, research, and development in science and engineering. Graduates participate as leaders on team projects Graduates conduct themselves in a professional and ethical manner in the workplace The third ABET criterion 3 focuses on Student Outcomes (SOs). ABET defines SOs as narrow statements that describe what students are expected to know and be able to do by the time of graduation. These relate to the skills, knowledge, and behaviors that students acquire in their matriculation through the program. Table 6 shows the PEV checklist for criterion 3. As can be seen, the main objective is for the PEV to determine if the students have achieved the eleven ABET outcomes, namely a-k. How this is demonstrated is up to the program. But, some factual assessment data and samples of student work demonstrating achievement of these outcomes will be needed. It has been suggested that a minimum of three measures be included in the assessment of the SOs, and there should be both direct and indirect measures, as suggested in Table 7. Programs are free to define their own SOs, but they must be mapped to the ABET a-k. In addition, there should be a clear relationship shown between the program SOs and the program PEOs, so some mapping of SOs to PEOs should be included in this section, as shown in Table 8. The PEV will be paying close attention to the assessment and evaluation procedures used to document that the SOs are being achieved. So samples of the evaluated student work should be retained and made available during the site visit. Assessment of the Student Outcomes is probably the most critical aspect of the entire ABET review process, so strong attention must be paid to this phase of the self-study.

5 3. STUDENT OUTCOMES Program has documented student outcomes that prepare graduates to attain the program educational objectives (a) ability to apply knowledge of math, engineering, and science (b) ability to design and conduct experiments, as well as to analyze and interpret data (c) ability to design system, component or process to meet needs within realistic constraints (d) ability to function on multi-disciplinary teams (e) ability to identify, formulate, and solve engineering problems (f) understanding of professional and ethical responsibility (g) ability to communicate effectively (h) broad education (i) recognition of need by an ability to engage in life-long learning (j) knowledge of contemporary issues (k) ability to use techniques, skills, and tools in engineering practice Additional outcomes articulated by the program Table 7: Examples of Direct and Indirect Measures for ABET Assessment Method Direct Indirect Oral Exit Interviews of Graduating Seniors Embedded Test Questions Pertaining to Specific Outcomes Faculty Evaluation of Student Portfolios Pertaining to Outcomes Senior Exit Outcomes Surveys Student Outcomes Focus Groups Classroom Observations of Student Performance by Faculty Evaluation of Student Course Work by Faculty Committee Compiled Results from FE Exam Employer Surveys of Student Performance During Co-op or Internship Cycles Starting Salary, FE Exam Rates, Graduate School Attendance, and Other Senior Exit Data External Advisory Committee (EAC) Student Outcomes Surveys Criterion 4: Continuous Improvement The fourth ABET criterion 4 pertains to continuous improvement. As shown in Table 9, programs are expected to gather assessment data pertaining to the SOs, to evaluate that data, and then to make changes in the program based on that evaluation. Thus, the PEV will want to know what improvements have been made to the curriculum and supporting resources, based on this ABET model. Feedback from students, alumni, and faculty will be useful in satisfying criterion 4. Highlighting new courses, laboratories, and other facilities will help demonstrate that criterion 4 is being met. It is important to show that some improvement is being regularly made, no matter how good the faculty find their current program to be.

6 Table 8: Mapping of Student Outcomes to Program Objectives Student Outcomes (a) an ability to apply knowledge of mathematics, science, and engineering (b) an ability to design and conduct experiments, as well as to analyze and interpret data (c) an ability to design a system, component, or process to meet desired needs (d) an ability to function on multidisciplinary teams (e) an ability to identify, formulate, and solve engineering problems (f) an understanding of professional and ethical responsibility (g) an ability to communicate effectively (h) the broad education necessary to understand the impact of engineering solutions in a global and societal context (i) a recognition of the need for, and an ability to engage in life-long learning (j) a knowledge of contemporary issues (k) an ability to use the techniques, skills, and modern engineering tools necessary for engineering practice. PEO 1: Practice Engineering PEO 2: Advanced Education and Research PEO 3: Leadership, Communication, and Teamwork PEO 4: Professional and Ethical 4. CONTINUOUS IMPROVEMENT Table 9: PEV list for Criterion 4 Regular use of appropriate, documented processes for assessing and evaluating the extent to which the student outcomes are being attained Results of evaluations systematically utilized as input for the continuous improvement of the program Other information, if available, used to assist in continuous improvement.

7 Criterion 5: Curriculum ABET criterion 5 is devoted to the program curriculum. Table 10 shows the PEV checklist for this criterion. The PEV needs to determine whether there is one year of mathematics and science in the curriculum. This usually means about credit hours devoted to math and science courses that are typically taught outside of engineering. The PEV also needs to determine if there is one and one-half years of engineering topics (45-48 credit hours) in the curriculum. This later requirement is usually taught inside the program by engineering faculty. In some cases, a program can argue for some math and science content inside of the engineering domain, but only a few of the required credits will be accepted. In addition, the PEV must assess the capstone design course(s) in the program and how well they incorporate student experiences in earlier courses. It is also important to show that the capstone design experience uses appropriate engineering standards. And of course, the entire required curriculum should be adequately mapped to the program SOs in some manner. Criterion 6: Faculty The sixth ABET criterion 6 deals with the program faculty, as shown in Table 11. The PEV needs to determine if there is sufficient number of faculty and if the faculty members possess the competencies needed to cover all curricular areas in the program. The PEV needs to assess faculty interaction with students in the areas of advising and career counseling. The faculty accomplishments need to be presented in the self-study. A professional development plan must be outlined in the self-study, including conference attendance and other faculty enrichment opportunities. The level of faculty interaction with industrial practitioners and employers should be included. If the program is large, the organization of faculty into smaller domain groups and the authority structure in the program must be outlined in the self-study. Table 10: PEV list for Criterion 5 5. CURRICULUM Devotes adequate attention and time to each component, consistent with the outcomes and objectives of the program and institution One year of college-level mathematics and basic (biological, chemical, and physical; some with experimental experience) sciences One and one-half years of engineering topics appropriate to the field of study (See criterion statement) General education component that complements the technical content and consistent with program and institutional objectives Culminates in a major design experience based on knowledge and skills acquired in earlier course work and incorporates appropriate engineering standards and multiple realistic constraints Table 11: PEV list for Criterion 6 6. FACULTY Sufficient number and competencies to cover all curricular areas Adequate levels of student-faculty interaction Adequate levels of student advising and counseling Adequate levels of university service activities Adequate levels of professional development Adequate levels of interaction with practitioners and employers Appropriate qualifications Sufficient authority for program guidance, evaluation, assessment, and improvement Overall competence of faculty

8 Criterion 7: Facilities ABET criterion 7 pertains to the program facilities. The PEV checklist is shown in Table 12. The PEV will want to tour the classrooms, laboratories, offices, and computing facilities. The PEV may want to review the program information technology (IT) support. The PEV may also want to view a few laboratory equipment set-ups, and will also be looking at the safety regulations that are in place within the program facilities. As many U.S. engineering programs are aging, safety in the laboratory has become a important focus on the ABET visits. Library services and supporting infrastructure are also a concern here. Criterion 8: Support ABET criterion 8 relates to the program financial resources for faculty, staff, and facilities. The PEV checklist for this criterion is shown in Table 13. The PEV will ask are there sufficient resources to support the teaching laboratories and to replace aging lab equipment? What level of support does the program receive from the college and the home institution? Criterion 9: Program Criteria In addition to the ABET criteria 1 to 8, there is a criterion 9 that pertains to the professional society criteria that represents the program s discipline. For example, a mechanical engineering program would correspond to the ASME criteria. Are there specific curricular needs for the degree? Are there any specific faculty qualifications needed? These society criteria are included in the ABET document Criteria for Accrediting Engineering Programs. There are approximately 28 member engineering societies that offer program criteria. Table 14 shows the brief PEV list. Some ABET member societies (for example ASEE) do not have criterion 9 needs, and the PEV in this case would ignore this criterion. Table 12: PEV list for Criterion 7 7. FACILITIES Adequate to support attainment of student outcomes and provide an atmosphere conducive to learning: classrooms, offices, laboratories, associated equipment Modern tools, equipment, computing resources and laboratories are available, accessible, and systematically maintained and upgraded Students provided appropriate guidance regarding the use of the tools, equipment, computing resources, and laboratories Adequate library services, computing infrastructure, and information infrastructure Table 13: PEV list for Criterion 8 8. SUPPORT Institutional support and leadership sufficient to assure quality and continuity of the program Institutional services, financial support, and staff adequate to meet program needs Sufficient to attract and retain a well-qualified faculty and provide for their professional development Sufficient to acquire, maintain, and operate infrastructure, facilities, and equipment Sufficient to provide an environment to attain student outcomes Table 14: PEV list for Criterion 9 9. PROGRAM CRITERIA Curricular topics (if any) Faculty qualifications (if any) Other (if any)

9 Accreditation Policy and Procedure Manual (APPM) The PEV must also make sure that the program adheres to certain ABET policies and procedures that are published in their manual (Table 15). The institution must release accreditation status of their programs and have that information readily available to the public. The engineering program names must be consistent with ABET requirements. And of course, safety is of paramount importance. Timeline ABET accreditation is a six-year cycle, and program faculty need to be cognizant about the processes needed for a successful ABET visit from one year to the next during this cycle. However, most programs will tend to gear up for ABET as the visit nears, rather than maintain constant vigilance. Experience suggests that a period of three years before the visit should be sufficient to have adequate time to prepare for the ABET visit. Table 16 shows a detailed timeline for a three-year preparation plan for ABET. This plan assumes that the program has defined the PEOs and SOs, and that the program s previous ABET visit had resulted in an NGR. Activity Table 16: A Three-Year Timetable for ABET Preparation. Frequency of Review Review PEO's Every Three Years Review SO's Every Three Years Interact With Program EAC Every Three Years S Student Outcomes Survey Each ester I Senior Exit Oral Interviews Each ester T Direct Measures of Student Work Each ester E Review of Student Portfolios Annually Results of FE Exam Annually V Senior Exit/Achievement Data Annually I Alumni Survey Biennially S Review Past Site Visit Report Once Per Visit Cycle I Prepare Course Notebooks Once Per Visit Cycle T Prepare Outcome Notebooks Once Per Visit Cycle Write Self-Study Once Per Visit Cycle Mock Site Visit Once Per Visit Cycle 1* 2 * ester 1 is the Fall semester 3-years before the ABET on-site visit. Table 15: PEV list for APPM ACCREDITATION POLICY AND PROCEDURE MANUAL II.A. Public release of information by the institution or program II.E.4 Program names must meet ABET requirements II.G.6.b.(1) Facilities adequate and safe for the intended purpose

10 The first task is for the program to review the PEOs and SOs, and together with input from appropriate constituents, the PEOs and SOs are updated as needed, while at the same time are appropriately mapped to the published ABET a-k. The various assessment processes are then initiated, using either a semester or annual time frame. These would include both the direct and indirect measures discussed earlier in Table 7, as well as student and alumni surveys. During evaluation of student work, both the evaluation results and the work itself should be saved for inclusion in the course and outcomes notebooks. About months before the site visit, the program faculty will start to get involved in earnest. The self-study needs to be started and the course notebooks need to be compiled. Some programs might also try to have a mock site visit the Fall semester before the ABET visit. Someone from outside the program who is an ABET PEV, could be asked to come in and evaluate the program s preparedness, using the current ABET evaluation forms. Conclusions This paper has overviewed ABET accreditation preparation by looking at what the PEV needs to do during the visit. The key emphasis here to study the PEV checklist form E062, and then write the self-study to easily accommodate that checklist. By looking at the desired results coming from the ABET on-site visit, including the pre-reading of the self-study, the program faculty can then trace backwards in time the processes and activities needed to be prepared for a successful ABET review. Using the suggested processes espoused in this paper, however, will certainly not guarantee a successful visit. The result of the ABET visit is still a function of the PEV assigned to the program and his/her own personal interpretation of the ABET criteria. Bibliography Abel, K.: Preparing an ABET Self-Study: Continuous Improvement the Second Time Around, Proceedings of the 2009 ASEE Annual Conference, Austin, TX, June Barr, R.: Student Outcomes Assessment: Direct and Indirect Measures, Proceedings of the 2007 ASEE Gulf-Southwest Section Annual Meeting, South Padre Island, TX, March Barr, R.: Reverse Engineering the ABET Process, Proceedings of the 2011 ASEE Gulf-Southwest Section Annual Meeting, Houston, Texas, March Chambers T. and Simon W.: Closing the Loop: Demonstrating Positive Program Changes as Part of the Continuous Improvement Process, Proceedings of the 2007 ASEE Gulf-Southwest Section Annual Meeting, South Padre Island, TX, March Engineering Accreditation Commission, Criteria for Accrediting Engineering Programs, Accreditation Board for Engineering and Technology (ABET), Baltimore, Maryland, latest version. Petersen, O., Williams, S., and Durant, E.: Understanding ABET Objectives and Outcomes, Proceedings of the 2007 ASEE Annual Conference, Honolulu, HI, June Shryock, K. and Reed, H.: ABET Accreditation: Best Practices for Assessment, Proceedings of the 2008 ASEE Gulf-Southwest Annual Conference, Albuquerque, NM, March Skvarenina, T.: Incorporating and Assessing ABET Soft Skills in the Technical Curriculum, Proceedings of the 2008 ASEE Annual Conference, Pittsburgh, PA, June 2008.

11 Biography Dr. Ronald E. Barr is Professor of Mechanical Engineering at the University of Texas at Austin, where he has taught since He previously taught at Texas A and M University. He received both his B.S. and Ph.D. degrees from Marquette University in 1969 and 1975, respectively. His teaching and research interests are in Biosignal Analysis, Biomechanics, Computer Graphics Modeling, and Engineering Education Scholarship. Barr is a recipient of the ASEE Chester F. Carlson Award, the Orthogonal Medal, and the EDGD Distinguished Service Award. Barr is a Fellow of ASME and ASEE, and served as ASEE President from He is a registered Professional Engineer (PE) in the state of Texas and is an ABET program evaluator (PEV) representing the member society ASEE.