University of Nevada, Reno, Mechanical Engineering Department ABET Program Outcome and Assessment

Transcription

1 2005 ABET Program Outcome and Assessment 3. Program Outcomes and Assessment Program Outcomes We define our program educational outcomes as statements that describe what students are expected to be able to do by the time of graduation from the program. The program educational outcomes directly support our program objectives and are consistent with (and largely are identical to) the ABET a-k outcomes. Our program educational outcomes are to develop: an ability to apply the principles of mathematics, science, and engineering an ability to design and conduct experiments, as well as to analyze and interpret data, an ability to identify and document desired needs and to design a system component, or process to meet desired needs within realistic constraints such as economic, environmental, social, political, ethical, health and safety, manufacturability, and sustainability an ability to function on multi-disciplinary teams an ability to identify, formulate, and solve engineering problems, an ability to explain professional and ethical responsibility and identify professional and ethical issues an ability to communicate effectively the broad education necessary to understand the impact of engineering solutions in a global, economic, environmental, and social context an ability to explain the need for, and an ability to, engage in life-long learning an ability to identify and analyze contemporary issues an ability to use the techniques, skills, and modern engineering tools necessary for engineering practice. Relation between Program Objectives (criterion 2) and Program Outcomes (criterion 3) Table B3.1 shows the correlation between the 11 ME program outcomes and 4 ME program objectives. The numbers indicate how much each outcome influences each particular objective. The weighting factors range from 0 (no influence) to 3 (high influence) and were arrived at by the ABET committee, ME faculty members, and the ME Advisory Board. The weighting factors are important in helping to identify which outcomes need to be addressed when a deficiency in an objective is detected. B3-1

2 Table B3.1: Correlation between ME program objectives and outcomes Outcome Objective Develop an ability to apply the principles of mathematics, science, and engineering Develop an ability to design and conduct experiments, as well as to analyze and interpret data Develop an ability to identify and document desired needs and to design a system component, or process to meet desired needs within realistic constraints Develop an ability to function on multidisciplinary teams Develop an ability to identify, formulate, and solve engineering problems, Develop an ability to explain professional and ethical responsibility and identify professional and ethical issues Develop an ability to communicate effectively Develop the broad education necessary to understand the impact of engineering solutions in a global, economic, environmental, and social context Develop an ability to explain the need for, and an ability to, engage in life-long learning Develop an ability to identify and analyze contemporary issues Develop an ability to use the techniques, skills, and modern engineering tools Proficient in performing entry-level mechanical engineering analysis and design Able to define, design, execute, and analyze experiments with minimal supervision Able to communicate effectively, verbally and in writing Successful in pursuit of graduate studies at UNR and at other institutions Note: (0=none, 1=low, 2=medium, 3=high) Mapping of Individual Courses to ME Program Outcomes and Objectives A course assessment form was developed to determine how individual courses mapped to both the eleven ME program outcomes (i.e., the ABET a-k criteria) and the four ME program objectives. The survey required faculty to rank how well each course addressed the ME program outcomes and objectives. The possible rankings included high (3), some (2), low (1) and not applicable (0). Table 3.2 summarizes the results by presenting the score for each outcome and objective. The entire listing of course assessment forms are presented in Appendix I-B. The only deficiencies identified based on Table B3.2 were: (1) not many courses include teamwork, (2) on average, professional and ethical responsibility is not covered well, (3) on average, the need for life-long learning is not covered well and (4) on average, the ability to identify contemporary issues are not covered well. At first glance, it appears that the ability to function on multi-disciplinary teams is not covered well (average 0.7). However, the low average is primarily due to the fact that not many courses require teamwork. For those courses which include teamwork, the average is quite satisfactory (2.1). B3-2

3 Table B3.2: Mapping between ME required courses and ME Program Outcomes and Objectives Average Response Average Response Outcome/Objective (0=N/A, 3=high) not including N/A responses (1=low, 3=high) Develop an ability to apply the principles of mathematics, science, and engineering Develop an ability to design and conduct experiments, as well as to analyze and interpret data Develop an ability to identify and document desired needs and to design a system component, or process to meet desired needs within realistic constraints Develop an ability to function on multi-disciplinary teams Develop an ability to identify, formulate, and solve engineering problems, Develop an ability to explain professional and ethical responsibility and identify professional and ethical issues Develop an ability to communicate effectively Develop the broad education necessary to understand the impact of engineering solutions in a global, economic, environmental, and social context Develop an ability to explain the need for, and an ability to, engage in life-long learning Develop an ability to identify and analyze contemporary issues Develop an ability to use the techniques, skills, and modern engineering tools Proficient in performing entry-level mechanical engineering analysis and design Able to define, design, execute, and analyze experiments with minimal supervision Able to communicate effectively, verbally and in writing Successful in pursuit of graduate studies at UNR and at other institutions B3-3

4 Assessment of Program Outcomes: Measurement Tools The mechanical engineering program utilizes a variety of assessment tools: 1. Standardized national tests (i.e. Fundamentals of Engineering (FE) Exam) All ME students are required to take the FE exam. While passing the exam is not required, there is anecdotal evidence that almost all students try to pass. Tables B3.4a and B3.4bindicates that our students are near the national pass rate, which supports this notion. Thus, the FE exam is an excellent tool for assessing our students mastery of engineering fundamentals. The ME faculty examine the results of the FE Exam and compare the performance our graduates to that of the national average. While year-toyear performance varies widely, consistent trends (e.g. higher or lower than national average) are considered clear indications of program strengths and weaknesses. Tables B3.4a and B3.4b summarize the AM and PM subject results, averaged over the test taken between April 1997 through April While not shown, the faculty further investigated the year-to-year results to identify weaknesses and strengths. We concluded that, relative to the national average, our students consistently (1) excel in controls, (2) under perform in strength of materials, and (3) under perform in engineering economics. 2. Written surveys and questionnaires a. College of Engineering course evaluations (completed by students) All students within the College of Engineering complete a course evaluation for every class at the end of each semester. The survey consists of 20 questions plus 2 sections for comments (see Appendix 1, paragraph (d)). As the survey asks students to evaluate how well the course helped them improved abilities in each of the 11 ABET a-k criteria, these questions map directly to the ME program outcomes. The results of the student course evaluations are summarized in Table 3.5. Of the 20 questions, we have limited our analysis to the 11 questions concerning the ME program outcomes and the overall rating of the instructor. The scores indicate a general satisfaction with the instructors and a general consensus that we are meeting the program outcomes. b. Course questionnaires (developed and implemented by faculty for specific courses) Many of the ME faculty implement their own course-specific evaluations during and/or at the end of each semester in order to gauge student progress and/or attitude. These surveys are often used by individual faculty to make mid-semester adjustments. c. Alumni feedback (questionnaire) An Alumni feedback questionnaire was developed and given to the graduating classes of 2002, 2003 and The survey was conducted in the Fall 2004 and again in the Spring The survey questions and the responses to these B3-4

5 surveys are presented in Appendix 1, paragraph (d). A total of 21 alumni responded. Overall the alumni responses to the survey were positive, however, more students than not felt that they were not ready to start their own business and more students than not felt that the CS201 (now CS135) course did not adequately provide them with programming skills. Of the favorable responses, the most favorable indicated that students felt well prepared to communicate verbally and orally, and felt well prepared to compete in the work place or graduate school with graduates from other institutions. Students also ranked communication the highest in their choice of excellent ME program objectives and outcomes. Of the favorable responses, students ranked the Design Capstone experience, laboratory and independent projects, and familiarity of statistics and linear algebra, on the lower side of the scale. Finally, another response on the lower side of the favorable scale indicated that the ME program may not have done as excellent a job as it could have done to prepare the students to do as well or better than their peers in the workplace. The difference between this response and the early observation that the students felt they could compete well could indicate that they also attribute other factors to their ability to compete. d. Employer feedback (questionnaire) An employer questionnaire was developed and given to employers identified by the ME faculty. The survey was conducted in the Fall 2004 and again in the Spring The survey questions and the responses to these surveys are presented in Appendix 1, paragraph (d). A total of 11 employers responded. Overall the employer responses to the survey were positive. Of the favorable responses, the most favorable indicated that UNR graduates were well prepared to express themselves orally and to use computers or other technology in the workplace. Regarding the ME education experience and program objectives the most favorable response indicated that the ME education experience does an excellent job of preparing individuals to be entry level engineers, and that proficiency in performing entry-level mechanical engineering analysis and design, and effective oral and written communication were excellent program objectives. Of the favorable responses, employers ranked success in pursuit of graduate studies, and broad education to understand the impact of engineering solutions in a global, economic, environmental, and social context on the lower side of the scale in terms of excellent program objectives and outcomes. They also ranked UNR ME graduates on the lower side of the favorable scale in the ability to develop solutions to work-related problems and think critically (including analyzing and evaluating information), and ranked the UNR ME program on the lower side of the favorable scale in its ability to prepare leaders in the work place. In regard to the questions about preferred employee abilities, oral and written communication and the ability to solve work related problems were at the higher side of the favorable scale while interest in continued education and use of quantitative methods were on the lower side of the favorable scale. B3-5

6 e. Course assessment forms (completed by faculty) As mentioned previously under mapping of individual courses to ME program outcomes and objectives, a course assessment form was developed to determine how individual courses mapped to both the eleven ME program outcomes (i.e., the ABET a-k objectives) and the four ME program objectives. Opportunities for improvement identified were (1) not many courses include teamwork, (2) on average, professional and ethical responsibility is not covered well, (3) on average, the need for life-long learning is not covered well and (4) on average, the ability to identify contemporary issues are not covered well. 3. Performance evaluations a. Student design projects (e.g., freshmen & capstone projects) Design projects are used by the faculty to assess a wide range of student abilities, including the ability to work on teams. b. Student labs (restricted elective labs) Laboratory experiences and reports are also used by the faculty to determine how well students are meeting the program outcomes. c. Student awards, activities, and competitions 4. Archival records Local, regional and national awards, activities and competitions are also excellent measures of student achievement because they are externally reviewed. Our student chapter of ASME is very active and competitive (e.g. UNR teams almost always place in the top 10 of the ASME Human Powered (HPV) competitions. a. Graduate school applications and admissions The percentage of graduates who go on to pursue graduate studies (in any discipline) is also used by the faculty as a direct measure how well prepared our students are. Data collected for recent graduates is shown in Table B3.3 below: Table B3.3 Graduate Admissions History Academic year Graduating seniors Number electing further professional study: (UNR + non-unr) Percentage of total 20% 36% 32% 28% Over this time period, approximately 30% of our graduating seniors chose to B3-6

7 continue professional study in law, medicine, science and engineering. Of this total, nearly half choose to remain at UNR. Course grades Course grades are a natural measure of student performance. 5. Job offers and acceptances Anecdotally, a typical graduating senior receives three or four job offers ranging between $40-50K. Student interviews a. Focus groups As part of a grant from the Hewlett Foundation, focus groups and attitude surveys were conducted in ME 150 in the Fall Also in the fall 2004 we began implementing focus group interviews for all seniors in our capstone design course (ME465). The goal of the focus group was to determine the perceived strengths and weaknesses of the ME program from the students perspective. A total of 26 students participated. Table B3.6 summarizes the results from a confidential survey given during the focus groups and Table B3.7 summarizes the results of the group discussion (based on the transcripts from the discussion). According to Table 3.6, students generally feel that we are doing a good job at preparing them for an engineering career. However, the data indicates that the students would like more contact with working professionals, coverage of ethics, better facilities and less lecturing. According to Table 3.7 students felt that teamwork and problem solving were strengths and that teaching and lack of design experiences were the major weaknesses. b. Exit interviews All seniors are required to complete an exit survey and are also invited to have lunch with the ME Department Chair. The results from the exit survey do not contradict any of the other assessment data. The students think that their communication skills were the strongest part of their education while the laboratory and independent design projects were the weakest. However, their opinions are much positive (see Q2,Q3, and Q4 in particular) than expressed BY THE SAME COHORT during a senior-level focus group. One faculty member is concerned about the validity of focus group processes. He believes that focus groups tend to suppress individual views by attempting to reach a consensus. He believes that surveys carried out anonymously are less intimidating; and like a secret ballot, may be more indicative of what individuals truly believe. 6. ME Advisory Board Feedback As indicated in Section A, several times per year the ME department and the ME Advisory Board meet to discuss the general status and direction of the department. Since the Advisory Board consists entirely of local employers, these B3-7

8 meetings provide the department with excellent feedback concerning the quality of our graduates and the perceived needs of industry. The Advisory Board also reviewed our program outcomes and objectives and provided feedback on how well we were meeting them. 7. ME faculty discussions and feedback The ME faculty regularly meets to discuss curricular issues. These discussions are a critical part of the assessment process. The ME faculty were polled to determine how the assessment tools described above correlated to the ME program educational outcomes. The ranking used ranged from 0 (no correlation at all) to 3 (very highly correlated). Table 3.9 shows the average ranking from the ME faculty. Also included in Table 3.9 are the Row Averages and Column Averages. The Row Averages can be interpreted as an indication of how well each outcome is assessed. The Column Averages can be interpreted as an indication of the utility of each assessment tool (in terms of assessing the ME Dept outcomes). B3-8

9 Table B3.4a: Fundamentals of Engineering Subject Results, Apr 1997 Apr 2004 AM Subject UNR ME % correct Chemistry Computers Dynamics Electrical Circuits Engineering Economics Ethics Fluid Mechanics Mat Sci/Str Mat Math Mechanics of Materials Statics Thermodynamics PM Subject Controls Computers Mechanical Design Dynamic Systems Energy Conversion Fluid Mechanics Fans, Pumps & Compressors Heat Transfer Measurements & Instrumentation Material Behavior & Processing Refrigeration & HVAC Stress Analysis Thermodynamics National % correct Table B3.4b: Recent Fundamental of Engineering Pass Rate Apr 01 Oct 01 Apr 02 Oct 02 Apr 03 Oct 03 Apr 04 ME students taking exam ME students passing exam 73% 83% 100% 75% 80% 85% 82% National Pass Rate 78% 87% 90% 87% 82% 85% 82% B3-9

10 Table B3.5: Student Evaluations for ME Department Required Courses for Spring and Fall 2004 Course Number Overall rating of instructor Develop an ability to apply the principles of mathematics, science, and engineering Develop an ability to design and conduct experiments, as well as to analyze and interpret data Develop an ability to identify and document desired needs and to design a system component, or process to meet desired needs within realistic constraints Develop an ability to function on multi-disciplinary teams Develop an ability to identify, formulate, and solve engineering problems, Develop an ability to explain professional and ethical responsibility and identify professional and ethical issues Develop an ability to communicate effectively Develop the broad education necessary to understand the impact of engineering solutions in a global, economic, environmental, and social context Develop an ability to explain the need for, and an ability to, engage in life-long learning Develop an ability to identify and analyze contemporary issues Develop an ability to use the techniques, skills, and modern engineering tools ME ME ME ME ME ME ME ME ME ME ME ME ME ME Column Average Note: (1=excellent, 3=Neutral, 5=poor). B3-10

11 Table B3.6: Summary of Results from Fall 2004 Focus Groups. Do you enjoy the study of engineering? Does the College of Engineering provide sufficient opportunities (e.g., coursework, extracurricular activities) to enhance your future professional development? Does your preparatory coursework in mathematics and science support your present engineering studies? Are the College of Engineering facilities adequate to support your practical coursework? Has your coursework acquainted you with the social and ethical responsibilities of professional engineers? Have your engineering skills been enhanced or complemented by the UNR core curriculum? Has the engineering curriculum enhanced or improved your ability to solve problems? Has the engineering curriculum enhanced improved your ability to work as a member of a team? Are you satisfied with the lecture method of instruction in your engineering courses? Do faculty regularly use other teaching methods such as team projects, projects-based learning, or small group discussions? Please rate the difficulty of the engineering curriculum at UNR (1 = Very easy; 5 = Very difficult) Please rate the usefulness of the UNR engineering curriculum in meeting your professional goals. (0 = N/A; 1 = Not valuable; 5 = Very valuable) Yes No No Response : 0 2: 1 3: 9 4: 12 5: 4 0: 0 1: 0 2: 5 3: 7 4: 11 5: 3 B3-11

12 Table B3.7: Focus Group Discussion Results (based on analysis of discussion transcripts) Thread (Strengths) Number of Occurrences Thread (Weaknesses) Number of Occurrences Communication skills 1 Funding 1 Hands on project 1 Heat transfer 2 Working in teams 4 Numerical methods 1 Problem solving 3 Computers 3 Design process 1 Class scheduling 4 Math 1 Advisement 2 John Leland 1 Poor teaching skills 6 Broad class range 1 Poor language skills TA 3 Great teachers 2 Lack of professor interest 2 Learned how to B.E. 1 No interaction between faculty and 1 students Instrumentation 1 Outdated technology 2 Interpreting languages 1 Insufficient to software 1 Pro-E 1 Need more design exper. 4 PowerPoint 1 Need instructions 2 Public speaking 1 Labs 2 Networking 1 Machine shop 1 Table B3.8: Senior Exit Survey Results for Fall 04-Spring 05 Question Average The overall education that you received in the UNR Core Curriculum 2.56 The overall education that you received in Mechanical Engineering 2.19 The faculty in Mechanical Engineering as teachers 2.78 Your knowledge of fundamentals in math/science/engineering 2.25 The design/capstone experience 2.28 The ME labs and independent projects 3.31 Your ability to communicate professionally (oral and written) 1.78 Ability to compete in the workplace/graduate school with grads from other institutions 2.19 Average 2.32 Note: (1=high, 3=neutral, 5=low) B3-12

13 Table B3.9: Mapping of ME program outcomes and assessment tools Assessment Tool Outcome FE exam results Student course evaluations Faculty feedback & discussion (e.g. replacement of MECH 491) Student awards, activities, competitions Alumni feedback Grad school apps, admissions Job offers, acceptances Focus groups/exit interviews Design projects & Labs (freshmen, capstone, restricted elective labs, etc) Employers feedback ME Advisory Board feedback Course grades Row Average Develop an ability to apply the principles of mathematics, science, and engineering Develop an ability to design and conduct experiments, as well as to analyze and interpret data Develop an ability to identify and document desired needs and to design a system component, or process to meet desired needs within realistic constraints Develop an ability to function on multidisciplinary teams Develop an ability to identify, formulate, and solve engineering problems, Develop an ability to explain professional and ethical responsibility and identify professional and ethical issues Develop an ability to communicate effectively Develop the broad education necessary to understand the impact of engineering solutions in a global, economic, environmental, and social context Develop an ability to explain the need for, and an ability to, engage in life-long learning Develop an ability to identify and analyze contemporary issues Develop an ability to use the techniques, skills, and modern engineering tools Column Average Note: (0=no correlation, 3=high correlation) B3-13

14 Assessment of Program Outcomes: Application of the Results The assessment data presented has already been used to implement several changes in the curriculum and is currently being used to plan a more ambitious restructuring of the design and computer programming curricula. 1) Based on the latest pedagogical research, faculty feedback, alumni feedback, exit surveys, and course evaluations, we eliminated ME 491 (2 credit laboratory course) and have created a suite of 1-credit laboratory courses that are associated with the restricted electives. This change essentially integrated the laboratory experience into the subject courses. 2) Based on the FE exam results and faculty feedback, engineering economics was added as a course component to ME ) Based on faculty discussions, alumni feedback, Advisory Board feedback and FE exam results, we are now investigating the replacement of CS201 (now CS135) with a an existing course ME 201 and renumber ME 402 to ME 202 such that the two course sequence focuses the fundamentals of structured programming (using LabView as the software package) and an in-depth knowledge of modern computational tools (MathCad with VISSIM, MatLab with SIMULINK, Excel). 4) Based on Advisory Board feedback, faculty feedback, alumni feedback, exit surveys, focus groups, FE exam results and course evaluations, we are in the process of completely overhauling the design curriculum (ME 150, ME 151, ME 351, ME 451 and ME 452). The discussions we are conducting include incorporating more depth in: a. Modern computation tools (e.g. solid modeling, FEA, fluid dynamics); b. Geometric dimensioning & tolerancing (GD&T); c. Kinematics of machinery (adding ME 343 Dynamics of Machinery as a required pre-requisite to ME 351 Mechanical Design) d. Design of experiments throughout the curriculum where appropriate (ME 150 Introduction to Engineering I, ME 151 Introduction to Engineering II, restricted elective laboratories, ME 311 Intrumentation, and ME 354 Manufacturing Processes); e. Principles of six sigma where appropriate (ME 150 Introduction to Engineering I, ME 151 Introduction to Engineering II, ME 351 Mechanical Design, ME 451 Mechanical System Design, and ME 452 Design Synthesis). 5) Based on student and faculty feedback, the facilities and laboratory equipment have been the target of a strong continuous improvement effort (See Section B6 Facilities and B7 Institutional Support for details). We ve: a. Consolidated undergraduate laboratories in PE 001, 002, and 113; b. Embarked on a major general housecleaning and disposal of obsolete or nonreparable equipment; c. Made an investment of approximately $105K since 2002 in laboratory equipment and experiment; and d. Changed from ProEngineer to Solid Works with COSMOS as the foundation of our modern computational tool set. B3-14

15 Assessment Materials Available for Review The faculty have assembled course packets for each course taught within the ME program for the academic year. In particular, samples representing what the faculty members consider to be excellent, good and fair work were compiled. These packets include: Student exams; Project reports; Capstone design reports; and Homework papers; collected during the course. We also have available for review: Senior exit surveys results; Alumni surveys results; Lunch with the Department Chair minutes; Employer surveys; Results and transcripts from the senior focus groups; and Minutes of the Program Educational Objective and Program Educational Outcomes review with the ME Advisory Board. Faculty minutes from the various curriculum review meetings. B3-15