Assessing Course Outcomes for a Freshman Engineering Computer Programming Course Robert Rabb, Monika Bubacz, Jason Howison, and Kevin Skenes The Citadel, rrabb@citadel.edu, mbubacz@citadel.edu, jhowison@citadel.edu, kskenes@citadel.edu Abstract - The Citadel School of Engineering has initiated a Bachelor s of Science in Mechanical Engineering program in the fall 2014. During the first semester various classes have been offered to freshmen, sophomores and juniors. The School of Engineering has two ABET accredited programs (Civil and Electrical) and will apply for accreditation of the new Mechanical program as soon as the first mechanical engineering students graduate, which is expected in May 2016. The new program courses have been prepared using the ABET engineering accreditation criteria and the new team of mechanical engineering faculty is working on collection, assessment and evaluation of the courses in order to provide a quality educational experience for students. The Mechanical Engineering program at The Citadel has employed a method to use graded event averages and standard deviations from student assignments, examinations, and projects into a multilevel assessment tool that provides a valuable measure of how well the students are achieving the course outcomes and, ultimately, the program outcomes. This paper will describe an evaluation tool used throughout the semester that allows assessment, analysis and continuous improvement that can be implemented before the end of semester. Each course outcome is evaluated by using embedded indicators which consist of selected graded events that are contained within the course. Each embedded indicator is described and the average grade, the standard deviation of the individual grades and the possible number of points for the graded event are recorded. This evaluation tool will also produce an archival record of all course assessment activities. As a spreadsheet it is easy to navigate, manage and adjust, and very transparent to evaluators as it shows all graded work and how those activities support the course outcomes. As a course planning tool, the matrix will show where there are shortfalls and strengths in the course content and will allow instructors to adjust requirements before the next term begins. This form of evaluation is being currently used by the ME faculty in, the new ME courses, and the outcomes of this new effort will be identified in May 2015. The course outcomes evaluation will provide a unified, consistent, efficient and transparent evaluation across all courses in the new program. Examples of course outcomes assessment results are presented. The paper also describes how the results of this assessment are used to modify course outcomes and improve course content within the program. Index Terms - Course, Embedded Indicator, Freshman Programming, Student Survey. THE NEW MECHANICAL ENGINEERING PROGRAM The new Mechanical Engineering Program of Study offers focused tracks in Power and Energy, Manufacturing, Aeronautical Systems, Materials (Composites), and Mechatronics. It is available to the cadet population as well as to the evening students transferring from partnering community and technical colleges (2+2 programs). The full-time evening Mechanical Engineering program mirrors the current full-time evening 2+2 programs in Civil and Electrical Engineering. The new program courses have been prepared using the ABET engineering accreditation criteria, and the new team of mechanical engineering faculty is working on collection, assessment and evaluation of the courses in order to provide a quality educational experience for students. The two-semester ME freshman sequence teaches basic engineering skills to prepare students for advanced courses, internships and careers in engineering. The first Freshman Engineering course MECH 101, Introduction to Mechanical Engineering, provides a broad overview of mechanical engineering disciplines. Many "undecided" ME freshmen use this courses to help them narrow down their interests and decide on a future focused track of study. Currently, MECH 102, Engineering Computer Applications, which broadly introduces the topics of technical graphics, computer-aided design, programming in MATLAB, engineering design and analysis, project management, ethics in engineering, teamwork and oral and written technical communication, is offered to the freshman and sophomore cadets and Thermo-Fluids, Computer Aided Design, and Engineering Materials are offered to the evening transfer juniors. A well-understood process of T1D-1
continuous data collection and course assessment and evaluation is crucial for the success of the new program. Also, early improvement and goal-oriented changes will keep the program effective in the long term. The new mechanical engineering courses are already thoroughly described and approved by the South Carolina Commission on Higher Education [1]. Each one has a list of course outcomes which are being used to evaluate the courses. Nordstrom and Petit discuss syllabus based assessment for ABET. As a course is developed and is taught, it is critical that each faculty member reviews and critiques the assessment instruments and assessment indicators used to evaluate the course. This ensures the validity of not only the course material, but the evaluation metric as well [2]. The course evaluation materials will be archived and used in the program evaluation process for future ABET accreditation. ENGINEERING COMPUTER APPLICATIONS A second semester freshman course, MECH 102 Engineering Computer Applications, was taught for the first time in the spring of 2015 by all of the MECH faculty, and the course outcomes were assessed at the end of the term. Approximately 70 students took the course as a prerequisite for a computer aided design course, both required courses in the mechanical engineering curriculum. The course is required of all Mechanical Engineering freshmen, and the description is detailed below: Foundations of computing to include software tools and engineering processes for mechanical engineers. Topics may include: structured programming (MATLAB), graphical drawings and 2D and 3D modeling of parts and assemblies, interface of 2D and 3D data with Computer Aided Manufacturing, simulation of rigid body motion, presentation software, and spreadsheets. Introduction to teaming and creativity. Since few, if none, of the freshman mechanical engineering students had experience with MATLAB, it was important to provide a well-structured way for them to learn and use the software. Frequent homework assignments were used to reinforce material from each chapter. Instructors shared homework assignments and collaborated weekly. The different instructors had different assignments, but the assignments covered the same topics and were approximately the same level of difficulty. By the end of the term, students were able to solve simple engineering and physics problems, systems of linear equations, interpolation problems, and curve fitting problems. Drawing and drafting in the course consisted of manual sketches of common engineering objects. The Course Outcomes listed in the syllabi are listed in Table I: Outcome TABLE I MECH 102 COURSE OUTCOMES 1 Write structured programs in MATLAB to solve engineering problems. 2 Create a variety of engineering plots and graphs using MATLAB. 3 Program simple numerical methods used for numerical integration. 4 Interpret an engineering drawing. 5 Create an engineering drawing that is consistent with professional engineering practice (including dimensioning and tolerancing). 6 Create sketches of typical engineered objects. The course is two credit hours with weekly meetings consisting of two hours of lecture and two hours of laboratory. The grading breakdown is in Table II. TABLE II COURSE GRADES Expected Performance Criteria % of Grade Reading Quizzes 5% Individual Homework 40% Exam 1 15% Exam 2 15% Laboratory Assignments * 10% Final Exam 15% Total 100% * Group Submission ASSESSMENT OF COURSE OUTCOMES The Mechanical Engineering faculty members developed, maintained, and promoted a supportive environment conducive to the recruitment, personal development, and retention of undergraduate ME students. A matrix was used to evaluate graded requirements against the MECH course outcomes. They realized that initial information about the MECH program would rely on surveys conducted at the end of the course and anecdotal information until a cohort of students graduated. Additionally, it would be difficult for students to provide feedback on a course and what they learned while still enrolled. The faculty wanted a way to objectively assess student performance and program outcomes and at the same time, improve their own course development. Without creating several new course evaluation models, they chose an abbreviated method based on a practice used by the United States Military Academy to assess their Mechanical Engineering Program and Engineering and Technology outcomes [3]. Each course outcome is evaluated by using embedded indicators which consist of selected graded events (homework, exams, and projects) that are contained within T1D-2
the course. Each embedded indicator is described and the average grade, the standard deviation of the individual grades and the possible number of points possible for the graded event are recorded. The instructor can enter data as each embedded indicator is completed or enter all the data at the end of the semester. Once all data is entered into the spreadsheet, it automatically computes the average and standard deviation for each course outcome. The resulting evaluation combines the strengths of objective evaluation (based on graded events and basic statistics) and subjective evaluation (based on faculty experience). Using the matrix to help course design and development, faculty should identify all embedded indicators and point values prior to starting the course. Chickering and Gamson argue that time on task and active learning leads to better understanding [4]. The matrix allows an instructor to quickly identify all graded tasks and activities in the course. Faculty can recognize if course outcomes are fairly balanced with requirements and assigned values. Over time, they can see deficiencies and trends. Although a faculty member can plan course material and time on tasks, it is the objective assessment of the matrix that shows what is learned versus what was covered. Students are asked in an end of course survey to evaluate their accomplishment of each course outcome. Knott and Matusovich noted that during or at the completion of the course, students may be occupied in the immediacy of the whole experience. For example, students may be distracted in the emotional experience of a challenging group member or stresses trying to learn content for a final exam to truly appreciate or understand what they have learned [5]. The matrix captures data immediately after the event. Combined with subjective survey data gathered at the end of the course, the two evaluation instruments provide a clearer picture to the instructor and program director. APPROACH First, student performance in the form of graded events (or embedded indicators) is combined with a subjective faculty rating to produce an assessment of each outcome in a course. Part of the course assessment matrix is shown in Figure 1. The course director or instructor enters embedded indicator data into the blue highlighted area after the graded event or at the end of the semester. Only two of the six course outcomes are shown in this figure. If an embedded indicator does not support the course outcome, no information is entered and any computed data under the outcome is deleted and highlighted in gray. The statistical data for each graded event is available from the instructors gradebook. At the end of the term, all the entered data allows the overall average and standard deviation to be computed for each course outcome. Embedded Indicators Indicator Description Avg Pts Stnd Dev Pts Pts Stnd Dev Stnd Dev Pts Pts Stnd Dev Stnd Dev Pts Earned % Possible Earned % Pts Possible Earned % Pts Possible HW 1 Math Functions 35.70 2.30% 40 35.70 2.30% 0.92 40 HW 2 Arrays 37.00 3.20% 40 37.00 3.20% 1.28 40 HW 3 Operations 37.30 2.60% 40 37.30 2.60% 1.04 40 HW 4 Scripts 37.50 1.70% 40 37.50 1.70% 0.68 40 Exam 1 83.50 8.50% 100 83.50 8.50% 8.50 100 Proj 1 Operations 43.20 5.70% 50 43.20 5.70% 2.85 50 43.20 5.70% 2.85 50 HW 5 Plots 34.30 4.10% 40 34.30 4.10% 1.64 40 34.30 4.10% 1.64 40 HW 6 Programming 36.00 2.80% 40 36.00 2.80% 1.12 40 36.00 2.80% 1.12 40 HW 7 User Functions 36.50 4.80% 40 36.50 4.80% 1.92 40 36.50 4.80% 1.92 40 HW 8 Polynomials 36.80 2.40% 40 36.80 2.40% 0.96 40 36.80 2.40% 0.96 40 Exam 2 87.16 4.30% 100 87.16 4.30% 4.30 100 87.16 4.30% 4.30 100 Proj 2 Programming 47.40 1.10% 50 47.40 1.10% 0.55 50 47.40 1.10% 0.55 50 HW 9 Symbolic Math 36.70 2.70% 40 36.70 2.70% 1.08 40 36.70 2.70% 1.08 40 HW 10 Eng Drawing 35.90 3.40% 40 Exam 3 133.80 6.80% 150 133.80 6.80% 10.20 150 133.80 6.80% 10.20 150 Totals 722.86 37.04 810 491.86 24.62 550 89.2% 89.4% Stnd Dev Max Level of Support 4.6% 5.00 4.5% 5.00 Objective Rating 4.46 4.47 Equiv. Percent 89.2% 89.4% FIGURE 1 MECH 102 COURSE OUTCOME ASSESSMENT (PARTIAL) Using the Likert Scale, Table III, the course director makes a subjective evaluation of how strongly the embedded indicators assess student success of the course outcome. The subjective evaluation of the Max Level of Support is reviewed by a committee of those familiar with the course. The subjective evaluation of the strength of the embedded indicators is multiplied by the embedded indicator percentage to produce a course director assessment of the course outcome. The subjective evaluation of embedded indicators looks at all the embedded indicators that support the particular course outcome. It is recognized that some of the embedded indicators support the outcomes better than others. The instructor may choose to adjust the embedded indicator (points, nature of activity, etc.). Rating TABLE III LIKERT SCALE Association 1 Strongly Disagree 2 Disagree 3 Neutral 4 Agree 1. Write structured programs in MATLAB to solve engineering problems 5 Strongly Agree 2. Create a variety of engineering plots and graphs using MATLAB Lastly, the assessment is compared to the end of the semester student course evaluation of the same course outcome, Figure 2. The student course evaluation is purely subjective and completed at the end of the semester. One limitation of the student assessment is the familiarity with more recent material in the course and the slight loss in expertise of material learned early in the semester. T1D-3
with a lower student assessment), the instructor should offer an explanation and plan of corrective action if necessary. As a new course is developed and offered, the faculty can expect some variation as the course is refined with each offering. While this method may not be perfect, it analyzes data from the instructor and students separately. This objective assessment of course outcomes with objective data from embedded indicators and student assessment of their accomplishment can produce a better evaluation of the course and areas for course improvement. Over time, historical data can track the effects of changes in a course. A review of each course outcomes assessment can lead to a rating of how well the students are achieving the MECH program outcomes. FIGURE 2 STUDENT COURSE OUTCOME ASSESSMENT A table of the most recent comparison of this assessment for each course outcome (second column) is shown with the student assessment of the course outcome (third column) in Table IV. TABLE IV COMPARISON OF COURSE OUTCOMES ASSESSMENT Course Outcome 1. Write structured programs in MATLAB to solve engineering problems. 2. Create a variety of engineering plots and graphs using MATLAB. 3. Program simple numerical methods used for numerical integration. 4. Interpret an engineering drawing. 5. Create an engineering drawing that is consistent with professional engineering practice. 6. Create sketches of typical engineered objects. Instr /Student 4.46 / 4.20 4.38 4.46 / 4.55 4.52 4.28 4.43 Remarks Students out of practice at end of course exam. done before final still reinforcing and practicing. The instructor and student assessments will rarely be perfect matches, so some margin of difference should be expected. When the margin is larger than 0.10 (especially DISCUSSION Although this matrix, like many other assessment tools, is neither completely objective nor precise, it does allow the new MECH faculty to evaluate their courses and determine where adjustments are needed in the courses and the overall program. A complete discussion of how this course was assessed and results is beyond the scope of this paper. However, a few examples are cited in the data presented in this paper. For instance, course outcome 1 in Figure 1 shows the course outcome is well supported by almost all embedded indicators. The outcome may be broadly articulated and should be rewritten to make it more directly measured by fewer embedded indicators. Likewise, course outcomes 4-6, that articulate engineering drawing, have the fewest embedded indicators (one homework and part of the final exam). Here is an opportunity to combine course outcomes or increase the number of embedded indicators. In Table III, Course Outcome 5 was rated lower by students than the course director. Since this was the final block of instruction in the course, many students had only one homework as feedback and were apprehensive about the final exam. The students completed the survey before the last exam. Combined with the lack of embedded indicators shown in Figure 1 and stated above, this is a focal point of discussion with instructors. Had the matrix assessment (course director assessment) and the student assessment not been looked at together, this discrepancy would likely be overlooked. CONCLUSIONS The ME Program used by Crawford [3] details a more thorough method to assess an entire program. Applying part of this assessment solely to courses has given the new faculty a more thorough tool to review their course structure and feedback. The Course Outcomes method presented in this paper is being used and will be refined by the new mechanical engineering program at The Citadel. So far, it has been well received by the instructors, T1D-4
allowing them to ensure a mix of embedded indicators, make changes, and easily see connections of different levels of assessment. Using objective student performance data (graded events) reduces preconception from pure selfassessment. Similarly, completing subjective portions of the matrix before entering graded data minimizes subjectivity and takes advantage of faculty experience. Combined with student assessment of the course outcomes, the matrix provides a more complete picture for conducting course assessment. REFERENCES [1] New Program Proposal, Bachelor of Science, Mechanical Engineering, The Citadel.,http://www.che.sc.gov/che_docs/ caameeting_sept05_ 2013/3a.pdf [2] Nordstrom, G. and Pettit, J., A Syllabus-Based and Evaluation Tool for ABET Program Accreditation, American Society of Engineering Education Annual Conference and Exposition, Louisville, KY, 2010. [3] Crawford, B., Using Student Performance and Faculty Experience to Assess a Mechanical Engineering Program, American Society of Engineering Education Annual Conference and Exposition, Honolulu, HI, 2007. [4] Chickering, A. and Gamson, Z., Applying the Seven Principles for Good Practice in Undergraduate Education, San Francisco, CA: Jossey-Bass, 1991. [5] Knott, T. and Matusovich, H., The Value of Interviews in the Longitudinal of a Course, in First Year Engineering Experience Conference, Pittsburgh, PA, 2012. AUTHOR INFORMATION Robert Rabb, Associate Professor, Mechanical Engineering, The Citadel, rrabb@citadel.edu Monika Bubacz, Associate Professor, Mechanical Engineering, The Citadel, mbubacz@citadel.edu Jason Howison, Assistant Professor, Mechanical Engineering, The Citadel, jhowison@citdel.edu Kevin Skenes, Assistant Professor, Mechanical Engineering, The Citadel, kskenes@citadel.edu T1D-5