Developing a Department-Wide Learning Assessment Program

Transcription

1 Developing a Department-Wide Learning Assessment Program Timothy L. Skvarenina 1 Session T1A Abstract - During the last several years, the North Central Association of Colleges and Schools (NCA) shifted the focus of its accreditation visits from resource-based to outcome-based. Faced with an accreditation visit by NCA in the fall of 1999, Purdue University began developing a university-wide learning assessment program. A council was formed at the University level, with committees in each of the schools within the University. The School of Technology was one of the leaders in developing an assessment program, which was to be expected as it is a teaching-based, rather than research-based school. Most of the faculty were reluctant to have new requirements added to their workload. Thus, it was necessary to work as a team to develop requirements that were acceptable or at least palatable to the faculty. This paper describes how an assessment program was developed in the Department of Electrical Engineering Technology (EET) at Purdue in the hope that it will provide help to others who are starting the process of developing an assessment program. Index Terms accreditation, assessment, faculty involvement, learning objectives. INTRODUCTION Purdue University is accredited as an institution of higher learning by the North Central Association (NCA) of Colleges and Schools. The School of Technology (SOT) at the West Lafayette, Indiana Campus consists of eight departments: Aviation Technology, Building Construction Management, Computer Technology, Electrical Engineering Technology, Industrial Technology, Mechanical Engineering Technology, Organizational Leadership and Supervision, and Computer Graphics. The department of Electrical Engineering Technology at West Lafayette has 23 full-time faculty members, serving approximately 470 students, and is accredited by the Technology Accreditation Commission of the Accreditation Board of Engineering and Technology (TAC of ABET). Traditionally, accreditation has been based on institutional input variables such as the number of books in the library, the numbers and amounts of grants received, and full-time equivalent faculty to student ratios. About five years ago, however, the emphasis of the NCA guidelines was changed to focus on outcome-based variables; i.e., assessment of student learning. This change in orientation proved to be quite challenging as there were no institutional models to follow that discussed or depicted how to define or measure student learning outcomes. To be accredited by the NCA, a university must now demonstrate that it meets the criteria shown in Appendix A. Criterion 1 requires that the University have goals for its academic programs, while criterion 3 is directly related to assessment. In particular, the commission considers the following as evidence of accomplishment of criterion 3:! assessment of appropriate student academic achievement in all its programs, documenting: " proficiency in skills and competencies essential for all college-educated adults; " completion of an identifiable and coherent undergraduate level general education component; " mastery of the level of knowledge appropriate to the degree granted. " control by the institution's faculty of evaluation of student learning and granting of academic credit.! effective teaching that characterizes its courses and academic programs The Associate Executive Vice-president for Academic Affairs at Purdue University was designated as the lead official for the NCA accreditation visit and for insuring that all 10 schools within the University were conducting assessment of student learning. She served as Chair of the University Assessment Council, which had representatives from all 10 of the schools. The University Assessment Council developed a statement of objectives for the academic programs at Purdue University, which reads as follows: Students at Purdue University are expected to acquire knowledge, develop the abilities to assess what they learn, and apply it effectively. To accomplish this, they must be able to read and think critically and to communicate - both orally and in writing - with clarity and precision. Developing competence in quantitative and scientific reasoning is equally necessary. They must also become aware of the cultural, social, political, and economic forces and the technologies that shape our world. In their area of specialization, Purdue students at all levels are expected to achieve depth of understanding of both the essential content and principal modes of inquiry and to become familiar with the ethical issues facing their chosen career fields. A Purdue education should prepare them for a lifetime of learning. This statement of objectives served as an umbrella for all of the academic programs on the campus. In other words, every program was expected to have a set of objectives that were traceable to the Purdue statement. 1 Purdue University, Electrical Engineering Technology, 1415 Knoy Hall, West Lafayette, IN tskvaren@purdue.edu T1A-11

2 The Assistant Dean of the School of Technology served on the University Council and also chaired the SOT Assessment Committee (SOTAC), which had representation from each department within the school. This committee had a two-fold purpose. One, to provide coordination between the departments, and two, to provide visibility of the assessment program to the rest of the faculty. Formal responsibility for assessment was assigned to the department heads, who in many cases delegated their authority to a member of the faculty. The priority status of assessment was conveyed regularly at the Dean's Academic Council meetings where, at regular intervals, department heads were expected to report on the progress of assessment in their respective departments. Formal responsibility was important for legitimizing a department's movement toward assessment and for supporting the implementation of assessment activities developed by the SOTAC. SOT ASSESSMENT COMMITTEE Overall, there were two phases of activity within the School of Technology Assessment Committee. While phase one was dominated by information seeking activities, phase two was dominated by action oriented activities. Essentially, the emphasis in the second phase was on deliverables. The committee concluded that the NCA had taken a Total Quality Management approach to assessing student learning outcomes and that it was the institution's responsibility for defining the features of their "products" and to provide evidence that they were satisfying their "customers" needs. In essence, the committee felt that NCA had deliberately designed a vague assessment framework that could fit all academic institutions based on their particular missions, goals, and objectives. In response to this, the SOTAC developed a departmental template for assessment, which became known as the "Minimum Assessment Model". The purpose of the model was to support a continuous quality improvement in an educational setting. An important conclusion of the SOTAC was that assessment that is not directly linked to the process of continuous quality improvement is of little or no value. The goal of assessment is not to produce a document, but rather to document that the process of continuous improvement is in-place and an active and integral part of the program. Nevertheless, documentation of the assessment process is required for accreditation by the North Central Association and other program specific accrediting agencies. The Minimum Assessment Model Components of the minimum assessment model are as follows: 1. A brief, one or two page, description of the department and its programs. 2. The Departmental Mission Statement. The mission of the department should set the stage for the learning outcomes of the programs offered by the department. 3. Learning outcomes for the degree and program option offered by that department. 4. The current curriculums and plans of study for degrees and programs offered by the individual department. 5. Documentation of the methods and techniques used to assess degree learning outcomes. 6. Course descriptions and learning outcomes for all courses that make up the current curriculums or programs. 7. Documentation of the methods and techniques used to assess course learning outcomes. Once the SOTAC had defined the minimum assessment model, the task of implementing it through the department faculties began. Developing a Department Assessment Program In addition to the university accreditation, our department is accredited by the Technology Accreditation Commission of the Accreditation Board for Engineering and Technology (TAC-ABET). Our accreditation visit was in the Fall of Although that visit was under the old TAC criteria, our next one will be under the new criteria. Appendix B shows two of the criteria for TAC-ABET accreditation. Criterion 1 requires sets forth the requirements for a technology program, while criterion 6 clearly requires an assessment program and continuous improvement program. The EET department has had, for many years, a curriculum committee with area teams (e.g., digital, analog, power and controls, etc.). The curriculum committee has the responsibility for course content as well as the overall plan of study. Among the things considered by the curriculum committee were course learning objectives and continuous improvement of courses. However, there was not a formalized assessment procedure that resulted in the type documentation required for NCA or for the new TAC-ABET requirements. While many of the faculty had learning objectives in their course syllabi, some did not. Very few faculty documented the changes they made to their courses and the reasons for doing so. The Department had a mission statement, but we did not have desired outcomes for graduates of the program. Thus, it was necessary to develop an assessment program within the department and a system for documenting the assessment results. Although most faculty in the EET department were conducting at least an informal assessment of their courses and making continual improvements to them, we were essentially starting from ground zero as far as formally documenting the assessment process and results. In addition, assessment at the program level was found to be well behind the assessment of individual courses in fact there were no curriculum goals corresponding to the University outcomes statement. Thus we realized that a variety of data would need to be collected or developed. In addition, it was evident that faculty training would be required to help them produce usable assessment materials. T1A-12

3 Our department normally holds a faculty meeting every month, so when it became clear that the faculty needed help in delivering the assessment products, we began putting assessment on every meeting s agenda. In fact some meetings were devoted to assessment, either in presenting material to the faculty or in having them work together to produce the materials required for the minimum assessment model. Educating the Faculty Many of the faculty stated that they did not understand what this assessment stuff was all about. Fortunately our curriculum is very project oriented, so it was possible to put it in terms they already understood the design process. Figure 1 shows the design process. When designing something, one should establish some specifications for how the item being designed will perform after it is built. Once the specifications are established, a design can be made that hopefully will meet the specifications. The item can then be built and, once built, can be tested to see if it indeed meets the specifications. If it does not meet some portion of the specs, then the design can be modified and the cycle repeated. course and for the curriculum as a whole. When presented with the design analogy, most of the faculty could see what was required; however, many did not know how to write behavioral objectives for their courses. To help the faculty with the writing of course objectives, it was really necessary to go back to basics as many of them have not had any formal education courses. Several presentations were made at faculty meetings covering Bloom s Taxonomy [2] and how to write proper learning objectives [3], and references to materials on the world wide web were provided so that faculty could find additional information. The goal of a behavioral objective is to express the desired outcome of the instruction in terms of what the recipient of the instruction should be able to accomplish. A learning objective contains three major components:! Conditions! A behavioral verb! Criteria The conditions should be a statement that expresses what is necessary for the behavior to be observed. The behavioral verb must state an action that can be observed and evaluated. This was a stumbling block for some of the faculty as they tended to use words like understand, comprehend, etc. Unfortunately, it is not possible to directly observe the students comprehension. Finally, the criteria indicates what level of achievement is required to state that the objective has been achieved. After the faculty had been provided with the required background, we were ready to begin working on the elements of the SOT minimum assessment model. FIGURE 1 THE DESIGN PROCESS The key point that was made to the faculty is that the process applies to their courses and to the curriculum just as much as to the design of a circuit. The italics under the boxes in Figure 1 shows how the process applies to education. To properly teach a course, one should begin with some course objectives. With the objectives in mind, one can then prepare the instructional materials (lectures, homework assignments, lab exercises, etc.). Then the instruction can be presented. Finally, to meet the requirements of our accreditation agencies, it is necessary to assess the results of the instruction and close the loop. Of course it is not immediately evident how that assessment of student learning can be conducted and documented. Thus, the concepts of direct and indirect measures of learning [1] were introduced to the faculty. Some were surprised to learn that grades, per se, were not a direct measure of student learning. Finally, the task at hand completion of the SOTAC minimum assessment model was explained. It was important for the faculty to understand that assessment occurs at both the course level and the curriculum level. Thus, objectives are required for each BUILDING THE DEPARTMENT ASSESSMENT MATERIALS In looking at the items for the minimum assessment model, it was clear that we needed to work particularly on items 3, 5, 6, and 7. That is, we needed to develop a set of goals for our curriculum and goals for each course and then we needed to begin assessing student learning against them. Developing goals for the curriculum Criterion 1 of the proposed TAC-ABET accreditation criteria (shown in Appendix B) provides a list of expected outcomes for a student who graduates from an engineering technology program. This, combined with the Purdue statement of student outcomes, provided a starting point for the development of our curriculum goals. In looking at the TAC-ABET criteria, many of the faculty felt that 11 goals were too many and that we needed to reduce them. Faculty also had difficulty with the non-specificity of the goals. This also required some education of the faculty. Specifically, goals are statements of broad outcomes that are not directly measurable [4]. Under the goals are objectives, which are more like the behavioral objectives discussed above. Several sessions were held with the faculty to develop and refine the T1A-13

4 curriculum goals and objectives. We found it very useful to conduct a workshop using the affinity method described in [4] to develop the objectives. The final statement of our program (curriculum) goals was: Our graduates will have: 1. The ability to communicate effectively in oral, written, visual, and graphical modes in both interpersonal and group environments 2. The attitudes, abilities, and skills required to adapt and adjust to rapidly changing technologies and a desire for life-long learning 3. The ability to think critically, and to identify, evaluate, and solve complex technical and non-technical problems 4. An understanding of all aspects of the design process including functional and esthetic considerations 5. A well developed sense of ethics and the personal discipline to succeed in their chosen profession 6. An understanding of the internationalization of technology and the marketplace and the value of cultural and personal diversity 7. The skills and attitudes necessary to work successfully as a member of a team Comparison of our goals to the Purdue Outcomes Statement and the TAC-ABET criteria, shows that we did in fact include each major item of those two documents in our goals. Developing learning outcomes for each course Responsibility for the learning outcomes for each course was assigned to the curriculum teams within the department. Each course coordinator was expected to develop a set of outcomes for his or her course and to coordinate them through the appropriate team. We found it helpful to hold a workshop during one of our faculty meetings in which each person was asked to write several outcomes for a course and then discuss them with the other members of their team. This helped to standardize the level of the outcomes between the teams and provided immediate feedback to the faculty members. All courses were required to have learning outcomes one year before our TAC-ABET accreditation visit. This allowed us to accumulate two semesters of assessment data prior to the visit. Assessment of the curriculum goals Because the curriculum goals aren t specific and because some involve looking at graduates performance over a period of time after they leave, we found that it was absolutely necessary to use a mix of direct and indirect methods of assessment at the curriculum level. Our first problem was to determine which courses were contributing to which program goals. One technique that proved useful was to create a spreadsheet with all of the courses listed down the first column and the curriculum goals across the first row. The faculty was then asked to check the goals for which they felt there course made a contribution to achieving the goal. Generally the faculty members played it straight, although one asked why he should check any of the boxes. Fortunately one senior faculty member stated that one would have to wonder why we would have any course that doesn t contribute to some of the curriculum goals. Once we had determined which courses were contributing to which goals, the faculty was asked to perform an assessment of their courses at the end of each semester. Most of the faculty wanted an indication of what was expected and it was generally felt that we should have a standard method of reporting our results. One of the other departments in the school had already promulgated a rather massive spreadsheet for keeping track of their assessment results. Our faculty felt that it was far too burdensome so we looked for something simpler that would still achieve the purpose of documenting the assessment and continuous improvement process. After some discussion we finally arrived at a simplified reporting method. Each faculty member was asked to answer the following four questions, with respect to each goal that was checked for his or her course: 1. What activities in your course contribute toward the achievement of the goal? 2. What measures (direct and indirect) do you use to measure the achievement of your students toward the goal/objective? 3. What are the results of your assessment? 4. What changes, if any, would you make next time to improve the level of attainment of the goal? The inputs from the faculty members provided some direct, and indirect, indication of the attainment of our curriculum goals, but clearly some goals, such as life-long learning require evaluation as the former students progress in their careers. Thus, we also used surveys of seniors, graduates, and employers to provide indirect assessment of the curriculum goals. We constructed a new survey for our graduating seniors last year that started by listing our curriculum goals and then asking them questions pertaining to those goals. For example, for written communication, we asked what their ability was when they arrived at Purdue, how they self assessed their ability now, and finally we asked which courses contributed to achieving the goal and which should have but did not. We have found from two semesters of surveys that there was dissatisfaction with several courses from other departments that pertained to this goal and we have opened a dialog with the departments that provide those courses. The final area that we as a faculty had to work on was the assessment of the course learning outcomes. Assessment of the course learning outcomes As with the curriculum goals, the faculty wanted a method of documenting the results of their assessment that would not be overly burdensome. Again we arrived at a simplified reporting method. Each faculty member was asked to create T1A-14

5 a report consisting of the learning objectives for the course they were assessing. After each objective they were asked to indicate how the objective is assessed and what the results of the assessment were. Finally they were asked to indicate what changes if any were required for the next offering of the course. While this part was more straightforward than the assessment of the curriculum goals, the faculty needed help in determining what was a good criteria. One technique that I had found useful in assessing the results of exams, for example, was to record the scores on every question of the exam, rather than just the total scores. That way, I could identify which course outcomes an exam question was testing and determine whether the desired learning was taking place. In the end, the faculty used all of the various components of their grading to aid in the assessment; i.e., quiz and exam results, laboratory performance. THE ASSESSMENT PROCESS With the creation of the assessment reports, it became evident that we needed some system for dealing with the materials being generated. We also needed to insure the results were being reviewed. As previously mentioned, our department has had a curriculum committee with teams for each of the areas. Thus it made sense to use that existing structure for the assessment process, rather than establishing a new assessment committee. As the assessment coordinator, I was made a member of the curriculum committee. Figure 2 shows a flowchart of the assessment process in our department. There are several components to the assessment process. At the end of each semester, students are given the opportunity to evaluate the course by answering a series of questions from the so-called Purdue cafeteria system. The cafeteria system contains a large number of questions that can be selected to personalize a survey. Traditionally those scores were provided only to the faculty member, although the member was required to include them when preparing a promotion and tenure document. As part of the assessment process and evaluation of teaching effectiveness at Purdue, it was decided that those scores would also be available to the department head. Thus, at the left side of Figure 2, there is a dotted closed loop between the faculty member and the head. As described above, each member must prepare assessment reports at the end of the semester. These are forwarded to the assessment coordinator, who puts them in a shared folder on our computer network, where they are then available to the curriculum teams. The teams are responsible for reviewing the results and reporting to the curriculum committee, which also receives inputs from the accreditation agencies, our industrial advisory board, and our industry liaison. Surveys are routed to the assessment coordinator for analysis and reporting to the curriculum committee, the head, and the faculty as a whole. All of the faculty are members of one or more of the curriculum teams so they all participate in the process. Changes to the curriculum, the FIGURE 2 FLOWCHART OF THE EET ASSESSMENT PROCESS curriculum goals, or the reporting method are voted upon by the entire department faculty, which is shown at the top. Thus, we believe we have achieved faculty ownership of the process. Conclusion When I became the assessment coordinator for the department and began telling the faculty what was going to be necessary, a couple of them approached me and said something like, can t you just do the paperwork and make it go away for us. I told them that wouldn t work as the accreditation criteria was quite specific that the faculty had to be involved in the assessment. I found that in order to get the faculty to take ownership of the assessment process, two things were necessary. First, we had educate them as to what was expected and how to do it. I have described some of the things we did to achieve that in this paper. Second, strong support from the department head and the administration are essential. In our case, the Dean was very supportive through the formation of the school assessment committee and through his dealings with the department heads. Our department head, in turn, made assessment a point of emphasis in faculty meetings and in year-end reviews for merit pay raises. That proved to be most helpful in getting the individual members of the faculty to actually do their assessments and report the results. We successfully passed our accreditation visit from TAC-ABET in the fall of 1998 and the university passed its visit from the NCA in the fall of The challenge now is to keep the faculty moving in the direction ACKNOWLEDGMENT I received strong support from Larry Hoffman, my department head, and Robert Herrick, assistant department head,. Without that support, getting the faculty to buy into the T1A-15

6 assessment process would have been very difficult, if not impossible. REFERENCES [1] (1996) Revised Sections of the Handbook of Accreditation, Criteria Three and Four. Chicago, Illinois. Commission on Institutions of Higher Education, North Central Association of Colleges and Schools [2] Linn, R. L. & Gronlund, N. E.. Measurement and Assessment in Teaching (7 th edition). Upper Saddle River, NJ. Prentice Hall (1995) [3] Kilik, R. How to Write Behavioral Objectives. [4] Rogers, G. M. & Bando, J. K. Stepping Ahead: An Assessment Plan Development Guide. Rose- Hulman Institute of Technology (1996) APPENDIX A The criteria for accreditation as an institution of higher learning by the North Central Association are as follows: 1. The institution has clear and publicly stated purposes consistent with its mission and appropriate to an institution of higher education. 2. The institution has effectively organized the human, financial, and physical resources necessary to accomplish its purposes. 3. The institution is accomplishing its educational and other purposes. 4. The institution can continue to accomplish its purposes and strengthen its educational effectiveness. 5. The institution demonstrates integrity in its practices and relationships. APPENDIX B Proposed TAC-ABET accreditation criteria Criterion 1. Students and Graduates An engineering technology program must prepare graduates who: a. demonstrate an appropriate mastery of the knowledge, techniques, skills, and modern tools of their disciplines, b. apply current knowledge and adapt to emerging applications of mathematics, science, engineering, and technology, c. conduct, analyze, and interpret experiments and apply experimental results to improve processes, d. apply creativity in the design of systems, components, or processes appropriate to program objectives, e. function effectively on teams, f. identify, analyze, and solve technical problems, g. communicate effectively, h. recognize the need for and possess the ability to pursue lifelong learning, i. understand professional, ethical, and social responsibilities, j. recognize contemporary professional, societal, and global issues and are aware of and respect diversity, and k. have a commitment to quality, timeliness and continuous improvement. Criterion 6. Assessment Programs must have written goals that, as a minimum, focus on the student body served, employer expectations, resource allocation, and other factors affecting the program. Programs are required to have plans for continuous improvement and evidence that the results are applied to further development and improvement of the program. Each program is required to demonstrate achievements through various methods including student outcomes assessment and employer feedback. Typical evidence may consist of student portfolios including project work and activity based learning; results of integrated curricula experiences; nationally- normed subject content examinations; recent graduate surveys that demonstrate graduate satisfaction with employment including career development activities, mobility opportunities, and appropriate job title; and employer surveys that demonstrate satisfaction with recent graduates. Programs also must demonstrate that their graduates are readily accepted into the workforce and are prepared for continuing education. T1A-16