FROM: TJ Rivard, Dean, School of Humanities & Social Sciences

Transcription

1 TO: Mary Blakefield, Associate Vice Chancellor FROM: TJ Rivard, Dean, School of Humanities & Social Sciences DATE: June 17, 2009 RE: General Education Assessment for Introduction In the spring of 2008, the General Education Assessment Committee tested the attached general education rubrics (appendix 2) on a number of classes to see if the information gathered would be valuable enough to assess the campus learning objectives (LO) in courses that met the general education requirements. This pilot also allowed the committee to see what logistical issues might be faced during implementation. In addition, the committee requested feedback from the faculty who participated in the pilot. Here are the conclusions of the pilot study: 1) The rubrics that the General Education Committee developed could be used but that we would have no way of knowing what the specific results would mean until we had two or more years of data to evaluate. As a result, we decided that the information gathered in would serve as a baseline against future years. 2) There was a concern that the rubrics would be interpreted too broadly to offer useful assessment data. While this could be true, we should be able to determine that across the data that we receive in any given year and across assessment years. Special attention should be given to erratic data and determine if adjusting the rubric will resolve the issue. 3) Because faculty in courses being assessed were asked to assess random assignments in their courses, some thought that faculty may skew the results by choosing specific students assignments that would show their courses in a positive light rather than choosing student work through a random selection. The pilot did not reveal that this was the case. 4) The pilot was implemented approximately two-thirds of the way through the semester in the hopes of capturing student assignments that would demonstrate their knowledge near the end of the term. Several faculty made the comment that the learning objectives that fit their particular classes were often best assessed in assignments during the first part of the course. In the fall, the committee opted to send out the rubrics earlier and remind faculty of their need to fill out the rubric at various points. In the fall, the assessment should have been sent out even earlier than it was.

2 5) The committee assigned specific courses to assess specific LO s, thinking that if we didn t we may get an uneven proportion of responses to LO s. However, many instructors found this difficult, because often they did not have assignments that addressed them. As a result, the committee decided to send out all of the rubrics in the fall to all of the faculty in General Education (with the exception of those teaching in math and science) and allow faculty to choose what it was they wanted to assess. 6) After consulting with our Higher Learning Commission mentor, we thought it best that we gather data in the fall rather than throughout the year and analyze it in the spring. The idea was that analysis and recommendations could be made during the spring to enable changes before the next fall. The timing will need to be worked on. As a result of these considerations, the committee felt fairly confident that we would be able to implement the General Education Assessment plan in the fall of 2008 (the pilot report can be seen in appendix 1). Method On October 28, 2008, the following was sent to all faculty teaching courses at the one hundred and two hundred level that counted toward general education (115 sections). While some courses taught at the three hundred level are accepted as meeting distribution requirements in general education, that is not their primary intent, so the Committee chose not to assess those courses. The IU East Assessment Academy has been working with the Higher Learning Commission on developing an assessment plan for General Education (not programs). We piloted the program last spring and are fully implementing it this fall. That means all 100 and 200 level courses that fit into the category of gen ed need to turn in an assessment rubric. This does not apply to courses in the Writing Program or SPCH- S121 courses; they already have an assessment process. All other 100 and 200 level courses should conduct an assessment. You have been included in this because you teach a 100 or 200 level class that counts toward general education at IU East. The IUE Assessment Academy decided that general education only addresses Learning Objectives 3-7. Since the Speech Program and the Writing Program assess Learning Objective 3, the only objectives that we need to concern ourselves with are Objectives 4 7. The Gen Ed Assessment Committee designed rubrics for each of those areas. Here is what we need instructors to do: 1) Read over the attached rubrics for each of the Learning Objectives

3 2) Choose ONE that you want to assess in your classroom. 3) Choose an assignment from the latter portion of the course: a paper, an exam if appropriate, or other assignment 4) Randomly select five examples of the assignment (e.g., the last five students on the roster; the first two, #14, and the last two; however you want) 5) Make five copies of the rubric (one for each of the randomly selected student works) 6) Circle the appropriate number on the scale based on your assessment of the piece for each item within the rubric (remember this is about what the students appear to be accomplishing, NOT about whether the teacher of the course is doing the course justice) 7) Send the scored rubrics from your classes with a cover sheet that identifies the course (not the section) and the nature of the assignment being assessed to TJ Rivard ML 238 by Dec. 15th. The data was collected and summarized through SPSS. Results Thirty-one of one hundred and fifteen sections returned the assessment rubrics. Five rubrics representing five sections were filled out incorrectly. The twenty-six remaining sections represented 145 students. * 24% of the students were assessed under LO 4. 68% of the students were assessed under LO 5. 12% of the students were assessed under LO 6. 5% of the students were assessed under LO 7. The chart on the next page illustrates the numeric data gathered from SPSS. Each of the learning objectives were broken into their component parts which is why LO 4 is broken into three parts, LO 5 into two, and so on. See Appendix 2 for details about the specific skills assessed under each LO. * One faculty member reported that she was unable to use any of the rubrics for her course (ENG-W 203 Creative Writing), because the LO that she assesses for and makes the most sense was LO 3 which we had determined would be assessed adequately under the Writing Program s assessment. This is clearly not the cases and will need to be reconsidered for the next round.

4 LO 4.1 LO 4.2 LO 4.3 LO 5.1 LO 5.2 LO 6.1 LO 6.2 LO 7.1 LO 7.2 Responses Mean Median Mode Std. Deviation The vast majority of instructors assessed LO 5: Educated persons should have the ability to develop informed opinions, to comprehend, formulate, and critically evaluate ideas, and to identify problems and find solutions to those problems. Effective problem solving involves a variety of skills including research, analysis, interpretation, and creativity. By contrast, very few assessed LO 7: Educated persons should be expected to have some understanding of and experience in thinking about moral and ethical problems. A significant quality in educated persons is the ability to question and clarify personal and cultural values, and thus to be able to make discriminating moral and ethical choices. Specific reasons for this disparity and the overwhelming choice to assess LO 5 might be able to be intuited given that the arts & sciences have a strong focus on LO 5, but it is difficult to say with any certainty the reason that the other LO s have so few numbers. Recommendations 1) Explore ways to encourage greater participation in the process. The need for a larger data set is essential if we are to make meaningful recommendations. 2) Perhaps identify specific courses for specific LO s rather than allow faculty to choose whatever LO they want to assess. Although, there allowing the choice may give us a better indication of what LO s are not being sufficiently addressed. 3) Create rubrics for LO 3 that are separate from what the Speech Coordinator and the Writing Program Director are doing so that courses that focus on writing or oral presentation can be assessed within general education as well. Conclusion In this initial effort, it appears that the rubrics may be giving us solid data; so it seems logical to continue with this instrument in the fall of However, we will

5 be unable to be certain of what the above results mean until we have two or three years of data to compare it to. Because of this, there cannot be any substantive recommendations for general education.

6 Appendix 1 Gen Ed Assessment Pilot Study of Data Gathered Spring 2008 April 4, 2008 General Education Committee: TJ Rivard, Chair; Roberta Roswell; Kumara Jayasuriya; Victoria Beck; Ange Cooksey; Frances Peacock. The General Education Committee developed rubrics for learning objectives four through seven. Learning Objective three is already being addressed through an assessment process developed by the Writing Program Director, Eddy Helton, and the Public Speaking Program Director, Jerome Mahaffey. Our goal was to find a way to collect data that would be useful in assessing the learning that was happening in general education courses. The rubric that we decided on for these objectives came out of a review of the best practices around the country. The rubric is based on a scale from one (emerging) to six (mastering), i.e.: 1-2 = emerging 3-4= developing 5-6 = mastering Each of these areas of the scale was assigned descriptors for each learning objective. Once this was complete, we identified a series of gen ed courses where we could pilot these rubrics to find out what sort of information we would discover. Faculty applied the rubric to a midterm project, paper, or exam in their own courses and returned the results to the committee. We were not only looking for data that, if it came from a larger sample size, would be useful in assessing data, we were also looking for information on the application of the rubric and the implementation of the rubric in particular courses. In other words, did we choose the right courses to assess these particular learning objectives? We worked from the premise that if the mean is three, that the average should fall close to that point. Victoria Beck provided the following analysis: 4) Educated persons should be able to relate computational skills to all fields so that they are able to think with numbers. At a minimum, students should be able to carry out basic arithmetical and algebraic functions; they should have a working concept of simple statistics; and they should be able to interpret and use data in various forms. 1) Class A: 4.33 N = 15 Range = 2 6 2) Class B: 4.26 N = 15 Range = 1 6 3) Class C: 4.00 N = 15 Range = 1 6

7 Students appear to do better with simple math problems (formulas, graphs, and tables) than more complex mathematical procedures (2 & 3). 5) Educated persons should have the ability to develop informed opinions, to comprehend, formulate, and critically evaluate ideas, and to identify problems and find solutions to those problems. Effective problem solving involves a variety of skills including research, analysis, interpretation, and creativity. 1) Class 1a: 5.96 Range = 4 6 Comprehension 2) Class 1b: 4.42 Range = 3 6 Evaluation 3) Class 2a: 4.07 Range = 3 6 Analysis 4) Class 2b: 3.96 Creativity Students tend to be better at reading comprehension and evaluation than analysis and creativity. 6) Educated persons should develop the skills to understand, accept, and relate to people of different backgrounds and beliefs. In a pluralistic world one should not be provincial or ignorant of other cultures; one's life is experienced within the context of other races, religions, languages, nationalities, and value systems. 1) Class 1: N = 6 2) Class 2: N = 6 Students appear to be somewhat better at analyzing and placing in context the contributions of diverse cultures to personal identity and local/global communities than comparing and contrasting. 7) Educated persons should be expected to have some understanding of and experience in thinking about moral and ethical problems. A significant quality in educated persons is the ability to question and clarify personal and cultural values, and thus to be able to make discriminating moral and ethical choices. Incomplete assessment. We only received results from one class, and based on the instructor s comments the course (P103) did not appear to be an appropriate class to test the rubric. Conclusion The rubrics that have been developed appear to be effective at one level and could give us valuable data over time. However, the implementation process may reflect instructor bias/error. Need to develop a method to norm the scoring across instructors.

8 Appendix 2 Assssessssment Rubriic for Learniing Objjectiive 4 Directions: For each of the following criteria below, assess the work by circling a numeric score. Educated persons should be able to relate computational skills to all fields so that they are able to think with numbers. At a minimum, students should be able to carry out basic arithmetical and algebraic functions; they should have a working concept of simple statistics; and they should be able to interpret and use data in various forms. 1. Interpret mathematical models such as formulas, graphs, tables, and schematics and draw inferences from them. Student lacks understanding of model development (chart, function, equation, inequality, etc) and the use of variables. Student can develop a mathematical model (chart, function, equation, inequality, etc) appropriate for the given data with correct use of variables. Student lacks ability to interpret the final answer (chart, function, equation, inequality, etc) and frequently reaches incorrect conclusion. Student can interpret the final answer for given data with correct use of variables (chart, function, equation, inequality, etc.) Student has conceptual understanding of the variables and their use in development of models (chart, function, equation, inequality, etc) appropriate for the given data. Student can consistently interpret the final answer (chart, function, equation, inequality, etc) and draw appropriate conclusions to reach the objective of the original problem. 2. Represent mathematical information symbolically, visually, numerically, and verbally. Student lacks understanding of data given and lacks abilities to organize the relevant data. Student lacks an understanding of information presented and/or has no coherent formation of thought. Student has difficulty stating the Student can understand data given and organize the relevant data into diagrams appropriate to the setting. Student has rote understanding of the information given (chart, sketch, formula, etc) Student understands data given and can organize it with consistent, correct, symbolic notation. Student understands the information given conceptually, can clearly express thoughts, and could explain to others.

9 solution in a symbolic, visual, numerical, or verbal setting using mathematical symbols. Student can state the solution in a symbolic, visual, numerical, or verbal setting using mathematical symbols. Student can clearly state the solution in a symbolic, visual, numerical, or verbal setting using mathematical symbols and has a clear understanding of the setting and solution. 3. Use a variety of mathematical methods (algebraic, geometric and/or statistical methods) to solve problems. Student lacks ability to manipulate data symbolically, visually, numerically, and verbally appropriate to the setting. Student makes frequent errors. Student can manipulate accurately the data symbolically, visually, numerically, and verbally appropriate to the setting with minimal errors. Student can manipulate the data accurately, consistently, and with conceptual understanding symbolically, visually, numerically, or verbally, appropriate to the setting. Student does not use variables appropriately. Student lacks understanding in carrying out the method to arrive at the correct solution. Student shows consistent misuse of signs and arithmetic operations. Solutions have significant errors. Student shows consistent misuse or nonuse of algebraic theorems, principles, or rules. Student cannot connect the solution to the initial question and cannot make a conclusion based upon the solution. Student can express the unknown mathematically and develop the equation, formula, graph, etc. Student can correctly use the equation/sketch/formula to arrive at the correct solution Student knows the correct manipulation of signs and arithmetic operations but lacks consistent use of them. Student can accurately use algebraic theorems, principles, or rules. Student can correctly answer the initial question and can accurately use the solution in a symbolic, visual, numerical, or verbal setting using mathematical symbols. Student can develop a clear path to the solution that is verbal, numeric and/or symbolic. Student has clear, logical sequence of steps through to the solution. Student shows consistent and accurate manipulation of signs and arithmetic operations. Student shows consistent and accurate use of algebraic theorems, principles, or rules Student expresses clear understanding of relationship between solution and the original problem.

10 Assssessssment Rubriic for Learniing Objjectiive 5 Directions: For each of the following criteria below, assess the work by circling a numeric score. Educated persons should have the ability to develop informed opinions, to comprehend, formulate, and critically evaluate ideas, and to identify problems and find solutions to those problems. Effective problem solving involves a variety of skills including research, analysis, interpretation, and creativity. 1. Comprehending, formulating, and critically evaluating problems or questions. Identifies the main problem or question but does not summarize or explain clearly or sufficiently. Fails to identify, summarize, or explain the main problem, or question. Represents the issues inaccurately or inappropriately. Successfully identifies and summarizes the main problem or question, but does not explain why/how it is a problem or creates a question. Clearly identifies the challenge and summarizes main problem or question and successfully explains why/how they are problems or questions; and identifies embedded or implicit issues, addressing their relationships to each other. 2. Finding solutions to those problems through research, analysis, interpretation and creativity. An inappropriate strategy is selected, or no strategy is selected. A strategy is selected, but may not be appropriate. Selects an appropriate strategy to solve the problem. No strategy is applied, or a strategy is applied incorrectly. The result is not evaluated. The solution is not presented. The selected strategy is partially or incorrectly applied. The result is not evaluated The solution is not correct and/or is not clearly presented. Correctly applies the selected strategy. Evaluates the result for correctness and plausibility. Clearly and fully presents a correct solution.

11 Assssessssment Rubriic for Learniing Objjectiive 6 Directions: For each of the following criteria below, assess the work by circling a numeric score. Educated persons should develop the skills to understand, accept, and relate to people of different backgrounds and beliefs. In a pluralistic world one should not be provincial or ignorant of other cultures; one's life is experienced within the context of other races, religions, languages, nationalities, and value systems. 1. Identifies and analyzes commonalities and differences among cultures through a variety of disciplines Mastering Lack of identification of commonalities and differences among cultures. Failure to analyze crossroads and common ground with regard to virtues and values. Failure to separate the compelling from the trivial in analysis. Partial identification of commonalities and differences among cultures. Attempts to analyze crossroads and common ground with regard to virtues and values. Begins to separate the compelling from the trivial in analysis. Readily Identifies commonalities and differences among cultures. Effectively analyzes crossroads and common ground with regard to virtues and values. Demonstrates keen ability to separate the compelling from the trivial in analysis. 2. Analyzes and places in context the contributions of diverse cultures to personal identity and local/global communities Mastering Failure to analyze and place in context the contributions of diverse cultures to personal identity and local/global communities. Limited ability to analyze and place in context the contributions of diverse cultures to personal identity and local/global communities. Demonstrates ability to analyze and place in context the contributions of diverse cultures to personal identity and local/global communities. Inability to recognize connections between micro-cultural contributions and meta-cultural development. Inability to personalize, localize and globalize impact of contributions of diverse cultures. Emerging ability to recognize connections between microcultural contributions and metacultural development. Attempts to personalize, localize and globalize impact of contributions of diverse cultures. Easily recognizes connections between micro-cultural contributions and meta-cultural development. Personalize, localizes and globalizes impact of contributions of diverse cultures.

12 Assssessssment Rubriic for Learniing Objjectiive 7 Directions: For each of the following criteria below, assess the work by circling a numeric score. Educated persons should be expected to have some understanding of and experience in thinking about moral and ethical problems. A significant quality in educated persons is the ability to question and clarify personal and cultural values, and thus to be able to make discriminating moral and ethical choices. 1. Recognizes, comprehends, and critically evaluates competing or conflicting values and moral or ethical issues. Recognizes the values or moral/ethical issues but does not summarize or explain clearly or sufficiently. Fails to recognize the relationship of values or moral/ethical issues to the main question or problem. Ignores the cultural context of the issue. Represents the issues inaccurately or inappropriately. May successfully identify and summarize the values or moral/ethical issues, but does not explain why/how the values or moral/ethical issues interact with the problem. Clearly identifies and summarizes main problem or question and successfully explains why/how they are problems or questions; and identifies embedded or implicit issues, addressing their relationships to each other. 2. Reaches a clear understanding of personal and cultural values through research, analysis, interpretation and creativity. A strategy is selected, but is not appropriate. An inappropriate strategy is selected, or no strategy is selected. No strategy is applied, or a strategy is applied The selected strategy is partially or incorrectly applied. Selects the most appropriate strategy to solve the problem. Correctly applies the selected strategy.

13 incorrectly. The result is not evaluated. The solution is not presented. The result is not evaluated. The solution is not correct and/or is not clearly presented. Evaluates the result for correctness and plausibility. Clearly and fully presents a correct solution.