Student Learning Outcome - The 15 Best Based Performance Criteria

Transcription

1 College of Liberal Arts & Sciences Department of Philosophy Philosophy M.A. August 16, 2014 David J. Buller, Chair Status Report 1

2 1. INTRODUCTION The Philosophy M.A. assessment plan submitted along with this status report is a new assessment plan, implemented in Fall Assessment results reported below concern the learning outcomes of the prior assessment plan (filed in Fall 2006). Some results gathered during the reporting period led to programmatic changes, but all of the results led to a reconsideration of the assessment plan itself. Explanations of the reasons for, and nature of, the changes to the program and the assessment plan will be provided in section 3. The learning outcomes for the prior assessment plan are stated in full detail in section 2 below, but an overview of those outcomes is as follows: Each graduate of the program will demonstrate: 1. Proficiency in writing analytical essays 2. Proficiency in symbolic logic 3. Broad, yet in-depth, knowledge of metaphysics and epistemology 4. Broad, yet in-depth, knowledge of ethics and metaethics 5. Broad, yet in-depth, knowledge of the related areas of philosophy of science, philosophy of language, and philosophy of mind Each of the outcomes above had a significant number of sub-outcomes or performance criteria, which are enumerated below. Student learning was measured by assigning one of the following scores to student work on each of the sub-outcomes or performance criteria: 0 = Does not meet expectation in any significant respect 1 = Partially meets expectation, but does not satisfy expectation for master s-level performance 2 = Meets expectation for master s-level performance Thus, the student-level target for each performance criterion was a score of 2. The program-level target was an average score, among program graduates collectively, of 1.75 on each performance outcome. An average score of 1.75 on each outcome mathematically guarantees that at least 75% of program graduates have met expectation for baccalaureate-level performance (received a 2 for their performance) on that outcome. In fact, assuming that all graduates receive only a 1 or a 2, an average of 1.mn indicates that mn% of graduates received a score of 2. Of course, not all graduates receive scores of 1 or 2 ; some receive scores of 0 on a performance outcome. Accordingly, the percentage of graduates who meet the department s expectation on each performance outcome is virtually always greater than mn% when the average is 1.mn. Thus, while the program-level target ensures that at least 75% of all graduates achieve the student-level target, which is what the department would like to Status Report 2

3 see, tracking the average score of program graduates collectively was simpler, hence more efficient, than tracking the number of graduates who achieved the student-level target. 2. REPORTED RESULTS Student Learning Outcome 1: Graduates will demonstrate proficiency in writing analytical essays. Program-level Target: An average score, among program graduates collectively, of 1.75 (on the 0-2 scale described in section 1) on each of the four performance criteria below. Assessment Method: Students will write comprehensive examination essays that: 1. Provide accurate exposition of the philosophical views discussed 2. Provide clear and thorough explanations of the philosophical views discussed 3. Provide strong and cogent arguments in support of philosophical theses 4. Critically engage opposing views and arguments Student-level Target: A student demonstrates achievement of an outcome by earning a score of 2 (on the 0-2 scale described in section 1). Analytical Essay Results, Fall 2010 Spring 2014 Graduates Sub-Outcome N Average Data Collection Cycle 1 69* * * 1.78 Every semester 4 122* 1.83 * Sample sizes were larger for this assessment method because each student was required to take three examinations, all of which were scored and included among the assessment results. Analysis of Results for Student Learning Outcome 1 The data do not indicate that program graduates have not been meeting departmental expectations for proficiency in analytical essay writing. Accordingly, the data indicate no need for programmatic change. However, the data, in themselves, also provide no compelling reason to believe that program graduates have been meeting departmental expectations in analytical essay writing. The problem is that the results concern essays that were written for comprehensive examinations, which are three-hour examinations during which students must answer three questions that were not made available to them in advance of the examination. In effect, then, student analytical writing performance was judged by scoring timed one-hour micro-essays in response to surprise writing prompts. Writing assignments of this nature are Status Report 3

4 not designed as an optimal showcase of student ability to provide strong and cogent arguments or critically engaging opposing views and arguments. As a result, the assessment scores may be a greater reflection of faculty judgments that student performances were adequate within the constraints of the task, rather than that student work actually met expectations for master s-level analytical writing. Thus, assessment of analytical writing needs to be embedded in the program at the points at which students produce sustained, wellresearched, reflective essays (e.g., by assessing term papers written for graduate seminars) Student Learning Outcome 2: Graduates will demonstrate proficiency in symbolic logic. Program-level Target: An average score, among program graduates collectively, of 1.75 (on the 0-2 scale described in section 1) on each of the six performance criteria below. Assessment Method: Students will complete exams in PHIL 505, Intermediate Logic, in which the student: 1. Clearly and accurately defines the fundamental logical concepts of validity, consistency, necessity, and logical equivalence 2. Demonstrates a precise knowledge of the implications of fundamental logical concepts and their logical relationships to one another 3. Accurately performs truth-functional computations to test for validity, consistency, necessary truth, and/or logical equivalence 4. Correctly renders English sentences in logical symbolism, including English sentences expressing necessary or sufficient conditions, limited quantity ( at least n and at most n ), and precise quantity ( exactly n and the x ) 5. Constructs proofs of validity, inconsistency, necessary truth, and/or logical equivalence 6. Constructs interpretations (semantic models) to prove invalidity and/or consistency Student-level Target: A student demonstrates achievement of an outcome by earning a score of 2 (on the 0-2 scale described in section 1). Symbolic Logic Results, Fall 2010 Spring 2014 Graduates Sub-Outcome N Average Data Collection Cycle Every fall semester (no data reported in Fall 2012) Status Report 4

5 Analysis of Results for Student Learning Outcome 2 Program graduates have been meeting departmental expectations of student learning in logic. No programmatic change is necessary. However, the fact that no data were reported in Fall 2012 indicates that the department needs to take steps to ensure greater faculty compliance with gathering and reporting assessment results Student Learning Outcome 3: Graduates will demonstrate broad, yet in-depth, knowledge of metaphysics and epistemology. Program-level Target: An average score, among program graduates collectively, of 1.75 (on the 0-2 scale described in section 1) on each of the six performance criteria below. Assessment Method: Students will write comprehensive examination essays that: 1. Understanding of central philosophical issues in metaphysics and epistemology 2. Knowledge of relevant literature in metaphysics and epistemology, associating important philosophical views with their proponents and critics Student-level Target: A student demonstrates achievement of an outcome by earning a score of 2 (on the 0-2 scale described in section 1). Metaphysics and Epistemology Results, Fall 2010 Spring 2014 Graduates Sub-Outcome N Average Data Collection Cycle Every semester Analysis of Results for Student Learning Outcome 3 The data indicate that program graduates have been meeting departmental expectations for knowledge of metaphysics and epistemology. Accordingly, the assessment results indicate no need for programmatic change. However, independent data did indicate a need for programmatic change. The department s terminal M.A. program is intended and designed to be a two-year program. But a study of program graduates from 2003 to 2013 showed that only 68% of graduates actually completed the program in two years. Indeed, the average time to degree completion within this period was five semesters, rather than the desired four. Without exception, program graduates who took longer than two years to complete the degree required the additional time in order to successfully pass the comprehensive examination. Graduates have been meeting the department s expectations of student learning, but roughly one-third of them have taken longer than the intended two years to do so. The design of the comprehensive examination was undermining the two-year design of the overall program. Thus, the design of the comprehensive examination requirement needed to be reconsidered. Status Report 5

6 Student Learning Outcome 4: Graduates will demonstrate broad, yet in-depth, knowledge of ethics and metaethics. Program-level Target: An average score, among program graduates collectively, of 1.75 (on the 0-2 scale described in section 1) on each of the fifteen performance criteria below. Assessment Method: Students will write comprehensive examination essays that: 1. Understanding of central philosophical issues in ethical theory and metaethics 2. Knowledge of relevant literature in ethical theory and metaethics, associating important philosophical views with their proponents and critics Student-level Target: A student demonstrates achievement of an outcome by earning a score of 2 (on the 0-2 scale described in section 1). Ethics and Metaethics Results, Fall 2010 Spring 2014 Graduates Sub-Outcome N Average Data Collection Cycle Every semester Analysis of Results for Student Learning Outcome 4 The data indicate that program graduates have been meeting departmental expectations for knowledge of ethics and metaethics. Accordingly, the assessment results indicate no need for programmatic change. However, as discussed above in the analysis of results for outcome 3, independent data regarding time to degree completion did indicate a need for programmatic change to the comprehensive examination requirement Student Learning Outcome 5: Graduates will demonstrate broad, yet in-depth, knowledge of the related areas of philosophy of science, philosophy of language, and philosophy of mind. Program-level Target: An average score, among program graduates collectively, of 1.75 (on the 0-2 scale described in section 1) on each of the thirteen performance criteria below. Assessment Method: Students will write comprehensive examination essays that: 1. Understanding of central philosophical issues in the philosophies of science, language, and mind Status Report 6

7 2. Knowledge of relevant literature in philosophy of science/language/mind, associating important philosophical views with their proponents and critics Student-level Target: A student demonstrates achievement of an objective by earning a score of 2 (on the 0-2 scale described in section 1). Science, Language, and Mind Results, Fall 2010 Spring 2014 Graduates Sub-Outcome N Average Data Collection Cycle Every fall semester Analysis of Results for Student Learning Outcome 5 The data indicate that program graduates have been meeting departmental expectations for knowledge of the related areas of philosophy of science, philosophy of language, and philosophy of mind. Accordingly, the assessment results indicate no need for programmatic change. However, as discussed above in the analysis of results for outcome 3, independent data regarding time to degree completion did indicate a need for programmatic change to the comprehensive examination requirement DECISIONS, ACTIONS, AND USE OF RESULTS For the most part, results to date show that program graduates are meeting the department s expectations for student learning. Analyses above, however, revealed two problems with the data. First, although assessment results show that graduates are meeting the department s expectations for student learning, independent data show that roughly one-third of program graduates fail to complete the program in the intended two years and that, in all such cases, additional time is required in order to successfully pass the comprehensive examination. These facts indicate the need for reconsideration of the design of the comprehensive examination. Second, assessment of analytical essay writing has been embedded in the comprehensive examination, which is not an optimal checkpoint for assessing student writing ability; analytical writing ability should be assessed with substantive student essays, not timed micro-essays. Programmatic Change During the reporting period, the comprehensive examination consisted of three sections, covering all the major areas of contemporary analytic philosophy: (1) Ethics and Metaethics, (2) Metaphysics and Epistemology, and (3) Philosophy of Science, Language, and Mind (SLaM). Students were required to pass each section of the examination, and each section was Status Report 7

8 administered as a timed three-hour examination during which students had to answer three questions that were not known to them in advance of the examination. At the time that the comprehensive examination was designed with its three-part structure, covering all the major areas of contemporary analytic philosophy, the department had no area requirements within the 30 credit hours required for the M.A. Students were not required to take any courses in any of the major areas of philosophy, since it was assumed that the comprehensive examination requirement itself would ensure that graduates had achieved the breadth of knowledge of the discipline that the department expected. The comprehensive examination was the gatekeeper to ensure breadth of knowledge of the discipline. However, the fact that course requirements didn t ensure that students received adequate preparation in all of the areas of the comprehensive examination undoubtedly contributed to the fact that roughly one-third of program graduates failed to pass the comprehensive examination within the intended two-year time to complete the degree program. Beginning with the Graduate Catalog, however, the department instituted area requirements. Within the 30 credit hours required for the M.A., students are now required to take 6 credit hours (two courses) within each of the areas covered by the sections of the comprehensive examination. Although intended to better prepare students for the comprehensive examination, these area requirements have now taken over the role of gatekeeper to ensure breadth of knowledge of the discipline. As a result, the comprehensive examination no longer needs to perform the function of ensuring breadth, which frees it to perform a different function within the M.A. program. Moreover, a review of degree requirements at other graduate programs in philosophy in the U.S. revealed that the multi-section comprehensive examination, which used to be a staple of graduate education in philosophy, is no longer required by nationally ranked programs. None of the other nationally ranked master s programs in philosophy, which form NIU s peer group, requires a multi-section examination covering all major areas of philosophy, and no top-50 doctoral program in philosophy requires such an examination. The discipline as a whole has clearly moved away from this sort of graduate degree requirement. However, many of the nationally ranked master s and doctoral programs do require a written examination in a student s chosen area of strength. Given all of the above, it made sense to scale back the scope of the comprehensive examination. Accordingly, effective in Fall 2014, the department changed the structure of its comprehensive examination from a multi-section examination covering multiple areas of philosophy to a single examination covering a single area of philosophy. The comprehensive examination is now a four-hour, three-question written examination in one of the following areas, as chosen by the student: 1. Ethics and Value Theory 2. Metaphysics Status Report 8

9 3. Epistemology 4. Philosophy of Science 5. Philosophy of Language 6. Philosophy of Mind Students are required to take the examination in the fall semester of the second year of full-time enrollment. In addition, at the end of each spring semester, a master list of ten questions for each examination is distributed to the graduate students. The fall and spring offerings of each examination the following year consist of three questions drawn from this list. Each master list of questions thus indicates departmental expectations of student learning in each area, and students have all summer between the first and second year in the program to study for the examination to be administered in the fall of the second year. Thus, rather than interpreting comprehensive to apply to the discipline as a whole, it becomes interpreted as applying to a sub-discipline within philosophy in which a student has particular strength and comprehensive knowledge. (The comprehensive examination in the Department of History is conceptualized in precisely this way.) With this new comprehensive examination requirement, students are encouraged to acquire and demonstrate particularly indepth knowledge of one area of philosophy in which they specialize. Moreover, with this new examination requirement, our program falls into line with the industry standard in graduate education in philosophy. By the time of the next program review, the department will have some data indicating whether this change to the comprehensive examination requirement is successful in increasing the percentage of graduates who complete the program within the desired two years. Changes to the Assessment Plan Of course, given that all M.A. student learning outcomes except those in symbolic logic were assessed through the comprehensive examination, the above change to the comprehensive examination requires a reconceptualization of the department s assessment plan. Student learning in the major areas of contemporary analytic philosophy can no longer be assessed via the comprehensive examination. Conversely, the new structure of the comprehensive examination is designed to assess different student learning outcomes than those assessed by the previous comprehensive examination. Finally, review of the analysis of learning outcome 1 above shows that assessment of student achievement in analytical writing needs to be decoupled from the comprehensive examination and embedded in programmatic checkpoints at which students submit substantive and sustained essays. Over the past year-and-a-half, the department undertook to overhaul the M.A. assessment plan to meet the four desiderata articulated above. The new assessment plan is best appreciated in the context of the new set of M.A. requirements, which are as follows: Status Report 9

10 Each student must pass a four-hour comprehensive examination in one of the following fields, as chosen by the student: 1. Epistemology 2. Ethics and Political Philosophy 3. Metaphysics 4. Philosophy of Language 5. Philosophy of Mind 6. Philosophy of Science Each student is allowed two opportunities to pass the comprehensive examination. In addition to the comprehensive examination, each student must complete 30 semester hours of graduate course work, at least 24 of which must be in philosophy. For students electing to write a thesis, the course work in philosophy will include 6 semester hours of PHIL 699, Thesis. All courses taken toward the completion of the degree are subject to the approval of the graduate adviser, and they must include: PHIL 505, Intermediate Logic, with a grade of B or better. Two courses in each of the following areas (18): Metaphysics and Epistemology (6) PHIL 510, Topics in Metaphysics or Epistemology (3) PHIL 570, Topics in Philosophy of Religion (3) PHIL 611, Epistemology (3) PHIL 612, Metaphysics (3) Ethics and Value Theory (6) PHIL 530, Topics in Ethics (3) PHIL 542, Theories of Value (3) PHIL 550, Topics in Social and Political Philosophy (3) PHIL 631, Ethical Theory (3) PHIL 642, Aesthetics (3) PHIL 651, Social and Political Philosophy (3) Philosophy of Science, Language, and Mind (6) PHIL 502, Philosophy of Logic (3) PHIL 504, Philosophy of Language (3) PHIL 561, Metaphysics of Science (3) PHIL 564, Philosophy of Physics (3) PHIL 602, Topics in Philosophy of Logic (3) PHIL 604, Topics in Philosophy of Language (3) Status Report 10

11 PHIL 660, Philosophy of Science (3) PHIL 663, Philosophy of Mind (3) One course in the following area (3): History of Philosophy (3) PHIL 520, Topics in the History of Philosophy (3) PHIL 521, Major Philosophers (3) PHIL 523, Medieval Philosophy (3) PHIL th Century Philosophy (3) PHIL 528, 20th Century Phenomenology (3) PHIL 529, 20th Century Analytic Philosophy (3) PHIL 582, American Philosophy (3) Electives (6) The requirements listed above are in service of the following new set of student learning outcomes (where the boldface type indicates what is intended to be the public face of the learning outcomes): Each graduate of the M.A. program will demonstrate: 1. Proficiency in formal logic by completing assignments in which they: a. Identify the metalogical relationships among the concepts of validity, consistency, logical truth, and logical equivalence b. Accurately perform truth-functional computations to identify logical properties of formulas in propositional logic c. Symbolize the logical form of English sentences expressing truth-functional compounds, monadic quantification, polyadic quantification with and without identity, and numerical quantity d. Construct derivations (by natural deduction) to prove validity in propositional logic, logical truth in propositional logic, validity in first-order predicate logic, and validity in first-order predicate logic with identity e. Construct truth trees to identify the logical properties of formulas and sets of formulas in propositional logic, first-order predicate logic, and first-order predicate logic with identity f. Construct interpretations to identify logical properties of formulas and sets of formulas in first-order predicate logic and first-order predicate logic with identity 2. Proficiency in philosophical writing by composing essays that: a. Provide strong and cogent arguments in support of philosophical theses b. Critically engage opposing views and arguments c. Display independent or creative thought Status Report 11

12 3. Knowledge of metaphysics and epistemology by writing essays that: a. Demonstrate knowledge of an appropriate body of relevant literature in metaphysics and epistemology b. Provide clear, accurate, and thorough explanations of the philosophical views discussed 4. Knowledge of ethics and social/political philosophy by writing essays that: a. Demonstrate knowledge of an appropriate body of relevant literature in ethics and social/political philosophy b. Provide clear, accurate, and thorough explanations of the philosophical views discussed 5. Knowledge of the related areas of philosophy of science, philosophy of language, and philosophy of mind by writing essays that: a. Demonstrate knowledge of an appropriate body of relevant literature in the philosophies of science, language, and mind b. Provide clear, accurate, and thorough explanations of the philosophical views discussed 6. Knowledge of a period, movement, or major figure in the history of philosophy by writing essays that: a. Demonstrate knowledge of an appropriate body of relevant literature of the period, movement, or figure b. Provide clear, accurate, and thorough explanations of the philosophical views discussed 7. In-depth knowledge of one major area of contemporary philosophy by writing comprehensive examination essays on a diverse range of topics in the area that: a. Demonstrate understanding of the central philosophical issues in the area and their significance b. Demonstrate mastery of the important philosophical views and arguments concerning these issues and how they relate to one another c. Identify proponents (authors and their works) of these views and arguments Learning outcome 7 is assessed via the new comprehensive examination, via a universal rubric that can be used by faculty no matter the examination area chosen by the student. Learning outcomes 2-5 are slightly modified versions of the learning outcomes that used to be assessed via the (old) comprehensive examination. Assessment of each of these outcomes is now embedded in coursework. Each term paper (culminating essay) that a student completes in a course taken in fulfillment of an area requirement will be scored with a universal rubric (applicable no matter the subject matter of the course) consisting of five performance criteria. Scores for three of these performance criteria will provide data regarding learning outcome 2 (proficiency in philosophical writing), and scores on the remaining two performance criteria Status Report 12

13 will provide data regarding student learning in the content area of the course (hence feeding outcome 3 for courses in metaphysics and epistemology, feeding outcome 4 for courses in ethics and political philosophy, and so on). Embedding assessment of writing and content mastery in coursework in this way will ensure higher quality data regarding student learning than had been acquired through assessment of student work on the previous comprehensive examination. Finally, given the new mechanism of coursework-embedded assessment of writing and content mastery, it becomes easy to acquire data regarding student learning in the remaining required area of coursework, history of philosophy (learning outcome 6). Despite this significant redesign of the assessment plan, virtually all of the legacy data gathered under the prior assessment plan can be funneled into the appropriate outcomes in the new assessment plan, thereby providing continuity of assessment of learning for current students in the program. Improving Compliance with Reporting Mandates Review of the analyses reported above also reveal a gap in data concerning student learning in logic, indicating that greater effort must be made to ensure faculty compliance with conducting assessment in PHIL 505, Intermediate Logic, and reporting the results to the department. During the period reported above, the assessment rubric for PHIL 505 was distributed each fall to the instructor of the course by the department s Office Support Specialist, with a request to complete the rubric with the required data and return it to the department at the end of the semester. Beginning in Fall 2014, the rubric will be distributed to the instructor of PHIL 505 by the department chair. The chair will provide detailed instructions about how to gather and report requested data, and the chair will follow up with reminders throughout the semester and an end-of-semester request to return the completed rubrics. Hopefully having the request come directly from the department chair will increase faculty compliance with reporting. 4. APPENDICES (ASSESSMENTS) The appendices to follow are the rubrics used to gather the data reported in section 2. Status Report 13

14 Appendices The Comprehensive Examination Rubric on the next page was designed as a universal rubric, useable no matter the subject section of the comprehensive examination being scored. Items 3-6 on the rubric provide data regarding learning outcome 1, analytical essay writing. Items 1-2 on the rubric provide data regarding content mastery in whatever the subject area of the section of the examination being scored. If a grader were scoring an examination in Metaphysics and Epistemology, scores on items 1 and 2 would provide data for learning outcome 3. If a grader were scoring an examination in Ethics and Metaethics, scores on items 1 and 2 would provide data for learning outcome 4. And so on.

15 Comprehensive Examination Rubric Exam: Semester: Faculty Name: Student ID Letter Demonstrates understanding of the central philosophical issues Demonstrates knowledge of the relevant literature, associating important philosophical views with their proponents and critics Provides accurate exposition of the philosophical views discussed Provides clear and thorough explanations of the philosophical views discussed Provides strong and cogent arguments in support of philosophical theses Critically engages opposing views and arguments GRADE Rubric Scores 0 = Does not meet expectation in any significant respect 1 = Partially meets expectation, but does not satisfy expectation for master's performance 2 = Meets expectation for master's-level performance Grades PH = Pass with Honors P = Pass F = Fail

16 Assessment Rubric PHIL 505, Intermediate Logic Student: Grader: Grade: 0 = Does not meet expectation in any significant respect 1 = Partially meets expectation, but does not satisfy expectation for master s-level performance 2 = Meets expectation for master s-level performance 1. Clearly and accurately defines the fundamental logical concepts of a. validity b. consistency c. necessity d. logical equivalence 2. Demonstrates a precise knowledge of the implications of fundamental logical concepts and their logical relationships to one another 3. Accurately performs truth-functional computations to test for a. validity b. consistency c. necessary truth d. logical equivalence

17 Logic Rubric/2 4. Accurately renders English sentences into logical symbolism, including English sentences expressing a. necessary or sufficient conditions b. limited quantity ( at least n and at most n ) c. precise quantity ( exactly n and the x ) 5. Correctly constructs proofs of a. validity b. inconsistency c. necessary truth d. logical equivalence 6. Correctly constructs interpretations to prove a. invalidity b. consistency