Undergraduate Degree Program Assessment Progress Report Cover Sheet

Transcription

1 Degree: BA in Political Science For Calendar Year: 2014 (Date submitted to college committee:) 2/23/2015 (Date posted on college assessment website:) By: Joseph Giammo, Rebecca Glazier, and Eric Wiebelhaus-Brahm Overall Rating: Respond to all six parts following the Degree Program Assessment Progress Report Instructions. Attach additional pages as needed. (NOTE: Parts 1 through 4 can be copied from the relevant sections of your assessment plan.) (1) Student learning goal(s) addressed this year: The mission of the Political Science Department is to provide students with both a substantive knowledge of politics and the skills necessary to make use of that knowledge effectively, not only as private citizens participating in a democracy, but also in a wide range of careers. The department has implemented a program of assessment to measure achievement of these objectives. Students in the department are exposed to both the theory and practice of politics. The department s faculty covers a broad range of substantive specializations, encompassing American politics, comparative politics, international relations, and political theory. Courses in the department also strive to instill in students a number of important skills, including critical thinking, analysis, writing, and the practical application of ideas to real world situations. The combination of substantive knowledge and practical skills instilled by courses in the department provide a solid background to help our students excel as leaders in a variety of fields, including politics, government service, law, and teaching. Our commitment to helping students develop transferable skills is seen in the field experience requirement for political science majors. Based on our mission, our learning goals match the following core competencies, which are stated in parentheses, as follows: Knowledge Goals Students majoring in political science will be exposed to a wide range of substantive information about the world of politics, covering subjects such as: 1) The history, characteristics, and impact of political institutions and systems, including international, national, state, and local governments (Historical PVC 2/07-1 -

2 Consciousness, Social and Cultural Awareness, International Awareness) 2) The behavior of individual actors at both the mass and elite levels (Social and Cultural Awareness, International Consciousness) 3) The processes involved in creating government policy and the impact of those policies (Social and Cultural Awareness, International Consciousness) 4) The debates over the normative foundations of the state, the scope of its activities, and the nature of justice (Ethical and Moral Consciousness, Critical Thinking) Skill Goals Students majoring in political science will develop practical and marketable skills which will help them to apply that knowledge to any number of career paths. These skills include: 5) Critical thinking the ability to read and evaluate primary texts, ideas, theories, and political analysis (Critical Thinking, Verbal Literacy) 6) Analysis/Research the ability to pose and answer questions effectively (Critical Thinking, Philosophy and Methods of Science, Information Technology) 7) Argumentation/Persuasion the ability to use logic and evidence to build a persuasive argument (Critical Thinking, Verbal Literacy, Ethical and Moral Consciousness, Information Technology) 8) Written and Verbal Communication the ability to communicate ideas clearly on paper or through oral presentations (Verbal Literacy, Information Technology) 9) Working in Groups the ability to effectively coordinate with others on a project (Critical Thinking, Ethical and Moral Consciousness) 10) Practical Applications the ability to take information gained through coursework and use it effectively in real world or simulated exercises (Critical Thinking, Verbal Literacy, Ethical and Moral Consciousness, Information Technology) (2) Learning outcomes/objectives for those goals addressed this year: How the specific learning objectives are implemented in the curriculum can be found by examining the syllabi of courses regularly taught by our faculty. These course syllabi are made available to the students at the start of the course, and clearly articulate which goals will be emphasized in the class. Sample syllabi are posted online as well and available on the department s website. As a means of assessing these goals, on course evaluation forms, students are asked to PVC 2/07-2 -

3 evaluate which of the above knowledge and skills goals were adequately addressed in the course. Faculty members are invited to discuss during annual review which goals were targeted in the most recent iteration of the course. Each course taught in the department is expected to clearly target at least 3 of the above goals, although many if not all routinely aim for 5 or 6 goals or more. Specific learning outcomes associated with each of these more general knowledge and skills goals will vary by course. For example, the most recent iteration of POLS 3320 American Presidency lists the following outcomes before moving on to a grid that matches course assignments with each goal and outcome: "Learning Goals, Assignments, Points, and Interaction Patterns," located in the syllabus section of Blackboard. Our learning goals are Goal #1: Students will know and understand the structure of the presidency and how this branch relates to the Congress and Federal Courts, along with how US policies relate to international political institutions and systems. (Knowledge 1) Goal #2: Students will master structural and cultural constraints and motivations of presidents, staff and advisors and how presidents connect to counterpart leaders in Congress and among mass and elite organizations, nationally and internationally. (Knowledge 2) Goal #3: Students will know and understand presidential decision processes for domestic and foreign/national security policy and also the impacts of those policies. (Knowledge 3) Goal #4: Students will demonstrate the ability to read and evaluate primary texts, ideas, theories and political analysis relating to the presidency. (Skill 5) Goal #5: Students will demonstrate the ability to communicate ideas clearly in writing in a variety of formats: blog postings, discussion postings and responses, short answer/discussion questions and in formal essays. (Skill 8) Goal #6: Promote Student Success through building online community, PVC 2/07-3 -

4 interactions with students, with coach and with the professor. Connect students to Blackboard support and verify in first week that students can use BB tools effectively. (Skill 9) This is an example of a syllabus in the department that is state of the art in terms of spelling out such objectives and outcomes. Not all syllabi are currently structured with such detail. A summary of the objectives and outcomes listed in political science syllabi--along with data on which objectives students see as met by our courses--is presented below. (3) Courses & activities where assessed: All courses offered in political science were included in the assessment. (4) Methods used: 1. Annual Review The annual review serves as the primary point of discussion of departmental assessment and for planning and assessing the results of curricular changes. Information gathered in the course of the annual review process is compiled and analyzed by the Departmental Assessment Committee to generate the annual assessment report. The Departmental Assessment Committee typically consists of three members of the department, under the leadership of the Departmental Assessment Leader. It is the duty of individual members of the Department to provide the meaningful data for the Annual Assessment Report, while the Assessment Committee drafts the report which is subsequently reviewed, discussed and approved by the entire Department council. During the review, we look at the spectrum of the materials provided by the faculty: syllabi, class grades, class materials, peer review reports, student evaluations, etc. Each faculty member reports with respect to the knowledge and skills goals and objectives that they teach in each class. Each faculty member discusses the range of evidence with respect to the actual results of student learning, most often through the presentation of question-specific and class averaged student evaluation scores, through course-specific narratives included as part of the annual review dossier, and by discussion of what worked, what did not, and what might need to be revised for the next iteration of the course. More specifically, the individual faculty provide the Assessment Committee with the following artifacts as components of the annual review dossier, which can be accessed by all members of the department for the week prior to, and following, the annual review meetings: Class syllabi and the report on how the learning objectives are embedded or operationalized in the class design; and Analysis of evidence regarding whether the objectives were met in the course of PVC 2/07-4 -

5 a particular class including: Course grade distribution; Assignments, tests, and activities; Course evaluations; and Other supporting material regarding the quality of class experience, including peer review, class surveys, student testimonies, etc. 2. Online Survey of Majors Since 2010, the department has implemented an online survey of majors for inclusion in the Political Science Program Assessment Report. The purpose of this survey is to access information about how well the department is doing teaching its core knowledge and skills goals and to learn more about whether, and to what degree, such goals are valued by our current majors and seen as useful to them in their careers. In 2013, we added a battery of questions targeting the delivery of advising in the department. Unfortunately, personnel changes prevented us from delivering the survey in 2014 as Dr. Andrew Drummond, who was responsible for the survey, was appointed associate dean of CSSC. The Committee feels the survey is valuable and plans to continue it in the future. The online survey tap attitudes toward the department s focus upon our stated learning goals using a series of likert scale questions asking respondents to indicate whether they strongly agree, mostly agree, neither agree, nor disagree, mostly disagree, or strongly disagree with specific statements about the department s success in fostering a chosen goal. In addition to these questions, respondents are offered multiple opportunities for open-ended responses, which are also tracked and reported. 3. Peer Classroom Faculty Visits Peer classroom faculty visits are performed on a volunteer basis for tenure track faculty, and on an obligatory basis for the adjunct taught-classes. The reports from peer visits are included in the assessment materials, taking the form of free response, open ended letters written from the reviewer to the instructor. The peer review process now extends to all faculty members in the department and has worked very well. Written record of peer reviews is based on a simple outline reporting the strengths, issues, and overall evaluation of the observed classes. 4. New Classroom Evaluation Forms The department met in November 2010 to discuss the creation of new course evaluation forms which were used for the first time in The new forms align questions with the curricular learning goals expressed in the department s mission. The results of the new evaluation procedures are included in this report. PVC 2/07-5 -

6 (5) What are the assessment findings? How did you analyze them? 1. Results of Departmental Annual Review The Political Science Department held its annual review meeting on Friday, February 6, As part of this process, all department members reviewed one anothers teaching portfolios, including syllabi, class grades, class materials, peer review reports, and student evaluations. During the meeting, each faculty member discussed the courses s/he taught in the previous year. This process allows the department to stay updated on the teaching approaches and innovations of its faculty and to offer constructive feedback. This review process illustrates just how committed the faculty in the department are to delivering a quality educational experience. One way to measure their success is through student evaluations. The evaluations indicate that students recognize and appreciate our faculty s commitment to teaching. The vast majority of all measures on these evaluations across all faculty in the department fell in the upper quintile of the 5 point measures we use to determine effectiveness on each particular measure. The 2014 annual review revealed a number of innovative teaching strategies used by the political science faculty, who have made a deep commitment to developing as teachers. For instance, Joe Giammo revised the International Studies senior seminar course to include more one-on-one meetings with students. This helped keep students accountable and on track with their research projects. Peggy Scranton implemented two new teaching strategies in her online classes. She switched from using a traditional textbook in Political Science 1310 to using an online publisher portal, Mindtap, which included multiple quizzes. As a result, she saw her retention numbers increase by nearly 20%. Peggy also used an automated grading program for writing assignments in POLS 1310, which was less successful. As a result, she will not be using it again. Our department s support of teaching innovation makes such experiments--even unsuccessful ones--possible. Rebecca Glazier is currently in the middle of a multi-year online teaching experiment to gauge the impact of rapport-building. She taught a travel course in the spring of 2014 and a research practicum course in the fall of 2014, both of which had experiential learning components. Eric Wiebelhaus-Brahm s International Relations course included an assignment where students applied the IR theories they learned to the fictitious events in the novel World War Z. Greg Schufeldt, our newest faculty member, joined the department in the fall of Greg s commitment to teaching fits well with our department culture. He included midterm evaluations in his fall courses and is preparing an online teaching experiment to determine how small group discussions impact participation. Additionally, faculty in the department are engaged with continuing education in their craft, such as attending or delivering material in ATLE and online faculty workshops, publishing in political science education journals, innovating in assessment and curricular reforms, taking part in peer evaluations, and revising simulations and other high impact experiential learning modules. In 2014, Rebecca published two pedagogical articles in peer-reviewed journals. These and other aspects of the department s commitment to teaching and student success are even more apparent when reading each faculty member s narrative about their past year in teaching, and their debriefings on each of their courses. 2. Results of Peer Visits PVC 2/07-6 -

7 The department began peer classroom visits in 2008, and these visits continued again in Each of the faculty took part in them again, as has been the case in the past, even though the program is completely voluntary. The peer visit program continues as an integral part of the department s culture of commitment to teaching and assessment. These peer reviews help faculty members see alternative approaches to teaching and help them get a fresh pair of eyes on their own teaching. In one peer-review, the reviewer recommended that a simulation be shifted from a student-professor interaction to a student-student interaction. Another peer review, this one of an online class, suggested some restructuring to make course navigation easier. This individual feedback is invaluable as we strive to improve our classes. 3) Results of Student Evaluations: Learning Objectives and Student Success When completing their evaluations, students are asked to identify which, if any, of the listed skills goals were met in the particular course. We can then compare the goals that students identified as having been met with those listed on the courses syllabi. This is the second time that the department has analyzed that data as part of the assessment report. The findings indicate that a significant majority of students believe that these goals have been met in their courses. For five of the eight goals, over 50% of all students completing evaluations in all courses felt that that goal had been met in both the spring and fall semesters, regardless of whether or not that goal was identified on the syllabus for that course. Critical thinking, which was the goal that most frequently appeared on course syllabi in the program, was the goal that students most frequently identified as having been met in their courses, with 79.2% of students choosing it in the spring and 73.5% choosing it in the fall. The three which had the lowest average (ethical and moral consciousness, working in groups, and information technology skills) were also the three that were least often identified on the syllabi. The average for all eight goals was remarkably consistent between the fall and the spring. When all answers were considered, regardless of the goals listed on course syllabi, students indicated that 55.4% of all possible skills goals had been met in their classes in the spring, compared to 55.7% in the fall. When only the choices which were actually listed on the course syllabi were included in the analysis, those improved to 71.8% and 71.4%, respectively. Given that some of remaining students were almost certainly simply skipping the question, rather than expressing an opinion that the goals had not been met, these are very high numbers and suggest a strong correlation between the course goals and the students perceptions of the skills which they are learning in their coursework in the department. PVC 2/07-7 -

8 Spring Mean Fall Mean Critical Thinking 79.2% 73.5% Analytical and Research Skills Argumentation and Persuasion Working in Groups Ethical and Moral Consciousness IT Skills Written and Verbal Communication Practical Application of Ideas Overall Mean Overall Mean Including Only Goals Identified on the Syllabi In addition to the skills goals analysis, the Assessment Committee also conducted a thorough examination of the responses given to a question on the evaluation form meant to tap student success concerns in each course. Q: What would have enabled you to be more successful in this class this semester? Background: At the department meeting where we formulated this question to add to our student evaluation forms, we deliberated for some time over wording. Our intention was to capture the reasons our students would cite about student success. We did not want to prompt students with examples of barriers to success that might influence their open-ended responses. Analysis: The first thing that stands out from the analysis is that a large percentage of students refrained from answering the question. In fact, in the spring semester only 40.3% did provide an answer to the question. The response rate was better in the fall, but even then was just barely above 50%. The low response rate was common across most of the department s course offerings: a majority of students provided an answer to the question in only a third of the courses in the spring and 46.2% of the courses in the fall. Spring Classes Fall Classes Percent of Students Providing Comments 40.3% 52.7% Percent of Classes with 50% or More of Students Providing Comments When students did answer the question, the majority of the responses were not about the professor or the course (although such answers comprised a significant plurality in the spring). Each semester, however 50% or more of the total comments were either about the student and/or the student s circumstances or were expressing satisfaction with the class and their level of success in it. PVC 2/07-8 -

9 Spring Classes Fall Classes Percent of Comments about the Professor or the Class Percent of Comments about the Student or Student's Circumstances Percent of Comments Expressing Satisfaction or Saying Nothing Else Was Needed Note: Some comments did not fit the three relevant categories (about the professor/class, about the student or student s circumstances, or expressing satisfaction with the class), so the percentages in the three categories do not add up to 100% in either semester. Overall, a sizeable minority of students wrote something in response to this question. Only about 17% of all students who turned in an evaluation, however, provided an answer that talked about either the professor or the class. By comparison, about 14% provided an answer that expressed satisfaction with the class and about 13% pointed to something about themselves, instead. The number of responses, however, is not as important as the content of those responses. Below are some typical response in each of the categories identified above: Comments about the Professor or the Class: Mandatory attendance Maybe having a quiz between tests to gain more points for a better grade. This may sound odd, but if the notes were not written on the board I could pay attention better, I felt like I was always playing catch up. If there were more hands-on and class discussions in class instead of lecture. More quizzes throughout the semester instead of 1 big test over everything. More in class time for major project work. Having more time in class to discuss. Better study notes. More specific study materials. If there were more quizzes over each chapter instead of having a big test. Perhaps more thorough review sheets for exams or pop quizzes to discourage slacking. Longer due dates/extensions for some assignments. Longer time to turn in extra credit. More homework over the material. Comments about the Student of the Student s Circumstances: If I would have studied more and did not take on a job Having more time to focus on the class. More studying Spending more time on reading and having a balanced schedule would have enabled me to be more successful in this class. Better study habits and an interest in the subject. Instead of staying quiet most of the time, actually engage in the discussion. Better attendance on my part. The more you show up the more you learn. PVC 2/07-9 -

10 Studying ahead of time. Doing required items like homework and readings. Having more time to study for exams. If I had been able to wake up on time to make it to every class! Not working 50 hours a week so I can afford a tutor. Better note taking and repetitive study habits. Read more along with taking good notes. Better health and a better memory. Comments Expressing Satisfaction or Saying Nothing Else Was Needed: Nothing. I feel quite enabled Not a thing. Nothing. [Instructor] did very well at facilitating the course. [Instructor] gave us all the tools needed to succeed. Nothing comes to mind. Overall a well balanced course. Nothing. Everything was well presented. This course has been fantastic. Nothing. Everything was amazing! Overall, I like the way the class is taught. There are many resources available and [Instructor] always encourages us to use them. Nothing. I was thoroughly engaged and enabled. Nothing more than what I got... an excellent instructor! Looking at the responses, a few patterns emerge. Those who talked about the instructor or the class were most commonly asking for either additional grades (homework assignments, quizzes, additional tests, etc.) or for more resources (study guides, additional time, etc.). As noted in last year s report, this suggests that some noticeable minority of students have an expectation that the class will be designed to fit their own expectations, rather than that they will need to adapt to the expectations of the class. This does indicate that the department needs to place a greater emphasis on changing those expectations, particularly in lower level classes, when such changes should have the greatest effect. The comments about the student or the student s circumstances tended to fall into three categories, mentioning a lack of time due to other obligations (jobs, family, etc.), a lack of effort by the student (failure to read or devote enough time to studying), or a lack of necessary skills (most commonly note taking or study skills). The first set of issues are the most difficult to actually do anything about. The student body at UALR is likely to always be one for whom other obligations are a common and constant challenge. The latter two, however, do point to areas where the department could do things to assist, both by emphasizing early on how much effort will be necessary to succeed and by providing opportunities, either in classes or in separate workshops, for students to improve upon their basic educational skills. It is, however, encouraging, that a significant number of students took the time to write an answer to express their satisfaction with the way that things went in that class. (6) What conclusions were drawn and what decisions were made as a result? How were stakeholder groups involved? PVC 2/

11 There is an established culture of teaching and commitment to student success in the Political Science Department. Faculty are reflective about their teaching, and the environment is one that promotes the open discussion of teaching methods and strategies for improving student success. The Assessment Committee is attempting to harness this energy and fold assessment into the existing culture by tapping into the various discussions (peer classroom visits, annual reviews, etc.) and by building a plan that generates a diverse set of data that can be analyzed systematically each year. Plan revisions must be iterative, as each effort to include a new type of data often reveals issues with the quality of the data or its delivery to the Assessment Committee. We focus in our conclusion on the most important issues to address going forward. Most important issues to address in 2015: The Assessment Committee has identified the most pressing issues to resolve for next year s assessment process. First, we plan to reinstitute the online survey of political science majors in This has been a useful tool to gain additional insights from students about how well they perceive that we are doing. Second, while we now have several mechanisms for gauging how faculty and students perceive that things are going, we need to add a more direct measure of what students are learning. To that end, the Assessment Committee is recommending that the department adopt a long-term program to begin digitally collecting student papers from lower level classes (Introduction to Political Science and American National Government specifically), storing them, and then matching them up with papers from those same students during their senior year. The Assessment Committee would then review these paired papers to look for progress by students in the department s knowledge and skills goals. Finally, department faculty will be asked to consider implementing changes to their courses to reflect the priorities identified in section three above. PVC 2/