Teagle Grant Project: Psychology Department Beloit College Alexis Grosofsky & Gregory Buchanan

Transcription

1 Teagle Grant Project: Psychology Department Beloit College Alexis Grosofsky & Gregory Buchanan 1. Overview. The psychology department s participation in the Teagle Grant Project began in 2007, but our work in this area is predated by our own in-house assessment of our learning goals and a self-study of our curriculum in preparation for psychology s move to Beloit College s program. In all, this has produced a range of data across multiple semesters that we will discuss in this report. 2. Information Collected. As part of a departmental assessment, members of the psychology department discussed the learning goals of our major in view of the learning goals for psychology majors developed by the American Psychology Association. We then constructed a grid to indicate where in our courses each of our goals was being met. With the help of the Registrar, we conducted a transcript analysis of our majors over the last 5 years looking at supporting courses, sequencing of courses, choice of cluster course and choice of upper level psychology course. In 2008, 14 graduating senior psychology majors participated in the CLA, and with the help of Cynthia Gray, we have been able to include these results in our analysis Over the course of the last 18 months, we have distributed multiple surveys to our majors: o A senior exit survey (class of 2009), which asked students to evaluate their experience with the major in terms of what they learned, how the major prepared them for the outside world, and what they believe is lacking in the major. The survey also asked seniors how well they think courses both in and outside of the major helped them achieve a variety of skills o A senior exit survey (class of 2008), which asked students to evaluate the psychology curriculum in terms of when they took the courses they did and why, what they found to the be the most and least rewarding/effective aspects of the major, and how confident they feel using the skills taught in the major o A survey given to all majors, which asked them to define the criteria set forth by the o Teagle Project and where within the major they believe these criteria are taught. A survey conducted by FACS, which asked students about their experience with FACS and whether or not they completed an internship o An open-ended question and answer session with graduating seniors (class of 2008) asking them about the psychology curriculum, what we teach, how we teach it, and the major requirements. 3. Results and Analysis. A. Psychology Department Learning Goals and Course Grid: Members of the Psychology department created a list of goals and associated objectives a few years ago as part of the College s attempt to be more systematic about assessment (see appendix A for our goals). We then created a grid in which we listed each goal with its objectives and indicated where in the curriculum we saw these objectives were being covered. The table below indicates which departmental goals best link to the areas of the Teagle grant we re assessing. Teagle evaluation items

2 Learning Goals Students who complete a major in psychology at Beloit will: acquire a knowledge base in psychological science be able to evaluate claims and evidence be proficient in research methods and statistics critical thinking quantitative literacy acquire professional communication skills demonstrate information literacy exhibit sound values and judgment apply psychological knowledge effectively civic engagement While some of these objectives are covered in the context of other goals, we will discuss only those that are most relevant to the 3 outcomes this grant is assessing. Quantitative reasoning: This is reflected in our be proficient in research methods and statistics goal. The courses contributing most to this most are: Statistics, Research Methods & Design, Advanced Topics courses and Senior Thesis. Critical thinking: This is reflected in our be able to evaluate claims and evidence goal. The courses contributing most to this are: Intro, Research Methods & Design, History & Systems, Senior Seminar, Advanced Topics courses, and Senior Thesis along with most of our content courses. Civic engagement This outcome is the hardest to see in our curriculum. It is most reflected in our apply psychological knowledge effectively goal. The courses that have the strongest contribution are Intro, Research Methods & Design, History & Systems, Senior Seminar, and Advanced Topics. Obviously internships are relevant to this outcome. The pattern of coverage is encouraging. These desired outcomes are addressed in our core courses the courses required of all majors. These courses, Introductory Psychology, Statistics, Research Methods & Design, History & Systems, and Advanced Topics span all levels of the curriculum. Thus, our students get repeated exposure to these topics during their years at Beloit. B. Student Definitions of Teagle Criteria: In a survey sent out to all psychology majors (See Appendix B), our students were asked to define Quantitative Reasoning, Critical Thinking, and Civic Engagement and to indicate where within the curriculum they felt these goals were being met. Psychology students correctly identify that quantitative reasoning involves thinking numerically and using numbers to solve problems. The majority of respondents also mentioned statistical significance and that it could be used to help make decisions, and support/refute hypotheses etc. That is, they recognize that it is more than just comparing numbers at face value. A smaller number of students identified that quantitative reasoning forms a part of logical thinking (figuring something out in sequence and in an unbiased fashion) and that quantitative reasoning can involve multiple methods. As expected, the majority of students indicated that it was in Statistics that they learned quantitative reasoning and about half also identified Research Methods. About 1/3 of respondents indicated that they learned quantitative reasoning skills across the curriculum because of the way the curriculum is

3 set up with interconnected classes, or that it comes up in any course that involves reading journal articles or doing research projects. In equal numbers, students identified three components of critical thinking: i) thinking deeply/intently about a problem; thinking outside of the box or more than just face value; ii) using multiple perspectives/tools, being interdisciplinary, and extrapolating conclusions; and iii) solving a problem logically, being objective, systematic and unbiased in one s approach. Students were less precise as to where they believed they were learning this approach. About half of them cited in-class assignments, but none referred to any specific assignments or classes, though both writing and discussing were cited. The remainder of the students identified individual classes as being sources of information/skills in critical thinking. Statistics and Research Methods were cited by ¼ of the respondents and 11 other courses were cited by at least one student. Given the very clear way in which students defined this term, their response to the second half of the survey is a bit disappointing. We suggest that perhaps it was unclear what we were asking and maybe students should be prompted to provide more specific examples of assignments that teach critical thinking (perhaps now with the prompts provided by their excellent definitions). Students had no difficulty defining civic engagement as doing something outside of the classroom that served some good purpose. Most commonly this was reaching out to the community at large through volunteering, service learning (internships) or sharing information. A few students defined community as also including the college community and one put it nicely as how the Psychology Department engages with the community and other departments. Four students indicated that civic engagement was a two-way street and that they could also learn from the community; and three students sharpened their definitions to include skills learned as psychology students. That is, civic engagement involves helping the community while utilizing a skill set or knowledge base specific to psychology majors (for example, giving a presentation about eating disorders at a local school). Students were split in terms of identifying where civic engagement is placed in our curriculum. About 1/3 cited volunteering or internships as the main vehicles for civic engagement for psychology majors, while almost ½ identified specific classes (Research Methods by 7 and Child Growth and Development by 4). Many students also gave their opinion on the need for civic engagement or the difficulty of finding internships or populations for running studies. These results are curious in a number of ways. First, our department has discussed having a service learning requirement (internship) as part of the major and we have always balked at the idea, largely because of the difficulty involved in finding internships during the school year. For the most part then, students who are getting their dose of civic engagement through internships are doing so outside of the major. Second, the type of engagement in Research Methods is identified by some students in their definition of engagement, though it doesn t fit with the fully endorsed definition (doing good). That is, students may be engaged with the community when they collect data for Research Methods or Advanced Research Seminar, but the benefit here is generally weighted towards them and not the community (one solution here is symposium or workshops etc. though less appropriate for Research Methods students). Third, unprompted responses about the need and importance of civic engagement means i) we might want to mine deeper on this question (what would students write if all were prompted) and ii) what can we do to increase civic engagement in psych (or is it enough to assume it s coming in elsewhere in the college?). In concluding this section, it is apparent that student definitions of quantitative reasoning, critical thinking, and civic engagement parallel the definitions of our faculty, but in some places fall short. Compared to faculty, students tend to emphasize the numeric aspects of quantitative reasoning ( doing sums ) as opposed to the problem solving utility of these numbers ( testing hypotheses, establishing cause and effect ). This important distinction is discussed in a later section.

4 C. Transcript Analysis: While the transcript analysis didn t really tell us anything new, it did give us the data that confirmed our suspicions. Things we confirmed via the transcript analysis included the following three observations: The average lag between students taking statistics and research methods is only half a semester. We re happy about this because it gives evidence that shows students are following our advice to take Research Methods & Design immediately after taking Statistics. The average number of Division 1 (Natural Sciences and Mathematics) units is very low. Students are required to take two to graduate and, with very few exceptions, that s all they take. We d like to see our students take more classes from the Natural Sciences and Mathematics division. One of the reasons we want to stay classified as a Social Science is that if we were to become a Natural Science then our students would take even fewer Natural Science classes outside the major. Student selections among our advanced topics and cluster courses tend to be fairly evenly distributed. The relatively even distribution reflects our requirement that each student take one course from each of our clusters. The distribution also reflects the fact that we try to rotate through our four clusters evenly. The areas most likely to be doubled up on are Clinical, Developmental, and to a lesser extent, Social. This follows student interest and reflects an avoidance of the experimental cluster which may be related to students avoiding Natural Science & Mathematics courses. Almost a quarter (22%) of our students double major which is in line with the college average (approximately 25%), while just over a quarter (28%) have a minor. The most popular majors to combine with Psychology are (alphabetically) Dance, Modern Languages, Philosophy, and Sociology. With the exception of Sociology (also a Social Science) these are all Arts & Humanities disciplines. This pattern reflects our students avoidance of the Natural Sciences & Mathematics. D. Senior Exit Questionnaires and Q & A: Graduating seniors from the classes of 2008 and 2009 completed two different surveys (see Appendices C & D). Given the similar nature of these surveys, the responses have been combined in this section. The most valuable and effective aspects of the major identified by students were the opportunity to conduct research, collaborating with faculty on research projects, and learning to think critically. Other positives endorsed by a smaller number of students were learning to read critically, doing class presentations (virtually ubiquitous in psychology classes), writing empirical papers, and learning APA format. Those students who completed research projects felt that they were well-prepared for the experience because of coursework that emphasized how to approach problems, conduct observations, and ask appropriate questions. On the negative side, students felt they needed more experience (mastery) with statistics, more integration of coursework across the discipline, and more opportunities for research and internships. Various surveys, including those given by FACS, indicate that about 50% of our majors complete an internship, while a smaller number (perhaps 20%) complete a research project other than the required project in Research Methods. When asked what they felt they had gained as the result of being a psychology major, the most common responses were: ability to look at a problem from different perspectives coupled with depth and breadth of knowledge in the discipline; the ability to read and think critically and scientifically; an understanding and appreciation of research; and a better understanding of themselves and others. All told these results are quite encouraging. Given the scientific nature of our discipline, we are pleased to see that our graduates are identifying these aspects of the discipline (research, critical thinking, reading, and writing) as being the most valuable. In light of the transcript analysis, however, these findings are also troubling. If the scientific aspects of

5 our discipline are the most important, and being able to view a problem from multiple perspectives is a plus, we are concerned that our students still avoid the natural sciences when there are obvious cognates between psychology and biology, chemistry and health sciences. Seniors in the class of 2009 were also asked where in the curriculum they were learning a variety of skills related to our department s learning goals outlined in paragraph A. These skills were: effective written and oral communication, logical thinking, quantitative reasoning, information literacy, problemsolving, judgment, and the ability to read critically. Overall, students indicated that they are being taught these skills across the curriculum i.e. in psychology courses, other courses, and co-curricular activities. The largest difference to emerge was for quantitative reasoning the majority of our majors are learning this skill in psychology classes (presumably Statistics and Research Methods) with less exposure to quantitative reasoning in other classes. This result is predictable, given the small number of natural science and mathematics classes that our students take as such it is also problematic given that these same students indicated that they need more exposure to quantitative methods. E. CLA Results. Some of the students who were psychology majors in the Spring 2007 and Spring 2008 semesters volunteered to take part in the College s pilot program with the Collegiate Learning Assessment (CLA). The volunteers were randomly selected to participate in one of the two sections of the CLA: the Performance Task or the Analytic Writing Task. According to the CLA Institutional Report, the Performance Task: requires students to use an integrated set of critical thinking, analytic reasoning, problem solving, and written communication skills to answer several open-ended questions about a hypothetical but realistic situation. In addition to directions and questions, each Performance Task also has its own document library that includes a range of information sources, such as letters, memos, summaries of research reports, newspaper articles, maps, photographs, diagrams, tables, charts, and interview notes or transcripts. Students are instructed to use these materials in preparing their answers to the Performance Task s questions within the allotted 90 minutes. The Analytic Writing Task requires students to: write answers to two types of essay prompts, namely: a Make-an-Argument question that asks them to support or reject a position on some issue; and a Critique-an-Argument question that asks them to evaluate the validity of an argument made by someone else. Both of these tasks measure a student s ability to articulate complex ideas, examine claims and evidence, support ideas with relevant reasons and examples, sustain a coherent discussion, and use standard written English. Unfortunately, only 14 senior psychology majors participated. After being assigned to one of the two tasks, we had nine majors take the Performance Task and only five take the Analytic Writing Task with only four of those five completing both subtasks. For what it s worth, the percentile ranks for the various tasks were: Performance Task (70 th percentile, N=9) and Analytic Writing overall (52 nd percentile, N=4). Analytic Writing subtasks: Makean-Argument (56 th percentile, N=5), Analytic Writing Break-an-Argument (49 th percentile, N=4). According to Cynthia Gray (Director of Institutional Research) Beloit students performed exceptionally well (among the best in the country) on the Performance Task, but only about average on the Analytic Writing task (and quite poorly on the Make-an-Argument portion specifically). Our results parallel those for the College.

6 This certainly gives us something to think about; however, given that it s based on only fourteen majors, we don t want to give it too much weight. F. Statistics and Research Methods. A consistent issue in our curriculum that predates our participation in the Teagle project is how we teach the quantitative and research aspects of the discipline. Through in-house discussions, an examination of how comparable psychology departments address this issue, and the ample feedback from students as part of this assessment, we believe we have a good grasp of what the problem is and we are currently pursuing various avenues of rectifying it. One of the most distressing things we hear on occasion is a student (worse, a major) saying that psychology is not a science. This perception of our discipline is nothing new (the early behaviorists lamented this, as have every generation of psychologists since). And while such pronouncements might be rare, many of our students indicate that they do feel weak in this particular aspect of the discipline the how do we know this? as opposed to the what do we know? Responses to our surveys indicate that students do appreciate the two courses that directly approach the process of psychology over the content (Statistics and Research Methods), but many of the same students called for statistics booster classes or felt that they were still unprepared to conduct independent research. Students also indicated that there were too few opportunities to practice their research skills. The members of our department also agree that even some of our best students struggle with various aspects of the research process including experimental design, data analysis, and reporting results. Finally, we are concerned about how student weakness in this area affects our learning goals (particularly critical thinking) in our 200 and especially our 300 level courses. Quantitative and qualitative research methods are addressed in our 200 level topic courses, but because of sequencing issues, in each class there are necessarily some students who have completed both Statistics and Research Methods, and some who have completed neither. In our 300 level Advanced Topics courses, however, these courses are pre-requisites and thus the expectation is that at this level there will be more emphasis on the research methods of the discipline. We have just begun a discussion about the extent to which our Advanced Topics courses are emphasizing and consolidating the information learned in Statistics and Research Methods. Additionally, we have made changes in our other senior capstone courses by first, increasing the frequency with which we offer Advanced Research Seminar (perhaps the best opportunity for our students to improve their proficiency with statistics and research methods) and increasing the unit value of this course from 0.5 to 1.0. As a full unit course it is possible to spend more time reviewing necessary material and more complex issues of design and analysis before students initiate their own research projects. While Advanced Research Seminar is only one possibility among our capstone requirements, and we expect that most students will continue to take Senior Seminar, providing the opportunity for more students to both review this important material and conduct independent research is an important first step. We are also in the process of organizing a forum to discuss the teaching of statistics and research methods at Beloit College. Currently there are some restrictions that prevent students from doubling-up on statistics and methods courses (for example, if a student who wishes to become a psychology major has taken MATH 106, he or she must audit our statistics course, but perhaps there is some argument that both courses be taken for credit to encourage our majors to get greater exposure to this area). Departmental assessment ideas: After participating in this grant, we would like to investigate two additional methods of evaluating our department and its majors. 1. We d like to create an assessment tool that allows us to see what our students can do with respect to critical thinking. This would be similar to the Collegiate Learning Assessment though on a much

7 smaller scale. We have already found an article that we think would be suitable. Our goal was to find a topic with broad appeal and one that was not in the primary literature; this will allow us to ask students to evaluate it at the beginning of their coursework in the major as well as at the end and compare their performance. Their evaluation will include designing an experiment. We propose to give an assessment like this before students take statistics, after students take the statistics/research methods sequence, and during their final semester. This will allow us to assess the development of their critical thinking specifically within psychology. 2. We want to discuss the pros and cons of using the Dunn et al. benchmarks for assessing our department. If we decide this would be helpful to us in general, and for our upcoming assessment in particular, we will use Vigorito s (2008) An Electronic workbook for assessing the Dunn et al. (2007) Quality benchmarks in undergraduate psychology programs. Dunn, D. S., McCarthy, M. A., Baker, S., Halonen, J. S., & Hill, G. Q., IV. (2007). Quality benchmarks in undergraduate psychology programs. American Psychologist, 62,

8 APPENDIX A Psychology Department Learning Goals Students who complete a major in psychology at Beloit will: Acquire a Knowledge Base in Psychological Science be aware of the major theories, concepts, principles, perspectives, and historical trends of the discipline achieve fluency in the language of the discipline understand the evolving nature of psychological knowledge Be Able to Evaluate Claims and Evidence be able to assess the strengths and weaknesses of different sources of knowledge recognize that knowledge is generated within a particular historical and cultural context be aware of common thinking fallacies and techniques to avoid them be appropriately skeptical of claims made by others Be Proficient in Research Methods and Statistics understand the scientific method and its key concepts (e.g., reliability, validity, operational definitions) be able to design and conduct studies using various research methodologies be able to use and interpret descriptive and inferential statistical procedures be able to use SPSS to analyze data comply with professional standards for the ethical treatments of research participants and laboratory animals Acquire Professional Communication Skills be able to communicate clearly in all standard professional modes (oral presentations, formal writing, and research posters) be able to summarize and synthesize psychological literature be able to communicate research findings clearly, using both words and numbers be proficient in the use of APA style Demonstrate Information Literacy be able to locate information found in text-based or electronic sources be able to evaluate the credibility of sources (e.g., peer-reviewed vs. non peer-reviewed) Exhibit Sound Values and Judgment recognize the necessity for ethical behavior in all aspects of the science and practice of psychology (e.g., avoid plagiarism and conflicts of interest) be open to new ideas and perspectives avoid bias and prejudice in one s thinking be able to disagree with ideas civilly and constructively tolerate ambiguity and acknowledge that most issues are complex and best understood within a multi-disciplinary framework Apply Psychological Knowledge Effectively be able to apply psychological principles to one s everyday life consult with professional psychologists when appropriate be able to articulate how psychological principles can be used to address social issues and inform public policy in a multicultural, global context

9 APPENDIX B The Psychology department is one of several departments working on assessment for a grant from the Teagle Foundation. We are tying to determine how much of our major contributes to development in three different areas: quantitative reasoning, critical thinking, and civic engagement. It would be very helpful for us to know what your thoughts are about how you define these terms as they may mean different things to students than to faculty or even different things from person to person. Please help us by (1) writing what each of these terms means to you, and (2) indicating how/what part(s) of the psychology curriculum, if any, help you develop these competencies. Please return this to Greg or Alexis before Fall break. Thanks again. Your definition Where/how you see the PSYC major contributing to development of this ability Quantitative reasoning Critical thinking Civic engagement Semester/year declared: Class year:

10 APPENDIX C 2008 PSYCHOLOGY DEPARTMENT GRAUDATE QUESTIONNAIRE Congratulations on graduating! Please take the time to answer the following questions, so that we can best evaluate our curriculum in light of your experiences with it. Please provide as much detail as possible, and also be as frank as possible. It will help us to know both the things you found most effective in your experience in the department, as well as the things you found least effective. Thank you for you thoughtful answers! Name: 1a) What cluster class did you take first and why? b) What cluster class did you take last and why? 2a) What were the most effective, rewarding, or otherwise beneficial aspects of your experience in the department? 2b) What were the least effective, rewarding, or otherwise beneficial aspects of your experience in the department?

11 3) What do you think you ve gained from your experiences in the department? 4a) If you were given free rein to redesign the department (including classes offered, requirements for the major etc.), what changes would you make and why? 4b) If you were given free rein to redesign the department (including classes offered, requirements for the major etc.), what would be sure to retain, and why? 5) All things considered, given present facts about the department, would you recommend that someone interested in majoring in psychology at Beloit College do so? Why or why not?

12 Please circle the number that most accurately reflects your answer to these questions: Strongly Strongly disagree agree 6. The courses I took in psychology promoted my critical thinking abilities 7. Studying psychology has increased my ability to analyze and evaluate arguments 8. Studying psychology has made me better at building arguments to support my claims 9. Studying psychology has contributed to my ability to find and challenge assumptions 10a. I feel confident of my ability to conduct searches using PsycLit b. I feel confident of my ability to conduct searches using J-STOR c. I feel confident of my ability to conduct searches using GOOGLE 11. The courses I took in psychology taught me how to analyze data 12. I feel confident that I could figure out how to use SPSS to analyze a dataset 13. I have a good understanding of the ethics involved in the research and practice of psychology 14. I feel confident of my ability to write in APA style 15. The courses I took in psychology taught me the scientific method (e.g., hypothesis testing, determining causality etc.) 16. Studying psychology has helped me see that it s important to be civically engaged

13 APPENDIX D 1. Do you read about topics related to this major just for fun? YES NO 2. Do you read about topics that are not assigned? YES NO For each of the following skills, please indicate how well you feel that the courses that you have taken for your major have helped you achieve these skills. Please use the following scale: 1 = very much; 2= somewhat; 3 = not at all; 4 = don t understand question 3. Effective written and oral communication Logical thinking Quantitative reasoning Information literacy Problem-solving Judgment Ability to read critically For each of the following skills, please indicate how well you feel that the courses that you have taken OUTSIDE OF THOSE for your major have helped you achieve these skills. Please use the following scale: 1 = very much; 2= somewhat; 3 = not at all; 4 = don t understand question 10. Effective written and oral communication Logical thinking Quantitative reasoning Information literacy Problem-solving Judgment Ability to read critically

14 Please tell us how the following types of learning have contributed to you educational accomplishments in a) your major, b) courses outside your major, c) co-curricular activities, d) summer learning or internships. Please mark 5 if they have contributed highly, 3 if somewhat, 1 if not at all, and 0 if you don t understand the question. Type of learning Major courses Other courses Co-curricular Summer/Internships Engaged learning Collaborative learning Interdisciplinary & Integrated learning Experiential learning International/Global perspectives Comments: 17. Please tell us the most valuable thing you have learned or learned to do in the major 18. If you spent time doing a major-related internship or research experience, in what ways did you find yourself well-prepared? In what ways did you find that your preparation was lacking? 19. As you prepare to graduate from Beloit College, what topics or techniques do you wish you learned more about as an undergraduate?