Psychology Major Assessment Report for

Transcription

1 Psychology Major Assessment Report for I. Brief Summary of Assessment Plan: Outcomes Assessed Goal 1: Knowledge Base, Outcomes 1 and 3-11 (Psychological Perspectives, Research Methods, Statistics, Biological Bases, Learning, Cognition, Development, Individual Differences, Psychological Disorders/Treatments, Social Psychology) ACAT: In our Assessment Plan, we said we would assess these outcomes by giving the Psychology Area Concentration Achievement Test (Psych ACAT) to our senior psychology majors in P457 Senior Seminar in Spring, Outcomes 1 and 3-11 identify specific content areas we have determined as important to our major. Because we were using the Psych ACAT to evaluate these outcomes, we did not need components or criteria. Each outcome is represented by one or more subtests of the ACAT as you can see in the table below. Outcomes Students will demonstrate basic psychological literacy in: 1. Major psychological perspectives ACAT Subtest (ACAT: history & systems) 3. Research Methods (ACAT: experimental design) 4. Statistics (ACAT: statistics) 5. Biological Bases (ACAT: physiological) 6. Learning (ACAT: human learning & cognition) 7. Cognition (ACAT: human learning & cognition) 8. Development (ACAT: developmental) 9. Individual Differences (ACAT: personality) 10. Psychological Disorders & Treatments (ACAT: Two sub-tests: abnormal and clinical & counseling) 11. Social Psychology (ACAT: social) Benchmarks for ACAT Assessment: We will use three benchmarks. 1) as a, students taking test will achieve 50 th percentile or higher on each subtest, 2) at least 75% of individuals will achieve 50 th percentile or higher on the subtests representing Outcomes 1, 3, 4, and 8 because those outcomes represent coursework that all psychology majors must take, and 3) at least 75% of individuals will achieve 50 th percentile or higher on at least one of the subtests representing each of the course clusters required for psychology majors. There are two course clusters in the major. Cluster 1 is P319 Personality, P320 Social, P303 Health, and P324 Abnormal. Cluster 2 is P325 Learning, P326 Neuroscience, and P335 Cognition. Majors must take two courses within each cluster. Thus, benchmark 3 is met when at least 75% of the students achieve 50 th percentile or higher on at least

2 one subtest that evaluates each cluster. History & System Mastery Tests: We also initially planned to assess Knowledge Base Outcomes 3-11 using a second measure: content mastery tests from P459 History & Systems (a required course taken by juniors and seniors). The tests were 10-item multiple-choice format, with a separate test for each content area. Students would need to earn 80% on either the original test or one of the two re-take tests (with the retakes containing the original 10 items + 10 new items) to achieve mastery. Data was combined from Fall, 2008 and Fall, However, the total number of students at the end of Fall, 2009 was still relatively small, so the decision was made to collect data on this one more time (i.e., Fall, 2010) before analyzing the data. A benchmark for this new measure needs to be created, and then data for this will be presented in the Assessment Report. Goal 2: Research Methods, Outcome 5, Design Basic Studies to Address Psychological Questions. Advanced Level. We had originally intended to conduct a pilot of the review process of this outcome at the advanced level during Summer, However, we ran out of time at the summer retreat to complete this and instead did the pilot assessment in Fall, We began with the following components, criteria, and benchmark (all but the last four of which were used for basic level assessment of this outcome). Our main purpose was to determine whether these components and criteria used at the basic level were also useable with senior-level work or whether the rubric would need to be modified. Components for Goal 2, Outcome 5 Design Basic Studies Research Question/Purpose/Objective (B, A) Clear specification of hypothesis (B, A) (i.e., does wording clearly indicate it is a hypothesis; says hypothesis, predict, etc. so can tell w/out guessing) Hypothesis appropriate to lit. review (B, A) (i.e., does it follow from their cited research?) Hypothesis appropriate to method (B, A) Clear specification of operational definitions for constructs found in hypothesis (B, A) (i.e., describes questions and response options; explains how responses are combined, if applicable) Criteria (i.e., Levels,Characteristics)* (Circle appropriate response) Present - Absent Clear - Somewhat Clear - Unclear Appropriate - Somewhat Appropriate - Inappropriate Appropriate - Somewhat Appropriate - Inappropriate Clear - Somewhat Clear - Unclear Sampling procedure (A) Research design (A) Plan for data analysis (A) Literature search (A) Criteria and benchmarks for these last 4 advanced components do not yet exist, but will be developed as part of the pilot assessment. *Note: Letters B and A following components identify which components are considered Basic Level (B) and which are considered Advanced Level (A).

3 Basic Level Benchmark: At least 75% of psychology majors will exhibit the following pattern of performance: Research Question (Present), Clear Hypothesis (at least Somewhat Clear), Hypothesis Appropriate to Lit. Review (at least Somewhat Appropriate), Hypothesis appropriate to method (Appropriate), and Clear Specification of Operational Definitions (at least Somewhat Clear). Goal 5: Ethics in Psychology, Outcomes 1 (Recognize necessity of having ethical standards/acting ethically), 4 (Understand what it means to be ethical in practice), and 5 (Apply ethical principles in either research or practice). In our Assessment Report, we described a 10-item multiple-choice measure of knowledge of ethical concepts, that had been piloted in the Spring 2009 term. We also described how we would add two items to this quiz (for ease of interpretation, and to assess each outcome equally) for the Spring 2010 administration. These changes were made and reviewed by the Psychology Faculty before administration. (Note: The nature of this assessment instrument does not lend itself to use of criteria.) Benchmark for meeting Goal 5, Outcomes 1, 4, and 5: The Psychology faculty agreed that 75% of students should correctly answer 75% of items written to address each outcome in order to meet the benchmark. Results are reported separately for each administration (Time 1 and Time 2). Goal 7: Quantitative Literacy, Outcome 2 (Select Appropriate Statistics). We originally intended to assess this outcome in two ways: 1) by reviewing the relevant section of the research proposal papers we were already collecting from Senior Seminar for use in evaluating Goal 2: Research Methods, Outcome 5 Design Basic Studies at Advanced Level, and 2) by using a small set of questions either embedded in a test or given as a quiz to students enrolled in the Fall, 2009 and Spring, 2010 sections of our K300 Statistics class (required of all psychology majors). Because the assessment of the Senior Seminar proposals for Goal 2 took longer than expected, we chose to forego that route of assessment until a future time and instead evaluated Goal 7, Outcome 2 based only on the items given to the K300 students. II. Assessment Methods Goal 1: Knowledge Base, Outcomes 1 and 3-11 (Psychological Perspectives, Research Methods, Statistics, Biological Bases, Learning, Cognition, Development, Individual Differences, Psychological Disorders/Treatments, Social Psychology) We currently use the Psychology Area Concentration Achievement Test (ACAT) to assess these outcomes. The ACAT allows us to choose the topic areas within psychology we want to include

4 on our students tests and provides data that shows us how our students performance compares to psychology students at other institutions that use the ACAT. Students take the Psych ACAT just before or just after Spring Break of their final spring semester of their Senior year in their P457 Senior Seminar class. Although the ACAT is now taken online, the tests are still taken during a class session and the faculty member teaching the seminar provides access codes and proctors the test. Psychology faculty reviewed and discussed the ACAT scores at a bi-weekly meeting during the fall semester Goal 2: Research Methods, Outcome 5 Design Basic Studies to Address Psychological Questions. Advanced Level. We began with a random sample of approximately 1/3 of the student papers from the P457 Senior Seminar classes from 2006, 2007, and The activity/object for the advanced level assessment of this outcome was the final draft of the Research Proposal Assignment from Senior Seminar (typically collected during the last week or two of the class). Copies of each paper were given to each full-time psychology faculty member to be evaluated using the criteria, and benchmarks described in Section I for the first five components. We intended to develop the criteria and benchmark for the last four outcomes at the end of this meeting. Faculty met as a to calibrate the P211 paper evaluations. We each read and evaluated the first five components of each paper individually, then compared evaluations. Goal 5: Ethics in Psychology, Outcomes 1, 4, and 5 During the Spring 2010 term, the measure was administered in P390 Helping Skills both 4 weeks before, and 3 weeks after, a class lecture covering central ethical concepts. The instructor informed students enrolled in the course that they would take an ungraded quiz over ethics at two points in the term. Thirteen psychology majors were enrolled in the course at Time 1, and 10 took the quiz. All thirteen psychology majors were still enrolled in the course at Time 2, and 11 took the quiz. Eight psychology majors took the quiz at both administrations. Goal 7: Quantitative Literacy, Outcome 2 (Select Appropriate Statistics). Two PSY K300 (Statistical Techniques) exams (Exam III and the Final Exam) were reviewed and eight questions were identified that pertained to students appropriate selection of inferential statistical procedures (one question had two parts that were evaluated here as separate questions, as each part involved selection of an appropriate statistical procedure). Class rosters from Fall 2009 and Spring 2010 were reviewed and psychology majors were identified. There were 17 psychology majors on the roster. However, two withdrew from the class prior to these two exams and two other students did not take exam III (they took a retake, with different questions). Thus, these four students were excluded, leaving 13 psychology majors for analysis. For each student, it was determined how many of the eight identified questions the student answered correctly.

5 III. Description of Assessment Results Goal 1: Knowledge Base, Outcomes 1 and 3-11 (Psychological Perspectives, Research Methods, Statistics, Biological Bases, Learning, Cognition, Development, Individual Differences, Psychological Disorders/Treatments, Social Psychology) Because the sample size for Senior Seminar is rather small each year, we tend to look for trends across years. Because we have used the ACAT as an assessment tool now for four years, the table below provides the data for each of those four years. Although this report does not require inclusion of previous data, we do so for our own benefit; it makes it easier and more reliable to look for patterns in the data. Our first benchmark was having our students (as a ) achieve at least the 50 th percentile in comparison to the national sample of other Psychology Majors who also took this test. This data is presented in the following table for each subtest/learning outcome. As you can see in the table below, in the past two years the overall performance of students has risen back up again over the 50 th percentile, after a somewhat usual year in 2008 when a snow day prevented the normal administration of the test. Although there continue to be a couple of content areas below benchmark levels in each of the past two years (i.e., 3 below in 2009 and 2 below in 2010), there are fewer areas below benchmark than in 2007 and 2008 (i.e., 5 below each year). The specific content areas in which we see below-benchmark performance vary from 2009 to 2010 and likely reflect differences in the courses students have completed, how recently they completed them, and the fact that there were two students whose performance was particularly poor (e.g., 18 th percentile and below). Benchmark 1 Results: Psychology Area Concentration Achievement Test (ACAT) subtest results in relation to outcomes for Goal 1: Knowledge Base Areas of Psychology Tested (i.e., ACAT subtests) 2006 n = n = n = n = n = 11 Subtest: History & Systems 45* 42* 20* 50 46* (Outcome 1: Perspectives in Psychology) Subtest: Experimental Design (Outcome 3: Research Methods) Subtest: Statistics (Outcome 4: Statistics) * 41* 60 Physiological (Outcome 5: Biological Bases) 71 47* 57 48* 68 Human Learning/Cognition (Outcomes 6 & 7: Learning & Cognition) Developmental (Outcome 8: Developmental) 53 48* 47* Personality (Outcome 9: Individual Differences) 48* 41* 35* 64 41* Abnormal (Outcome 10: Disorders & Treatments) Clinical/Counseling 54 46* (Outcome 10: Psychol. Disorders & Treatments) Social (Outcome 11: Social Psychology) * 46* 63 OVERALL PERFORMANCE * 62 59

6 Note: * indicate benchmarks not met Our second benchmark is that at least 75% of individuals will achieve 50 th percentile or higher on the subtests representing Outcomes 1, 3, 4, and 8 because those outcomes represent coursework that all psychology majors must take. As you can see in the Benchmark 2 data below, we did not meet this benchmark for any of these four outcomes, though we came closest to the benchmark on the developmental outcome (for which we met the benchmark last year). Benchmark 2: 2010 Results Outcome 1: Psychological Perspectives: 55% at 50 th percentile (as compared to 56% last year) Outcome 3: Research Methods: 55% at 50 th percentile or higher (as compared to 44% last year) Outcome 4: Statistics: 55% at 50 th percentile or higher (as compared to 22% last year) Outcome 8: Developmental: 64% at 50 th percentile or higher (as compared to 89% last year) Our third benchmark is that at least 75% of individuals will achieve 50 th percentile or higher on at least one of the subtests representing each of the course clusters required for psychology majors. There are two course clusters in the major. Cluster 1 is P319 Personality, P320 Social, P303 Health, and P324 Abnormal (which represent knowledge Outcomes 9, 10, and 11). Cluster 2 is P325 Learning, P326 Neuroscience, and P335 Cognition (which represent knowledge Outcomes 5, 6, and 7). Majors must take two courses within each cluster. Thus, benchmark 2 is met when at least 75% of the students achieve 50 th percentile or higher on at least one subtest that evaluates each cluster. The results for the third benchmark (see below) are very encouraging. The benchmarks were met for each cluster. Thus, students seem to be learning the material from the clusters. Benchmark 3: 2010 Results Cluster 1: 91% had at least 1 of the 3 tests at 50 th percentile or higher Cluster 2: 82% had at least 1 of the 2 tests at 50 th percentile or higher Goal 2: Research Methods, Outcome 5, Design Basic Studies at the advanced level. We piloted an assessment of this outcome on a subset of research proposals from several old sections of Senior Seminar on Sept. 25. Our main purpose in doing so was to determine whether the rubric we were using for this goal at the basic level was useable with senior-level work or whether the rubric would need to be modified. We found the original rubric worked poorly at the senior level and that faculty evaluations varied considerably when using that rubric. We made a number of changes to the criteria and components of the rubric that made it more useful, but decided it would be most useful to begin anew next academic year with a different set of student papers for the actual assessment.

7 Goal 5: Ethics in Psychology, Outcomes 1 (Recognize necessity of having ethical standards/acting ethically), 4 (Understand what it means to be ethical in practice), and 5 (Apply ethical principles in either research or practice). Results of P390 Helping Skills Ethics Quiz No student achieved a total score below 75% on this quiz, and 92% was the modal individual score at both administrations. The table below indicates the percentage of students who correctly answered each item at each administration, and the percentages of students who answered 75% of the items correctly for each outcome. It is organized according to the outcomes that the quiz items were intended to address. The benchmark of 75% of the students correctly answering 75% of the items for each outcome was met. Narrative discussion follows. Category - Items Sp 2010 Time 1 (N=10) Percent of students getting 75% correct at Time 1 Sp 2010 Time 2 (N=11) Percent of students getting 75% correct at Time 2 Outcome 1: Recognize necessity of having ethical standards/acting ethically 80% 81% -Who holds responsibility for enforcement of ethical 92% 100% principles, and what sanctions can be placed against helpers -Ethical standards relevant to peer/institutional review of research, and supervision/licensure of practice -How past difficulties resolving ethical dilemmas necessitated the development of ethical guidelines -Purpose of education on ethics for research and practice 77% 77% 100% 54% 92% 92% Outcome 4: Understand what it means to be ethical in practice 90% 100% -Ethical standard of competence and its relation to 92% 100% helping -Avoiding violation of ethical standard regarding 92% 100% personal problems and conflicts of helpers -Need to inform clients about limits to confidentiality 100% 100% -Need for formal releases of information from clients, in order to release documents to parties other than the client Outcome 5: Apply ethical principles in either research or practice -Recognize an example of ethically-sound preparation for working with diverse populations -Recognize violation of ethical standard regarding multiple relationships in clinical practice -Recognize a clinical circumstance requiring breaking client confidentiality -Distinguish parties who can provide expert testimony in legal matters, from those who cannot 100% 100% 100% 100% 100% 100% 92% 92% 100% 100% 92% 85%

8 These data indicate that students completing the Psychology program appear to understand the need for ethical guidelines in research and practice, have a basic understanding of some core ethical principles in psychology, and can apply some of those principles to hypothetical situations. Performance on this quiz was very strong both at Time 1 and Time 2, indicating the likelihood that psychology students are exposed to ethical principles previous to taking this 300-level course (which most students complete in their junior or senior year). Goal 7: Quantitative Literacy, Outcome 2 (Select Appropriate Statistics). Of the 13 psychology majors for which data were analyzed, all students answered at least half (4 of 8) of the questions correctly and 10 out of 13 (76.9%) answered at least 75% (6 of 8) questions correctly. Giving our common benchmark of at least 75% of students getting a 75% or higher, these results put us just above our benchmark. IV. Using Assessment for Program Improvement Goal 1: Knowledge Base, Outcomes 4-11 (Statistics, Biological Bases, Learning, Cognition, Development, Individual Differences, Psychological Disorders/Treatments, Social Psychology). We have seen some sustained improvement on this goal this year. For example, we met Benchmark 1 for overall performance on the ACAT test as well as the benchmark for eight of the ten individual tests (with the exception of history and systems and personality). This year 8 of the 11 students took History and Systems. Additionally, we met the benchmarks for both of the sets of Cluster courses (Learning, Neuroscience, and Cognition; Outcomes 5, 6, & 7), and (Personality, Social, Health, and Abnormal; Outcomes 9, 10, & 12) for the second year in a row. Although the Personality, Social, Health, and Abnormal cluster meets Benchmark 2, For personality, which is offered every other year, we have noted a pattern over the last two years that is connected to when the class is offered relative to the ACAT (i.e., better scores in years when personality is offered and is thus fresher in students minds). However, personality scores have been below benchmark on Benchmark 1 in a number of past years. We have also seen some dramatic improvement in both Benchmarks 1 and 2 for statistics. Last year, we were becoming concerned with the ACAT score pattern for statistics, for which we had not met benchmark two years in a row. However, this year, we met both benchmarks relevant for statistics. This may be due to random error and individual differences among students. It may also have helped that this year in Senior Seminar students were given a decision tree handout to help them identify appropriate statistics for their research proposals. We plan to continue using the decision tree in this class in the future. Although we saw improvements for statistics, we again did not meet benchmarks for Outcomes 1, 3, 4, & 8 (Psychological Perspectives, Research Methods, Statistics, and Development, respectively). It has occurred to us that expecting 75% of our students to get higher than 50 th percentile on a nationally normed test is perhaps more of an aspirational goal rather than a realistic one, particularly given our students wide range of ability levels. We

9 agreed, however, that if 75% of our students couldn t achieve the 40 th percentile, we would be more concerned and want to try to do something about it. We believe that setting a new benchmark of 75% of individuals achieving at least 40 th percentile is more fitting given that large variation in student ability levels. ). In future reports, we will start reporting both the percentage to reach 40 th and percentage to reach 50 th percentile. In addition, if students can perform well on another measure of knowledge, such as the mastery tests given in P459 History & Systems, we would be less concerned about our program s ability to serve content knowledge goals. We will look at the mastery test data in our Assessment Report. It also may be time for us to return to a discussion of potential changes to our research and statistics curriculum. Goal 2: Research Methods, Outcome 3, Design Basic Studies to Address Psychological Questions. Advanced Level. Now that we have developed a more useful rubric for the advanced level for the first five components, we will be using the revised rubric to do a more formal assessment of this outcome with proposals from more recent sections of the course for our assessment report. We will also need to develop criteria for the last four components before doing the actual assessment. Goal 5: Ethics in Psychology, Outcomes 1 (Recognize necessity of having ethical standards/acting ethically), 4 (Understand what it means to be ethical in practice), and 5 (Apply ethical principles in either research or practice). Given that student performance exceeded benchmark, it seems further assessment of these particular outcomes may not be warranted in the near future, allowing us to focus on other outcomes for assessment. It may be wise to administer this measure again in a few years, just to ensure that the pattern is maintained over time. Goal 7: Quantitative Literacy, Outcome 2 (Select Appropriate Statistics). The results are encouraging. Our students are demonstrating mastery of this fairly advanced skill (knowing when to apply statistical procedures) during the class. We will continue to emphasize the selection of statistical procedures in the PSY K300 class (e.g., during lectures, in a class activity where students practice this skill, in the context of a final survey project assignment). Future assessment of research proposal papers from Senior Seminar (PSY P457) will help us better assess the extent to which students retain this understanding beyond the class and can demonstrate these skills in a research context toward the end of their program. V. Dissemination of Results An electronic copy of this report will be posted on the psychology website.