Undergraduate Degree Program Assessment Progress Report Cover Sheet

Transcription

1 Degree: BA Psychology For Calendar Year: 2012 (Date submitted to college committee:) (Date posted on college assessment website:) April 1, 2013 By: Roger Webb Overall Rating: Respond to all six parts following the Degree Program Assessment Progress Report Instructions. Attach additional pages as needed. (NOTE: Parts 1 through 4 can be copied from the relevant sections of your assessment plan.) (1) Student learning goal(s) addressed this year: Beginning two years ago, the Psychology Department shifted our emphasis in assessment to a focus on student success. Student success has been given by the Chancellor, Provosts and the Arkansas Department of Higher Ecucation as UALR and the State's top priority. This year, the Department was gratified to see the University join us and make student success the basis for assessment grants. Reports on how many Psychology majors are on UALR's books vary considerably in the 500 to 900+ range. For the current semester, we advised 368 majors, a countable number. These numbers are large enoungh to suggest that if the Psychology Department were to increase the number of majors who completed degrees, it would have a noticable impact on University student success statistics. (2) Learning outcomes/objectives for those goals addressed this year: The primary outcome was whether a senior psychology student had completed his/her degree. This was explored as a function of when the student achieved senior status and measures of academic success. We also gave the ACAT test to 52 students in Senior Synthesis. (3) Courses & activities where assessed:. Most of the data to be discussed were collected as part of the Department's Senior Synthesis course (PSYC4100), a one SSCH course designed solely for the purpose of collecting assessment data, or from the records of students taking PSYC4100 in past years (4) Methods used: PVC 2/07-1 -

2 With the assistance of Brad Patterson and Sylvester Cartwright, we were able to obtain records from the BOSS system on students having been registered in PSYC4100 since (There are more from the days before T-numbers still to explore.) The report generated from BOSS included whether the student had graduated, the degree granted, whether they had taken particular courses, ACT scores and GPA. We also administered the ACAT test to 52 students in PSYC4100 and collected survey data. We also got data from BOSS on samples of students we had surveyed while they were enrolled either in PSYC2300 (introductory level) or an upper level Psychology course. PVC 2/07-2 -

3 (5) What are the assessment findings? How did you analyze them? The overall ACAT test results were a good replication of the results obtained the last time we gave the test a few years back. Our overall score (449 on CEEB scale, mean 500, SD 100) was roughly a half a standard deviation below the mean of the norms, but given that our students' average ACT scores (21.14) are even further below the mean on ACT norms for college graduates (est. mean 25), the results suggest that we are doing about as well as one would expect. The subtest scores did give us a different result from that seen several years ago: Social Psychology came out as the top subscale score, a result never seen before. See Section 6 for a discussion of this outcome and other observations on the ACAT. We were able to obtain degree outcome data, GPAs, ACT scores, and whether some key courses had been passed on 828 students who had taken PSYC4100 since The data are clearly incomplete for the early years, but become stable after In the years since 2003 and excluding last year, 85.4% of psychology students who took the course had graduated. That means, however, that 14.6% of students achieving senior standing had still not completed their degrees. This is the most important finding in this year's report. Analyses are still in progress and additions to the database are still underway. Our efforts so far have concentrated on what factors might predict non-graduation. For the PSYC4100 group, the most interesting outcomes consist of negative findings. We can find no evidence that GPA, ACT, ACAT or MAPP (a standardized test from ETS that is used widely on a national level) scores predict degree completion. To put it bluntly, no academic measures predict student success for this group. There is some evidence that GPA and ACT scores predict graduation for students from whom we collected survey data in our introductory Psychology course, and minimal, but significant, predictions from GPA for students surveyed in upper level courses (6) What conclusions were drawn and what decisions were made as a result? How were stakeholder groups involved?. The ACAT data from this year's class (n = 52) were analyzed together with data collected in previous years (n = 307), and the overall results have not changed much. The mean comprehensive scores have hardly budged (449 this year, 457 in the past, an effect size of.08 SD). The subtest scores have changed a bit. Statistics used to be our top score, probably reflecting the fact that we require two courses and most Psychology programs only one. This year, Social Psychology was the clear winner. The score for Social was 500, and that score did not correlate as highly with the total as the other scores, suggesting something perhaps serious teaching--was going on. The bigger problem with the ACAT is that it correlates very highly with the ACT. A factor analysis on the old data using the subscales and the ACT test found a first factor that accounted for over 90% of the total variance in the scores. This year s first factor accounted for 86%, but the N is too small to be taken seriously in the second decimal place. Anytime one sees a cognitive test with a first factor accounting for 90% of the variance, it is clearly a measure of g or general cognitive ability. In plain English, it is an PVC 2/07-3 -

4 IQ test. While the Department has felt it worthwhile to give a nationally recognized assessment test from time to time (we have also used the MAPP test) to maintain external credibility, the truth is that these tests tell us very little about how our students are doing; they just tell us who they are. Both the ACAT and MAPP appear not to be worth the time and money spent collecting the scores. The graduation data are much more interesting and will be the focus for the remainder of this discussion. The great retention bloodletting at UALR occurs in the first year of enrollment. The small fraction of students who begin their first semester and are back the next year in good standing can be shocking, though those numbers seem to be improving. (We are presently looking for more current data on that question and also whether the lottery scholarship program has had any impact.) We see some of the beginner s effect in our data since both GPA and ACT scores do predict which of our PSYC2300 (intro level) students eventually graduate. Once we get people enrolled as Psychology majors and take over their advising, however, the majority graduate. Robert Corwyn s analysis of the 2005 cohort of majors found that 67% had a degree within six years of matriculation--the standard of student success used by ADHE and in most national studies. We also see that effect in our current data. Of students tested in our upper level courses, GPA, but not ACT, does differ significantly between students who graduate or do not, but the difference is so small that it would not appear to have much predictive power. When we get students to be seniors the PSYC4100 sample GPA and the various tests (ACT, ACAT, MAPP) do not show a statistically significant difference between successful and non successful students. Statistical significance is the lowest form of statistical evidence; if we cannot demonstrate a statistically significant difference, no further analysis is going to find anything important or interesting. As we consider this finding, we suppose we should have assumed that students who have achieved senior status were capable of doing the work. Still, the complete lack of positive predictions from academic measures is a little surprising. Thus we have our problem: well over ten percent of students who achieve senior status and are apparently planning on graduating do not actually graduate and none of the overall measures of academic competence predict who those students are. We are presently pursuing two data driven approaches to understanding this problem. We have some data on which courses students have taken. For example, the non graduating group tends not to have taken the second semester of our required statistics sequence. Since many of our students are math phobic and put that course off as long as possible, the finding is not surprising. So far the grade distribution in that course of students who enroll does not seem to distinguish successful from non successful students, suggesting that an inability to pass the course is not the operative hurdle. That question, however, requires further study. Professor Corwyn is working on the use of new analytic techniques that will make it easier to look at course sequences. PVC 2/07-4 -

5 We are also puzzled by the 13.4% of PSYC4100 students who graduated, but whose degrees were not listed as Psychology (that is 15.7% of the successful students). In this report, we have counted any four year degree as a success, but do not know what to make of our apparent non-majors. How many have a double major, etc.? The Psychology department also has student responses to a number of non academic questions collected over years that we think may reveal more of nature of our problem. We have self report data on such questions as whether the student s family expected them to go to college, whether they took the college prep courses in high school, whether they are getting financial support from their family, whether they have children and such. In the national literature, for example, family financial support is currently the single biggest factor in predicting graduation. Taking college prep and AP tests has always shown major predictive power (probably reflecting family assumptions of college education) and having children is a negative factor. A secondary report of a recent study claimed that women double their lifetime earnings by delaying child bearing ten years (and we are looking for the actual study). So far these data have been difficult to integrate with the academic data. Professor Webb who has been directing the collection of these data deeply regrets his lack of foresight in not developing a multiyear database from the time PSYC4100 was first required and in changing surveys from year to year in response to the concern of the day. There are also problems with how we asked some questions. For example, we asked about family support indirectly by asking who is responsible for paying for your college education? The answer I am completely responsible, which the majority of our students picked would seem to exclude family support, but inconsistencies call that into question. Another question suggested a higher level of family help, but still nowhere near a majority. (There are a fairly high percentage of out of range responses in our data set, suggesting that some students were not very careful in filling out surveys.) A third line of attack comes from anecdotal accounts from advising sessions and complaints from students. These may be our most telling data, but they are obviously nonsystematic and difficult to summarize. The difficulties that our students have with finances, employment, and children run through the stories, and it not clear how we can do much about that. There are related complaints about the scheduling of classes and the difficulty of getting into online courses. Many students have made what appear to be maladaptive decisions about majors and minors that have delayed their gradations for no apparent good reason. (We could use a survey of people making hiring decisions in the community as to whether a degree with two or more majors or minors makes a job candidate more or less attractive. Our assumption is that such inefficient educational planning would appear as a negative to a potential employer.) The difficulties in completing graduation requirements that appear to come from the limited offering of key courses on campus and the inaccessibility of online courses are issues over which we have some control. The Department can look at how often and PVC 2/07-5 -

6 when key courses are offered, though our response for some key courses will be limited by lack of staff. Making the online courses more accessible and the management of those courses more flexible would be a major plus. Several students we have talked with have moved out of town, need one or two more courses to graduate, and cannot get into a section of an online course. We have also seen enrollment in online courses arbitrarily limited by individual professors. Surely, there are ways to guarantee enrollment in those courses for someone that close to graduation. Presently, all our online courses are taught by full time faculty members, and that is a maladaptive, self imposed, limitation on the Department s part. The History Department, for example, has online instructors who do not even live in the U.S. We can clearly expand our offerings without expanding our faculty, though in the long run that strategy has already caused much of our trouble. Allowing the Department to teach virtually all our lower level courses, and many upper level courses, with part timers has led to the serious understaffing of the Psychology Department. PVC 2/07-6 -