1 Analysis of Student Retention Rates and Performance Among Fall, 2013 Alternate College Option (ACO) Students A Study Commissioned by the Persistence Committee Policy Working Group prepared by Jeffrey T. Luftig, Ph.D. and distributed by The Office for Performance Improvement April, 2015
2 Table of Contents Purpose & Groups....... 3 Analysis of Retention Rates Among ACO Students...... 5 Analysis of Retention Rates ACO vs Non-ACO Students..... 8 Analysis of Probation & Dismissal Rates ACO vs Non-ACO Students..... 15 Analysis of Academic Performance ACO vs Non-ACO Students...... 20 Analysis of Academic Performance Among ACO Groups.. 26 Analysis of Differences in cumulative GPA Variability. 32 Analysis of Differences in cumulative GPA Central Tendency.. 33 Summary of Results: Retention, Probation, and Academic Performance 39 Program and Major Transition Through Time 41 ACO Business Students.. 41 ACO Engineering Students.... 45 Summary of Results: Student Transitions Through Time... 50
3 The purpose of this study was to respond to requests for analyses associated with the relative performance of students admitted to the College of Arts & Sciences (A & S) under the Alternate College Option (ACO). As defined by a 2006 report issued by Perry Sailor and the PBA, these are students who apply to the colleges of business, engineering, or architecture and planning, are not admitted to these colleges, but are offered (and accept) admission to arts and sciences. These students are known as ACOs, for alternate college option. They are typically less prepared academically than. students who are admitted to their chosen colleges. For this study, students in the Fall, 2013 cohort (not including transfer students) who were identified as ACO students were tracked for retention and scholastic performance through the beginning of the Spring, 2015 semester. It should be noted that, although all ACO students in A & S are included in the category of Open Option, there are many more Open Option (Undeclared Major on Entry) students than ACO students. Because there are significant differences among these two groups of students, a conscious decision was made to evaluate the retention and performance of ACO students as a singular group; that is, not combined with other Open Option students. A second study and report will present the results of retention rate and scholastic performance among Open Option students in A & S, excluding ACO students. The data related to students identified as ACO students, and their program affiliation, was provided by Academic Advising Center (AAC) staff, and merged with the Fall 2013 cohort student demographic, retention, and performance data as provided by the CU Boulder Institutional Research (IR) group. For the Fall, 2013 cohort, the ACO students represented 15.9% of the entire cohort, and 23% of the Fall 2013 A & S cohort (not including Open Option / Undeclared Major students):
4 Within the ACO Group, the distribution of ACO students appeared as shown in the table and bar graph below: Because the subgroup size for the ACO Music group was so small, subsequent analyses related to retention and performance differences among the ACO student groups omitted the ACO-Music students as a subgroup.
5 Analysis of Retention Rates Among ACO Students The cumulative retention rates for the entire Fall, 2013 cohort in all schools and colleges for the first two academic years appeared as shown in the three tables which follow, on the left. The same data, for only A & S, appears in the tables to the right.
6 As shown by these tables, A & S retention rates were lower in each of the three semesters following the Fall 2013 semester; additionally, the difference between the overall campus rate and A & S rates increased with each semester through time. These same data can also be evaluated on a semester-by-semester basis; that is, without calculating the cumulative loss in students. These data for the entire campus (to the left) and for only A & S (to the right) follow. Note that the first semester Spring 2014 is not shown because it is unchanged.
7 Having established that the retention rates for the Fall 2013 cohort in A & S are greater than for the students in the other schools and college on campus (taken on the whole), the next step was to assess the degree of influence that: 1) ACO student retention rates had on the cumulative retention rates recorded for A & S; and to determine 2) Whether significant differences exist in cumulative retention rates among the three ACO groups evaluated. As with many of the analyses we perform for cohort data, there is an issue of whether statistical significance and associated statistical tests of significance can be applied to these data. Classical statistical theory dictates that inferential statistical tests be conducted on data that represent a random sample or subgroup of the research population of interest. Clearly, we are utilizing the entire cohort (N rather than n) for these analyses, so these data cannot be argued to represent a random sample in time. However, many researchers would argue that the cohort, even taken in its entirety, might be interpreted as representing a random sample of cohorts through time (N Tx); and that given this interpretation, inferential statistics may be applied. If the reader is determined to employ the former interpretation, then any observed difference in the descriptive statistics represents a true difference (although it still may not be an important difference; practically speaking). If the reader interprets the data utilizing the latter interpretation of random sampling theory, then the tests of significance presented can assist in determining whether the observed differences are indeed consequential; or simply due to sampling error (chance).
8 Analysis of Retention Rate Differences: ACO Students versus All Other Students in A & S The three analyses which follow reflect the cumulative retention rate differences between all ACO students and all other students in A & S, and the differences between the three categories of ACO students in the Fall 2013 cohort. As shown by these results, no significant differences are present in the first semester retention rate (Spring 2014) between ACO and Non-ACO students; or within the ACO student groups. However, significant differences are observed starting with the cumulative retention rates for the Fall 2014 semester; although not from ACO group-to-group. Differences observed in retention rates increase for the cumulative retention rate differences by Spring, 2015; and approach statistical significance when comparing the individual ACO group results. All analyses which follow include only students admitted to A & S in the Fall 2013 cohort as ACO students; as Open Option students, or as students with a declared major in A & S.
Spring 2014 Results 9
10
Fall 2014 Results (cumulative) 11
12
Spring 2015 Results (cumulative) 13
14
15 Analysis of Probation and Dismissal Rate Differences: ACO Students versus All Other Students in A & S The analyses which follow reflect: 1) the first semester probation rate differences between all ACO students and all other students in A & S, and the differences between the three categories of ACO students in the Fall 2013 cohort; and 2) the second semester academic dismissal rate differences between all ACO students and all other students in A & S, and the differences between the three categories of ACO students in the Fall 2013 cohort. As shown by these results, there are statistically significant differences in the first semester probation rate (Fall 2013) and second semester (Spring 2015) academic dismissal rate between ACO and Non-ACO students. Significant differences between the ACO groups are observed for first semester probation rates ( at α = 0.10). For second semester dismissal rates, meaningful differences between the groups are observed; although at the subgroup sizes available, not statistically significant at α = 0.10 ( p = 0.154).
16
17
18
19
20 Analysis of Differences in Student Academic Performance: ACO Students versus All Other Students in A & S The next step in assessing the differences in performance between ACO students and A & S students who were not ACO students was to evaluate differences in academic performance as measured by cumulative GPA indices at the end of each of the first 3 semesters available: Fall, 2013, Spring, 2014, and Fall, 2014 (Spring, 2015 results were not available at the time this report was prepared. Two primary comparisons were of interest and requested: 1) Is there a significant difference between ACO student performance and non-aco student performance in A & S? 2) Is there a significant difference in student performance among the three primary ACO student groups? The cumulative GPAs for the groups of students evaluated appeared as follows for the semesters of interest:
21
The recorded data may also be viewed for differences in central tendency and variability using Box-and-Whisker plots: 22
23 The first step in comparing the two groups is to test for the normality of the two data sets. This is necessary as tests for independent variances are not robust from departure from normality. As expected, the cumulative GPAs for the six groups were uniformly non-normal (cumulative GPAs are almost always found to be negatively skewed and are often not mesokurtic). Fall, 2013 cumulative GPAs ACO_NonACO n Skewness p-value Kurtosis p-value A & S Not ACO 3,070-1.053 0.000* 0.966 <.02* ACO Student 904-0.792 0.000* 0.319 >.10 Spring, 2014 cumulative GPAs ACO_NonACO n Skewness p-value Kurtosis p-value A & S Not ACO 2,916-0.863 0.000* 0.703 <.02* ACO Student 862-0.776 0.000* 0.706.05-.02* Fall, 2014 cumulative GPAs ACO_NonACO n Skewness p-value Kurtosis p-value A & S Not ACO 2,570-0.732 0.000* 0.779 <.02* ACO Student 737-0.754 0.000* 1.505 <.02* As a result, Levene s Improved test approach for differences in variability (versus variance) was selected for comparison purposes. This approach conducts a t-test on the mean differences in the Absolute Deviation from Medians for the three pairs of cumulative GPA data.
As illustrated by the results above, the variability of cumulative GPA values for the Fall 2013 and Spring 2014 semesters for ACO and non-aco students are equal; statistically speaking. For the Fall, 2014 semester, the variability of the cumulative GPAs for the non-aco students is statistically significantly greater than for the ACO students (all groups combined). 24
25 Moving on to the comparison of differences in the average cumulative GPAs for the two groups, within each of the three semesters, we employ the exact (Fisher) t-test for the means for the first two semesters (the exact t-test is robust to departure from normality when sample or subgroups sizes are large), and the approximate t-test for the Fall 2014 semester given the unequal variability observed for the two groups (the exact t-test is not robust from departure from homogeneity of variability when sample sizes are unequal). As shown by the results of the three t-tests, the mean cumulative GPA values recorded by non-aco students were statistically significantly higher than for the ACO students in every one of the three semesters. Additionally, the importance (statistical) of the differences increases with each subsequent semester. The ω 2 values for the differences observed each semester are 4.2%, 5.7%, and 6.0%; respectively.
26 Analysis of Differences in Student Academic Performance: Variability Among ACO Student Groups Having established that Non-ACO students perform, academically-speaking, at a superior level than ACO students taken as a single group, the next step in the analysis was to evaluate whether differences existed in performance among the three ACO groups (Business, Engineering, and Journalism). The cumulative GPA indices recorded for students in the three groups for the Fall 2013 semester appeared as follows (the Non-ACO Student results are provided for comparison purposes).
27
The descriptive statistics for the Spring 2014 and Fall 2014 semesters for the three ACO groups were generated as well, and are presented in the summary table below. Again, the Non-ACO student results are presented for comparative purposes. 28
29
30
31 The same general sequence as was employed and previously illustrated for the two group comparisons was repeated in order to conduct statistical comparisons for the three ACO groups. Testing for the normality of the three populations represented by the ACO subgroups yielded similar results as expected; specifically, that we have sufficient statistical evidence to reject the hypothesis of normality for the cumulative GPA distributions. ACO_Group n Skewness p-value Kurtosis p-value ACO - Business 369-0.754 0.000* 0.252 >.10 ACO Engineering 470-0.771 0.000* 0.209 >.10 ACO - Journalism 65-1.406 0.000* 2.269.05-.02* As a result, we employ Levene s Improved test using Absolute Deviations from subgroup Medians (ADMs) to test the comparative variability among the three groups; within each of the three semesters. Instead of a t-test, however, we use Welch s Oneway ANOVA rather than the standard (Fisher) Oneway ANOVA (Fisher s ANOVA is not robust from departure from homogeneity of variability when sample sizes are unequal, and the conservative approach here is to use Welch s ANOVA). If significant differences are identified, the Games-Howell post-hoc tests are applied (unequal variance alternatives to the Tukey tests). For the means analysis, if unequal variability is found in the group comparisons, then Welch s ANOVA is employed, followed by Games-Howell post hoc analyses if necessary`. If the variability among the three goups was inferred to be equal, Fisher s ANOVA was employed for the comparison of the three groups, followed by Tukey tests for post-hoc analyses if required. In order to reduce the probability of a Type II Error ( β ), the Type I Error level was established at α = 0.10.
32 Tests for Differences in Variability As shown by these data, the variability of cumulative GPAs varied only for the Spring 2014 semester, with all three groups reflecting a difference in variability. As a reminder, the three standard deviation (s) values for the ACO groups were s Engrg = 0.800; s Bus = 0.704; and s Jour = 0.555. When comparing the Non-ACO student variability (s Non-ACO = 0.75) in this semester to the three ACO groups, the variability of the Non-ACO student cumulative GPAs was equal to Engineering and Business, and significantly greater than for Journalism (p = 0.008).
33 Tests for Differences in Central Tendency (Mean cumulative GPAs) The line graphs which follow reflect the mean (average) cumulative GPAs for the three ACO student groups for each of the three semesters evaluated. Results for A & S students who were not in any of the ACO groups are presented for comparison purposes.
34
35
36 Comparing the three ACO groups only, the results of the analysis reflected no statistically significant difference in the mean cumulative GPAs for the Fall 2013 or Fall 2014 semesters: Given these results, there were no post-hoc analyses justified. On the other hand, when comparing the mean cumulative GPAs for the three ACO groups at the end of the Spring, 2014 semester, statistically significant effects were indeed identified (although due to the relatively high variability associated with the cumulative GPA values, the importance of the mean differences is relatively low, with ω 2 1%):
37 Following this Oneway ANOVA with Games-Howell post-hoc tests to identify the source(s) of the variability resulted in the following outcomes: indicating that on a statistical versus observational basis the mean cumulative GPAs in the Spring 2014 semester (i.e. after the first academic year) were equivalent for the ACO Engineering and ACO Business groups; both of which were significantly less (statistically) than for the ACO Journalism group. Of course, when adding the Non-ACO groups to the three semester analyses, all of the ANOVAs reflected significant differences in the mean cumulative GPAs recorded:
Utilizing a combination of Tukey and Games-Howell post-hoc tests where each are appropriate, the mean cumulative GPAs in every one of the three semesters are statistically significantly higher for Non-ACO students than for ACO students in any of the three groups: 38
39 Summary of Results: Retention, Probation, and Academic Performance Fall 2013 Cohort - Retention rates for all students in the College of Arts & Sciences tend to be lower than for the entire campus, with all schools and colleges combined; - For the first three semesters of their academic experience, retention rates for ACO students in any group were lower than for Non-ACO students in the College of Arts and Sciences, with sequential differences of 0.5%, 3.1%, and 6%, respectively; - Beginning in the Fall 2014, meaningful and statistically significant differences in cumulative retention rates among the ACO Business, ACO Engineering, and ACO Journalism groups become apparent, with: o Journalism (84.8%) > Business (81.3%) > Engineering (76.8%) for Fall 2014, versus a Non-ACO retention rate of 82.3% for the same period; and o Journalism (81.8%) > Business (74.6%) > Engineering (70.8%) for Fall 2014, versus a Non-ACO retention rate of 79.1% for the same period; - First semester (Fall 2013) Probation Rates for ACO students (25.8%) as a singular group were significantly higher than for Non-ACO students (14.6%) in the College of Arts & Sciences; - First semester probation rates between the three ACO groups were all higher than for the Non-ACO students (14.6%), with ACO Business (29.1%) > ACO Engineering (24.64%) > ACO Journalism (16.67%); - Second semester (Spring 2014) academic dismissal rates for ACO students as a singular group (11.8%) were significantly higher than for Non-ACO students (6.0%); - Second semester academic dismissal rates for the ACO Enginnering and ACO Business groups (12.7% and 12.0%, respectively) were significantly higher than for Non-ACO students (6.0%), which was higher than for the ACO Journalism students (4.5%); - Clearly, ACO Engineering and ACO Business student performance in the context of retention, probation, and dismissal rates negatively impact the overall indices for the College of Arts & Sciences, and uniformly so; - Non-ACO students, as compared to the ACO students evaluated as a single group, performed at a superior level of academic performance, as measured by cumulative GPAs in each of the first three semesters, recording mean GPA differences of 0.43, 0.44, and 0.39, respectively;
- Observationally, mean cumulative GPAs for the ACO Business and ACO Engineering students were always lower than for the ACO Journalism students, at a statistically significant difference, at the end of the second semester (Spring 2014); finally - Even as compared to the highest of the mean cumulative GPAs recorded by any of the ACO groups (Journalism), Non-ACO students uniformly performed at a significantly higher level than ACO students. 40
41 Program and Major Transition Through Time: ACO - Business and ACO Engineering Students Fall 2013 Cohort The final request for analysis as associated with this study was to determine the transitional pattern for students admitted to A & S as ACO students within their first two years at CU Boulder. For this analysis, only the ACO Business and ACO Engineering students were evaluated, as ACO Journalism students no longer exist as an option, so understanding their transitional pattern would provide little useful information insofar as future decisions are concerned. ACO - Business Tracking the 374 students who were admitted to the ACO Business group in the Fall 2013 cohort revealed the following results:
As shown by these data, once ACO Business students transitioned to a program outside of A & S, they tended to remain in that school or college. The summary results illustrating the status of the 374 ACO Business students after their first two academic years at CU Boulder are reflected by the bar graph below. 42
43 As shown by these summary results, only 11% of the ACO Business students were registered in the Leeds School of Business by Spring 2015; which is less than half the percentage of students from this group who are no longer attending the University (27.8%). For those students still registered in A & S, the distribution of declared / undeclared majors appears as follows:
Focusing on the majors that accounted for 2% or more of the students in those categories: 44
45 ACO - Engineering Tracking the 487 students who were admitted to the ACO Engineering group in the Fall 2013 cohort revealed the following results:
46
47 One of the significant differences in the transition patterns of the ACO Engineering group as compared to the ACO Business group is that after transitioning to other schools or colleges, there was more movement by the students; either to different programs, or leaving CU Boulder. The final tally for the current status of the ACO Engineering group after the first two years of their academic career is reflected on the bar chart which follows.
48 As compared to the ACO Business group, a higher percentage of ACO Enginnering students are currently enrolled in the CEAS (17.9%) versus the 11% previously identified. On the other hand, a higher percentage of ACO Engineering students have left CU (30.6%) than left from the ACO Business group (27.8%). For those ACO Engineering students still enrolled in A & S for the Spring 2015 semester, the distribution of majors is illustrated on the following bar graph:
Focusing on the majors that accounted for 2% or more of the students in those categories: 49
50 Summary of Results Program and Major Transition Through Time: ACO - Business and ACO Engineering Students Fall 2013 Cohort - Only 11% of the ACO Business students were registered in the LSB by the beginning of the Spring 2015 semester; - 27.8% of the ACO Business students had left CU Boulder by the beginning of the Spring 2015 semester; - The most frequently selected majors in the College of A & S for ACO Business students who did not transfer into other schools or colleges were, in order of preference, Communication, Economics, International Affairs, and Psychology; - Only 17.9% of the ACO Engineering students were registered in the CEAS by the beginning of the Spring 2015 semester; - 30.6% of the ACO Engineering students had left CU Boulder by the beginning of the Spring 2015 semester; and - The most frequently selected majors in the College of A & S for ACO Engineering students who did not transfer into other schools or colleges were, in order of preference, Computer Science, Integrative Psychology, and Environmental Studies.