REPORT ON ETS EPP TEST OF GENERAL EDUCATION SKILLS AND KNOWLEDGE

Transcription

1 REPORT ON ETS EPP TEST OF GENERAL EDUCATION SKILLS AND KNOWLEDGE Fall 2013 Spring 2014 Prepared by The Center for Teaching & Assessment of Learning Kevin R. Guidry, Ph.D. Senior Research Analyst Kathleen Langan Pusecker, M.S. Director of Educational Assessment Prepared: August 2014

2 Executive Summary At the direction of the Office of the Provost, the Center for Teaching & Assessment of Learning administered the short form of the ETS Educational Proficiency Profile (EPP) to firstyear and senior students at UD in the academic year to examine their general education skills and knowledge. The short form of the EPP is a 40-minute multiple choice test that includes questions about critical thinking, reading, writing, and mathematics. Of the 657 UD students who have taken the EPP, 307 students who completed more than 75% the test in 2010 and 329 completed more than 75% in Overall, we have some confidence that these two samples are relatively representative of the populations from which they were drawn based on their demographics, particularly their gender and race/ethnicity. The results of these tests are generally positive. In general, UD students who took the EPP performed better than students at other institutions who have taken the EPP. More importantly, UD students made statistically significant gains in every area tested by the EPP when comparing average results from first-year students to senior students in the same cohort indicating that students are learning general education skills and knowledge during their time at UD. EPP results also confirm that students who began in 2013 were more academically qualified than those who began in Test results also indicate that the UD undergraduate education has remained broadly consistent during the past four years as both groups of senior students performed similarly despite performing differently as first-year students. Based on these test results, our experience administering the EPP for a second time, experience with general education at the University of Delaware, and our broader knowledge of U.S. higher education, we recommend that UD:

3 1. Adopts a cross-college, cross-discipline method of assessing general education skills and knowledge that is more actionable. EPP scores are not actionable given their vagueness and opacity. Smaller, more targeted assessments grounded in coursework or disciplinary work may be more appropriate and useful. Similarly, electronic portfolios or expanded transcripts where students are required to upload work that they feel indicates mastery of the general education goals to be assessed by UD faculty (e.g., CTAL Assessment Scholars using AAC&U rubrics) may be more viable, actionable, and accepted by UD constituents. 2. Places a greater emphasis on assessing samples of students that reflect UD priorities and needs. For example, if we are particularly interested in large-scale, longitudinal work then we should ensure we are assessing representative samples that are ideally the same ones at different points in time e.g., assess the same group of students as first-year students and as seniors. This could also include targeted assessments of specific groups of students e.g., quantitative reasoning skills of non-stem students. This work would be greatly enabled if UD were to adopt a regular, scheduled time for the structured collection of assessment data and reflection on how it can improve university practices like the Assessment Day held twice annually at James Madison University.

4 Table of Contents Executive Summary... 2 Table of Contents... 4 Introduction... 6 Study Sample and Representativeness... 6 Test Results... 9 Between-university comparisons All four-year institutions Public research universities Within-university comparisons First-year-2010-cohort-to-senior-2013-cohort First-year-cohort-to-first-year-cohort Senior-cohort-to-senior-cohort Implications Recommendations Appendix A : ETS Proficiency Profile Levels Reading and Critical Thinking Level Level

5 Level Writing Level Level Level Mathematics Level Level Level Appendix B : All EPP-Participating 4-Year Institutions... 32

6 Introduction At the direction of the Office of the Provost, the Center for Teaching & Assessment of Learning administered the short form of the ETS Educational Proficiency Profile (EPP) to firstyear and senior students at UD in the academic year to examine their general education skills and knowledge. The short form of the EPP is a 40-minute multiple choice test that includes questions about critical thinking, reading, writing, and mathematics. We also administered the EPP in 2010 so we are able to compare responses from first-year students and senior students in the same cohort. Study Sample and Representativeness In total, 657 UD students have completed the EPP. Once we remove the students who completed less than 75% of the test questions 1 we have 307 students who completed the test in 2010 and 329 who completed it in as summarized in Table 1. Overall, we have some confidence that these two samples are relatively representative of the populations from which they were drawn based on their demographics, particularly their gender and race/ethnicity. Table 1: UD EPP Sample First-year students Entering first-year students (0 credits) First-year students with more than zero but less than 30 credits Senior students Other 30 7 Total This removes only 21 (3%) of the students from the analysis, 10 (3%) from 2010 and 11 (3%) from 2013-

7 In terms of gender and race/ethnicity, the students who have taken the EPP are roughly comparable to the broader UD student population. Table 2 displays the gender of UD students and EPP test-takers with Table 3 displaying their race/ethnicity. The percentages of students in each of these categories are comparable and were improved in the administration. Some of the differences in the race/ethnicity numbers, particularly the overrepresentation of Asian students, are likely due to differences in how these data are collected for the EPP and for other UD purposes especially since the EPP does not distinguish between U.S. citizens and international students. Table 2: Gender of UD Undergraduate Population and EPP Sample n Percentage n Percentage Female UD 9,067 57% 9,081 58% EPP % % Male UD 6,820 43% 6,676 42% EPP % % Unknown UD n/a n/a n/a n/a EPP 11 4% 1 < 1%

8 Table 3: Race/ethnicity of UD Undergraduate Population and EPP Sample n Percentage n Percentage White UD 12,322 78% 12,789 76% EPP % % African-American UD 797 5% 834 5% EPP 6 2% 14 4% Hispanic UD 891 6% 1,144 7% EPP 11 3% 19 6% Asian 2 UD 648 4% 718 4% EPP 22 7% 26 8% Native American UD 21 < 1% 13 < 1% EPP 1 < 1% 1 < 1% International 2 UD 524 3% 723 4% EPP n/a n/a n/a n/a Other 2 UD 684 4% 650 4% EPP 27 9% 16 5% In both 2010 and , we recruited first-year and senior students to take the EPP. For first-year students in both rounds of testing, we asked faculty teaching First-Year Seminar (FYS) courses if we could administer the test during one of their class sessions in the fall. We targeted faculty teaching a diverse set of FYS courses including some that were largely aimed at students with particular declared majors and those with a high percentage of University Studies students. When recruiting senior students in 2010, we relied largely on open recruitment of UD students via posters. In , we changed this procedure to be similar to our first-year student process except we focused on capstone classes instead of FYS classes. As shown in Table 4, we were not entirely successful in completely representing the student body in terms of the UD college in which they have a declared major. However, this is a flawed and difficult comparison because the EPP has a limited selection of majors that a student can choose and the 2 These categories are not directly comparable between the UD population and EPP sample because of different categorizations and questions.

9 data files do not include the written Other: responses from the students. Therefore, it is possible that much of the discrepancy between the study sample and the population is hidden in the nearly one-third of students who selected that option. For example, one of the senior capstone classes to whom we administered the EPP was a sports management course, a major in the Alfred Lerner College of Business & Economics but one not listed in the EPP for students to select. Table 4: College of UD Undergraduate Population and EPP Sample EPP UD EPP UD College of Agriculture & Natural 1% 4% 2% 4% Resources Alfred Lerner College of Business & 6% 18% 0% 17% Economics College of Arts & Sciences 24% 39% 33% 37% College of Education & Human 5% 6% 2% 6% Development College of Engineering 5% 11% 15% 14% College of Earth, Ocean, & Environment 6% 1% 1% 2% College of Health Sciences 3% 12% 9% 14% No answer/other 30% n/a 31% n/a Office of University Studies 21% 7% 9% 7% Test Results There are two groups of results that are returned to institutions whose students participate in the Educational Proficiency Profile. First, norm-referenced scaled numeric scores for each skill area are included in the data files. These scores range from 100 to 130 and are not comparable to one another but are intended to be compared with the scores of other students. Second, criterion-referenced proficiency levels for each skill area are included in the data files. Students are rated using these proficiency levels as proficient, marginal, or not proficient at each

10 level of each skill area. More detailed information on the construction of these proficiency levels can be found on ETS s website at One minor methodological note about student class level is necessary. In the EPP, students were asked to identify how many course hours they had completed: None entering freshman, Fewer than 30 semester hours or fewer than 45 quarter hours, semester hours or quarter hours, semester hours or quarter hours, or More than 90 semester hours or more than 145 quarter hours. ETS reports only allow us to select all students or students who selected only one of those response options so between-university comparisons are somewhat stilted. However, for within-university comparisons we are able to use raw data and create a derived class variable that combines the first two response options into a more representative first-year response category and retain the final response option as the senior response category. We omit students who selected the remaining response options as they were mistakenly included in the study sample, an artifact of how we recruited senior student participants by focusing on capstone classes in which a handful of junior students were also enrolled. Between-university comparisons. The reporting tools offered by ETS permit us to select peer comparison groups and receive summary statistics for those groups. We can make these comparisons at the student and the institution level. However, this tool is limited in several ways. First, the group of institutions that can be included in our selection is somewhat limited as the EPP is only one of many ways to assess general education goals and other institutions are free to employ other methods e.g., the Collegiate Learning Assessment, electronic portfolios. Second, we are constrained by how we can select the student groups to be compared in that we

11 can only choose to compare all of our students (who have taken the test) to all of the students at our peer institutions (who have taken the test) or we can select one particular (limited) class level as described above. In this report, we compare UD students to two other groups of students. First, we compare UD students to students at four-year colleges and universities who have taken the EPP since July Second, we compare UD students to students at universities that are somewhat comparable to UD. All four-year institutions. As shown in Table 5, students at several hundred colleges and universities have participated in the EPP since Although this group of colleges and universities may not be completely representative of the entire population of four-year colleges and universities in the United States, there are enough of them to make a comparison between these groups of students and UD students a useful one. The entire listing of these institutions is included as Appendix B. Table 5: EPP Four-year Colleges and Universities Entering first-year student with 0 credits Less than 30 credits but more than 0 credits Senior students Number of institutions Total number of students 167,916 51, ,711 As shown in the following tables, UD students fare well in all areas included in the EPP when compared against students at all of the four-year colleges and universities who have administered the EPP. We caution, however, that this may not be a very meaningful comparison since we do not know anything about the methods by which students at other institutions were selected to participate nor do we know the purposes for which these institutions administered the

12 EPP. For those reasons, we only supply basic descriptive statistics in these three tables and the difference between the student-level averages without any statistical analysis that may lead readers to lend more weight to these comparisons than is appropriate. Additional tables list the percentages of students who were classified in each of ETS s proficiency levels. Table 6: Entering First-Year Student Score Comparisons With All EPP Four-Year Institutions University of Delaware (n=276) All four-year institutions (n=137,875 3 ) Mean Standard Deviation Mean Standard Deviation Difference Critical Thinking Reading Writing Math Humanities Social Sciences Natural Sciences Total This is a weighted total that corrects for institutional size so institutions with very large samples are not disproportionately represented in the reported statistics.

13 Table 7: Entering First-Year Student Proficiency Comparisons With All EPP Four-Year Institutions All four-year institutions University of Delaware (n=276) (n=137,875) Not Proficient Marginal Proficient Not Proficient Marginal Proficient Critical Thinking 80% 16% 4% 88% 10% 2% Reading, Level 2 42% 26% 32% 61% 17% 21% Reading, Level 1 11% 21% 68% 29% 23% 48% Writing, Level 3 56% 32% 12% 78% 17% 5% Writing, Level 2 34% 37% 28% 58% 29% 12% Writing, Level 1 4% 26% 71% 19% 32% 49% Mathematics, Level 3 69% 23% 8% 85% 11% 4% Mathematics, Level 2 36% 27% 37% 58% 23% 19% Mathematics, Level 1 14% 20% 66% 31% 27% 41% Table 8: First-Year Students With Some Credits Score Comparisons With All EPP Four-Year Institutions University of Delaware (n=95) All four-year institutions (n=37,915 3 ) Mean Standard Deviation Mean Standard Deviation Difference Critical Thinking Reading Writing Math Humanities Social Sciences Natural Sciences Total

14 Table 9: First-Year Students With Some Credits Proficiency Comparisons With All EPP Four- Year Institutions University of Delaware (n=276) All four-year institutions (n=137,875) Not Proficient Marginal Proficient Not Proficient Marginal Proficient Critical Thinking 80% 16% 4% 86% 11% 3% Reading, Level 2 48% 22% 29% 60% 17% 24% Reading, Level 1 14% 23% 63% 29% 22% 49% Writing, Level 3 57% 26% 17% 77% 18% 5% Writing, Level 2 33% 37% 31% 58% 29% 13% Writing, Level 1 8% 21% 71% 21% 31% 48% Mathematics, Level 3 74% 20% 6% 84% 11% 5% Mathematics, Level 2 34% 35% 32% 58% 22% 20% Mathematics, Level 1 14% 22% 64% 34% 25% 41% Table 10: Senior Student Score Comparisons With All EPP Four-Year Institutions University of Delaware (n=228) All four-year institutions (n=134,871 3 ) Mean Standard Deviation Mean Standard Deviation Difference Critical Thinking Reading Writing Math Humanities Social Sciences Natural Sciences Total

15 Table 11: Senior Student Proficiency Comparisons With All EPP Four-Year Institutions University of Delaware (n=276) All four-year institutions (n=137,875) Not Proficient Marginal Proficient Not Proficient Marginal Proficient Critical Thinking 54% 30% 17% 74% 19% 7% Reading, Level 2 25% 19% 57% 42% 19% 39% Reading, Level 1 7% 10% 82% 17% 17% 66% Writing, Level 3 45% 32% 23% 65% 26% 9% Writing, Level 2 18% 43% 39% 43% 36% 22% Writing, Level 1 3% 12% 85% 12% 25% 63% Mathematics, Level 3 47% 30% 23% 74% 17% 9% Mathematics, Level 2 18% 21% 61% 45% 25% 31% Mathematics, Level 1 4% 14% 82% 22% 23% 55% Public research universities. Although a comparison of UD students against all students at four-year institutions is a reasonable starting point, a more useful comparison is one made against students at peer institutions. Although we tried to use preexisting lists of peer and comparison groups constructed by the Office of the Provost, the number of institutions that have participated in the EPP and the ETS-imposed requirement that comparison groups to be of a minimum size of ten required us to expand our selections to include all public institutions with the same Basic Carnegie Classification of RU/VH: Research Universities (very high research activity); these institutions and the number of students who participated in the EPP are listed in Table 12.

16 Table 12: EPP Peer Institutions Entering firstyear student with 0 credits Less than 30 credits but more than 0 credits Senior students Arizona State University - Tempe, AZ Clemson University, SC 5,258 1,490 1,771 Mississippi State University, MS 2, Indiana University - Bloomington, IN Montana State University, MT Mississippi State University, MS ,840 Montana State University, MT North Carolina State University, NC Purdue University - West Lafayette, IN University of Alabama at Birmingham, AL 2, University of Alabama in Huntsville, AL University of Cincinnati, OH University of Georgia, GA University of Illinois, IL University of Missouri - Columbia, MO ,133 University of South Carolina - Columbia, SC University of South Florida - Tampa, FL University of Tennessee - Knoxville, TN Wayne State University, MI Total 11,880 3,820 9,633 The next set of tables compares the scores and proficiency levels of UD students against the average scores and proficiency levels of students at all of the four-year public research universities that have participated in the EPP. As before, UD students fare well in all areas included in the EPP. We again caution, however, that this may not be a very meaningful comparison since we do not know anything about the methods by which students at other institutions were selected to participate nor do we know the purposes for which these institutions administered the EPP.

17 Table 13: Entering First-Year Student Score Comparisons With EPP Public Research Universities University of Delaware (n=276) EPP Peers (n=9,622 4 ) Mean Standard Deviation Mean Standard Deviation Difference Critical Thinking Reading Writing Math Humanities Social Sciences Natural Sciences Total Table 14: Entering First-Year Student Proficiency Comparisons With EPP Public Research Universities University of Delaware (n=276) EPP Peers (n=9,622) Not Proficient Marginal Proficient Not Proficient Marginal Proficient Critical Thinking 80% 16% 4% 76% 19% 5% Reading, Level 2 42% 26% 32% 40% 22% 38% Reading, Level 1 11% 21% 68% 13% 17% 70% Writing, Level 3 56% 32% 12% 60% 29% 11% Writing, Level 2 34% 37% 28% 37% 40% 24% Writing, Level 1 4% 26% 71% 6% 23% 70% Mathematics, Level 3 69% 23% 8% 70% 20% 10% Mathematics, Level 2 36% 27% 37% 36% 27% 37% Mathematics, Level 1 14% 20% 66% 14% 21% 64% 4 This is a weighted total that corrects for institutional size so institutions with very large samples are not disproportionately represented in the reported statistics.

18 Table 15: First-Year Students With Some Credits Score Comparisons With EPP Public Research Universities University of Delaware (n=95) EPP Peers (n=3,230 4 ) Mean Standard Deviation Mean Standard Deviation Difference Critical Thinking Reading Writing Math Humanities Social Sciences Natural Sciences Total Table 16: First-Year Students With Some Credits Proficiency Comparisons With EPP Public Research Universities University of Delaware (n=276) EPP Peers (n=3,230) Not Proficient Marginal Proficient Not Proficient Marginal Proficient Critical Thinking 80% 16% 4% 70% 23% 6% Reading, Level 2 48% 22% 29% 35% 21% 44% Reading, Level 1 14% 23% 63% 11% 16% 73% Writing, Level 3 57% 26% 17% 56% 31% 13% Writing, Level 2 33% 37% 31% 30% 42% 28% Writing, Level 1 8% 21% 71% 6% 20% 75% Mathematics, Level 3 74% 20% 6% 60% 25% 14% Mathematics, Level 2 34% 35% 32% 25% 27% 48% Mathematics, Level 1 14% 22% 64% 11% 14% 76%

19 Table 17: Senior Student Score Comparisons With EPP Public Research Universities University of Delaware (n=228) EPP Peers (n=7,793 4 ) Mean Standard Deviation Mean Standard Deviation Difference Critical Thinking Reading Writing Math Humanities Social Sciences Natural Sciences Total Table 18: Senior Student Proficiency Comparisons With EPP Public Research Universities University of Delaware (n=276) EPP Peers (n=7,793) Not Proficient Marginal Proficient Not Proficient Marginal Proficient Critical Thinking 54% 30% 17% 60% 27% 13% Reading, Level 2 25% 19% 57% 27% 19% 53% Reading, Level 1 7% 10% 82% 8% 13% 79% Writing, Level 3 45% 32% 23% 52% 33% 16% Writing, Level 2 18% 43% 39% 27% 41% 32% Writing, Level 1 3% 12% 85% 4% 18% 78% Mathematics, Level 3 47% 30% 23% 53% 26% 21% Mathematics, Level 2 18% 21% 61% 23% 24% 54% Mathematics, Level 1 4% 14% 82% 8% 15% 77% Within-university comparisons. ETS also supplies us with the ability to create reports that compare groups of UD students to other groups of UD students. There are essentially four groups of UD students who have taken the EPP: first-year 5 and senior students in 2010 and firstyear 5 and senior students in Comparing first-year and senior students who took the 5 For purposes of these within-university comparisons, first-year students are those who indicated on the EPP that they had either no credit hours or fewer than 30 semester hours, two different response options combined into category for these analyses.

20 test in the same year is a comparison that yields little actionable, useful information because those are different groups of students who were admitted in different years under different conditions and have had different experiences at UD. However, the other three comparisons may yield useful information. First-year-2010-cohort-to-senior-2013-cohort. The most meaningful comparison we can make with these data is to compare the scores of first-year students with those of senior students who are in the same class. Only three students who took the test in 2010 and but this remains a meaningful comparison since the sizes of the sample and the sampling methods make us relatively confident in our ability to generalize these scores to the entire class. Table 19, illustrated in Figure 1, shows that this cohort of UD students made significant gains in areas tested by the EPP with nearly all areas showing a large effect size. Table 20, illustrated in Figure 2, shows similar information using the ETS proficiency scores. Table 19: First-year-2010-cohort-to-senior-2013-cohort Score Comparisons 2010 (n=185) (n=136) Mean Standard Deviation Mean Standard Deviation Difference p Effect size Critical < Thinking Reading < Writing < Math < Humanities Social < Sciences Natural < Sciences Total < Table 20: First-year-2010-cohort-to-senior-2013-cohort Proficiency Comparisons 2010 (n=185) (n=136)

21 Not Proficient Marginal Proficient Not Proficient Marginal Proficient Critical Thinking 88% 10% 2% 56% 30% 14% Reading, Level 2 54% 22% 24% 24% 22% 54% Reading, Level 1 15% 25% 60% 7% 10% 82% Writing, Level 3 60% 32% 8% 43% 31% 26% Writing, Level 2 42% 35% 23% 20% 43% 37% Writing, Level 1 8% 27% 65% 4% 13% 83% Mathematics, Level 3 81% 16% 4% 43% 34% 23% Mathematics, Level 2 45% 30% 25% 13% 19% 68% Mathematics, Level 1 22% 24% 55% 1% 11% 88% Figure 1: First-year-2010-cohort-to-senior-2013-cohort Score Comparisons Critical Thinking Reading Writing Math Humanities Social Sciences Natural Sciences First-year students (2010) Senior students ( )

22 Figure 2: First-year-2010-cohort-to-senior-2013-cohort Proficiency Comparisons 100% 90% 80% 70% 60% 54% 60% 82% 65% 83% 68% 55% 88% 50% 40% 37% 30% 20% 10% 0% 2% 14% 24% 8% 26% 23% 4% 23% 25% First-year students (2010) Senior students ( ) First-year-cohort-to-first-year-cohort. Comparing the first-year cohort in 2010 to the firstyear cohort in 2013 provides some information about the levels of preparation and ability of these two different groups of students who have not spent much time at UD. As shown in Table 21, UD students who were admitted in 2013 scored significantly higher than students admitted in Effect sizes for these differences were small or medium for each subscale with a large cumulative effect size for the total scaled score. This is consistent with the small increase in average SAT scores for incoming (first-time first-year) students in 2013 compared to their peers in 2010.

23 Table 21: First-year-cohort-to-first-year-cohort Score Comparisons 2010 (n=185) 2013 (n=186) Mean Standard Deviation Mean Standard Deviation Difference p Effect size Critical Thinking < Reading < Writing < Math < Humanities Social Sciences < Natural Sciences < Total < Table 22: First-year-cohort-to-first-year-cohort Proficiency Comparisons 2010 (n=185) (n=186) Not Proficient Marginal Proficient Not Proficient Marginal Proficient Critical Thinking 88% 10% 2% 73% 21% 6% Reading, Level 2 54% 22% 24% 33% 29% 38% Reading, Level 1 15% 25% 60% 9% 18% 73% Writing, Level 3 60% 32% 8% 53% 28% 19% Writing, Level 2 42% 35% 23% 26% 39% 34% Writing, Level 1 8% 27% 65% 2% 22% 76% Mathematics, Level 3 81% 16% 4% 60% 29% 11% Mathematics, Level 2 45% 30% 25% 25% 28% 47% Mathematics, Level 1 22% 24% 55% 7% 17% 76% Senior-cohort-to-senior-cohort. Since the 2010 and 2013 first-year student groups were relatively comparable with small but significant increases for the 2014 cohort, it is useful to compare the two senior cohorts to see if there are similar differences. Table 23 shows that only one subscale, math, is significantly different with the cohort scoring significantly higher with a medium effect size.

24 Table 23: Senior-cohort-to-senior-cohort Score Comparisons 2010 (n=92) (n=136) Mean Standard Deviation Mean Standard Deviation Difference p Effect size Critical Thinking Reading Writing Math < Humanities Social Sciences Natural Sciences Total Table 24: Senior-cohort-to-senior-cohort Proficiency Comparisons 2010 (n=92) (n=136) Not Proficient Marginal Proficient Not Proficient Marginal Proficient Critical Thinking 50% 29% 21% 56% 30% 14% Reading, Level 2 26% 14% 60% 24% 22% 54% Reading, Level 1 8% 10% 83% 7% 10% 82% Writing, Level 3 47% 35% 18% 43% 31% 26% Writing, Level 2 16% 41% 42% 20% 43% 37% Writing, Level 1 2% 11% 87% 4% 13% 83% Mathematics, Level 3 52% 25% 23% 43% 34% 23% Mathematics, Level 2 27% 23% 50% 13% 19% 68% Mathematics, Level 1 10% 17% 73% 1% 11% 88% Implications The picture painted by these test scores is generally positive. Between-university comparisons show that the UD students who took the EPP, students who are broadly representative of their respective classes, performed better than students at other institutions who have taken the EPP. The differences narrow when comparing UD students to students at other public Research Universities (very high research activity) universities and in one instance, first-

25 year students with some credits (e.g., transfer students, first-year students with AP credits), UD students received lower scores. However, these are not very informative comparisons because we do not know the sampling methods of these other institutions or the purposes for which they administered the EPP. Moreover, very few of our peers have participated in the EPP making it challenging to determine how our students perform relative to students at comparable universities. It is more useful and informative to compare the EPP results of different groups of UD students to inform and confirm our understandings and intuitions of UD students and a UD education. First, UD students made statistically significant gains in every area tested by the EPP when comparing average results from first-year students to senior students in the same cohort. In the broadest sense, this indicates that students are indeed learning general education skills and knowledge during their time at UD. Second, the EPP confirms that students who began in 2013 were more academically qualified than those who began in 2010 (as already known by their higher average SAT scores). Third, senior students in 2010 and had EPP scores that were statistically indistinguishable with only their math scores being significantly higher for senior students. This may be interpreted as an indication that the UD undergraduate education has remained broadly consistent during the past four years as both groups of students rose to the same levels of proficiency in these general education skills despite starting from slightly different places. Recommendations Based on these test results, our experience administering the EPP for a second time, experience with general education at the University of Delaware, and our broader knowledge of U.S. higher education, the Center for Teaching & Assessment of Learning makes the

26 recommendations below. Although these recommendations would likely be carried out or coordinated by CTAL, we require specific guidance and charges from the Office of the Provost, ideally following extensive consultation with faculty, to implement these university-wide, multiyear assessments. 1. Adopt a cross-college, cross-discipline method of assessing general education skills and knowledge that is more actionable. EPP scores are not actionable given their vagueness and opacity. Smaller, more targeted assessments grounded in coursework or disciplinary work may be more appropriate and useful. Similarly, electronic portfolios or expanded transcripts where students are required to upload work that they feel indicates mastery of the general education goals to be assessed by UD faculty (e.g., CTAL Assessment Scholars using AAC&U rubrics) may be more viable, actionable, and accepted by UD constituents. Work that directly involves faculty such as the Critical thinking Assessment Test (CAT) would also be an effective, viable means of assessing a significant portion of our goals. 2. Place a greater emphasis on assessing samples of students that reflect UD priorities and needs. For example, if we are particularly interested in large-scale, longitudinal work then we should ensure we are assessing representative samples that are ideally the same ones at different points in time e.g., assess the same group of students as first-year students and as seniors. This could also include targeted assessments of specific groups of students e.g., quantitative reasoning skills of non-stem students. This work would be greatly enabled if UD were to adopt a regular, scheduled time for the structured collection of assessment data and reflection on how it can improve

27 university practices like the Assessment Day held twice annually at James Madison University.

28 Appendix A: ETS Proficiency Profile Levels 6 Reading and Critical Thinking Level 1. To be considered proficient at Level 1, students should be able to: recognize factual material explicitly presented in a reading passage understand the meaning of particular words or phrases in the context of a reading passage Level 2. To be considered proficient at Level 2, students should be able to: synthesize material from different sections of a passage recognize valid inferences derived from material in the passage identify accurate summaries of a passage or of significant sections of the passage understand and interpret figurative language discern the main idea, purpose or focus of a passage or a significant portion of the passage Level 3. To be considered proficient at Level 3, students should be able to: evaluate competing causal explanations evaluate hypotheses for consistency with known facts determine the relevance of information for evaluating an argument or conclusion determine whether an artistic interpretation is supported by evidence contained in a work 6 Copied from

29 evaluate the appropriateness of procedures for investigating a question of causation evaluate data for consistency with known facts, hypotheses or methods recognize flaws and inconsistencies in an argument Writing Level 1. To be considered proficient at Level 1, students should be able to: recognize agreement among basic grammatical elements (e.g., nouns, verbs, pronouns and conjunctions) recognize appropriate transition words recognize incorrect word choice order sentences in a paragraph order elements in an outline Level 2. To be considered proficient at Level 2, students should be able to: incorporate new material into a passage recognize agreement among basic grammatical elements (e.g., nouns, verbs, pronouns and conjunctions) when these elements are complicated by intervening words or phrases combine simple clauses into single, more complex combinations recast existing sentences into new syntactic combinations Level 3. To be considered proficient at Level 3, students should be able to: discriminate between appropriate and inappropriate use of parallelism discriminate between appropriate and inappropriate use of idiomatic language recognize redundancy

30 discriminate between correct and incorrect constructions recognize the most effective revision of a sentence Mathematics Level 1. To be considered proficient at Level 1, students should be able to: solve word problems that would most likely be solved by arithmetic and do not involve conversion of units or proportionality. These problems can be multistep if the steps are repeated rather than embedded. solve problems involving the informal properties of numbers and operations, often involving the Number Line, including positive and negative numbers, whole numbers and fractions (including conversions of common fractions to percent, such as converting "1/4" to 25 percent). solve problems requiring a general understanding of square roots and the squares of numbers. solve a simple equation or substitute numbers into an algebraic expression. find information from a graph. This task may involve finding a specified piece of information in a graph that also contains other information. Level 2. To be considered proficient at Level 2, students should be able to: solve arithmetic problems with some complications, such as complex wording, maximizing or minimizing and embedded ratios. These problems include algebra problems that can be solved by arithmetic (the answer choices are numeric). simplify algebraic expressions, perform basic translations, and draw conclusions from algebraic equations and inequalities. These tasks are more complicated than

31 solving a simple equation, though they may be approached arithmetically by substituting numbers. interpret a trend represented in a graph, or choose a graph that reflects a trend. solve problems involving sets; problems have numeric answer choices. Level 3. To be considered proficient at Level 3, students should be able to: solve word problems that would be unlikely to be solved by arithmetic; the answer choices are either algebraic expressions or numbers that do not lend themselves to back-solving solve problems involving difficult arithmetic concepts, such as exponents and roots other than squares and square roots, and percent of increase or decrease generalize about numbers (e.g., identify the values of (x) for which an expression increases as (x) increases) solve problems requiring an understanding of the properties of integers, rational numbers, etc. interpret a graph in which the trends are to be expressed algebraically or one of the following is involved: exponents and roots other than squares and square roots, percent of increase or decrease solve problems requiring insight or logical reasoning

32 Appendix B: All EPP-Participating 4-Year Institutions Entering firstyear student with 0 credits Less than 30 credits but more than 0 credits Institution Abilene Christian University, TX Alabama A&M University, AL 1, Alabama State University, AL 3,310 1, Albertus Magnus College, CT Alcorn State University, MS Alice Lloyd College, KY American Intercontinental University, IL American Public University, WV ,174 American Sentinel University, CO Anderson University - South Carolina, SC 2, Andrews University, MI 0 0 1,377 Antioch University McGregor, OH Aquinas College (MI), MI Arizona State University - Tempe, AZ Senior students Arkansas State University, AR Armstrong Atlantic State University, GA 0 0 3,870 Asbury University, KY Ashford University, IA 1,255 2,198 2,364 Athens State University, AL Austin Peay State University, TN 0 0 1,007 Ave Maria University, FL Azusa Pacific University, CA 1, Baldwin Wallace University, OH Bauder College, GA Belhaven University (MS), MS Bellarmine University, KY Bemidji State University, MN Benedict College, SC 868 1,140 0 Bennett College for Women, NC Bethel College, IN Bethel University, TN 1, Biola University, CA Bloomsburg University, PA 7,384 1,222 2,028 Blue Mountain College, MS

33 Entering firstyear student with 0 credits Less than 30 credits but more than 0 credits Institution Bluffton University, OH Bowie State University, MD Bradley University, IL Brenau University, GA Brescia University, KY Brewton-Parker College, GA Bridgewater College, VA Bryan College, TN Cairn University, PA California University of Pennsylvania, PA Campbell University, NC 1, Capella University, MN Capital University, OH 2, Cazenovia College, NY Charleston Southern University, SC Charter Oak State College, CT Cheyney University of Pennsylvania, PA Senior students Christian Brothers University, TN Citadel, The, SC Clarion University of Pennsylvania, PA Clark Atlanta University, GA 2, Clayton State University, GA 1, Clemson University, SC 5,258 1,490 1,771 Coastal Carolina University, SC College of Charleston, SC College of New Jersey, The, NJ College of New Rochelle, The, NY College of the Ozarks, MO Colorado Mesa University, CO Colorado State University- Global Campus, CO Colorado State University-Pueblo, CO Columbia College (MO), MO Concordia College, MN Concordia University (CA), CA Concordia University Chicago, IL 1, Concordia University Wisconsin, WI

34 Entering firstyear student with 0 credits Less than 30 credits but more than 0 credits Institution Concordia University, MI Coppin State University, MD Covenant College, GA Daemen College, NY Dallas Baptist University, TX Delaware Valley College, PA Denison University, OH DeVry University, IL Dickinson State University, ND 1, Dordt College, IA 1, East Stroudsburg University, PA 2, Eastern Mennonite University, VA Eastern New Mexico University, NM Eastern University, PA 1, Eckerd College, FL ECPI University, NC ,335 Edinboro University of Pennsylvania, PA Elizabeth City State University, NC 1, Endicott College, MA 1, Erskine College, SC Everglades University, FL Excelsior College, NY Felician College - Lodi, NJ Ferrum College, VA 1, Fisher College, MA Fisk University, TN Senior students Florida Agricultural and Mechanical 2, ,673 University, FL Florida Gulf Coast University, FL Florida International University, FL 2, ,004 Florida Southern College - Lakeland, FL Forman Christian College, non-us Fort Hays State University, KS Fort Lewis College, CO Francis Marion University, SC Friends University, KS Gardner-Webb University, NC

35 Entering firstyear student with 0 credits Less than 30 credits but more than 0 credits Institution George Fox University (OR), OR Georgetown College, KY Governors State University, IL Grambling State University, LA ,216 Grand Canyon University, AZ Guilford College, NC Harrison College - Indianapolis, IN Henderson State University, AR Hesser College, NH High Point University, NC Holy Family University, PA Houghton College, NY Houston Baptist University, TX Howard Payne University, TX Humboldt State University, CA Huston-Tillotson University, TX Indiana State University, IN Indiana University - Bloomington, IN Indiana University - System Office, IN 1, instituto Tecnologico y de Estudios Superiores de, non-us Senior students Jacksonville State University, AL Jarvis Christian College, TX Judson College, AL Kaplan University, IL Keystone College, PA Kutztown University of Pennsylvania, PA Kuyper College, MI LaGrange College, GA Lake Superior State University, MI 2, Lamar University, TX Lambuth University, TN Lander University, SC 2, ,461 Lee University, TN 0 0 1,283 LeTourneau University, TX Lewis-Clark State College, ID Liberty University, VA

36 Entering firstyear student with 0 credits Less than 30 credits but more than 0 credits Institution Limestone College, SC ,256 Lincoln Memorial University, TN Lindenwood University, MO Lock Haven University of PA, PA Loyola University New Orleans, LA Maharishi University of Management, IA Mansfield University, PA Maranatha Baptist University, WI Marian University (IN), IN Marietta College, OH Mary Baldwin College, VA Massachusetts Maritime Academy, MA 1, Senior students McNeese State University, LA Metropolitan State College of Denver, CO Midland University, NE Midway College, KY Milligan College, TN Minnesota State University Moorhead, MN Minnesota State University-Mankato- Economics, MN Minnesota State University, Mankato, MN Misericordia University, PA Mississippi College, MS 0 0 1,260 Mississippi State University, MS 2, ,840 Mississippi Valley State University, MS Missouri Southern State University, MO 2, ,323 Missouri State University, MO ,299 Missouri University of Science and Technology, MO Missouri Western State University, MO 0 0 2,846 Montana State University - Billings, MT Montana State University, MT Montana Tech of the University of Montana, MT Morehouse College, GA 1,

37 Entering firstyear student with 0 credits Less than 30 credits but more than 0 credits Institution Morgan State University, MD Mount Vernon Nazarene University, OH Neumann University, PA New Jersey Institute of Technology, NJ Newberry College, SC Nicholls State University, LA Norfolk State University, VA North Carolina A&T State University, NC North Carolina State University, NC North Carolina Wesleyan College, NC Northeastern Illinois University, IL Northeastern State University, OK Northern Arizona University, AZ 1, Northern Michigan University, MI Northwest Missouri State University, MO Senior students Norwich University, VT Nova Southeastern University - Law School, FL Oakland City University, IN Oglala Lakota College, SD Ohio Christian University, OH 2, Oklahoma Baptist University, OK Oklahoma State University, OK Ottawa University- Phoenix, AZ Ottawa University, KS Pace University - New York, NY Pacific Union College, CA Palm Beach Atlantic University, FL Patrick Henry College, VA Patten University, CA Paul Quinn College, TX Pfeiffer University, NC ,018 Philander Smith College, AR Point Loma Nazarene University, CA Point University, GA Prairie View A&M University, TX

38 Entering firstyear student with 0 credits Less than 30 credits but more than 0 credits Institution Presentation College, SD Providence College, RI Purdue University - West Lafayette, IN Senior students Queens University of Charlotte, NC Quinnipiac University, CT 3, Randolph-Macon College, VA Regent University, VA Reinhardt University, GA Rivier University, NH Robert B. Miller College, MI Rocky Mountain College, MT Rogers State University, OK Saint Augustines University, NC Saint Joseph's College (IN), IN Saint Leo University, FL 2, Saint Mary of the Woods University, IN Saint Mary's University, TX Saint Peter's College, NJ Saint Philips College, TX Sam Houston State University, TX Schreiner University, TX Seattle University, WA Shawnee State University, OH Shenandoah University, VA Shepherd University, WV Shorter University, GA Slippery Rock University of PA, PA South Carolina State University, SC South College, TN Southeast Missouri State University, MO 9,783 1,983 4,720 Southeastern University, FL 1, ,652 Southern Adventist University, TN 0 0 1,337 Southern Illinois University Carbondale, IL Southern Illinois University Edwardsville, IL Southern Utah University, UT

39 Entering firstyear student with 0 credits Less than 30 credits but more than 0 credits Institution Southern Vermont College, VT Southern Wesleyan University, SC Southwest Baptist University, MO Senior students Southwestern Assemblies of God University, TX Southwestern Christian College, TX Southwestern College, KS Spalding University, KY Spelman College, GA 1, Spring Hill College, AL St. Andrews University, NC Stephen F. Austin State University, TX 1, ,204 Sterling College, KS Stevenson University, MD Strayer University, DC SUNY at Binghamton, NY Talladega College, AL Tarleton State University, TX Taylor University, IN Temple University, PA Tennessee State University, TN 0 0 3,640 Tennessee Wesleyan College, TN Texas A&M University - Central Texas, TX Texas A&M University - San Antonio, TX Texas A&M University - Texarkana, TX Texas A&M University Kingsville, TX Texas A&M University-Commerce, TX Texas Tech University Health Sciences Center, TX Texas Wesleyan University, TX The New School, NY Thomas Edison State College, NJ ,118 Thomas More College, KY 1, Total 167,916 51, ,711 Touro College, CA