ACCUPLACER ENGLISH VALIDATION SUMMARY

Transcription

1 OFFICE OF INSTITUTIONAL RESEARCH ACCUPLACER ENGLISH VALIDATION SUMMARY The following brief summarizes the efforts of Santa Monica College to validate their use of a secondparty assessment test. The college uses the ACCUPLACER English Placement Test to place students into different levels of the English sequence. The brief contains the results of three different validation efforts that are required by colleges based on the guidelines mandated by the California Community College Chancellor s Office (CCCCO). Since the ACCUPLACER exam is currently approved by the CCCCO, the college must conduct a study addressing three standards: Content validity, cut-score validity and disproportionate impact. All three studies were conducted during the spring 2014 term. Content Review In March 2014, nine English faculty, including the department chair, reviewed a sample version of the ACCUPLACER English Placement exam. The exam is a used to place students in two different English levels, ENGL 21A (1-level below transfer) and ENGL 1 (transfer level). Students that do not take the exam, or do not meet the minimum score for placement into ENGL 21A may enroll in ENGL 85. The faculty were chosen based on their understanding of the exit and entrance skills of the required courses in the English sequence. Faculty were given the sample test, allowed time to discuss the major topics covered, and then asked to rate each question privately. For each test item faculty were asked to rate the importance of the knowledge or skill being measured for successful acquisition of the knowledge and skill taught in the course. They were instructed to use a four-point scale where 4= Critically important, 3 = Important, 2 = Slightly important and 1 = Not relevant. The ACCUPLACER English test has two components, a Sentence Skills section, and Reading Comprehension section. The following table presents the average scores for each section based on the rating of each individual question.

2 Sentence Skills Item ENGL 21 (Avg) Pre-req Skills ENGL 1 (Avg) Pre-req Skills 1 3 Yes 4 Yes 2 2 Yes 3 Yes 3 1 No 3 Yes 4 1 No 2 Yes 5 1 No 3 Yes 6 1 No 2 Yes 7 1 No 3 Yes 8 2 Yes 4 Yes 9 2 Yes 3 Yes 10 2 Yes 3 Yes 11 2 Yes 3 Yes 12 2 Yes 4 Yes 13 1 No 3 Yes 14 2 Yes 4 Yes 15 2 Yes 4 Yes 16 2 Yes 4 Yes 17 2 Yes 4 Yes 18 2 Yes 3 Yes 19 2 Yes 3 Yes 20 3 Yes 4 Yes 21 3 Yes 4 Yes 22 2 Yes 4 Yes 23 2 Yes 3 Yes 24 2 Yes 4 Yes 25 3 Yes 4 Yes Total Pre-req Skills Pre-req Skills Reading Skills Item ENGL 21 (Avg.) ENGL 1 (Avg.) 1 2 Yes 4 Yes 2 2 Yes 3 Yes 3 2 Yes 3 Yes 4 2 Yes 3 Yes 5 3 Yes 4 Yes 6 3 Yes 4 Yes 7 3 Yes 4 Yes 8 3 Yes 4 Yes 9 3 Yes 4 Yes 10 3 Yes 3 Yes 11 3 Yes 4 Yes 12 2 Yes 3 Yes 13 2 Yes 3 Yes 14 2 Yes 3 Yes 15 3 Yes 4 Yes

3 16 2 Yes 3 Yes 17 3 Yes 4 Yes 18 3 Yes 4 Yes 19 2 Yes 3 Yes 20 2 Yes 4 Yes 21 2 Yes 3 Yes 22 2 Yes 4 Yes Total The majority of items in both sections of the exam were rated as important for the successful acquisition of the knowledge and skills in ENGL 21A and ENGL 1. Study on Cut-Score Validity (On-Going) In order to validate the current cut scores, the college measured the consequential validity of its placement results. In order to show that there is consequential validity to the current cut-scores, faculty and students in English courses were surveyed on the appropriateness of student placement. The CCCCO has stated that at least 75% of faculty assessments and 75% of student self-assessments should show that students are adequately placed. Faculty Survey During the spring 2014 semester a randomly selected group of faculty teaching ENGL 85, ENGL 21A and ENGL 1 were sent an online survey. The survey contained a list of students in their course that had used the placement exam within the last year. The instructions for faculty were the following: This part of the validation is asking you to rate the extent to which you believe the students listed are placed appropriately. Not all students in your section will be listed. Simply mark each student as: 1. Unprepared for the course. Probably should have been placed into a lower level. 2. Adequately prepared for the course. Student was placed into the appropriate level. 3. Over-prepared for the course. Probably should have taken a higher level course. You should base your decision on their performance on quizzes, test or homework.

4 The following table shows the results of the spring 2014 faculty survey: Total Assessed Adequately Prepared Percent Should be placed higher Percent ENGL % 0 0.0% ENGL 21A % 1 4.5% ENGL % 4 5.2% Total % 5 4.5% While all course levels meet the 75% threshold, the numbers of responses in both ENGL 21A and ENGL 1 were low. Student Survey During the spring 2014 semester a survey was sent out to over 800 students enrolled in ENGL 85, ENGL 21A and ENGL 1 courses. These students must have been placed into those courses within the last year, and must not have had a previous ENGL enrollment on record. The instructions for students participating in the survey were: Please select the statement that best describes how you feel about your current ENGL course: I should be in a lower level course this class is too difficult for me I belong in this course this course is about the right level of difficulty for me I should be in a higher level course this course is too easy for me The following table shows the results of the spring 2014 student survey: Should be in Total Responses Belong in course Percent higher level course Percent ENGL % % ENGL 21A % % ENGL % % Total % % Only about 69.4% of students felt that they were placed in the correct course. No students surveyed felt they should be placed in a lower course. The response rates for ENGL 21A and ENGL 85 were lower than expected; there were less than 40 responses at each level.

5 Future Plans While over 25 surveys were sent to faculty and over 600 student surveys were ed out, only 23 and 121 surveys were completed, respectively. The college has decided to continue to survey faculty and students in ENGL 85, ENGL 21A and ENGL 1 through the fall 2014 semester in order to increase the validity of the sample and results. Disproportionate Impact Assessment validations require that an evaluation be conducted to determine whether implementation of a specific assessment instrument will have a disproportionate impact on particular groups of students described in terms of race/ethnicity, gender and age. Disproportionate impact occurs if the placement rate in the upper level courses in a sequence for a particular group is less than 4/5ths or 80% of the placement rate for the group with the highest placement rate. The following tables contain data on students that used the placement test and then enrolled in an English course during the fall and spring terms from spring 2011 through fall Upper Level Courses Gender English 85 English 21A English 1 Total Eligible Female 3.0% 56.3% 40.8% 100% 7167 Male 2.1% 50.8% 47.1% 100% % of Placement Rate for Males 1.7% 40.6% 37.7% 80.0% Upper Level Courses Race/Ethnicity English 85 English 21A English 1 Total Eligible Asian/PI 2.4% 45.5% 52.1% 100% 1191 Black 2.3% 65.2% 32.5% 100% 1260 Hispanic 3.0% 64.1% 32.9% 100% 5714 Two or more 1.5% 42.1% 56.4% 100% 592 White 2.0% 34.5% 63.5% 100% 3202 Unknown 2.5% 54.0% 43.6% 100% % of Placement Rate for White 1.6% 27.6% 50.8% 80.0%

6 Upper Level Courses Age group English 85 English 21A English 1 Total Eligible 19 and younger 2.5% 55.9% 41.5% 100% to % 48.3% 48.9% 100% to % 41.6% 56.2% 100% to % 38.6% 58.8% 100% to % 37.1% 61.4% 100% and up 0.0% 52.5% 47.5% 100% % of Placement Rate for 19 and younger 2.0% 44.7% 33.2% 80.0% The data show that black and Hispanic students make up a disproportionately smaller percentage of students placed into ENGL 1. The data also shows that students age 50 and up make a disproportionately smaller percentage of students placed into ENGL 1. The assessment office and the faculty leadership in math and English, have been aware that a number of minority groups have been disproportionately placed into lower level courses. In order to mitigate the effects of this a number of initiatives were started. The assessment office has begun a campaign to encourage students to prepare for the assessment tests. Under the banner of Prep 2 Test, the college has created a dedicated webpage with videos stressing both the importance of the placement tests, and the importance of preparation for success. The English department has taken part in creating study guides to help students prepare for the English exam. Additionally a web app is also in development that will contain sample, or prep, questions. As part of a larger focus on non-cognitive skills, the college has begun implementation of an online assessment tool called the Success Navigator. The tool measures a number of noncognitive skills, and will be used as a multiple measures supplement to the assessment test. Counselors will be able to move students that tested into ENGL 21A into ENGL 1. Finally, the English department is piloting a course where students that tested into ENGL 21A will be able to enroll in ENGL 1 as long as they participate in a supplemental course. These initiatives aim to both increase the number of students that test directly into ENGL 1, and failing that, increase the number of students that are eligible to enroll into ENGL 1 through alternative means.