Using Survey-Based Assessment to Inform First-Year Student Interventions in the Center for Academic Excellence, University of Wisconsin-Madison

Transcription

1 Pa Her, Advisor, Pathways Student Academic Services Aygul Hoffman, Tutoring Coordinator Steve Kosciuk, Data Consultant Statistician The Center for Academic Excellence (CAE) at the University of Wisconsin-Madison (UW-Madison) works with first-generation, low-income, and underrepresented students of color to make their Wisconsin Experience a reality. Launched by the College of Letters and Science Office of Student Academic Affairs in 2010, the Center for Academic Excellence promotes the values of a liberal arts education and finds new ways for students to make a difference locally, nationally, and globally. The Center is comprised of a few different programs: The Academic Advancement Program (AAP), the longest-standing academic support program on campus The Pathways Student Academic Excellence, a two-year academic enrichment program Summer Collegiate Experience (SCE), a high-impact, first-year experience summer program Four student-service coordinators who work with four underrepresented populations on campus Pre-law, pre-health advising, networking, and enrichment CAE offers college-readiness initiatives, academic advising, instructional support, and other information to help students transition to campus after high school and to make decisions about their majors and careers. Our mission is to make the Wisconsin Experience 1 a reality for first-generation and underrepresented students of color on campus. The essential components of the Center are high-impact, integrative learning experiences, community building, mentoring, and intrusive (high-touch) advising. Students in AAP participate in SCE the summer before their first fall term. In the fall, they engage with other students and the faculty in smaller class sizes through the First-Year Interest groups or engage in research as they enter fall semester. Pathways students also have a choice of living in the residential learning communities in their first year. Given the nature of the high-touch approach we exercise with our students, getting to know the students and what they bring to college, as well as anticipating their challenges, are crucial for the work of the Center s advisors. The first encounters with the students during their first semester are critical, both to establish a close studentadvisor connection and to guide the students decisions about course enrollment, social life, and career choices. This is also the time to help students build ties to the campus community and resources. The College Student Inventory (CSI) provides us with a tool that supports these goals for the first semester. In addition, the Mid-Year Assessment (MYSA) provides us with an opportunity to assess the impact of our interventions on the students and continue building the right path for the student. Most important, the MYSA tells 1 The Wisconsin Experience describes what's unique about getting a degree from UW Madison together, we create and apply learning inside and outside the classroom to make the world a better place. UW Madison produces graduates who are creative problem solvers, able to integrate empirical analysis and passion, seek out and create new knowledge and technologies, adapt to new situations, and engage as world citizens.

2 Page 2 us about the impact of the intentional support interventions such as first-year interest groups, research, and summer program, as well as the effects of intrusive academic advising and mentoring components of the Center. The Center began using the Noel-Levitz retention tools in 2010, when all AAP students took the CSI before the start of the first summer term and the MYSA at the end of the fall semester. In 2011, both the CSI and MYSA were administered to all students in the Center. Both assessments were repeated for 2012 CAE incoming freshman cohort. Although we have not been very prescriptive in the use of the survey reports by advisors, we see three threads in our use of the data collected from the CSI and MYSA: 1. The advisors individual use of student and advisor reports during one-on-one meetings with students. 2. An extensive analysis of the aggregate results has developed an overall picture of SCE and CAE students preparedness for college, and informed the development of first-year programming. 3. In addition, linking student responses to student record data has revealed some significant contrasts in firstyear outcomes disaggregated by the survey responses. The Center s task force looked at such contrasts and found that they were informative for developing our programming and practice. First Thread: Using individual student reports in the advising practice To gather qualitative data to illuminate and assess our use of the CSI and MYSA student reports in the context of one-on-one advising, we conducted an advisor roundtable discussion. The discussion revealed two modalities in how Center advisors used student reports: as a resource that informed the initial meetings with students, or as a platform for a more detailed discussion of the specific scales with the student. Most advisors used the student report in the first modality, perusing the student survey responses primarily to shape the conversation with the student and treating certain critical survey scores as flags signaling areas of concern to watch more closely as the semester progressed. On the other hand, some advisors felt comfortable discussing details of the reports with students, even addressing the scales where a student scored low. In these cases, advisors found it important to discuss the why behind any particular score, e.g., whether the score was consistent with a student s understanding of the scale or not and how the score might best be interpreted. Interestingly, the data from the surveys, (whether discussed or not with the students), tended to reflect a student s particular attributes/desires/motivations characteristic at the point when the survey was conducted. For instance, advisors reported that indicators such as math/science confidence, verbal/writing confidence, desire to finish college, receptivity to assistance, and others were consistent with students attitudes, behaviors, and actions. Nevertheless, there were a few cases in which students felt the survey scores did not reflect their personalities and intentions. Any high-touch advising practice requires the development of a trusting student-advisor relationship. In certain contexts, advisors may feel that showing students their reports might undermine the foundation of the relationship with the student, especially if the advisor does not feel that discussion of the low scores on the student report is appropriate, before building a trusting relationship with the student.

3 Page 3 Another theme noticed in our practice is the simple lack of time to go through the report with the student and discuss it in depth, as suggested by the creators of the tool. Thus, even though advisors find value in the reports in terms of framing their interactions with students and adding awareness about the particular challenges students may experience, other, more pressing decisions in their college career (such as enrolling in classes, checking on the program requirements, etc.) may take precedence over devoting the whole session to discussing the report. In summary, although advisors have found value in the individual student reports, they tend to feel more comfortable using aggregate reports to inform the design of programmatic interventions for the whole cohort. Nevertheless, the Center advisors do think that there should be continued discussion around the best use of the data. For instance, one suggestion that generated considerable discussion was to educate the students in a group setting about the concepts from the CSI prior to sharing the results and discussing them with advisor. Second Thread: Developing programming for student success The Center has also used the Noel-Levitz data in the contexts of the enrollment management and the development of programming for first and second-year students. One CAE committee oversees the development of programming for first and second-year students to assist them in their transition from high school to college, to connect students with campus resources, and to help them advance academically by successfully completing their courses. This committee investigated Noel-Levitz data to understand individual students needs in aggregate, which, in turn, helped identify common programming needs of the sister programs: Pathways Student Academic Services and the Academic Advancement Program. In addition, the Noel-Levitz data have helped to identify those campus partners whose services are best matched to the needs of CAE students. The committee began looking at the aggregate data in 2011, when CSI and MYSA were administered to all Center students. In spring of academic year, MYSA results drove our programming, by addressing students challenges with study habits, writing skills, and making career choices. We have become even more explicit in the use of aggregate data in the fall of During the beginning of the fall semester, the Noel-Levitz task force compiled aggregate data based on student responses on the CSI. Disaggregating the data by various groups of students helped the enrollment management committee better understand the needs of all Center students. For example, it was useful to develop cohort profiles based on student demographic information, such as high school GPA, parental education, degree sought, etc. In addition, the task force analyzed the relationships between key variables and the Noel-Levitz scales, such as students gender and their receptivity to various kinds of assistance or social engagement. These analyses informed the design and planning of a series of co-curricular programming events in fall 2012 focused around students personal and academic interests and needs. For example, many students reported a strong need for help with selecting an academic program and discussing qualifications for occupations; many also indicated a need for assistance with time management and study skills. The first event simply welcomed students to the UW campus and to their CAE community through an informal gathering in the student union. The second event partnered with a campus tutorial service to speak to students on the importance of seeking tutorial services early on and the role tutorial services play in helping students to have a successful first year. This event also focused on time management and study skills. The third event led by the Career Exploration Center facilitated students understanding of a major and the relationship between major, career, and student interests, values, and goals. Finally, the last event of the semester centered on the skills employers seek and how students could improve or acquire such skills in their first year. At this last event, CAE staff also took time to celebrate and acknowledge student accomplishments thus far.

4 Page 4 For spring semester 2013, the co-curricular events will focus on other areas of interest to students as indicated in their CSI responses, such as financial aid, scholarships, jobs, study abroad, research and volunteer opportunities, and resume building. Third Thread: Analyzing aggregate patterns in Noel-Levitz survey responses As mentioned in the previous section, the aggregate Noel-Levitz reports were useful for determining, on average, what support services students wanted most, such as exams, study, time management, math, and writings skills. Similarly, other survey scales indicated that many students wanted help with financial aid and finding part-time jobs. In addition, contrasts in responses by demographic factors, such as gender and first-generation college status, also guided first-year programming. For example, many strategic planning discussions wrestled with the issues surrounding the fact that male students tend to be less receptive to academic assistance, as well as social engagement opportunities, than female students. Interestingly, both of these two receptivity scales were discovered to be highly correlated with academic outcomes, in particular, students cumulative GPA after the first two semesters. Since database development, management, and analysis were essential for this aspect of using the Noel-Levitz surveys, the process is described below in greater detail. Student IDs were uploaded to our Noel-Levitz account data, and so it was a simple matter to link or join student responses to their academic records. This allowed us to examine survey responses by data elements such as ACT or Placement scores, GPAs, and course grades. The data for this project were extracted from several data views maintained in the university data warehouse The Microsoft Access application within Microsoft Office was used to extract the student data and join it with the Noel-Levitz survey responses, which were extracted with the download utilities on the Noel-Levitz Web site. Student-level data from several sources were collated by a single MS-Access query that served as the data source for an Excel pivot table. Of course, student confidentiality is protected by FERPA, and we maintained strict compliance with the federal regulations. We also took great care to protect the student data via encryption and storage on secure servers. The main thrust of this effort was to investigate in what ways survey responses were related to student academic outcomes. The Excel pivot table utility allowed our Noel-Levitz task force to quickly discover relationships between survey responses and academic outcomes. For example, using Excel pivot tables it was easily determined that the relationship between, say, receptivity to academic assistance and cum GPA/Credits existed across all levels of prior achievement as measured by ACT scores. This sort of aggregate analysis of the Noel-Levitz survey responses linked to student data was able to inform both the use of Noel-Levitz responses in one-on-one student-advisor meetings as well as cohort programming. For example, prior to meeting with a student, an advisor could quickly check whether a given student was flagged by a low receptivity score and structure the conversation with the student accordingly. Likewise, the cohort programming team could structure feedback from cohort co-curricular activities to see whether they were effective for students with low receptivity scores. The Center for Academic Excellence hopes to continue to build on the knowledge and experience using the College Student Inventory and Mid-Year Student Assessment in our student support practice. These are the goals that we have established for the continued use of the data and adjusting it to our practice:

5 Page 5 - Continue developing first and second-year co-curricular programming based on student needs and interests data; - Continue educating our students about study skills, finding best ways to integrate the teaching of study skills into their day-to-day academics; - Continue broadening leadership and involvement opportunities for students in their first and second years, and beyond; - Create events and opportunities that foster student-faculty interaction; - Use high-touch advising with students identified as at-risk by Noel-Levitz data and, given low grades or dropped courses, discuss their progress at staff meetings; - Continue having conversations with advisors about finding best ways to utilize individual student reports in advising practice; - Further develop the student records component of our use of the Noel-Levitz surveys; - Continue educating ourselves and our staff about the Noel-Levitz research nationwide and find best practices that match the Center s needs; - Track benchmarks indicating student persistence and success. We are also hoping to broaden the scope of our use of the students individual and aggregate data to extend it to our campus partners to address the students needs and challenges in their first year and beyond. We look forward to continued use of the CSI and MYSA tools and to find new and better ways to engage our students, to develop their talents, and to be their best supporting resource. By partnering with our colleagues across the nation, we hope to further develop our mission to make the Wisconsin Experience a reality for firstgeneration and underrepresented students of color on campus.