Summary of Counseling Program SLO Assessment Fall 2012 Review & Spring 2013 Plans (Compiled from data submitted by February 13, 2013) Summary prepared by Kathleen Mitchell Table of Contents Introduction p. 1 Sample Highlights/Benefits p. 2 Sample SLOs p. 4 Summary Improvements p. 4 Data Summary p. 5 Fall 2012 Review p. 6 Spring 2013 Plans p. 8 The counseling process enhances knowledge and promotes change. In Fall 2012, 100% of student counseling programs across the college conducted assessments, with the majority assessing three components of the counseling process: the individual counseling session, the educational planning process, and topical workshops including college orientations. The individual counseling sessions were assessed to determine the extent student decision-making and knowledge were enhanced. Criteria used to determine successful decision-making were students ability to modify behavior to avoid placement on academic and progress probation, and their utilization of support services to manage personal obstacles and enhance learning. Successful acquisition of knowledge was assessed by students demonstrating adherence to deadlines for graduation petition, and by students accurately completing and filing transfer documentation and required certification for benefits and financial aid. Educational planning is a process that leads to an educational plan. Through counselor guidance and expertise, students construct a pathway through their education that may include skill development and self knowledge opportunities as well as certificate, graduation, and transfer requirements. The educational plan includes services recommended by the counselor that will support student learning and success. Counseling faculty measured student knowledge and understanding of the educational planning process and its relationship to academic and career objectives. Success criteria included the percentage of students who through survey results indicated knowledge of both the purpose and content of the educational planning process and of student support services. Workshops are a form of group counseling. Workshops can be topical, such as those introducing students to employment information or to particular transfer requirements; they can be process-based as in those that facilitate discussions on accommodation or preventing academic or progress probation. Additionally, orientations are generally groups of students gathered together to learn from a counselor about college programs and services. Counseling faculty measured the format, content, and delivery of workshops and orientations. Assessment included reviewing the persistence and academic performance over a semester or more of those students who attended various orientation formats. Through surveys and direct observation, counselors assessed student knowledge of workshop topics. 1
Assessment results were shared through department meetings, individual conferences, and email. Changes included updates to SLOs and assessments, revisions to counseling methods and workshop structure, and increased faculty dialogue. The majority of counseling programs will conduct assessments in the Spring by using direct observation and surveys. Data will be analyzed, shared at department meetings and by email, and changes and revisions to the assessment process are anticipated. HIGHLIGHTS Sample self-described highlights of & benefits from SLO assessment Comparisons of student persistence and performance of those who attended the 2010 Enhanced Orientation to those who participated in the one hour and online orientations concluded that those who attended the Enhanced Orientations retained more units and completed more classes the following semester than those within the other groups. These data confirm that the Enhanced Orientations contribute greatly to student success. The SLO process has reinforced the department's commitment to utilize data to effectively improve our transfer counseling and services to students. Our findings indicated that MRSD faculty are very effective (86-100%) in increasing MRSD students knowledge and understanding of Ed Plans. From the assessment of student knowledge of services, students were on average familiar with 8.3 types of services, an increase of 2.5 services (or 73%) per student. The core counseling services reached very high rates Counseling (97%), Educational Plans (96%), Tutoring (90%), Scholarships (87%) & Academic Classes (88%). Not surprisingly, other less familiar services on the pre-test received significant gains on the post-test; for example, increase in Book Loan awareness (32% to 75%), Book Voucher & Bi-lingual services (28% to 66%), and our least familiar service of Outreach (13% to 48%). We have renewed appreciation for how the SLO process enables us to reflect on improving our services to students and on impacting student learning. We are benefitting from collaboration. We are benefitting from data rather than impressions. We have to look at our counseling methods and methods of delivery of services to try to understand better why some students do not avail themselves of any recommended services that would predictably help them. The SLO process has reaffirmed our commitment to evaluate services. Moreover, we use data gathered from assessments to make program changes to increase student awareness of their values, skills, and interests. Additionally, we use the SLO process to more effectively prepare students for the world of work. The SLO process has enabled me to focus on those critical issues that are imperative for students to understand as they work toward fulfilling their graduation requirements for an Associates Degree and preparing for transfer. Through the workshops and counseling sessions, students have multiple pathways to accurate academic information and support in clarifying their questions. The collaboration between counseling faculty and instructional faculty is strengthened. Consequently, student learning is improved. As a result of the SLO activities, the Vet counselors affirmed that our work is effective in assisting students. However, we noted that surveys we used could be strengthened to gather information from 2
both new and continuing students. As a result, our program has become more inclusive of the various needs presented by our student population. Based on my observations of students and on my analysis of the data collected from surveys, I have gained insight about the information needed by athletes to facilitate their learning and success. I plan to expand the orientation to include additional topics such as utilization of technology to obtain records and academic progress. Collaboration with coaches is crucial for strengthening support for athletes. Student athletes benefit from consistent and frequent counseling interactions. Additionally, to facilitate my on-going professional development, I would benefit greatly from more interaction with counselors and faculty across the college community. Increased dialogue among staff and faculty in EOPS on program improvement. Improved customer services. Adopted new counseling practices. Sharing among counselors of best practices used in individual counseling sessions. Although attendance was low during the initial pilot of grad petition workshops, those who participated gained a tremendous amount of information and felt very much supported by counselors. Those students who began the workshop with hesitancy about the grad petition process left the workshop feeling more confident. Moreover, those who felt somewhat confident before the workshop were strengthened through the process and information and completed more accurate grad petitions. Through dialogue between colleagues about these workshops, more ideas surfaced for ways to improve services to students. Counselors learned from each other through this process, became more familiar with the wealth of expertise within counseling, and strengthened relationships. This process gave me the impetus to move forward on ideas that have been with me for a long time. The individual conferences with other faculty supervisors of graduate student interns, and the development of the "What's Up" module within the training have had the greatest impact on student learning. The positive outcomes include: more knowledge of what students pay attention to during presentations, how we need to conduct outreach, and how we can promote single stop through instructing and counseling faculty. The data revealed that when probation students meet with a counselor, they succeed at a greater rate than those who do not see a counselor. I have modified the content of the course as a result of awareness I have gained through the SLO process. These modifications will result in improved student learning and understanding of methods to avoid being placed on probation in the future. Students have become more aware of the changing landscape of work and how to better prepare for this new environment not only by acquiring and developing skills but also by remaining curious and persistent throughout their lives. 3
Sample Student Learning Outcomes Counseling Session Educational Planning Workshops & Orientations Students are able to By the end of the OPT workshop, navigate the educational students will gain knowledge of and student support the qualifications, procedures, services to enhance their and eligibility for applying for success. Students are able OPT. to make decisions related to their educational and personal goals. Our first outcome is students will identify barriers and will be able to know where to seek help so that possible barriers can be eliminated and help students remain in school. Our second outcome is to measure satisfaction of students who use the Single Stop Office and institute new procedures to improve the program. By the end of a counseling session or workshop on TAG, students will gain knowledge and understanding of the process of completing a TAG online, of the minimum GPA and course requirements, deadlines, and of the differences and similarities among UC campus requirements. Within five weeks of reviewing learning disability test results and qualification for accommodation(s), a student will begin to use at least one recommended accommodation. By the end of a counseling appointment with a Veteran s counselor, students will have acquired knowledge about benefits and services available, educational planning, appropriate referrals for added personal and emotional support, registration information, and transfer information. SLO: Increase the number of under-represented students who transfer to four-year colleges and universities. EOPS Counseling SLO: Student will explain the purpose of an educational plan for establishing and achieving an educational goal. After completing the presentation, students were able to clearly articulate and demonstrate the required steps to complete their educational goals. Summary Improvements Made As a Result of Analyses By participating in the employment recruitment workshop, students will gain knowledge of internship opportunities, the needed qualifications, requirements, and the application process and internship expectations. Upon completion of the ESL Credit Orientation, students will be able to navigate educational and student support services to enhance their success. By the end of the Graduation Petition Preparation Workshop, students will become knowledgeable about requirements for graduation, the information to bring to their counseling appointments such as transcripts from other colleges, ideas about a major, unit requirements, and catalog year. Strengthen successful interventions Revise workshop content to increase clarity Increase number of student focus groups Gather student demographics Explore additional methods to prevent academic and progress probation Add additional slides to presentation for clarity Expand range of workshop topics to meet student needs Add new components to orientation to increase student knowledge Collect larger sample of student responses Create second survey to reach students not included in initial survey Revise student satisfaction survey Assess rubrics and pre/post surveys Use informed percentages to report student learning 4
DATA Summary SLO Overview Student Counseling Programs Program Program Assessment Stage Academic Counseling 4 CalWORKs Education and Training 4 Career Development Counseling 4 DSPS Counseling 5 DSPS Counseling Part II 5 EOPS Counseling 5 Enhanced Orientation Program 4 Financial Aid Counseling 5 Gateway to College Program 5 Instructional Liaison Counseling 4 International Student Counseling 4 Multicultural Retention Services 4 New Student Counseling Department (NSCD) Student Satisfaction Survey Fall 2012 4 New Student ESL Credit Orientation 4 Probation Caseload 3 Puente Program 4 Single Stop USA 3 Student Athlete Counseling 1 Transfer Counseling 5 University Graduate Intern Training and Development 4 Veteran Counseling 4 Totals Stage 5: 6 Stage 4: 12 Stage 3: 2 Stage 2: 0 Stage 1: 1 Total: 21 Program Assessment Stage 1. SLOs are developed and regularly updated. 1 4% 2. SLO assessments are developed and in use. 0 0% 3. Assessment data/results are being analyzed and discussed. 2 9% 4. Changes are being implemented and program will be reassessed. 12 52% 5. Program has undergone at least one full closed -loop cycle, and assessment is continual. 6 26% FINDINGS: 87% of REPORTED COUNSELING PROGRAMS HAVE ONGOING ASSESSMENT; 26% are at stage 5 closed-loop CQI. Where do you keep evidence/documentation of outcomes assessment? Assessment website 16 73% Department binders (on file) 18 82% Faculty binders (on file) 9 41% Other 15 68% 5
FALL 2012 REVIEW Did you conduct SLO assessments (measurements) for this program in Fall? Yes 17 74% What assessment method(s) did you use? Quizzes/exams (for workshops) 2 12% Assignments based on rubrics (for workshops) 0 0% Direct observation of students 4 24% Satisfaction surveys 2 12% Self-evaluation surveys 5 29% Focus groups 0 0% Other 12 71% OTHER INCLUDES: Pre and post tests How many students completed your assessments? Numbers of students assessed ranged from 10-380 depending on the program assessment plan. In Fall, did you review/discuss/analyze these data and plan changes? Yes 17 74% With which of the following did you share your data for review and analysis? Faculty and staff within the same counseling program. 16 94% Faculty and staff from other counseling programs. 6 35% Faculty and staff from across the college 3 18% Faculty and staff from other colleges 1 6% Students 7 41% No consultation planned 0 0% Other 7 41% Through which of the following did you share your data for review and analysis? Department meetings/discussions 16 94% Individual conference 11 65% Teleconferencing or skype 0 0% Email 14 82% Phone 8 47% Website review and commentary 6 35% Shared documents files 10 59% None 0 0% Other 4 24% Did you make any program changes in Fall to improve student learning? Yes 13 57% 6
Which of the following changes did you complete in Fall? Updated program SLOs 9 69% Updated assessments 9 69% Used new or revised counseling methods 8 62% Used new or revised resources within a counseling session 8 62% Provided additional resources for outside a counseling session 6 46% Used new equipment or supplies 1 8% Developed/increased faculty and staff dialogue opportunities 12 92% Developed/increased faculty and staff mentoring opportunities 4 31% Created or expanded repository (website or hands on) of shared resources (news links, book or journal references, online tutorials, webinar series) 3 23% Offered seminars or workshops on counseling skills and methods and/or new content 2 15% Engaged in outside professional development about best practices 5 38% Improved communication 9 69% Other 4 31% What other SLO-assessment activities did you complete Fall semester? Samples: MRSD surveyed 316 students, who prior to meeting with MRSD counselors, completed a pretest evaluating their knowledge & understanding of MRSD services. Following the counseling session, all 316 students completed a post-test. Students are required to attend a follow-up counseling appointment as an extension of the workshop. Within the session, students, along with their counselor, will complete the Status, Application, and Referral form, which is submitted to the international students' admissions and records unit to close the loop. Within the admissions and records units, students receive the application for OPT. Another SLO assessment activity was the Engineering 10A Workshop that was measured by a six item pre and post test that included questions about the number of students who transfer from CCSF, the steps involved in transfer, the minimum requirements, and the TAG Program. The assessment also expanded the choice of responses to a range of four competency areas of knowledge. SPRING 2013 PLANS Will you be conducting assessments (measurements) Spring semester? Yes 15 65% What assessment method(s) will you use? Quizzes/exams (for workshops) 5 33% Assignments based on rubrics (for workshops) 1 7% Direct observation of students 6 40% Satisfaction surveys 6 40% Self-evaluation surveys 5 33% Focus groups 4 27% Other 5 33% 7
Will you also be reviewing and analyzing data this semester? Yes 19 83% With whom do you plan to share your data for review and analysis? Faculty and staff within the same counseling program. 18 95% Faculty and staff from other counseling programs. 8 42% Faculty and staff from across the college 5 26% Faculty and staff from other colleges 3 16% Students 10 53% No consultation planned 0 0% Other 7 37% How do you plan to share your data for review and analysis? Department meetings/discussions 17 89% Individual conference 12 63% Teleconferencing or skype 0 0% Email 15 79% Phone 8 42% Website review and commentary 8 42% Shared documents files 11 58% None 0 0% Other 3 16% Will you also be implementing changes Spring semester? Yes 11 48% Which of the following changes do you plan to complete in Spring? Update program SLOs 3 27% Update assessments 5 45% Use new or revised counseling methods 4 36% Use new or revised resources within a counseling session 4 36% Provide additional resources for outside a counseling session 5 45% Use new equipment or supplies 2 18% Develop/increase faculty and staff dialogue opportunities 5 45% Develop/increase faculty and staff mentoring opportunities 0 0% Create or expand repository (website or hands on) of shared resources (news links, book or journal references, online tutorials, webinar series) 1 9% Offer seminars or workshops on counseling skills and methods and/or new content 3 27% Engage in outside professional development about best practices 2 18% Improve communication 1 9% Other 3 27% FINDINGS: Biggest difference between Fall changes and Spring plans is the increase in almost all areas of planned changes. 8