REPORT OF THE COMPREHENSIVE ASSESSMENT PLAN

Transcription

1 Assessment Plan 1 EXECUTIVE SUMMARY REPORT OF THE COMPREHENSIVE ASSESSMENT PLAN (FORMATIVE AND SUMMATIVE EVALUATIONS) COUNSELOR EDUCATION PROGRAM Marriage and Family Therapy, Mental Health Counseling, School Counseling Counselor Education and Supervision DEPARTMENT OF EDUCATIONAL AND HUMAN SCIENCES UNIVERSITY OF CENTRAL FLORIDA

2 Assessment Plan 2 Table of Content Comprehensive Assessment Plan: Overview Figure 1: Overview of Components.3 Figure 2: Cycle of Evaluation..4 Comprehensive Assessment Plan: Timeline Formative Assessment Timeline..5 Summative Assessment Timeline 6 Comprehensive Assessment Plan: Instruments Program Evaluation Instruments..7 Outcome Questionnaire (OQ-45.2)..7 Counselor Self-Efficacy Scale (CSES) 7 Heintzelman (Kuch-Robinson Inventory) 7 Counselor Competencies Scale 7 Intro Version 7 Techniques Version.8 Practicum Version 8 Practicum and Internship Application.8 Student Review Rubric 9 Site Supervisor Evaluation Form.9 Marriage and Family Therapy Version...9 Mental Health Counseling Version..9 School Counseling Version..9 Specialty Standards Knowledge Assessments.9 Marriage and Family Therapy...10 Mental Health Counseling.10 School Counseling.10 Doctoral Program...10 Counselor Preparation Comprehensive Examination (CPCE).. 10 Florida Teacher Certification Examinations (Subject Area Exam in Guidance & Counseling)...10 Comprehensive Portfolio...10 Graduating Graduate Student Survey 11 Institutional Effectiveness Reports 11 Table 1: Program Evaluation and Assessment Matrix...12 Comprehensive Assessment Plan: Remediation...13 Comprehensive Assessment Plan: Data Review 16 Table 2: Data Result Plan..16 Marriage and Family Assessment Plan and Results..17 Mental Health Counseling Assessment Plan and Results..32 School Counseling Assessment Plan and Results..45 Comprehensive Assessment Plan: Curricular and Program Modifications..62

3 Assessment Plan 3 Comprehensive Assessment Plan: Overview All of the Counselor Education faculty are actively involved in the Comprehensive Assessment Plan. The Assessment Plan is systematic and ongoing from year to year, using multiple methods of assessment. The assessments include evaluations of current students academic and personal/professional development, level of learning based on students accomplishment of student learning outcomes, personal/professional development in counseling identity, knowledge, advocacy, relationship skills, research, and scholarship. Graduates are evaluated by assessing knowledge of student learning outcomes, accomplishments, and employer/site supervisor evaluations. Faculty evaluate the curriculum, programs, coursework, admissions process, and current student functioning. Site supervisors and board members evaluate current students, program outcomes, and current/future direction of the programs. Figure 1 provides an overview of components of the Comprehensive Plan. Figure 1 Program Mission Evaluation of Students and Student Learning Outcomes CACREP Standards Program Objectives Course Objectives & Measurement of Student Learning Outcomes 1. The CACREP Standards are used to inform each of the aforementioned processes. This is noted above by placing the CACREP Standards in the central position and noting how the Standards inform each area. 2. The Counselor Education Program Mission is aligned with CACREP standards and informs the development of the Program Objectives. 3. The Program Objectives are infused into the Objectives for each course, both at the Core and Specialty levels. Course Objectives are aligned with the CACREP standards and inform the development of student learning outcomes (SLOs).

4 Assessment Plan 4 4. The SLOs are aligned with CACREP standards and inform the measurement and evaluation processes. 5. Outcomes from the SLO measurement and evaluation processes are aligned with CACREP standards and are used to inform the evaluation of the overall Program. The ultimate goal of this process is directed at students achieving the Counselor Education Program Mission. The specific cycle of evaluation is presented in figure 2. Figure 2 Assessments of Current Students Executive Report Completed, Distributed Follow-Up of Graduates Program Faculty Retreat: Review Findings and Suggest Changes Follow-Up with Employers Program Faculty Review and Assessment Follow-Up with Advisory Board/Site Supervisors

5 COMPREHENSIVE ASSESSMENT PLAN: TIMELINE Formative and Summative Assessments Formative Assessments Time-Line Assessment(s) Assessed by Orientation Outcome Questionnaire (OQ-45.2) Research Supervisor (Fall, Spring) Counselor Self-Efficacy Scale (CSES) Heintzelman (Kuch-Robinson Inventory) Assessment Plan 5 MHS 5005 Counselor Competencies Scale (Intro Version) Intro Instructor & Doc Students Intro Course Skills, Behaviors, and Professional Dispositions (Fall, Spring) MHS 6401 Counselor Competencies Scale (Techniques Version) Techniques Instructors Techniques (mid) Skills, Behaviors, and Professional Dispositions (Fall, Spring) MHS 6401 Counselor Competencies Scale (Techniques Version) Techniques Instructors Techniques (final) Skills, Behaviors, and Professional Dispositions (Fall, Spring) March Student Grades and Progression though Program Faculty Advisors October Practicum and Internship Applications June Practicum Orientation Outcome Questionnaire (OQ-45.2) Research Supervisor (Fall, Spring, Summer) Counselor Self-Efficacy Scale (CSES) Heintzelman (Kuch-Robinson Inventory) MHS 6803 Counselor Competencies Scale (Practicum Version) Practicum Instructors Practicum I (mid) Skills, Behaviors, and Professional Dispositions (Fall, Spring, Summer) MHS 6803 Counselor Competencies Scale (Practicum Version) Practicum Instructors Practicum I (final) Skills, Behaviors, and Professional Dispositions (Fall, Spring, Summer) Ongoing and Student Review Session All Faculty April Student Review Rubric December

6 Assessment Plan 6 Summative Assessments Time-Line Assessment(s) Assessed by Internship I (pre) Outcome Questionnaire (OQ-45.2) Research Supervisor (Fall, Spring, Summer) Counselor Self-Efficacy Scale (CSES) Heintzelman (Kuch-Robinson Inventory) Internship I (mid) Site Supervisor Evaluation Form Site & University Supervisors (Fall, Spring, Summer) Specialty Standards Knowledge Assessments (All) Research Supervisor Internship I (final) Site Supervisor Evaluation Form Site & University Supervisors (Fall, Spring, Summer) Comprehensive Final (CPCE) (All) Clinic Administrator Florida Teacher Certification Examinations (SC) Internship II (mid) Site Supervisor Evaluation Form Site & University Supervisors (Fall, Spring, Summer) Internship II (final) Site Supervisor Evaluation Form Site & University Supervisors (Fall, Spring, Summer) Comprehensive Portfolio (All) Faculty Advisors Prof. Identity, Practice, & Program Area Standards Accomplishments Report Form Faculty Advisors Graduation Certification for Graduation Faculty Advisors (Fall, Spring, Summer) Student Degree Audit Ongoing Graduating Graduate Student Survey Track Coordinators Results reported in Fall, Spring, Summer September Institutional Effectiveness Reports Track Coordinators MFT, MHC, SC, DOC Covers Summer, Fall, and Spring Semesters April Faculty Retreat and Advisory Board Meeting Faculty Review of Findings and Modifications Recommendations and Modifications July Annual Comprehensive Assessment Plan Report All Faculty Submitted/Available to: Faculty Students Administrators Graduates Supervisors/Board Members Personnel in cooperating agencies

7 Assessment Plan 7 Comprehensive Assessment Plan: Instruments The following instruments are used by the Counselor Education Program as a means of formative and summative assessment of their students. A brief description of each instrument is offered as well as where these instruments can be located (or where more information may be found). See Table 1 on page 12 for a summary and timeline of how the Program uses these formative and summative assessments. Program Evaluation Instruments The purpose of these assessments is to evaluate the efficacy of the Counselor Education program in preparing students to become competent and ethical counseling professionals and to enhance counseling gatekeeping procedures. Students identity will be kept confidential (i.e. names or other identifying information are not be used in any report). o Outcome Questionnaire (OQ-45.2) The OQ-45 is a 45-item self-report outcome/tracking instrument designed for repeated measurement of client progress through the course of therapy. The OQ-45 measures functioning in 3 domains: Symptom distress (heavily loaded for depression and anxiety), Interpersonal functioning, and Social Role. It enables clinicians to assess functional level and change over time. The Program uses this instrument as a means of assessing students level of distress and functioning. o Counselor Self-Efficacy Scale (CSES) The CSES consists of 20 items regarding knowledge and skill competencies related to the practice of individual and group counseling and therapy. The instrument uses a 5-point Likert-type scale corresponding to responses indicating the degree of agreement regarding respondents' confidence in their counseling abilities. o Kuch-Robinson Heintzelman Inventory (Altruism scale) This 40-item self-report inventory measures the various influences that motivate graduate students to pursue a career within the field of counseling. These instruments are distributed to students during the (a) New Student Orientation, (b) Practicum Orientation, and (c) first semester of internship. Counselor Competencies Scale The Counseling Competencies Scale assesses students Skills and Practices-based Student Learning Outcomes (based up the CACREP Core and Specialty Area Curricular Standards). More specifically, this instrument is designed to assess students in the domains of counseling skills, professional dispositions, and professional behaviors. This instrument is administered at least five times during students time in the program (seven for MFT and MHC students) and is developmental in nature: once in MHS 5005 Intro to the Counseling Profession, twice during MHS 6401 Techniques of Counseling, twice during Practicum I (for MFT, MHC, and SC students) and twice during Practicum II (for MFT and MHC students; same version as that used for Practicum I). o Intro Version The Intro Version of the Counseling Competencies Scale assesses beginning skills, dispositions, and behaviors: Counseling Skills Nonverbal skills, questions, and reflections of content Professional Dispositions

8 Assessment Plan 8 Ethics, professionalism, self-awareness/self-understanding, emotional stability & selfcontrol, motivation, multicultural competency, openness to feedback, boundaries, flexibility & adaptability, congruence & genuineness Professional Behaviors Attendance & participation, knowledge & adherence to university and counseling policies, knowledge of professional literature, consultation o Techniques Version The Techniques Version of the Counseling Competencies Scale assesses intermediate skills, dispositions, and behaviors: Counseling Skills Nonverbal skills, encouragers, questions, reflections (content, feelings, meaning, summarizing), confrontation, goal setting, focus of counseling, facilitating a therapeutic environment Professional Dispositions Ethics, professionalism, self-awareness/self-understanding, emotional stability & selfcontrol, motivation, multicultural competency, openness to feedback, boundaries, flexibility & adaptability, congruence & genuineness Professional Behaviors Attendance & participation, knowledge & adherence to university and counseling policies, knowledge of professional literature, consultation o Practicum Version The Practicum Version of the Counseling Competencies Scale assesses advanced skills, dispositions, and behaviors Counseling Skills Nonverbal skills, encouragers, questions, reflections (content, feelings, meaning, summarizing), confrontation, goal setting, focus of counseling, facilitating a therapeutic environment Professional Dispositions Ethics, professionalism, self-awareness/self-understanding, emotional stability & selfcontrol, motivation, multicultural competency, openness to feedback, boundaries, flexibility & adaptability, congruence & genuineness Professional Behaviors Attendance & participation, knowledge & adherence to university and counseling policies, record keeping, knowledge of professional literature, application of theory to practice, case conceptualization, consultation, psychosocial & treatment planning, appraisal, referral These instruments are distributed to students during the courses noted above. Practicum and Internship Application The Practicum and Internship Application is a means for Advisors to track students progress in the Program. Student must complete an application for each semester of their clinical experiences (practicum and internship) and submit a Degree Audit with their application. Advisors assess student progress by assuring that students: (a) have completed the mandatory prerequisite courses, (b) have earned a grade of B or better in MHS 5005 and MHS 6401, (c) have completed a minimum of 27 completed credit hours, and (d) have maintained a minimum GPA of 3.0 in the Counselor Education program.

9 Assessment Plan 9 This instrument can be viewed in the Student Corner Important Downloads section of the Counselor Education website. Student Review Rubric As stated in the Student Handbook, the Program faculty conduct a systematic developmental assessment of each students progress throughout the program, including consideration of their academic performance, professional development, and personal development. Additional information can be viewed in Counselor Education Student Handbook (pp ) which is located in the Student Corner Important Downloads section of the Counselor Education website. Site Supervisor Evaluation Form The Site Supervisor Evaluation Form ( Intern Eval ) aligns with the CACREP Specialty Area Standards and is aimed at assessing Skills and Practices-based Student Learning Outcomes in the domains of counseling skills, professional dispositions, and professional behaviors. This instrument is administered two to four times during students time in the program: twice during MHS 6830: Counseling Internship I (for MFT and MHC students) or SDS 6947: Internship in Professional School Counseling (for SC students), and twice again during MHS 6830: Counseling Internship II (for MFT and MHC students) or SDS 6947: Internship in Professional School Counseling (for SC students who extend their internship into two semesters). o Marriage and Family Therapy Version The Marriage and Family Therapy Version of the Intern Eval assesses students in the following domains: Counseling and the Therapeutic Alliance, Legal & Ethical Behaviors, Professionalism (Personal Behaviors and Site-Specific Behaviors), Intervention & Treatment Planning, Assessment Skills and Practices, Multicultural Competency, and Program Evaluation. o Mental Health Counseling Version The Mental Health Counseling Version of the Intern Eval assesses students in the following domains: Counseling and the Therapeutic Alliance, Legal & Ethical Behaviors, Professionalism (Personal Behaviors and Site-Specific Behaviors), Intervention & Treatment Planning, Assessment Skills and Practices, Diagnostic Skills and Practices, Multicultural Competency, and Program Evaluation. o School Counseling Version The School Counseling Version of the Intern Eval assesses students in the following domains: Individual Counseling, Group Counseling, Classroom Guidance Curriculum, Consultation Services, Crisis Management/Interventions, Program & Service Coordination, Academic Advising, Programmatic Evaluation & Accountability, Career Counseling & Development, Multicultural Counseling & Development, Student Assessment & Appraisal, and Ethical Practice and Advocacy. These instruments are distributed to students during their internship experiences. Specialty Standards Knowledge Assessments These instruments are based upon the CACREP Core and Specialty Area standards and are administered to students at the beginning of their internship experience (or for doctoral students, during their comprehensive final exam). Students will take both the Core Standards Knowledge

10 Assessment Plan 10 Assessment and the Specialty Area Standards Knowledge Assessment that matches their Program track (i.e. MFT, MHC, SC, & Doc). o Marriage and Family Therapy The Marriage and Family Therapy Specialty Standards Knowledge Assessment measures students knowledge related to (a) Foundations, (b) Counseling, Prevention, and Intervention, (c) Diversity and Advocacy, (d) Assessment, and (e) Research and Evaluation. o Mental Health Counseling The Mental Health Counseling Specialty Standards Knowledge Assessment measures students knowledge related to (a) Foundations, (b) Counseling, Prevention, and Intervention, (c) Diversity and Advocacy, (d) Assessment, (e) Research and Evaluation, and (f) Diagnosis. o School Counseling The School Counseling Specialty Standards Knowledge Assessment measures students knowledge related to (a) Foundations, (b) Counseling, Prevention, and Intervention, (c) Diversity and Advocacy, (d) Assessment, (e) Research and Evaluation, (f) Academic Development, (g) Collaboration and Consultation, and (h) Leadership. o Doctoral Program The Doctoral Program Specialty Standards Knowledge Assessment measures students knowledge related to (a) Supervision, (b) Teaching Research and Scholarship, (c) Counseling, (d) Leadership, and (e) Advocacy. These instruments are distributed to students during their internship experiences. Counselor Preparation Comprehensive Examination (CPCE) The CPCE is an objective test developed by the Center for Credentialing & Education (CCE), an affiliate of the National Board for Certified Counselor (NBCC). The CPCE is a knowledge-based examination that reflects the eight core curriculum areas approved by CACREP (Human Growth and Development, Social and Cultural Foundations, Helping Relationships, Group Work, Career and Lifestyle Development, Appraisal, Research and Program Evaluation, and Professional Orientation and Ethics. The exam (which consists of 160 items with 20 items per CACREP area) is a summative evaluation that measures pertinent and professional knowledge acquired by students during the Counselor Preparation Program. For more information, visit: Florida Teacher Certification Examinations (Subject Area Exam in Guidance & Counseling) School Counseling students must successfully complete the Florida Teacher Certification Examinations (FTCE) prior to beginning their internship experience. More specifically, students need to pass (a) the FTCE General Knowledge Examination, (b) the FTCE Subject Examination (Guidance & Counseling), and (c) the FTCE Professional Education Examination. For more information, visit: Comprehensive Portfolio The Comprehensive Portfolio is where students demonstrate their accomplishments related to professional identity, professional practice, and program area standards in the Specialty Area

11 Assessment Plan 11 domains (MHC, MFT, SC). Students must demonstrate competence by collecting and submitting evidence related to their: o Professional identity [assessed through: (a) a Professional Reflection Paper, (b) a Professional Resume, (c) Evidence of Attendance at two conferences, (d) evidence of Professional Membership, and (e) submission of the Accomplishments Report Form], o Professional practice [assessed through: (a) submission of four final Counseling Competencies Scale instruments and (b) submission of final Site Supervisor Evaluation Forms], and o Program area standards [assessed through: (a) submission of Evidence of passing the CPCE (for all tracks) and the Florida Teacher Certification Exam (FTCE) (for SC students only) and (b) completion of the Graduate Student Survey. This instrument can be viewed in the Student Corner Important Downloads section of the Counselor Education website. Accomplishments Report Form The Accomplishments Report Form. In terms of professional identity, the Program has implemented the collection of data using the. This document collects information related to students publications, presentations, awards, association memberships, and community involvement. Completion of this form is a component of the Master s students final portfolio and the doctoral students annual evaluation process. This instrument can be viewed in the Student Corner Important Downloads section of the Counselor Education website. This instrument can be viewed in the Student Corner Important Downloads section of the Counselor Education website. Graduating Graduate Student Survey The Counselor Education program conducts formal follow-up studies of program graduates through the completion of the Graduating Graduate Student Survey that is conducted through the College of Education and analyzed and reported by the Office of Operational Excellence and Assessment Support (OEAS). This is done every semester and the results are available to the public on the OEAS website. The questions that relate to the major aspects of the program relate to the coursework (in and outside of the program), clinical experiences, and CACREP s eight common core areas. These instruments are distributed to graduates through the College of Education and OEAS. Institutional Effectiveness Reports The Institutional Effectiveness Reports document the outcomes of the Program s systematic evaluation process, to include descriptions of program modification plans and implementations. Effectiveness plans are monitored by a university committee and the results of the evaluation are reviewed at the university level each year. The official report is distributed via the UCF Assessment website and the Counselor Education website (under Program Evaluation). Current students, program faculty, institutional administrators, personnel in cooperating agencies, and the public are granted access to the Counselor Education Program s evaluation plan in this way.

12 RS = Research Supervisor Table 1: Program Evaluation and Assessment Matrix Assessment Instrument New Student Orientation Intro Techniques Practicum Orientation Practicum I (Mid) Practicum I (Final) Practicum II (Mid) Practicum II (Final) Last Internship (Mid) CPCE Demographics RS Administers During Orientation RS Administers During Orientation Counseling Competencies Scale (CCS) Instructor Administers Instructor Administers Instructor Administers Instructor Administers Instructor Administers Instructor Administers Counselor Preparation Comprehensive Examination (CPCE) Counselor Self-Efficacy Scale (CSES) RS Administers During Orientation RS Administers During Orientation Instructor Administers Clinic Coordinator Administers Heintzelman (Kuch-Robinson Inventory) RS Administers During Orientation RS Administers During Orientation Instructor Administers Outcome Questionnaire (OQ-45.2) RS Administers During Orientation RS Administers During Orientation Instructor Administers

13 Assessment Plan 13 Comprehensive Assessment Plan: Remediation The purpose of assessment is to ascertain whether students are attaining the appropriate amount of education and training that will (a) lead to the necessary knowledge, skills, and professional behaviors to prepare them to become competent and ethical counseling professionals and/or (b) enhance counseling gatekeeping procedures. As such, it is the Faculty s desire to work with students to move them toward their desired professional goals. Recognizing that not everyone moves in the desired direction at the same speed (hence our developmental approach to education and assessment), and recognizing that the counseling profession may not be suitable for everyone, the Faculty have designed remediation plans in the event that assessment results in a student being unable to demonstrate sufficient progress to move forward. Whereas the reader is encouraged to read the entire Counselor Education Remediation Plan (as found in the Counselor Education Student Handbook), this narrative outlines specific remediation procedures for each level of assessment that is used as a part of the Comprehensive Assessment Plan. Program Evaluation Instruments The purpose of these assessments is to evaluate the efficacy of the Counselor Education program itself. Thus we do not use the results of any of these assessments in the evaluation of individual students. No remediation procedures are tied to the Program Evaluation Instruments. Counselor Competencies Scale (All Versions) The Counselor Education Faculty have established four courses as those requiring that students earn the grade of B or better in order to move forward in the program. These courses are MHS 5005 Intro to the Counseling Profession, MHS 6401 Techniques of Counseling, MHS 6803 Practicum, and MHS 6830: Counseling Internship (for MFT and MHC students) or SDS 6947: Internship in Professional School Counseling (for SC students). A large portion of the grade in the MHS 5005, MHS 6401, and MHS 6803 courses (see info regarding MHS 6830/SDS 6947 below) is to perform at the level of Meets Expectations / Demonstrates Competencies (score of 6 ) in all domains (skills, dispositions, & behaviors) of the Counseling Competencies Scale (CCS). If students do not demonstrate such proficiency by the end of the semester in which they are taking the course (i.e. MHS 5005, MHS 6401, and MHS 6803), they will receive a maximum grade of C and will need to repeat that course. For remediation then, students will need to retake the course in which they received a C for their lack of competence. Further, any student earning a C in MHS 6803 will necessarily be placed on a remediation plan that outlines the procedures necessary to succeed. This may include the need for personal counseling, time management skills, and/or additional assignments (e.g., creating transcripts, watching recorded sessions of others, increased supervision). As for MHS 6803, students may only retake the course one time beyond what is required for their program track. More information can be found in the Counselor Education Student Handbook. Practicum and Internship Application Remediation related to the Practicum and Internship Application would only result when students have not completed the mandatory prerequisite courses, have not earned a grade of

14 Assessment Plan 14 B or better in MHS 5005 and MHS 6401, have completed less than 27 completed credit hours, and/or have a GPA that has fallen below 3.0. Such remediation may include completing the mandatory courses, retaking coursework, and/or taking additional coursework. Student Review Rubric As stated in the Student Handbook, the Faculty conduct a systematic developmental assessment of each students progress throughout the program, including consideration of their academic performance, professional development, and personal development. When Faculty express concerns regarding a student in any of these areas, three options are considered: (a) the student is identified as meeting adequate progress but will receive feedback regarding any deficits through his/her advisor, (b) the student is recommended for remediation (details to be determined by the faculty), or (c) the student is recommended for dismissal from the program. Details related to Faculty concerns resulting from the Student Review process can be viewed in Counselor Education Student Handbook (pp ). Site Supervisor Evaluation Form As noted above in the section related to the Counseling Competencies Scale (CCS), the Faculty have established MHS 6830: Counseling Internship (for MFT and MHC students) and SDS 6947: Internship in Professional School Counseling (for SC students) as B or better courses. A large portion of the grade in the MHS 6830 and SDS 6947 courses is to perform at the level of Meets Expectations / Demonstrates Competencies (1) in all knowledge, skills, and dispositions as noted on the Site Supervisor Evaluation Form. All students must be at this level at the conclusion of the semester of internship. If students score Below Expectations on any one of the indicated Specific Descriptors found under the Disposition/Behavior headings, they will be marked as Below Expectations (0) for the entire Disposition/Behavior. This would necessitate a written narrative regarding this item in the Narrative Feedback from Site Supervisor section at the end of the assessment. This will focus the student on necessary developmental issues and trigger a review of this student by the University Supervisor. If students do not demonstrate such proficiency by the end of the semester in which they are taking the course, they will receive a maximum grade of C and will need to repeat that course. For remediation then, students will need to retake the course in which they received a C for their lack of competence. Further, any student earning a C in MHS 6830 will necessarily be placed on a remediation plan that outlines the procedures necessary to succeed. This may include the need for personal counseling, time management skills, and/or additional assignments (e.g., creating transcripts, watching recorded sessions of others, increased supervision). As for MHS 6830, students may only retake the course one time beyond what is required for their program track (unless they have received prior approval by their advisor). More information can be found in the Counselor Education Student Handbook.

15 Assessment Plan 15 Specialty Standards Knowledge Assessments These instruments have been recently developed and tested for their initial validity and reliability. Passing rates have not yet been established, but once they are, a passing score will be necessary before students can register and take any other comprehensive exam (e.g., the CPCE and/or the FTCE). Remediation for scores less than the passing score may include (but are not limited to): retaking the entire exam, retaking a portion of the exam, completing additional work in the area(s) that were not passed (e.g., writing a paper on the subject area that was not passed), and/or retaking a course that matches the area that was not passed. Counselor Preparation Comprehensive Examination Since successful completion of all portions of the Counselor Preparation Comprehensive Examination (with a passing score of 80 +/- the standard error of the mean) is a requirement for graduation, those students not passing the exam must retake it until they pass. Additional remediation policies can be viewed in Counselor Education Student Handbook s description of the CPCE. Florida Teacher Certification Examinations Since successful completion of all portions of the Florida Teacher Certification Examinations (i.e. the General Knowledge Examination, the Subject Examination in Guidance & Counseling, and the Professional Education Examination) with a passing score (currently at 70%) are a requirement for graduation, those students not passing the exam must retake it until they pass. Comprehensive Portfolio The only reason that students would not successfully complete the Comprehensive Portfolio is if they failed to complete one of the necessary components. Remediation then would involve the advisor identifying the incomplete component and ensuring that the student complete that component in a satisfactory manner. Accomplishments Report Form The only reason that students would not successfully complete the Accomplishments Report Form is if they failed to complete the form. Remediation then would involve the advisor identifying the incomplete form and ensuring that the student complete the form in a satisfactory manner. Graduating Graduate Student Survey Since completion of the Graduating Graduate Student Survey is optional, there are no remediation efforts tied to the lack of student completion of this assessment. Institutional Effectiveness Reports The purpose of the Institutional Effectiveness Reports is to document the outcomes of the Program s systematic evaluation process. No remediation procedures are tied to the Institutional Effectiveness Reports.

16 Assessment Plan 16 Comprehensive Assessment Plan: Data Review The following Institutional Effectiveness Reports indicate the data that resulted from our last comprehensive assessment. Table 2 indicates which of the assessment instruments are included in Institutional Effectiveness Reports. Instrument Program Evaluation Instruments (OQ-45.2, CSES, Kuch-Robinson Inventory are not reported) Table 2: Data Result Plan Reported in If not, why? IE? No Used for Program Evaluation If not, where? Available for analysis Sample Provided? No due to confidentiality Counselor Competencies Scale Yes N/A N/A N/A Practicum and Internship Application No Advisor tracks Kept on File N/A Student Review Rubric No Advisor tracks Kept on File No due to confidentiality Site Supervisor Evaluation Form Yes N/A N/A N/A Specialty Standards Knowledge Assessments No New Assessment Kept on File Initial psychometrics provided Counselor Preparation Comprehensive Examination (CPCE) Florida Teacher Certification Examinations (Subject Area Exam in Guidance & Counseling) Yes N/A N/A N/A Yes N/A N/A N/A Comprehensive Portfolio Yes N/A N/A N/A Graduating Graduate Student Survey Yes N/A N/A N/A The Institutional Effectiveness Reports for the Marriage and Family, Mental Health, and School Counseling program follow this page.

17 Menu Home > Assessment Plan and Results User: ltomei Role: DRC Chair Help Logout X Assessment Plan and Results Plan Year: Program/Unit: Marriage and Family Therapy - M.A. IE Assessment Rubrics Assessment DRC Chair Instructions Assessment for Unit Heads View/Submit Results Review Results Review Program/Unit: Marriage and Family Therapy - M.A. DRC: College of Education Year: DRC Chair: Lance J Tomei Due Date: 09/30/2011 Coordinator(s): Reviewer(s): Mark Young, K. Dayle Jones Lindsay Archambault Quick Links: Mission: The mission of the Master of Arts (M.A.) in Marriage and Family Therapy is to educate and prepare graduate students to work in private practice, agencies and other settings as marriage and family therapists. Graduates of the program are expected to have a sense of professional identity, acquire requisite skills and knowledge to work with couples and families, and attain licensure and become leaders in the profession. The Marriage and Family Therapy program emphasizes the blending of research and theory with practical applied experience. It also emphasizes and reflects the urban and multicultural nature of our community, as well as general trends in specializations within the field of counseling. The program involves intensive clinical training and field work. Assessment Process: The assessment strategy of the Marriage and Family Therapy program involves systematic assessment of two key areas of student performance: knowledge and skills. We assess these areas of student performance with both formative and summative measurements. Knowledge comprises information deemed as fundamental for student success in the marriage and family therapy field, which includes the following content areas: Human Growth and Development, Social and Cultural Foundations, Helping Relationships, Group Work, Career and Lifestyle Development, Appraisal, Research and Program Evaluation, and Professional Orientation and Ethics. Student knowledge will be appraised using scores on a nationally normed test called the Counselor Preparation Competency Exam (CPCE) (see attachment) and scores on the student portfolio rubric (see attachment). Both instruments are administered during students' final semester before graduation. 11:16:05 AM]

18 Skills refer to those demonstrable counseling skills that students need to master to become proficient as a marriage and family therapist. Skills will be evaluated using the Counseling Competencies Scale (CCS) (see attachment) at midpoint and at final of MHS 6803 Practicum in Counselor Education, which students usually take in their fourth semester in the program. In addition, the Intern Final Evaluation (see attachment) will be completed by the Internship site supervisor during MHS 6830 Counseling Internship, which is usually taken in the sixth or seventh semester in the program. To measure students own perceptions about their knowledge and skills level, we have also developed program specific questions for students to complete as part of the "Graduating Graduate Student Survey" (an indirect measure) (see attachment). This survey will analyze students' opinions about the degree to which they feel adequately prepared to become a highly qualified marriage and family therapist. These questions were designed based on the required core curricular areas outlined by the Council of Accreditation for Counseling and Related Educational Programs (CACREP). Students complete this survey when they file their intent to graduate form. The assessment strategy is managed by the coordinator of the Marriage and Family Therapy program as well as the research instructor in charge of all program evaluation within the counselor education program. These efforts to continually enhance program quality directly support UCF Strategic Goal #2. Top Outcome: 1 Students in the M.A. program in Marriage and Family Therapy will demonstrate knowledge deemed as fundamental for student success in the marriage and family therapy field. Measure: 1.1 On the Counselor Preparation Competency Exam (CPCE), students' mean score will equal or exceed the mean score of the CPCE national norm on each of the eight (8) Subscales and the Total Score. Result: Each year, the Center for Credentialing and Education provides national descriptive statistics for the CPCE (see Attachment). The CPCE is a standardized assessment instrument utilized by over 190 universities and colleges and is designed to assess counseling students knowledge of counseling information viewed as important by counselor preparation programs. The CPCE consists of eight content areas similar to the areas of competency identified by the Council of Accreditation of Counseling and Related Academic Programs (CACREP). These content areas include (1) Human Growth and Development, (2) Social and Cultural Foundations, (3) Helping Relationships, (4) Group Work, (5) Career and Lifestyle Development, (6) Appraisal, (7) Research and Program Evaluation, and (8) Professional Orientation and Ethics. For 2011, the national average for the total CPCE score was (n = 1398). Marriage & family therapy tudents total average CPCE scores was Out of 17 marriage and family therapy students who took the CPCE, students' scores were below the mean CPCE scores on all but one subscale: Professional Orientation and Ethics (see table below). The table below compares the mean scores between the national norm group and UCF students. Comparison of CPCE National Mean Scores & UCF Marriage & Family Therapy Student Mean Scores CPCE National Mean Scores (N=5927) Content Area 1. Human Growth and Development Social and Cultural Foundations Helping Relationships Group Work Career and Lifestyle UCF Mean Scores (N=17) 11:16:05 AM]

19 Development Appraisal Research and Program Evaluation Professional Orientation and Ethics TOTAL Review: Measure: 1.2 Ninety percent (90%) of students in the Marriage and Family Therapy program will receive a score of satisfactory or better on each area of their portfolio. Result: Of the student portfolios (N=8) reviewed during the academic year, all (100%) scored a satisfactory or better on each competency area of their portfolio. The mean score for the students was Students present their individual portfolio during their internship class, which is typically the semester they graduate. Portfolios are graded by the advisor and the internship instructor and are evaluated on seven areas of competency based on accreditation standards by the Council of Accreditation of Counseling and Related Academic Programs (CACREP). These areas include the following: Professional Identity and Ethics; Social and Cultural Diversity and Human Development Across the Lifespan; Use of Technology and Assessment Skills; Clinical, Consultation and Communication Skills; Specialty (play therapy, family therapy, career development); Personal Growth and Wellness; and Research and Program Evaluation. The portfolio is scored using a rubric (see Attachment) which evaluates each competency area using a Likert Scale from 0 to 3 as follows: 0 = No Evidence: Demonstrates no evidence of knowledge, skills and dispositions in this area. 1 = Emerging: Demonstrates evidence of limited knowledge, skills and dispositions in this area. 2 = : Demonstrates evidence of consistent knowledge, skills and dispositions in this area. 3 = Exceptional: Demonstrates evidence of strong knowledge, skills and dispositions in this area. Review: Reflective Statement: As a measure of whether our students are demonstrating knowledge of the counseling field, students' scores were below the mean CPCE scores on all but one subscale: Professional Orientation and Ethics. Our program needs to evaluate the purpose for these lowered scores and examine strategies to improve the scores. Regarding the portfolio data, the results that 100% of students scored a 1 (meets expectations) or better were not very useful. A new portfolio and rubric were developed last year and will be implemented this 11:16:05 AM]

20 year, with the hope of collecting more useful data to inform our program. Reflective Statement Review: Overall Outcome Results Review Comment: Attachments: Portfolio Evaluation.pdf CPCE National Statistics Fall 2011.pdf Top Outcome: 2 Students in the Marriage and Family Therapy progam will demonstrate basic counseling skills required for all marriage and family therapists in their practicum experience in an educational training facility. Measure: 2.1 Fifty percent (50%) of students in the Marriage and Family Therapy progam will score at least a 6.0 (meets expectations) on the Counseling Competencies Scale (CCS) measured at midpoint in MHS 6803 Practicum I. Result: The Counseling Competencies Scale (CCS) is a 32-item instrument designed to measure counseling competencies within three proposed factors: (a) counseling skills, (b) professional dispositions, and (c) professional behaviors (please see the attached copy of the CCS). We used the counseling skills section of the CCS only, which contains 12 items that evaluate such basic counseling skills as paraphrasing, questioning, encourages, reflecting feelings, confrontation and goal setting. The CCS contains five supervisor-rater evaluation response categories that include the following: 8 = Exceeds Expectations. The counseling student demonstrates strong knowledge, skills, and dispositions in the specified counseling skill(s), professional disposition(s), and professional behavior(s). 6 = Meets Expectations. The counseling student demonstrates consistent and proficient knowledge, skills, and dispositions in the specified counseling skill(s), professional disposition(s), and professional behavior(s). 4 = Near Expectations. The counseling student demonstrates inconsistent and limited knowledge, skills, and dispositions in the specified counseling skill(s), professional disposition(s), and professional behavior(s). 2 = Below Expectations. The counseling student demonstrates limited or no evidence of the knowledge, skills, and dispositions in the specified counseling skill(s), professional disposition(s), and professional behavior(s). 0 = Harmful. The counseling student demonstrates harmful use of knowledge, skills, and dispositions in the specified counseling skill(s), professional disposition(s), and professional behavior(s). On the CCS Counseling Skills subscale, the table below documents the frequencies of counselors' scores at midterm. Of the 23 students, over 50% scored at least a 6.0 or better in each skill (e.g., nonverbal, encouragers, etc.) except for reflecting meaning and summarizing. MFT CCS Counseling Skills Subscale Item Frequencies at Midterm (N=23) 8 - Exceeds Expectations % 6 - Meets Expectations % 4 - Near Expectations % 2 - Below Expectations % 0 - Harmful % 1. Nonverb :16:05 AM]

21 2. Encourger Questions Paraphrasing Reflecting Feelings Reflecting Meaning Summary Confront Goals Focus Empathy Respect Review: Measure: 2.2 Seventy five (75%) of students in the Marriage and Family Therapy progam will score at least a 6.0 (meets expectations) on the Counseling Competencies Scale (CCS) measured at the final in MHS 6803 Practicum I. Result: On the CCS Counseling Skills subscale, the table below shows the frequencies of marriage and family therapy students' scores (N=20) at final. This shows that for each skill (e.g., nonverbal, encourager, etc.), well over 75% of students scored at least a 6 or better. MFT CCS Counseling Skills Subscale Item Frequencies at Final (N=20) 8 - Exceeds Expectations % 6 - Meets Expectations % 4 - Near Expectations % 2 - Below 0 - Expectations % Harmful % 1. Nonverbals Encourager Questions Paraphrasing Reflecting 11:16:05 AM]

22 Feelings Reflecting Meaning Summary Confront Goals Focus Empathy Respect Review: Reflective Statement: As a measure of our students' counseling skills, the CCS scores at midterm practicum found that over 50% of students met the cut-off score of "meets expectations" (6) on each skill, except for reflecting meaning and summarizing. These two skills, particularly reflecting meaning, are advanced counseling skills. At final practicum, over 75% of students scored at least at "meets expectations" on all skills, exceeding the stated outcome measure of 75%. Two years ago, we reduced these cut-off scores because they were unrealistically high for students to attain. At this point, we may need to re-evaluate these cut scores to determine if they need raising. Reflective Statement Review: Overall Outcome Results Review Comment: In future tables, if you report percentages, please report them as whole numbers, not the equivalent decimals. Your column heading is "%," so a decimal value would technically equate to less than 1 percent. I.e., where you report.30 percent (which is less than one percent), you mean 30 percent. Also, your report an N of 20 for Measure 2.2, but most rows add up to 19, one to 18, and none to 20. Measure 2.2: the N is 20 but only one row has data for 20. This should have been explained (or you could have added a "no data for this criterion" column to ensure your row totals always added up to 20). Attachments: Counseling Competencies Scale.pdf Top Outcome: 3 Students in the Marriage and Family Therapy progam will demonstrate skills as well as feel prepared to work effectively in the field of marriage and family therapy. Measure: :16:05 AM]

23 Ninety percent (90%) of students in the Marriage and Family Therapy progam will agree or strongly agree that their program helped them attain knowledge and skills (as indicated in the common core areas identified by CACREP) as measured by the Graduating Graduate Student Survey. Result: The Graduating Graduate Student Survey (GGSS) assesses the extent to which Marriage & Family Therapy graduates agree or disagree that the UCF Counselor Education prepared them to become a highly qualified counselor. It also evaluates graduates' perceptions that the program helped them to attain knowledge in the "common core areas" as identified by our accrediting body (CACREP). Sixteen (16) graduates completed the GGSS. Please see the table below showing GGSS results. From students who completed the GGSS (N=16), 90% or more of students agreed or strongly agreed in 9 out of 12 areas asssessed. The three areas with the lowest scores included the Item #4 portfolio, Item #7 human growth & development, Item #8 career development, and Item #11 assessment and Item #12 research. Graduating Graduate Student Survey Results (N=16) ITEMS Strongly Agree Agree Strongly Neutral Disagree Disagree N/A For items 1 to 4, to what extent do you agree or disagree that the following elements of your UCF Counselor Education program adequately prepared you to become a highly qualified counselor: 1. Coursework in the Counselor Education program? 7 (44%) 8 (50%) 1 (6.2%) Coursework outside the Counselor Education program? 7 (44%) 7 (44%) 1 (6%) 1 (6%) 3. Practicum/Internship? 12 (75%) 3 (19%) 1 (6%) Portfolio? (25%) (31%) (44%) For items 5-12, to what extent do you agree or disagree that the UCF Counselor Education program helped you to attain the experiences and knowledge as indicated below in the common core areas identified by CACREP: 5. Professional Identity & Ethics 11 (69%) 5 (31%) Social and Cultural Diversity 9 (56%) 5 (31%) 1 (6%) 1 (6%) Human Growth and Development (44%) (44%) (13%) Career Development (50%) (31%) (19%) Helping Relationships 12 (75%) 3 (19%) 1 (6%) Group Work 9 (56%) 5 (31%) 1 (6%) 1 (6%) Assessment (25%) (50%) 12. Research and Program Evaluation 5 (31%) 10 (63%) AVERAGE (PERCENTAGE) for ITEMS 5 TO 12 ONLY 8.88 (68%) 3.63 (28%) 3 (19%) 1 (6%) (6%) (.03%) :16:05 AM]

24 Review: Measure: 3.2 Ninety percent (90%) of students in the Marriage and Family Therapy progam will receive a rating of 1 or better on a 3-point scale (0=below expectations, 1=meets expectations, 2=exceeds expectations) on the Intern Final Evaluation Form as completed by the internshipby the internship site supervisor in students' final semester of MHS 6830 Internship in Counseling. Result: During their final two semesters of the Marriage & Family Therapy program, students enroll in MHS 6830 Internship in Counselor Education, which consists of field experience in a counseling setting. At the end of the semester, internship site supervisors complete the Intern Final Evaluation (see Attachment) rating the student on several competency areas using a 3 point scale (0=below expectations, 1=meets expectations, 2=exceeds expectations). Of the students evaluated (N=5), 100% scored a 1 (meets expectatons) or better on the Intern Evaluation Form. Review: Reflective Statement: Students received excellent evaluations on their Intern Final Evaluation form (as evaluated by their site supervisor); however, only 5 forms were made available to the IE coordinator by internship instructors. This reflects the ongoing problem of lost data due to use of paper forms. Efforts will be made to make these form electronic so no data is lost. Furthermore, based on the results of the GGSS, at least 90% of students either agreed or strongly agreed that the UCF Marriage & Family Therapy program prepared them to become a highly qualified counselors in all areas except four: portfolio, human growth & development, career development, and assessment. From this information, the program needs to further investigate the purpose for these results and decide on ways to improve students' perception of these program areas. Reflective Statement Review: Overall Outcome Results Review Comment: Table for Measure 3.1: In addition to items 4, 7, 8, and 11, items 2, 6 and 10, though close, also fell short of 90% agree or strongly agree responses. That means only 5 out of 12 areas add up to 90% or more agreeing or strongly agreeing. The narrative indicates that 9 out of 12 items are adding up to over 90%. 11:16:05 AM]

25 Attachments: Graduating Graduate Student Survey (GGSS).pdf Changes to Academic Process: Modify Frequency or Schedule of Course Offerings Make Technology Related Improvements Make Personnel Related Changes Implement Additional Training Revise Advising Standards or Process Revise Admission Criteria Other implemented or planned change No Changes to Academic Process Explain EACH item checked above: No major changes to the academic process were made this year. Changes to Curriculum: Revise and/or Enforce Prerequisites Revise Course Sequence Revise Course Content Add Course Delete Course Other implemented or planned change No Changes to Curriculum Criteria: Please comment on implemented and planned changes Clear statement of change(s) Description of how changes created improvements; make suggestions for future cycles Review: Revision or explanation needed Review Comments: Explain EACH item checked above: No major changes to the curriculum were made this year. Changes to Assessment Plan: Revise Student Outcome Statement Revise Measurement Approach Collect and Analyze Additional Data and Information Change Method of Data Collection Other Planned Changes Plan has been reviewed and no change made No Changes to Assessment Plan Explain EACH item checked above: The Counselor Education program faculty will review the Counseling Competency Scale (CCS) cut scores to determine if they need to be raised. If so, this would change measures for 11:16:05 AM]

26 outcome 2. Furthermore, the CCS has been implemented into (1) MHS 5005 Introduction to the Counseling Profession and (2) MHS 6401 Techniques of Counseling to support students learning outcomes and evaluate programmatic effectiveness. We will begin evaluating CCS scores in these two classes in the next year to fully utilize the CCS as a formative measure. The Counselor Education program faculty will implement new portfolio assessment requirements and rubric, as well as a new Intern Final Evaluation form within the next year. We hope these new instruments will provide more useful data for analyzing our program outcomes. Curriculum/Course-related Assessment Methods: Capstone Course Capstone Project or Performance Evaluation Case study / Simulation Course-embedded Questions Portfolio Rating Scale / Scoring Rubric (yields a grade) Assessment Rubrics (student demonstrates proficiency) Lab Journals / Reports Observation (focused on specific program outcomes) Other method Explain EACH item checked above: 1. Portfolio: Students present their individual portfolio during their internship class, which is typically the semester they graduate. Portfolios are graded by the advisor and the internship instructor and are evaluated on seven areas of competency based on accreditation standards by the Council of Accreditation of Counseling and Related Academic Programs (CACREP). These areas include the following: Professional Identity and Ethics; Social and Cultural Diversity and Human Development Across the Lifespan; Use of Technology and Assessment Skills; Clinical, Consultation and Communication Skills; Specialty (play therapy, family therapy, career development); Personal Growth and Wellness; and Research and Program Evaluation. The portfolio is scored using a rubric, which evaluates each competency area using a Likert Scale from 0 to 3 as follows: 0 = No Evidence: Demonstrates no evidence of knowledge, skills and dispositions in this area. 1 = Emerging: Demonstrates evidence of limited knowledge, skills and dispositions in this area. 2 = : Demonstrates evidence of consistent knowledge, skills and dispositions in this area. 3 = Exceptional: Demonstrates evidence of strong knowledge, skills and dispositions in this area. 2. Counseling Competencies Scale (CCS): The CCS is a 32-item instrument designed to measure counseling competencies within three proposed factors: (a) counseling skills, (b) professional dispositions, and (c) professional behaviors (please see the attached copy of the CCS). We used the counseling skills section of the CCS only, which contains 12 items that evaluate such basic counseling skills as paraphrasing, questioning, encourages, reflecting feelings, confrontation and goal setting. The CCS contains five supervisor-rater evaluation response categories that include the following: 11:16:05 AM]

27 0 = Harmful. The counseling student demonstrates harmful use of knowledge, skills, and dispositions in the specified counseling skill(s), professional disposition(s), and professional behavior(s). 2 = Below Expectations. The counseling student demonstrates limited or no evidence of the knowledge, skills, and dispositions in the specified counseling skill(s), professional disposition(s), and professional behavior(s). 4 = Near Expectations. The counseling student demonstrates inconsistent and limited knowledge, skills, and dispositions in the specified counseling skill(s), professional disposition(s), and professional behavior(s). 6 = Meets Expectations. The counseling student demonstrates consistent and proficient knowledge, skills, and dispositions in the specified counseling skill(s), professional disposition(s), and professional behavior(s). 8 = Exceeds Expectations. The counseling student demonstrates strong knowledge, skills, and dispositions in the specified counseling skill(s), professional disposition(s), and professional behavior(s). 3. Intern Final Evaluation: During their final two semesters of the mental health counseling program, students enroll in MHS 6830 Internship in Counselor Education, which consists of field experience in a mental health counseling setting. At the end of the semester, internship site supervisors complete the Intern Final Evaluation rating the student on several competency areas using a 3 point scale: 1 = Below Expectations / Insufficient / Unacceptable (0) = the mental health counseling intern demonstrates inconsistent and limited knowledge, skills, and dispositions in the specified counseling skill(s), professional disposition(s), and professional behavior(s). Remediation is necessary. 2 = Meets Expectations / Demonstrates Competencies (1) = the mental health counseling intern demonstrates consistent and proficient knowledge, skills, and dispositions in the specified counseling skill(s), professional disposition(s), and professional behavior(s). A beginning mental health counselor should be at this level at the conclusion of his or her internship. 3 = Exceeds Expectations / Demonstrates Competencies (2) = the mental health counseling intern demonstrates strong (i.e., exceeding the expectations of a beginning mental health counselor) knowledge, skills, and dispositions in the specified counseling skill(s), professional disposition(s), and professional behavior(s). Examinations/Tests: Standardized: Nationally-normed Exam State-normed Exam Other Explain EACH item checked above: Counselor Preparation Comprehensive Examination (CPCE): The CPCE is a standardized assessment instrument utilized by over 190 universities and colleges and is designed to assess counseling students knowledge of counseling information viewed as important by counselor preparation programs. The CPCE consists of eight content areas similar to the areas of competency identified by the Council of Accreditation of Counseling and Related Academic Programs (CACREP). These content areas include Human Growth and Development, Social and Cultural Foundations, Helping Relationships, Group Work, 11:16:05 AM]

28 Career and Lifestyle Development, Appraisal, Research and Program Evaluation, and Professional Orientation and Ethics. Local: Post-test Only Pre-post Test Other exam or test Explain EACH item checked above: Surveys: Institution (UCF): UCF Graduating Student Survey (Seniors or Graduate student) Alumni Survey Student Satisfaction Survey First Destination Survey Employee Survey Explain EACH item checked above: Graduating Graduate Student Survey (GGSS): The GGSS assesses the extent to which mental health counseling graduates agree or disagree that the UCF Counselor Education prepared them to become a highly qualified counselor. It also evaluates graduates' perceptions that the program helped them to attain knowledge in the "common core areas" as identified by our accrediting body (CACREP). Local: Alumni Survey (Department or Program; not UCF) Customer Satisfaction Survey Exit and Other Interviews Explain EACH item checked above: Other Survey(s): National Survey State Survey Other Survey Explain EACH item checked above: 11:16:05 AM]

29 Miscellaneous Assessment Methods: Advisory Board Focus Group Institutional Data Student Records Accreditation Reviews (e.e. SACS, NCATE, ABET) Other Explain EACH item checked above: Institutional Effectiveness Assessment Results Rubric *If programs or units fail to provide any input, their results will be evaluated with "No effort (0)." Beginning (1) Emerging (2) Meets Expectations (3) Accomplished (4) Exemplary (5) Indicators: 1. Complete and relevant data are provided for all measures OR if data are incomplete or missing, an explanation is provided 2. Data reporting is thorough (see below ) i.e.,populations are defined;sampling methods and response rates are provided with survey data, etc. 3. Results for each measure indicate whether the target for that measure has been met 4. Reflective statements are provided either for each outcome or aggregated for multiple outcomes 5. Implemented and planned changes are included and are linked to assessment data, or if no changes are reported, an explanation is provided 6. Assessment instruments are attached or linked to if not proprietary Additional Indicators: 7. Includes description of how the assessment process has been useful to your program or unit 8. Includes description of how IE Assessment has resulted in quality improvement initiatives 9. Data collection and analysis are used to assess the impact of implemented changes, demonstrating a fully closed loop process Summary of Assessment Process: Think about the last few years and describe evidence-based changes that have taken place because of assessment. Also address other factors that have caused changes to be made (e.g., state mandate, accreditation review recommendations). Over the last few years, we've made several changes to our assessment program as we Review Criteria: (Examples: Could you be more specific? Has your benchmark remained at this level too long?) Review: 11:16:05 AM]

30 worked to improve data collection procedures. We added new instruments to the assessment plan, including the Counseling Competencies Scale (CCS), the Graduating Graduate Student Survey (GGSS), and a new Intern Evaluation Form. We will be implementing a new assessment instrument (i.e., a new portfolio rubric) within the next year in an effort to obtain more useful data to inform our program. We have extended the use of the CCS by administering it not only in MHS 6803 Practicum, but also using it in MHS 5005 Introduction to Counseling and MHS 6401 Techniques in Counseling. We hope to make this instrument truly formative and to be able to evaluate scores from these additional classes in the next couple of years. The Counselor Education program will also determine if the Counseling Competencies Scale (CCS) cut scores for practicum need revising. Since it is so late in the reporting cycle, we are not requesting corrections to the reporting errors noted above. Overall, this is a solid report despite those minor errors. The data reported are accurate--the problem is with the narrative summaries associated with the data table for Measure 3.1. Also, please read the prompt carefully for the Summary of Assessment Process statement. This statement should be more results oriented than process oriented (regardless of the title). Also, this is where you should be reporting other sources of information used to inform change (e.g., advisory group input, changes in national standards, etc.). Top Site maintained by Operational Excellence and Assessment Support Terms of Use Privacy Policy Webmaster 11:16:05 AM]

31 Menu Home > Assessment Plan and Results User: ltomei Role: DRC Chair Help Logout X Assessment Plan and Results Plan Year: Program/Unit: Counselor Education - Mental Health - M.A. U IE Assessment Rubrics Assessment DRC Chair Instructions Assessment for Unit Heads View/Submit Results Review Results Review Program/Unit: Counselor Education - Mental Health - M.A. DRC: College of Education Year: DRC Chair: Lance J Tomei Due Date: 09/30/2011 Coordinator(s): K. Dayle Jones Reviewer(s): Lindsay Archambault Quick Links: Mission: The Master of Arts (M.A.) in Counselor Education, Mental Health Counseling Track, is designed to educate and prepare individuals interested in working as mental health counselors in community mental health settings, institutions, hospitals, schools, and private practice. Nationally accredited by the Council for the Accreditation of Counseling and Related Education Programs (CACREP), the mental health counseling program emphasizes the blending of research and theory with practical applied experience. It also emphasizes and reflects the urban and multicultural nature of our community, as well as general trends in specializations within the field of counseling. The program involves intensive clinical training and field work. Assessment Process: The assessment strategy of the M.A. program in Mental Health Counseling involves systematic assessment of two key areas of student performance: knowledge and skills. We will assess areas of student performance with both formative and summative measurements. Knowledge comprises information deemed as fundamental for student success in the counseling field and includes such content areas as Human Growth and Development, Social and Cultural Foundations, Helping Relationships, Group Work, Career and Lifestyle Development, Appraisal, Research and Program Evaluation, and Professional Orientation and Ethics. Student knowledge will be appraised using scores on a nationally normed test called the Counselor Preparation Competency Exam (CPCE) (see attachment) and scores on the student portfolio rubric (see attachment). Both instruments are administered during students' final semester before graduation. Skills refer to those demonstrable counseling skills that students need to master to become proficient as a mental health counselor. Skills will be evaluated using the Counseling Competencies Scale (CCS) (see attachment) at midpoint and at final of MHS 6803 Practicum in Counselor Education, which students usually take in their fourth semester in the program. In addition, the Intern Final Evaluation (see attachment) will be completed by the 11:15:22 AM]

32 Internship site supervisor during MHS 6830 Counseling Internship, which is usually taken in the sixth or seventh semester in the program. To measure students own perceptions about their knowledge and skills level, we have also developed program specific questions for counselor education students to complete as part of the "Graduating Graduate Student Survey" (an indirect measure) (see attachment). This survey will analyze students' opinions about the degree to which they feel adequately prepared to become a highly qualified mental health counselor. These questions were designed based on the required core curricular areas outlined by the Council of Accreditation for Counseling and Related Educational Programs (CACREP). Students complete this survey when they file their intent to graduate form. The assessment strategy is managed by the coordinator of the mental health counseling program as well as the research instructor in charge of all program evaluation within the counselor education program. These efforts to continually enhance program quality directly support UCF Strategic Goal #2. Top Outcome: 1 Students in the M.A. program in Counselor Education, Mental Health Counseling Track, will demonstrate knowledge deemed as fundamental for student success in the counseling field. Measure: 1.1 On the Counselor Preparation Competency Exam (CPCE), students' mean score will equal or exceed the mean score of the CPCE national norm on each of the eight (8) Subscales and the Total Score. Result: Each year, the Center for Credentialing and Education provides national descriptive statistics for the CPCE (see Attachment). The CPCE is a standardized assessment instrument utilized by over 190 universities and colleges and is designed to assess counseling students knowledge of counseling information viewed as important by counselor preparation programs. The CPCE consists of eight content areas similar to the areas of competency identified by the Council of Accreditation of Counseling and Related Academic Programs (CACREP). These content areas include (1) Human Growth and Development, (2) Social and Cultural Foundations, (3) Helping Relationships, (4) Group Work, (5) Career and Lifestyle Development, (6) Appraisal, (7) Research and Program Evaluation, and (8) Professional Orientation and Ethics. For 2011, the national average for the total CPCE score was (n = 1398). Mental health conseling students total average CPCE scores was Out of 27 mental health counseling students who took the CPCE, students' scores exceeded the mean CPCE scores on all but two subscales: (1) Career and Lifestyle Development and (2) Research and Program Evaluation (see table below). The table below compares the mean scores between the national norm group and UCF students. Comparison of CPCE National Mean Scores & UCF Mental Health Counseling Student Mean Scores CPCE National Mean Scores (N=1398) Content Area 1. Human Growth and Development Social and Cultural Foundations Helping Relationships Group Work Career and Lifestyle Development Appraisal Research and Program Evaluation Professional Orientation and Ethics TOTAL UCF Mean Scores (N=27) 11:15:22 AM]

33 Review: Measure: 1.2 Ninety percent (90%) of students in the M.A. program in Counselor Education, Mental Health Counseling Track, will receive a score of satisfactory or better on each area of their portfolio. Result: Of a sample of 17 students in the mental health counseling program who submitted their portfolio for review during the academic year, all (100%) scored a satisfactory or better on each competency area of their portfolio. The mean score for the mental health counseling students was 2.98 according to the portfolio rubric. Students present their individual portfolio during their internship class, which is typically the semester they graduate. Portfolios are graded by the advisor and the internship instructor and are evaluated on seven areas of competency based on accreditation standards by the Council of Accreditation of Counseling and Related Academic Programs (CACREP). These areas include the following: Professional Identity and Ethics; Social and Cultural Diversity and Human Development Across the Lifespan; Use of Technology and Assessment Skills; Clinical, Consultation and Communication Skills; Specialty (play therapy, family therapy, career development); Personal Growth and Wellness; and Research and Program Evaluation The portfolio is scored using a rubric (see Attachment) which evaluates each competency area using the following 4 -point scale: 0 = No Evidence: Demonstrates no evidence of knowledge, skills and dispositions in this area. 1 = Emerging: Demonstrates evidence of limited knowledge, skills and dispositions in this area 2 = : Demonstrates evidence of consistent knowledge, skills and dispositions in this area. 3 = Exceptional: Demonstrates evidence of strong knowledge, skills and dispositions in this area. Review: Reflective Statement: As a measure of whether our students are demonstrating knowledge of the counseling field, our students' CPCE scores exceeded the national mean scores on all subscales except (a) Career and Lifestyle Development and (b) Research and Program Evaluation. We further reviewed CPCE scores over the last 5 years. As seen in the table below, all subscales scores except for Appraisal declined between the and academic years. Furthermore, attached is a chart of the CPCE total scores over the last 5 years. Upon review, we noticed that total scores increased from year 1 to 3, and then declined over the last 2 years. Our program needs to evaluate the purpose for these lowered scores and examine strategies to improve the scores. Mental Health Counseling CPCE Subscale Scores to Content Area (N=30) (N=34) (N=35) (N=18) (N=27) 1. Human Growth & Development Social & Cultural Foundations :15:22 AM]

34 3. Helping Relationships Group Work Career & Lifestyle Development Appraisal Research & Program Evaluation Professional Orientation & Ethics TOTAL Regarding the portfolio data, the results that 100% of students scored a 1 (meets expectations) or better were not very useful. A new portfolio and rubric were developed last year and will be implemented this year, with the hope of collecting more useful data to inform our program. Reflective Statement Review: Overall Outcome Results Review Comment: Excellent follow-up analysis of longitudinal data related to this Outcome. Attachments: Portfolio Evaluation.pdf CPCE National Statistics Fall 2011.pdf MH CPCE Total Scores Chart pdf Top Outcome: 2 Students in the M.A. program in Counselor Education, Mental Health Counseling Track, will demonstrate basic counseling skills required for all mental health counselors in their practicum experience in an educational training facility. Measure: 2.1 Fifty percent (50%) of students in the M.A. program in Counselor Education, Mental Health Counseling Track, will score at least a 6.0 (meets expectations) on the Counseling Competencies Scale (CCS) measured at midpoint in MHS 6803 Practicum I. Result: The Counseling Competencies Scale (CCS) is a 32-item instrument designed to measure counseling competencies within three proposed factors: (a) counseling skills, (b) professional dispositions, and (c) professional behaviors (please see the attached copy of the CCS). We used the counseling skills section of the CCS only, which contains 12 items that evaluate such basic counseling skills as paraphrasing, questioning, encourages, reflecting feelings, confrontation and goal setting. The CCS contains five supervisor-rater evaluation response categories that include the following: 8 = Exceeds Expectations. The counseling student demonstrates strong knowledge, skills, and dispositions in the specified counseling skill(s), professional disposition(s), and professional behavior(s). 11:15:22 AM]

35 6 = Meets Expectations. The counseling student demonstrates consistent and proficient knowledge, skills, and dispositions in the specified counseling skill(s), professional disposition(s), and professional behavior(s). 4 = Near Expectations. The counseling student demonstrates inconsistent and limited knowledge, skills, and dispositions in the specified counseling skill(s), professional disposition(s), and professional behavior(s). 2 = Below Expectations. The counseling student demonstrates limited or no evidence of the knowledge, skills, and dispositions in the specified counseling skill(s), professional disposition(s), and professional behavior(s). 0 = Harmful. The counseling student demonstrates harmful use of knowledge, skills, and dispositions in the specified counseling skill(s), professional disposition(s), and professional behavior(s). On the CCS Counseling Skills subscale, the table below documents the frequencies of counselors' scores at midterm. Of the 27 students, 83% of students scored at least a 6.0 or better. CCS Counseling Skills Subscale Item Frequencies at MIDTERM (N=27) Reflect Reflect Freq Frequency NonverbEncourag Quest Para Feelings MeaningSumm ConfrontGoalFocus EmpathRespect% 8 Exceeds 8 Expectations % 6 Meets 6 Expectations % 4 Near 4 Expectations % 2 Below Expectations Harmful Review: Measure: 2.2 Seventy five (75%) of students in the M.A. program in Counselor Education, Mental Health Counseling Track, will score at least a 6.0 (meets expectations) on the Counseling Competencies Scale (CCS) measured at the final in MHS 6803 Practicum I. Result: On the CCS Counseling Skills subscale, the table below shows the frequencies of counselors' scores (N=27) at final. This shows that 97% of students scored at least a 6 or better. CCS Counseling Skills Subscale Item Frequencies at FINAL (N=27) Frequency 8 Exceeds Expectations 6 Meets Expectations Reflect Reflect Freq NonverbEncour Quest Para Feelings MeaningSumm ConfrontGoals Focus EmpathRespect% % % 4 Near Expectations % 2 Below Expectations :15:22 AM]

36 0 Harmful Review: Reflective Statement: As a measure of our students' counseling skills, 83% of students met the cut-off score of "meets expectations" (6) on the CCS at practicum midterm, which surpassed our outcome measure of 50% meeting the cut score. At final practicum, 97% of students scored at least at "meets expectations," also exceeding the stated outcome measure of 75%. Two years ago, we reduced these cut-off scores because they were unrealistically high for students to attain (previously, the midterm CCS cut-off score was 75% and the final score was 90%). At this point, we may need to reevaluate these cut scores to determine if they need raising. Reflective Statement Review: Overall Outcome Results Review Comment: Attachments: Counseling Competencies Scale.pdf Top Outcome: 3 Students in the M.A. program in Counselor Education, Mental Health Counseling Track, will demonstrate skills as well as feel prepared to work effectively in the field of mental health counseling. Measure: 3.1 Ninety percent (90%) of students in the M.A. program in Counselor Education, Mental Health Counseling Track, will agree or strongly agree that the UCF Counselor Education program helped them attain knowledge and skills (as indicated in the common core areas identified by CACREP) as measured by the Graduating Graduate Student Survey. Result: The Graduating Graduate Student Survey (GGSS) assesses the extent to which mental health counseling graduates agree or disagree that the UCF Counselor Education prepared them to become a highly qualified counselor. It also evaluates graduates' perceptions that the program helped them to attain knowledge in the "common core areas" as identified by our accrediting body (CACREP). Twelve (12) graduates completed the GGSS. Please see the table below showing specific results on the GGSS. A total of 40 mental health counseling students completed the GGSS, and 90% or more agreed or strongly agreed in 9 out of 12 areas asssessed. The three areas with the lowest scores included the Item #4 portfolio, Item #11 assessment and Item #12 research. Mental Health - Graduating Graduate Student Survey Results (N=40) Strongly Strongly 11:15:22 AM]

37 ITEMS Agree Agree Neutral Disagree Disagree N/A For items 1 to 4, to what extent do you agree or disagree that the following elements of your UCF Counselor Education program adequately prepared you to become a highly qualified counselor: 1. Coursework in the Counselor Education program? (63%) (33%) 2 (5%) Coursework outside the Counselor Education program? (38%) (44%) Practicum/Internship? (80%) 4. Portfolio? 9 (23% 13 (33%) 6 (15%) (20%) (33%) 2 (5%) 2 (5%) 1 (2.5%) 1 (2.5%) For items 5-12, to what extent do you agree or disagree that the UCF Counselor Education program helped you to attain the experiences and knowledge as indicated below in the common core areas identified by CACREP: 5. Professional Identity & Ethics 6. Social and Cultural Diversity 7. Human Growth and Development 8. Career Development 9. Helping Relationships 10. Group Work 11. Assessment 12. Research and Program Evaluation 32 (80%) 23 (58%) 16 (40%) 16 (40%) 31 (79%) 23 (58%) 11 (28%) 11 (28%) 8 (20%) (38%) 2 (5%) (50%) 20 (50%) 4 (10%) (10%) (18%) 1 (3%) (43%) 20 (51%) 17 (43%) (18%) 11 (28%) 1 (3%) (2.5%) 0 0 Review: Measure: 3.2 Ninety percent (90%) of students in the M.A. program in Counselor Education, Mental Health Counseling Track, will receive a rating of 1 or better on a 3-point scale (0=below expectations, 1=meets expectations, 2=exceeds expectations) on the Intern Final Evaluation Form as completed by the internship site supervisor in students' final semester of MHS 6830 Internship in Counseling. Result: During their final two semesters of the mental health counseling program, students enroll in MHS 6830 Internship in Counselor Education, which consists of field experience in a mental health counseling setting. At the end of the semester, internship site supervisors complete the Intern Final Evaluation (see Attachment) rating the student on several competency areas using a 3 point scale (0=does not meet expectations; 1 = meets expectations; 2 = exceeds expectations). All students (N = 6; 100%) evaluated with the Intern Final Evaluation form received an average rating of 1 or 11:15:22 AM]

38 better on the Intern Final Evaluation Form. Review: Reflective Statement: Students received excellent evaluations on their Intern Final Evaluation form (as evaluated by their site supervisor); however, only 6 forms were made available to the IE coordinator by internship instructors. This reflects the ongoing problem of lost data due to use of paper forms. Efforts will be made to make these form electronic so no data is lost. Based on the results of the GGSS, at least 90% of students either agreed or strongly agreed that the UCF Mental Health Counseling program prepared them to become a highly qualified counselors. However, there were three areas in the program in which fewer than 90% of students felt that they were prepared: Research, asssessment, and the portfolio. From this information, the program needs to further investigate the purpose for these results and decide on ways to improve students' perception of these program areas. Reflective Statement Review: Overall Outcome Results Review Comment: Results for Measure 3.1: Some minor errors noted. You indicate 12 respondents to the GGSS in one location and 40 in another. You have data for 40. Also, the data table is labeled when, in fact, the data are for Attachments: Graduating Graduate Student Survey (GGSS).pdf Intern Final Evaluation.pdf Changes to Academic Process: Modify Frequency or Schedule of Course Offerings Make Technology Related Improvements Make Personnel Related Changes Implement Additional Training Revise Advising Standards or Process Revise Admission Criteria Other implemented or planned change No Changes to Academic Process Explain EACH item checked above: No major changes to the academic process were made this year. Changes to Curriculum: Revise and/or Enforce Prerequisites Revise Course Sequence Criteria: Please comment on implemented and planned changes Clear statement of change(s) Description of how changes created improvements; make suggestions for future cycles Review: Revision or explanation needed 11:15:22 AM]

39 Revise Course Content Add Course Delete Course Other implemented or planned change No Changes to Curriculum Review Comments: Good reporting overall. Explain EACH item checked above: No major changes to the program curriculum were made this year. Changes to Assessment Plan: Revise Student Outcome Statement Revise Measurement Approach Collect and Analyze Additional Data and Information Change Method of Data Collection Other Planned Changes Plan has been reviewed and no change made No Changes to Assessment Plan Explain EACH item checked above: The Counselor Education program faculty will review the Counseling Competency Scale (CCS) cut scores to determine if they need to be raised. If so, this would change measures for outcome 2. Furthermore, the CCS has been implemented into (1) MHS 5005 Introduction to the Counseling Profession and (2) MHS 6401 Techniques of Counseling to support students learning outcomes and evaluate programmatic effectiveness. We will begin evaluating CCS scores in these two classes in the next year to fully utilize the CCS as a formative measure. The Counselor Education program faculty will implement new portfolio assessment requirements and rubric, as well as a new Intern Final Evaluation form within the next year. We hope these new instruments will provide more useful data for analyzing our program outcomes. Curriculum/Course-related Assessment Methods: Capstone Course Capstone Project or Performance Evaluation Case study / Simulation Course-embedded Questions Portfolio Rating Scale / Scoring Rubric (yields a grade) Assessment Rubrics (student demonstrates proficiency) Lab Journals / Reports Observation (focused on specific program outcomes) Other method Explain EACH item checked above: 1. Portfolio: Students present their individual portfolio during their internship class, which is 11:15:22 AM]

40 typically the semester they graduate. Portfolios are graded by the advisor and the internship instructor and are evaluated on seven areas of competency based on accreditation standards by the Council of Accreditation of Counseling and Related Academic Programs (CACREP). These areas include the following: Professional Identity and Ethics; Social and Cultural Diversity and Human Development Across the Lifespan; Use of Technology and Assessment Skills; Clinical, Consultation and Communication Skills; Specialty (play therapy, family therapy, career development); Personal Growth and Wellness; and Research and Program Evaluation. The portfolio is scored using a rubric, which evaluates each competency area using a Likert Scale from 0 to 3 as follows: 0 = No Evidence: Demonstrates no evidence of knowledge, skills and dispositions in this area. 1 = Emerging: Demonstrates evidence of limited knowledge, skills and dispositions in this area. 2 = : Demonstrates evidence of consistent knowledge, skills and dispositions in this area. 3 = Exceptional: Demonstrates evidence of strong knowledge, skills and dispositions in this area. 2. Counseling Competencies Scale (CCS): The CCS is a 32-item instrument designed to measure counseling competencies within three proposed factors: (a) counseling skills, (b) professional dispositions, and (c) professional behaviors (please see the attached copy of the CCS). We used the counseling skills section of the CCS only, which contains 12 items that evaluate such basic counseling skills as paraphrasing, questioning, encourages, reflecting feelings, confrontation and goal setting. The CCS contains five supervisor-rater evaluation response categories that include the following: 0 = Harmful. The counseling student demonstrates harmful use of knowledge, skills, and dispositions in the specified counseling skill(s), professional disposition(s), and professional behavior(s). 2 = Below Expectations. The counseling student demonstrates limited or no evidence of the knowledge, skills, and dispositions in the specified counseling skill(s), professional disposition(s), and professional behavior(s). 4 = Near Expectations. The counseling student demonstrates inconsistent and limited knowledge, skills, and dispositions in the specified counseling skill(s), professional disposition(s), and professional behavior(s). 6 = Meets Expectations. The counseling student demonstrates consistent and proficient knowledge, skills, and dispositions in the specified counseling skill(s), professional disposition(s), and professional behavior(s). 8 = Exceeds Expectations. The counseling student demonstrates strong knowledge, skills, and dispositions in the specified counseling skill(s), professional disposition(s), and professional behavior(s). 3. Intern Final Evaluation: During their final two semesters of the mental health counseling program, students enroll in MHS 6830 Internship in Counselor Education, which consists of field experience in a mental health counseling setting. At the end of the semester, internship site supervisors complete the Intern Final Evaluation rating the student on several competency areas using a 3 point scale: 1 = Below Expectations / Insufficient / Unacceptable (0) = the mental health counseling intern demonstrates inconsistent and limited knowledge, skills, and dispositions in the specified counseling skill(s), professional disposition(s), and professional behavior(s). Remediation is necessary. 2 = Meets Expectations / Demonstrates Competencies (1) = the mental health counseling intern demonstrates consistent and proficient knowledge, skills, and dispositions in the specified counseling skill(s), professional disposition(s), and professional behavior(s). A beginning mental health counselor should be at this level at the conclusion of his or her internship. 3 = Exceeds Expectations / Demonstrates Competencies (2) = the mental health counseling intern demonstrates strong (i.e., exceeding the expectations of a beginning mental health counselor) knowledge, skills, and dispositions in the specified counseling skill(s), professional disposition(s), and professional behavior(s). 11:15:22 AM]

41 Examinations/Tests: Standardized: Nationally-normed Exam State-normed Exam Other Explain EACH item checked above: Counselor Preparation Comprehensive Examination (CPCE): The CPCE is a standardized assessment instrument utilized by over 190 universities and colleges and is designed to assess counseling students knowledge of counseling information viewed as important by counselor preparation programs. The CPCE consists of eight content areas similar to the areas of competency identified by the Council of Accreditation of Counseling and Related Academic Programs (CACREP). These content areas include Human Growth and Development, Social and Cultural Foundations, Helping Relationships, Group Work, Career and Lifestyle Development, Appraisal, Research and Program Evaluation, and Professional Orientation and Ethics. Local: Post-test Only Pre-post Test Other exam or test Explain EACH item checked above: Surveys: Institution (UCF): UCF Graduating Student Survey (Seniors or Graduate student) Alumni Survey Student Satisfaction Survey First Destination Survey Employee Survey Explain EACH item checked above: Graduating Graduate Student Survey (GGSS): The GGSS assesses the extent to which mental health counseling graduates agree or disagree that the UCF Counselor Education prepared them to become a highly qualified counselor. It also evaluates graduates' perceptions that the program helped them to attain knowledge in the "common core areas" as identified by our accrediting body (CACREP). Local: 11:15:22 AM]

42 Alumni Survey (Department or Program; not UCF) Customer Satisfaction Survey Exit and Other Interviews Explain EACH item checked above: Other Survey(s): National Survey State Survey Other Survey Explain EACH item checked above: Miscellaneous Assessment Methods: Advisory Board Focus Group Institutional Data Student Records Accreditation Reviews (e.e. SACS, NCATE, ABET) Other Explain EACH item checked above: Institutional Effectiveness Assessment Results Rubric *If programs or units fail to provide any input, their results will be evaluated with "No effort (0)." Beginning (1) Emerging (2) Meets Expectations (3) Accomplished (4) Exemplary (5) Indicators: 1. Complete and relevant data are provided for all measures OR if data are incomplete or missing, an explanation is provided 2. Data reporting is thorough (see below ) i.e.,populations are defined;sampling methods and response rates are provided with survey data, etc. 3. Results for each measure indicate whether the target for that measure has been met 4. Reflective statements are provided either for each outcome or aggregated for multiple outcomes 5. Implemented and planned changes are included and are linked to assessment data, or if no changes are reported, an explanation is provided 6. Assessment instruments are attached or linked to if not proprietary Additional Indicators: 7. Includes description of how the assessment process has been useful to your program or unit 11:15:22 AM]

43 8. Includes description of how IE Assessment has resulted in quality improvement initiatives 9. Data collection and analysis are used to assess the impact of implemented changes, demonstrating a fully closed loop process Summary of Assessment Process: Think about the last few years and describe evidence-based changes that have taken place because of assessment. Also address other factors that have caused changes to be made (e.g., state mandate, accreditation review recommendations). Over the last few years, we've made several changes to our assessment program as we worked to improve data collection procedures. We added new instruments to the assessment plan, including the Counseling Competencies Scale (CCS), the Graduating Graduate Student Survey (GGSS), and a new Intern Evaluation Form. We will be implementing a new assessment instrument (i.e., a new portfolio rubric) within the next year in an effort to obtain more useful data to inform our program. We have extended the use of the CCS by administering it not only in MHS 6803 Practicum, but also using it in MHS 5005 Introduction to Counseling and MHS 6401 Techniques in Counseling. We hope to make this instrument truly formative and to be able to evaluate scores from these additional classes in the next couple of years. The Counselor Education program will also determine if the Counseling Competencies Scale (CCS) cut scores for practicum need revising. Review Criteria: (Examples: Could you be more specific? Has your benchmark remained at this level too long?) Review: Your ongoing utilization of CCS and associated trend analysis does a good job of "closing the loop." Your Summary of Assessment Process statement does not really respond to the prompt. It does not talk about implemented changes and other catalysts for change as suggested (e.g., haven't new CACREP standards resulted in some changes?). That said, good report overall. We have tracked student scores on the CPCE for the last 5 years, and we've seen a recent decline in scores over the last one to two years. Our program needs to evaluate the purpose for these lowered scores and examine strategies to improve the scores. Top Site maintained by Operational Excellence and Assessment Support Terms of Use Privacy Policy Webmaster 11:15:22 AM]

44 Menu Home > Assessment Plan and Results User: ltomei Role: DRC Chair Help Logout X Assessment Plan and Results Plan Year: Program/Unit: School Counseling for Certification IE Assessment Rubrics Assessment DRC Chair Instructions Assessment for Unit Heads View/Submit Results Review Results Review Program/Unit: School Counseling for Certification DRC: College of Education Year: DRC Chair: Lance J Tomei Due Date: 09/28/2012 Coordinator(s): Glenn Lambie Reviewer(s): Lindsay Archambault Quick Links: Mission: The mission of the School Counseling for Certification Track in the Counselor Education Program is to promote the development of knowledgeable, skilled and reflective educational professionals who are effective change agents and advocates for youth, families, and communities of varying abilities, cultural heritages and socioeconomic backgrounds. In addition, it is primary that the School Counseling for Certification Track in the Counselor Education Program students feel capable of, and committed to, assuming leadership roles in the fields of school counseling and education. The School Counseling for Certification Track in the Counselor Education Program provides students with the knowledge, dispositions, and skills needed to be ethical and effective professional school counselors in diverse settings and meets the certification requirements to be a school counselor in the State of Florida. Furthermore, the School Counseling for Certification Track in the Counselor Education Program goal is to be a national leader in the preparation of high quality and effective professional school counselors as measured by national, state, and local data collection sources (e.g., Counselor Preparation Competency Examination, Florida Teacher Certification Examinations, Counseling Competencies Scale, internship site supervisor evaluation forms, and alumni survey data). This institutional effectiveness plan supports the UCF Strategic Goals: 2, 4, and 5. Assessment Process: The assessment strategy of the School Counseling for Certification Track in the Counselor Education Program 11:14:02 AM]

45 involves systematic assessment of three key counseling areas: (a) knowledge; (b) skills, dispositions and professional behaviors; and (c) ability to ethically and effectively implement their training. The School Counseling for Certification Track in the Counselor Education Program assesses areas of student performance with both formative and summative measurements. Knowledge comprises information deemed as fundamental for student effectiveness in the counseling field and includes such content areas as (a) Human Growth and Development, (b) Social and Cultural Foundations, (c) Helping Relationships, (d) Group Work, (e) Career and Lifestyle Development, (f) Appraisal, (g) Research and Program Evaluation, and (h) Professional Orientation and Ethics. Student knowledge will be appraised using scores on a nationally normed test called the Counselor Preparation Competency Exam (CPCE, 2011), which is administered towards the conclusion of students' program of study. To measure students own perceptions about their knowledge and skills level, we have also developed program specific questions for the students in the School Counseling for Certification Track in the Counselor Education Program to complete as part of the "Graduating Graduate Student Survey" (an indirect measure). The Graduating Graduate Student Survey analyzes students' opinions about the degree to which they feel adequately prepared to become a highly qualified school counseling professional. Survey questions were designed based on the required core curricular areas outlined by the Council of Accreditation for Counseling and Related Educational Programs (CACREP, 2009). Students complete the Graduating Graduate Student Survey when they file their intent to graduate form. The skills, dispositions and professional behaviors refer to those demonstrable abilities that students need to master to become proficient as a professional school counselor. Skills, dispositions and professional behaviors will be evaluated using the Counseling Competencies Scale (CCS; Swank, Lambie, & Witta, 2011) at three points during their academic program: (a) MHS 5005: Introduction to the Counseling Profession (formative assessment), (b) MHS 6401: Techniques in Counseling (formative assessment), and (b) MHS 6803: Practicum in Counselor Education (summative assessment). In addition, during students internship experience (SDS 6947: Internship in Professional School Counseling) their school-based site supervisor completes a comprehensive assessment of their performance based on CACREP (2009) Standards, the12 Florida Educator Accomplished Practices, and the Florida Department of Education Competencies and Skills: Guidance and Counseling Pk-12 (13th ed., 2010; summative assessment). Students ability to ethically and effectively implement their training refers to students ability to find employment in the school counseling field after graduation, which is evaluated through the use of alumni survey data. Furthermore, employers of our graduates will be surveyed and asked to rate their satisfaction level with the knowledge, skills, dispositions, and professional behaviors of our graduates (UCF College of Education Employment and Retention data). The School Counseling for Certification Track in the Counselor Education Program assessment strategy is managed by the school counseling program director, as well as the research associate in charge of all program evaluation within the counselor education program. Top Outcome: 1 The School Counseling for Certification Track in the Counselor Education Program students will demonstrate knowledge required for ethical and effective professional school counselors. Measure: 1.1 Ninety percent (90%) of the School Counseling for Certification Track in the Counselor Education Program students who sit for the Counselor Preparation Competency Examination (CPCE) will earn a passing score per the established programmatic benchmark on their first attempt. Result: 1.1 Students are required to take the Counselor Preparation Comprehensive Examination (CPCE) to assess their content knowledge of the field of counseling. The CPCE is giving three times a year and is taken the last semester of a student s internship experience. Results for , indicated that 100.0% of the students successfully passed the programs set benchmark score on their first attempt (N = 15). In addition, the measures of central tendency indicated that our School Counseling for Certification Track in the Counselor Education Program students scored higher than the national average on the CPCE (N = 15; M = 11:14:02 AM]

46 92.33; Mdn = 91.00; SD = 7.14; Range = ( ). Specifically, our students scored higher than the national average on the CPCE for the academic year (N = 1,382; M = 91.75; SD = 14.95; Range = 84 [ ]). More specifically the examination is broken down into eight domains ([C1] Human Growth and Development, [C2] Social and Cultural Foundations, [C3] Helping Relationships, [C4] Group Work, [C5] Career and Lifestyle Development, [C6] Appraisal, [C7] Research and Program Evaluation, and [C8] Professional Orientation and Ethics. The maximum score per each domain is Our School Counseling for Certification Track in the Counselor Education Program students descriptive statistics per domain was: C1 - Human Growth and Development UCF (N = 15, M = 11.20, SD = 2.48, Range 8 [ ]) National Sample (N = 1,382, M = 12.51, SD = 2.55, Range 14 [ ]) C2 - Social and Cultural Foundations UCF (N = 15, M = 11.13, SD = 1.96, Range 7 ( ) National Sample (N = 1,382, M = 10.58, SD = 2.32, Range 2.64 [ ]) C3 - Helping Relationships UCF (N = 15, M = 12.07, SD = 1.66, Range 6 [ ]) National Sample (N = 1,382, M = 11.54, SD = 2.30, Range 14 [ ]) C4 - Group Work UCF (N = 15, M = 11.43, SD = 1.81, Range 7 [ ]) National Sample (N = 1,382, M = 12.64, SD = 2.44, Range 13 [ ]) C5 - Career & Lifestyle Development UCF (N = 15, M = 12.60, SD = 1.92, Range 7 [ ]) National Sample (N = 1,382, M = 11.05, SD = 2.32, Range 15 [ ]) C6 Appraisal UCF (N = 15, M = 10.20, SD = 1.90, Range 6 [ ]) National Sample (N = 1,382, M = 9.70, SD = 2.51, Range 15 [ ]) C7 - Research & Program Evaluation UCF (N = 15, M = 10.73, SD = 2.02, Range 6 [ ]) National Sample (N = 1,382, M = 11.13, SD = 2.85, Range 15 [ ]) C8 - Professional Orientation & Ethics UCF (N = 15, M = 12.87, SD = 2.07, Range 8 [ ]) National Sample (N = 1,382, M = 12.61, SD = 2.47, Range 15 [ ]) Review: Measure: 1.2 Ninety percent (90%) of the School Counseling for Certification Track in the Counselor Education Program students who sit for the Florida Teacher Certification Examination / Subject Area Examination in Guidance and Counseling will receive a passing score per the Florida Department of Education benchmarks on their first attempt. 11:14:02 AM]

47 Result: 1.2 The Florida Teacher Certification Examination (FTCE) is required by the Florida State Department of Education for certification in school counseling. All students are required to successfully complete the three FTCE Examinations ([a] General Knowledge, [b] Professional Educator, and [c] Subject: Guidance and Counseling). In , 96.0% (N = 25) of the students who completed these three examinations received passing scores on all three assessments on their first attempt. Regarding the FTCE Subject: Guidance and Counseling examination (minimum score of 70.0% needed to pass examination per Florida State Department of Education), the measures of central tendency were: N = 18, M = 83.06, Mdn = 82.87, SD = 3.75, Range = ( ). The FTCE Subject: Guidance and Counseling examination has 10 subscale scores that include: 1. Knowledge of counseling (85.00% correct responses) 2. Knowledge of activities and programs for addressing current concerns (88.56% correct responses) 3. Knowledge of student assessment (87.78% correct responses) 4. Knowledge of career development and postsecondary opportunities (79.29% correct responses) 5. Knowledge of consultation, collaboration, and coordination (83.33% correct responses) 6. Knowledge of professional, ethical, and legal considerations (81.25% correct responses) 7. Knowledge of academic advisement (76.19% correct responses) 8. Knowledge of research, program evaluation, and follow-up (84.92% correct responses) 9. Knowledge of appropriate technology (82.41% correct responses) 10. Knowledge of social and cultural diversity (81.94% correct responses) Therefore, the UCF School Counseling for Certification Track in the Counselor Education Program students scored above the minimum competency score (70.0%) in all 10 domain of the FTCE Subject: Guidance and Counseling examination. Review: Measure: 1.3 Ninety percent (90%) of the School Counseling for Certification Track in the Counselor Education Program students will receive a score of exceptional or satisfactory their programmatic professional portfolio as judged by a panel of faculty. Result: 1.3 Students are required to demonstrate that they have met specific competencies ([a] Professional Identity, [b] Social and Cultural Diversity of Human Development across the Lifespan, [c] the Use of Technology and Assessment Skills, [d] Clinical Consultation and Communication Skills, [e] Specialty, [f] Personal Growth and Wellness, and [g] Research and Program Evaluation) during their time in the School Counseling for Certification Track in the Counselor Education Program. To evaluate the development of the students counseling competencies, students are required to construct a comprehensive portfolio and present it during their final semester of internship (SDS 6947: Internship in Professional School Counseling). In , 100% (N = 13) of these students successfully demonstrated mastery of these competencies per their portfolio review and presentation, scoring at either the exceptional or satisfactory level per the programmatic evaluation rubric. Review: 11:14:02 AM]

48 Measure: 1.4 Ninety percent (90%) of the School Counseling for Certification Track in the Counselor Education Program students completing the Graduating Counselor Education Student Survey will agree or strongly agree that the UCF Counselor Education Program supported their development of their knowledge and skills to be ethical and effective school counselors as identified with the Council for Accreditation of Counseling and Related Educational Programs (CACREP, 2009) Standards. Result: 1.4 Students in the School Counseling for Certification Track in the Counselor Education Program are asked to complete the Graduating Counselor Education Student Survey at the conclusion of their program of study(sds 6947: Internship in Professional School Counseling). Of the 15 students graduating in the School Counseling for Certification Track in the Counselor Education Program in , 9 students (60%) completed the Graduating Counselor Education Student Survey. Within the survey, the students were asked to rate their graduate preparation is 16 Academic Preparation and Competencies domains and 17 University and Program Service domains on a five point Likert scale (5 = Excellent, 4 = Strong, 3 = Average, 2 = Weak, and 1 = Poor). The measures of central tendency per the 33 domains in the Graduating Counselor Education Student Survey are presented below: 16 Academic Preparation and Competencies 1. Developmental counseling (M = 4.0, SD =.71, range = 2 [3 to 5]) 2. Personality/Counseling theories (M = 4.1, SD =.78, range = 2 [3 to 5]) 3. Individual counseling (M = 4.2, SD =.83, range = 2 [3 to 5]) 4. Group counseling (M = 4.0, SD = 1.00, range = 3 [2 to 5]) 5. Diversity issues and multicultural counseling (M = 3.4, SD =.73, range = 2 [2 to 4]) 6. Assessment/measurement and evaluation (M = 3.78, SD =.83, range = 2 [3 to 5]) 7. Career/vocational assessment and counseling (M = 4.2, SD =.83, range = 2 [3 to 5]) 8. Consultation (M = 3.33, SD = 1.12, range = 3 [2 to 5]) 9. Research and statistics (M = 3.56, SD =.73, range = 2 [3 to 5]) 10. Professional, ethical, and legal issues (M = 4.22, SD =.67, range = 2 [3 to 5]) 11. Crisis Intervention (M = 3.44, SD =.88, range = 2 [2 to 4]) 12. Case management and service provision (M = 3.33, SD =.87, range = 2 [2 to 4]) 13. Mental health strategies and interventions (M = 3.56, SD =.73, range = 2 [2 to 4]) 14. Identification and needs of exceptional education students (M = 3.22, SD =.83, range = 2 [2 to 4]) 15. Identification and needs of individuals with addictions (M = 3.56, SD =.88, range = 3 [2 to 5]) 16. Use of technology (M = 3.67, SD =.87, range = 3 [2 to 5]) 17 University and Program Services 1. Academic program advising by faculty (M = 3.89, SD = 1.17, range = 3 [2 to 5]) 2. Practicum/internship placement and supervision (M = 4.56, SD =.73, range = 2 [3 to 5]) 3. Facilitation of collegiality among students & faculty (M = 4.11, SD =.78, range = 2 [3 to 5]) 4. Facilitation of personal growth and self-evaluation (M = 4.44, SD =.73, range = 2 [3 to 5]) 5. Preparation for subject area/comprehensive exam (M = 3.89, SD =.78, range = 2 [3 to 5]) 6. Accessibility of information via website, handbooks, etc. (M = 4.33 SD =.87, range = 2 [3 to 5]) 7. Feedback on academic progress and skill development (M = 4.56, SD =.73, range = 2 [3 to 5]) 8. Physical facility/classrooms (M = 4.56, SD =.73, range = 2 [3 to 5]) 9. Library and media resources (M = 4.56, SD =.73, range = 2 [3 to 5]) 10. Personal adjustment counseling (M = 4.33, SD =.71, range = 2 [3 to 5]) 11. Career assistance (M = 3.88, SD =.64, range = 2 [3 to 5]) 11:14:02 AM]

49 12. Extracurricular activities (M = 3.88, SD =.99, range = 2 [3 to 5]) 13. Opportunities to socialize with other students (M = 4.63, SD =.74, range = 2 [3 to 5]) 14. Dissemination of information via the student listserv/ (M = 4.44, SD =.73, range = 2 [3 to 5]) 15. Job placement/letters of recommendation (M = 4.33, SD =.71, range = 2 [3 to 5]) 16. Chi Sigma Iota International Honor Society in Counseling (M = 4.00, SD =.76, range = 2 [3 to 5]) 17. Orientation sessions (Interested Student/New Student) (M = 4.22, SD =.67, range = 2 [3 to 5]) Review: Reflective Statement: All four summative measures (CPCE, FTCE [General Knowledge, Professional Educator, and Subject: Guidance and Counseling], Comprehensive Portfolio, and the Graduating Counselor Education Student Survey) support that the School Counseling for Certification Track in the Counselor Education Program students scored at levels demonstrating mastery of knowledge required for professional school counselors both on the state (FTCE) and national (CPCE and Professional Portfolio) levels. However, per the CPCE domain specific results, a need for improvement in the areas of: (C1) Human Growth and Development: UCF (N = 15, M = 11.20, SD = 2.48, Range 8 [ ]), National Sample (N = 1,382, M = 12.51, SD = 2.55, Range 14 [ ]); (C4) Group Work: UCF (N = 15, M = 11.43, SD = 1.81, Range 7 [ ]), National Sample (N = 1,382, M = 12.64, SD = 2.44, Range 13 [ ]), and (C7) Research & Program Evaluation: UCF (N = 15, M = 10.73, SD = 2.02, Range 6 [ ]), National Sample (N = 1,382, M = 11.13, SD = 2.85, Range 15 [ ]) may be necessary. In addition, the CPCE domains of (C4) Group Work and (C7) Research & Program Evaluation were identified as curricular areas of concerns per the IE data report. Therefore, the Counselor Education Program faculty will work to revise the MHS 6500: Group Procedures & Theories in Counseling and the EDF 6481: Fundamentals of Graduate Research in Education curriculum. Nevertheless, the School Counseling for Certification Track in the Counselor Education Program students demonstrated competency in the knowledge required for ethical and effective professional school counselors per the four summative evaluation measures. Furthermore, the Counselor Education Program faculty members will continue to work to strengthen the data collection processes for the Graduating Counselor Education Student Survey data. Reflective Statement Review: Overall Outcome Results Review Comment: Attachments: Top Outcome: 2 The School Counseling for Certification Track in the Counselor Education Program students will demonstrate the skills, dispositions, and professional behaviors required for ethical and effective professional school counselors. 11:14:02 AM]

50 Measure: 2.1 Seventy five percent (75%) of the School Counseling for Certification Track in the Counselor Education Program students completing MHS 5005: Introduction to the Counseling Profession during this academic year will score at the "Exceeds Expectations / Demonstrates Competencies (8)" or Meets Expectations / Demonstrates Competencies (6) or "Near Expectations / Developing Toward Competencies (4)" on the Counseling Competencies Scale (CCS; Swank, Lambie, & Witta, 2011), which aligns with the Council for Accreditation of Counseling and Related Educational Programs (CACREP, 2009) Standards (formative assessment) Result: 2.1 The School Counseling for Certification Track in the Counselor Education Program students enrolled in MHS 5005: Introduction to the Counseling Profession are evaluated by their course instructor with the Counseling Competencies Scale (CCS, Swank, Lambie, & Witta, 2012), which aligns with the Council for Accreditation of Counseling and Related Educational Programs (CACREP, 2009) Standards. The CCS is used to evaluated counseling students in 32 domains: (1) Nonverbal Skills, (2) Encouragers, (3) Questions, (4) Reflection(a), (5) Reflection(b), (6) Advanced Reflection (meaning), (7) Advanced Reflection (summarizing), (8) Confrontation, (9) Goal Setting, (10) Focus of Counseling, (11) Facilitating Therapeutic Environment(a), (12) Facilitating Therapeutic Environment(b), (13) Professional Ethics, (14) Professionalism, (15) Selfawareness & Self-understanding, (16) Emotional Stability & Self-control, (17) Motivation to Learning & Grow / Initiative, (18) Multicultural Competencies, (19) Openness to Feedback, (20) Professional & Personal Boundaries, (21) Flexibility & Adaptability, (22) Congruence & Genuineness, (23) Attendance & Participation, (24) Knowledge & Adherence to University & Counseling Site Policies, (25) Record Keeping and Task Completion, (26) Knowledge of Professional Literature, (27) Application of Theory to Practice, (28) Case Conceptualization, (29) Seek Consultation, (30) Psychosocial & Treatment Planning, (31) Appraisal, and (32) Referral. In , 87.0% (N = 23) of the School Counseling for Certification Track in the Counselor Education Program students enrolled in MHS 5005: Introduction to the Counseling Profession successfully demonstrated the required counseling competencies (counseling skills, professional dispositions, and professional behaviors) as measured by the CCS (e.g., scored at Exceeds Expectations / Demonstrates Competencies or Meets Expectations / Demonstrates Competencies or Near Expectations / Developing towards Competencies ) per the course instructor s CCS. Review: Measure: 2.2 One hundred percent (100%) of the School Counseling for Certification Track in the Counselor Education Program students completing MHS 6401: Techniques of Counseling during this academic year will score at the "Exceeds Expectations / Demonstrates Competencies (8)" or Meets Expectations / Demonstrates Competencies (6) or "Near Expectations / Developing Toward Competencies (4)" on the Counseling Competencies Scale (CCS; Swank, Lambie, & Witta, 2011), which aligns with the Council for Accreditation of Counseling and Related Educational Programs (CACREP, 2009) Standards (formative assessment). For students to progress to MHS 6803: Practicum in Counselor Education, they must score at the "Exceeds Expectations / Demonstrates Competencies (8)" or Meets Expectations / Demonstrates Competencies (6) or "Near Expectations / Developing Toward Competencies (4)" levels per their instructor's final CCS evaluation. Result: 11:14:02 AM]

51 2.2 The School Counseling for Certification Track in the Counselor Education Program students enrolled in MHS 6401: Techniques of Counseling are evaluated by their course instructor with the Counseling Competencies Scale (CCS, Swank, Lambie, & Witta, 2012), which aligns with the Council for Accreditation of Counseling and Related Educational Programs (CACREP, 2009) Standards. For students to progress to MHS 6803: Practicum in Counselor Education, they must score at the Exceeds Expectations / Demonstrates Competencies or Meets Expectations / Demonstrates Competencies or Near Expectations / Developing towards Competencies levels per their instructor s final CCS evaluation. Specifically, the CCS is used to evaluated counseling students in 32 domains: (1) Nonverbal Skills, (2) Encouragers, (3) Questions, (4) Reflection(a), (5) Reflection(b), (6) Advanced Reflection (meaning), (7) Advanced Reflection (summarizing), (8) Confrontation, (9) Goal Setting, (10) Focus of Counseling, (11) Facilitating Therapeutic Environment(a), (12) Facilitating Therapeutic Environment(b), (13) Professional Ethics, (14) Professionalism, (15) Selfawareness & Self-understanding, (16) Emotional Stability & Self-control, (17) Motivation to Learning & Grow / Initiative, (18) Multicultural Competencies, (19) Openness to Feedback, (20) Professional & Personal Boundaries, (21) Flexibility & Adaptability, (22) Congruence & Genuineness, (23) Attendance & Participation, (24) Knowledge & Adherence to University & Counseling Site Policies, (25) Record Keeping and Task Completion, (26) Knowledge of Professional Literature, (27) Application of Theory to Practice, (28) Case Conceptualization, (29) Seek Consultation, (30) Psychosocial & Treatment Planning, (31) Appraisal, and (32) Referral. In , 96.2% (N = 26) of the School Counseling for Certification Track in the Counselor Education Program students enrolled in MHS 6401: Techniques of Counseling successfully demonstrated the required counseling competencies (counseling skills, professional dispositions, and professional behaviors) as measured by the CCS (e.g., scored at Exceeds Expectations / Demonstrates Competencies or Meets Expectations / Demonstrates Competencies or Near Expectations / Developing towards Competencies ) per the course instructor s CCS. Review: Measure: 2.3 One hundred percent (100%) of the School Counseling for Certification Track in the Counselor Education Program students completing MHS 6803: Practicum in Counselor Education during this academic year will score at the "Exceeds Expectations / Demonstrates Competencies (8)" or Meets Expectations / Demonstrates Competencies (6) on the Counseling Competencies Scale (CCS; Swank, Lambie, & Witta, 2011), which aligns with the Council for Accreditation of Counseling and Related Educational Programs (CACREP, 2009) Standards (summative assessment). For students to progress to SDS 6947: Internship in Profesional School Counseling, they must score at the "Exceeds Expectations / Demonstrates Competencies (8)" or Meets Expectations / Demonstrates Competencies (6) levels per their instructor's final CCS evaluation. Result: 2.3 The School Counseling for Certification Track in the Counselor Education Program students enrolled in MHS 6803: Practicum in Counselor Education are evaluated by their course instructor with the Counseling Competencies Scale (CCS, Swank, Lambie, & Witta, 2012), which aligns with the Council for Accreditation of Counseling and Related Educational Programs (CACREP, 2009) Standards. For students to progress to SDS 6947: Internship in Professional School Counseling, they must score at the Exceeds Expectations / Demonstrates Competencies or Meets Expectations / Demonstrates Competencies levels per their instructor s final CCS evaluation. Specifically, the CCS is used to evaluated counseling students in 32 domains: (1) Nonverbal Skills, (2) Encouragers, (3) Questions, (4) Reflection(a), (5) Reflection(b), (6) Advanced Reflection (meaning), (7) Advanced Reflection (summarizing), (8) Confrontation, (9) Goal Setting, 11:14:02 AM]

52 (10) Focus of Counseling, (11) Facilitating Therapeutic Environment(a), (12) Facilitating Therapeutic Environment(b), (13) Professional Ethics, (14) Professionalism, (15) Self-awareness & Self-understanding, (16) Emotional Stability & Self-control, (17) Motivation to Learning & Grow / Initiative, (18) Multicultural Competencies, (19) Openness to Feedback, (20) Professional & Personal Boundaries, (21) Flexibility & Adaptability, (22) Congruence & Genuineness, (23) Attendance & Participation, (24) Knowledge & Adherence to University & Counseling Site Policies, (25) Record Keeping and Task Completion, (26) Knowledge of Professional Literature, (27) Application of Theory to Practice, (28) Case Conceptualization, (29) Seek Consultation, (30) Psychosocial & Treatment Planning, (31) Appraisal, and (32) Referral. In , 90.9% (N = 22) of the School Counseling for Certification Track in the Counselor Education Program students enrolled in MHS 6803: Practicum in Counselor Education successfully demonstrated the required counseling competencies (counseling skills, professional dispositions, and professional behaviors) as measured by the CCS (e.g., scored at Exceeds Expectations / Demonstrates Competencies or Meets Expectations / Demonstrates Competencies ) per the course instructor s CCS. Review: Measure: 2.4 One hundred percent (100%) of the School Counseling for Certification Track in the Counselor Education Program students completing SDS 6947: Internship in Professional School Counseling during this academic year will be rated by their School-based Site Supervising School Counselor as "proficient (demonstrating professional competency at expectation of a beginning professional school counselor) or "advanced proficient" (demonstrating professional competency beyond expectation of a beginning professional school counselor) each benchmark aligning with Council for Accreditation of Counseling and Related Educational Programs (CACREP, 2009) Standards, the12 Florida Educator Accomplished Practices, and the Florida Department of Education Competencies and Skills: Guidance and Counseling Pk-12 (13th ed., 2010). Result: 2.4 During the the School Counseling for Certification Track in the Counselor Education Program students internship experience (SDS 6947: Internship in Professional School Counseling), school-based site supervisors evaluate the student-interns based upon the 12 Florida Educator Accomplished Practices Benchmarks (FEAPs, [a] Assessment/ Research, [b] Communication, [c] Continuous Improvement, [d] Critical and Creative Thinking, [e] Diversity, [f] Ethics and Professionalism, [g] Human Development and Learning, [h] Knowledge and Presentation of Subject Matter, [i] Learning Environment, [j] Planning, [k] Role of the Professional School Counselor, and [l] Technology), and the FL-DOE Competencies and Skills: Guidance and Counseling PK-12 (13th ed.; 2010), and the Council for Accreditation of Counseling and Related Educational Programs (CACREP, 2009) Standards. In , 100% (N = 15) of the students received a score of Advanced Proficient (Student-Intern demonstrates this professional competency beyond expectation of a beginning professional school counselor) or Proficient (Student-Intern demonstrates this professional competency at expectation of a beginning professional school counselor) on the benchmark requirements per each of the 12 FEAPs, the 10 FL-DOE Guidance and Counseling Competencies, and CACREP Standards from their school-based site supervisors. Review: 11:14:02 AM]

53 Reflective Statement: The two formative assessment evaluations (CCS during MHS 5005: Introduction to the Counseling Profession and MHS 6401: Techniques of Counseling) and two summative assessment evaluations (CCS during MHS 6803: Practicum in Counselor Education and SDS 6947: Internship in Professional School Counseling school-based site supervisors evaluate of the student-interns) supported that the students in the School Counseling for Certification Track in the Counselor Education Program demonstrated the skills, dispositions, and professional behaviors required for ethical and effective professional school counselors both on the state (FEAPs, FL-DOE Guidance and Counseling Competencies) and national (CACREP, 2009) levels. However, per these evaluations and previous data, the Counselor Education Program faculty continue to make curricular modification and enhancements (e.g., application and modification of the Counselor Competencies Scale and revised SDS 6947: Internship in Professional School Counseling site supervisor evaluation of student-intern form per new accreditation standards) over the past three-years. Reflective Statement Review: Overall Outcome Results Review Comment: Attachments: CCS Introduction Course Form PDF.pdf CCS Techniques Course Form PDF.pdf CCS Practicum - Complete Form pdf CES 2012 CCS EFA Article.pdf CCS CORE Article for IE Report.pdf UCF PSC INTERNSHIP HANDBOOK Final PDF.pdf Top Outcome: 3 The School Counseling for Certification Track in the Counselor Education Program students will ethically and effectively implement the training received in their graduate degree program. Measure: 3.1 During the School Counseling for Certification Track in the Counselor Education Program students internship experience (SDS 6947: Internship in Professional School Counseling), 100% of school-based internship site supervisors will evaluate that they are very confident or confident in their student-interns employability and ability to delivery effective service to all stakeholders. Result: 3.1 The School Counseling for Certification Track in the Counselor Education Program students during their internship experience (SDS 6947: Internship in Professional School Counseling) are evaluated by their school-based internship site supervisor relating to their employability and effectiveness of service delivery. In , 100% (N = 15) of the school-based internship site supervisors indicated that they were very confident or confident in the student-interns employability and effectiveness of service delivery. Review: 11:14:02 AM]

54 Measure: 3.2 Seventy percent (70%) of the School Counseling for Certification Track in the Counselor Education Program graduates will be hired as school counselors in the state of Florida per the UCF College of Education Employment and Retention data. Result: 3.2 Fifty-five percent of the School Counseling for Certification Track in the Counselor Education Program graduates (N = 11) were hired as school counselors in the state of Florida per the UCF College of Education Employment and Retention data. In addition, 67% of the School Counseling for Certification Track in the Counselor Education Program graduates (N = 8) who were employed in Year 1, are still employed as school counselors in the state of Florida per the UCF College of Education Employment and Retention data. Review: Measure: 3.3 Sixty percent (60%) of the School Counseling for Certification Track in the Counselor Education Program alumni will report being "postive" regarding their overall experience in the Counselor Education Program per the UCF Graduate Student Survey Results (General Report) and the UCF Graduate Student Survey Results (Program Specific Survey for Counselor Education) data. Result: The UCF Graduating Graduate Student Survey Results for the Counselor Education Program ( ) supported that alumni were positive (100.0%, N = 43) regarding their overall experience in the Counselor Education Programs ( In general, how would you rate your overall graduate experience at UCF ). In addition, the UCF Graduating Graduate Student Survey Results Program Specific Survey for Counselor Education ( ; N = 44) supported that alumni strongly agree or agree (97.7%; n = 43) that the Counselor Education Program supported their development to become highly qualified counselors ( Question 2: To what extent do you agree or disagree that the following elements of your UCF Counselor Education program adequately prepared you to become a highly qualified counselor Coursework in the counselor education program? ). In should be noted the both UCF Graduating Graduate Student Survey Results reports present aggregate data for all three Counselor Education Program tracks (Marriage & Family Counseling, Mental Health Counseling, and School Counseling); however, the results supported the alumni were positive regarding their learning experiences in the Counselor Education Program. Review: Reflective Statement: Goals related to the domain of the ethical and effective implement of training received in the School Counseling for Certification Track in the Counselor Education Program appeared to have been met per the UCF Graduating Graduate Student Survey Results for the Counselor Education Program ( ), the UCF 11:14:02 AM]

55 Graduating Graduate Student Survey Results Program Specific Survey for Counselor Education ( ), and the school-based internship site supervisor evaluations (SDS 6947: Internship in Professional School Counseling) data. In addition, the Counselor Education Program faculty members plan to continue strengthening the alumni survey data collection process to increase the potential responses rates and clarity in the survey items. Reflective Statement Review: Overall Outcome Results Review Comment: Attachments: Changes to Academic Process: Modify Frequency or Schedule of Course Offerings Make Technology Related Improvements Make Personnel Related Changes Implement Additional Training Revise Advising Standards or Process Revise Admission Criteria Other implemented or planned change No Changes to Academic Process Explain EACH item checked above: Academic processes appear to be functioning effectively Criteria: Please comment on implemented and planned changes Clear statement of change(s) Description of how changes created improvements; make suggestions for future cycles Review: Review Comments: Overall, very well done, but a couple of editing observations: "SDS 694" is indicated as the internship course in Changes to Assessment plan--an obvious typo. This page should also serve as a checklist for instruments that were used in the plan and results. You also failed to check some items that are used in support of your plan (e.g., state-normed exam, portfolio). Given the overall quality of this report, these are minor editing issues. Changes to Curriculum: Revise and/or Enforce Prerequisites Revise Course Sequence Revise Course Content Add Course Delete Course Other implemented or planned change No Changes to Curriculum Explain EACH item checked above: Will examine the curriculum for MHS 11:14:02 AM]

56 6500: Group Procedures & Theories in Counseling and the EDF 6481: Fundamentals of Graduate Research in Education to make enhancements per the CPCE data for both & Changes to Assessment Plan: Revise Student Outcome Statement Revise Measurement Approach Collect and Analyze Additional Data and Information Change Method of Data Collection Other Planned Changes Plan has been reviewed and no change made No Changes to Assessment Plan Explain EACH item checked above: The Counselor Education Program faculty members plan to strengthen the administration and scoring processes related to the Counseling Competencies Scale (Swank, Lambie, & Witta, 2012); supporting the psychometric soundness of the instrument and the validity of our data. The Counselor Education Program faculty members plan to update & revision the SDS 694: Internship in Professional School Counseling site supervisor evaluation of student-interns (summative assessment) per new accreditation standards and rubric enhancement Curriculum/Course-related Assessment Methods: Capstone Course Capstone Project or Performance Evaluation Case study / Simulation Course-embedded Questions Portfolio Rating Scale / Scoring Rubric (yields a grade) 11:14:02 AM]

57 Assessment Rubrics (student demonstrates proficiency) Lab Journals / Reports Observation (focused on specific program outcomes) Other method Explain EACH item checked above: The Counselor Education Program faculty members plan to update and enhance their formative assessment rubrics that are integrated throughout the curriculum to support programmatic accountability and effectiveness Examinations/Tests: Standardized: Nationally-normed Exam State-normed Exam Other Explain EACH item checked above: Local: Post-test Only Pre-post Test Other exam or test Explain EACH item checked above: Surveys: Institution (UCF): UCF Graduating Student Survey (Seniors or Graduate student) Alumni Survey Student Satisfaction Survey First Destination Survey 11:14:02 AM]

58 Employee Survey Explain EACH item checked above: The Counselor Education Program faculty members plan to strengthen both their Graduating Counselor Education Student Survey and their Counselor Education -- School Counseling Alumni Survey, increasing assess to meaningful data to support continous programmatic improvement Local: Alumni Survey (Department or Program; not UCF) Customer Satisfaction Survey Exit and Other Interviews Explain EACH item checked above: Other Survey(s): National Survey State Survey Other Survey Explain EACH item checked above: Miscellaneous Assessment Methods: Advisory Board Focus Group Institutional Data Student Records Accreditation Reviews (e.e. SACS, NCATE, ABET) Other Explain EACH item checked above: NCATE & CACREP reviews 11:14:02 AM]

59 Institutional Effectiveness Assessment Results Rubric *If programs or units fail to provide any input, their results will be evaluated with "No effort (0)." Beginning (1) Emerging (2) Meets Expectations (3) Accomplished (4) Exemplary (5) Indicators: 1. Complete and relevant data are provided for all measures OR if data are incomplete or missing, an explanation is provided 2. Data reporting is thorough (see below ) i.e.,populations are defined;sampling methods and response rates are provided with survey data, etc. 3. Results for each measure indicate whether the target for that measure has been met 4. Reflective statements are provided either for each outcome or aggregated for multiple outcomes 5. Implemented and planned changes are included and are linked to assessment data, or if no changes are reported, an explanation is provided 6. Assessment instruments are attached or linked to if not proprietary Additional Indicators: 7. Includes description of how the assessment process has been useful to your program or unit 8. Includes description of how IE Assessment has resulted in quality improvement initiatives 9. Data collection and analysis are used to assess the impact of implemented changes, demonstrating a fully closed loop process Summary of Assessment Process: Think about the last few years and describe evidence-based changes that have taken place because of assessment. Also address other factors that have caused changes to be made (e.g., state mandate, accreditation review recommendations). The Counselor Education-School Counseling Program faculty members employ multiple data collection sources to support their programmatic effectiveness, accountability, and ongoing improvement. Specifically, the Counselor Education-School Counseling Program faculy members: 1. Use assessment data per UCF institutional effectiveness report to strengthen programmatic effectiveness 2. Collect data at programmatic transition points to assess students' development throughout their program of study 3. Gather data on faculty members' "recency of experience" (for state approved programs), support faculty members' experience in schools 4. Collect data from multiple survey to gain insight about programmatic effectiveness and student satisfaction 5. Review student remediation data for all counselor educationschool counseling students 6. Gather retention data for counselor education-school counseling graduates 7. Update & modify the counselor education-school counseling curriculum per new CACREP (2009) and NCATE (2008) Standards 8. Update & modify the counselor education-school counseling curriculum per new state accreditation standards (FEAPs) 9. Review counselor education-school counseling program Review Criteria: (Examples: Could you be more specific? Has your benchmark remained at this level too long?) Review: An exemplary report that well demonstrates how your program uses assessment data to monitor and enhance program quality and student learning outcomes. Well done! 11:14:02 AM]

60 enrollment data to inform programmatic decision-making 10. Collect data from the counselor education-school counseling program advisory group to inform programmatic decision-making The counselor education-school counseling program plans to continue to strengthen our programmatic assessment processes to enhance the effectiveness of our pedagogy and the development of ethical & effective school counselors. Top Site maintained by Operational Excellence and Assessment Support Terms of Use Privacy Policy Webmaster 11:14:02 AM]

61 Assessment Plan 62 Comprehensive Assessment Plan: Curricular and Program Modifications The following curricular and programmatic modifications were made as a result of our formal and informal comprehensive assessment plan. These occurred over time, so continued changes and modifications are grouped together in order to see the development of the ideas as they occurred. Curricular/Programmatic Changes: Clinical Experiences (Practicum) Students must submit their final CCS from previous courses Faculty o CCS from Intro, Techniques, Practicum and Internship must be submitted to the instructor on the first day of class for the next course in that line-up. Example: A student will submit their final CCS from Techniques to the instructor of their 1 st semester of Practicum. All Faculty agreed this will be a requirement. Dr. Jones suggested we add it to the syllabus, Dr. Young agreed. The participation grade in Practicum was replaced with points assigned to a passing score on the CCS (which was assigned 25% of the final grade in the class). o If students do not score at the appropriate level (6 or above), then they will not be awarded the 25 points, resulting in a maximum score of 75% in the course (or a C). Will be discussed further Practicum Ideas for Summer (due to less weeks to see clients) - Dr. Hagedorn o Start seeing clients the first week of the semester o Extend the hours for each session to 1.25 or 1.50 hours o Use Fridays as a flex day would need supervision CCS Scoring Dr. s Lambie and Hundley discussion on best policy to evaluate students through CCS scores. o For those students who are working with young child clients, it is challenging to evaluate their skills with a CCS. Therefore, the faculty agreed that every practicum student must have at least one adolescent/adult client. o Have maintained the two C policy and anyone receiving a non-passing CCS must have a remediation plan To expose students to the skills, professional dispositions, and professional behaviors necessary to be successful in the program, the faculty adapted the CCS to a version used in the MHS 5005 Intro to the Counseling Profession course in In order to successfully pass the Intro course, students must meet expectations in all domains on the CCS Intro Version. This has resulted in the Intro course becoming a B or Better course, meaning that students must earn a grade of B or better in order to move forward in the Program. As a means to help students prepare for practicum, the faculty adapted the CCS to a version used in the MHS 6401 Techniques of Counseling course (internally known as prepracticum ) in In order to successfully pass the Techniques course, students must meet expectations in all domains on the CCS Techniques Version. This has resulted in the Techniques course becoming another B or Better course.

62 Assessment Plan 63 Curricular/Programmatic Changes: Clinical Experiences (Internship) Internship hours change per the State of Florida Dr. Daire o MFT students are required to obtain 180 relational hours. Internship applications will note the new requirement. MFT students should choose a site that will support this path. Changes will be made to the handbook. Dr. Young will send these changes to Joyce. The faculty designed the Site Supervisor Evaluation Form in response to the lack of useful site supervisor feedback that resulted in the older form. This newer version, tied to CACREP Skills and Practices standards, has proven more useful in identifying interns competencies and areas for growth. Curricular/Programmatic Changes: Doctoral Program PhD. Clinical Experience needed Faculty Decision o The Faculty agreed to discuss options to require two years of clinical experience for admission to our Doctoral Program. o The exception to the policy If applicants do not have the required two years of experience and the Faculty agreed to advocate for admission; they will tell the applicant they will be required to get clinical experience during their program. o Form needed to verify 2-year pre-candidacy clinical experiences requirement. Dr. Hagedorn agreed to create the document. o Dr. Hagedorn presented each attendee a copy of the proposed PhD Student Clinical Experiences document. Final decisions to implement these changes by next fall needs clarification. Curricular/Programmatic Additions: New Programs The call for an Addictions Certificate in Counseling Currently this is only available as a undergraduate certificate in Social Work. Dr. Jones and Dr. Hagedorn will re-submit our proposal for Certificate in Addictions Counseling. Curricular/Programmatic Additions: New/Revised Documentation The new Portfolio Rubric was presented for discussion. Faculty discussed the new format and will submit their comments to Dr. Hagedorn within two weeks. This will be put forward for approval at our next meeting. Upon acceptance of the new revised Portfolio Rubric, Dr. Butler will present it to all new students in the Intro Class beginning this FALL The Program of Study needs to be updated to reflect the 27 credit hours required to take practicum and align with CACREP Standards. Review decision about Conference attendance (date was set at Fall 2013) and Portfolio Update Dr. Hagedorn o Dr. Hagedorn will distribute new Programs of Study (with the change to the Child/Adolescent class) and new Portfolio Rubrics showing the changes for conference attendance. These changes will start with spring graduates. All students must attend two conferences, with one conference attendance being at the UCF Counseling Conference. Implications of Comprehensive Portfolio Results o All students passed the Comprehensive Portfolio process. The Portfolio process was significantly revised as a result of feedback from students and alumni (as noted in the

63 Assessment Plan 64 Graduating Graduate Student Survey). Whereas the old Portfolio was designed to ascertain if students demonstrated competency in CACREPs 8 core curricular domains, the new Portfolio is a better measure of students student learning and performance on professional identity, professional practice, and program area standards. Curricular/Programmatic Additions: Assessment Results CPCE Test Score Updates - Dr. Hagedorn o Faculty decided to adjust the passing score to 73 due to the new CPCE test and the national test results recording much lower scores. No national averages would be available until later in November. The recommendation from CPCE was to allow the lower passing grades till further research can be completed. The vote was unanimous by the faculty. Further adjustments will be made. Implications of CPCE Results o UCF s CPCE mean scores in the domain of Career and Lifestyle Development dropped below the national mean scores in this domain last year (for the MHC students). The instructor of SDS 6347: Career Development (Dr. Stacy Van Horn) is using doctoral students from her MHS7340: Advanced Career Development to enhance the students learning environment. o UCF s CPCE mean scores in all domains but Professional Orientation and Ethics dropped below the national mean scores last year (for the MFT students). The creation and use of CORE Standards Knowledge Assessment will assist faculty in determining the reason for these lower scores. o UCF s CPCE mean scores in the domain of Group Work dropped below the national mean scores in this domain last year (for the SC students). The revision and standardization of the MHS 6511 Group Counseling syllabus was done to ensure that all students receive the same information in this course (given that multiple sections are offered each year). CPCE procedures for students not passing exam o Faculty agreed to require student to answer standardized essay questions to demonstrate the student knows and has mastered this skill. o Re-take the failed portion (s) The faculty agreed if student passes the test specific to their area of need, they will go forward with having met the requirements of the program for graduation. Curricular/Programmatic Additions: Admissions Discussion continued with changing admission to once a year Dr. Lambie o Would help with sequencing of courses o Moving to a cohort model o We were in agreement to move to the next step: get data related to how many students we would need to admit to maintain our FTE and current faculty o Undergraduate minor might help with the FTEs o A meeting will happen with Drs. Hagedorn, Robinson, and Young and Lambie Evaluate the other top programs in the US and determine criteria Benchmarking report Mark has this and will provide for the meeting How does our program impact the college (FTEs, grant)

64 Assessment Plan 65 Chart of FTE growth since 2000 (start of doc program) from Lance or Grant Review application deadline changes Dr. Hagedorn o The Faculty voted to change the deadlines for admission FALL September 15th SPRING February 15th. These changes will have to be posted on the Graduate and Counselor Education Websites.