ASSESSMENT OF STUDENT LEARNING COUNSELING MA PROGRAM DEPARTMENT OF COUNSELING PSYCHOLOGY AND COMMUNITY SERVICES

Transcription

1 2012 ASSESSMENT OF STUDENT LEARNING COUNSELING MA PROGRAM DEPARTMENT OF COUNSELING PSYCHOLOGY AND COMMUNITY SERVICES We are currently seeking accreditation from the Masters in Psychology and Counseling Accreditation Council (MCAC), and our intent is to have our assessment plan in-line with their standards. MCAC standards ask that training programs provide evidence of promoting competence in several domains (MPCAC Manual, pp ). In line with those domains, students who complete the on-campus master s program in counseling should demonstrate: 1) Knowledge and skills in professional (counselor) identity, ethical behavior, and social justice practices. 2) Knowledge and skills in human development and wellness; including physiological/biological aspects of human development and wellness. 3) Knowledge and skills in multicultural counseling theory and practice. 4) Knowledge and skills in counseling and consultation theory and practice 5) Knowledge and skills in group theory and practice. 6) Knowledge and skills in career counseling and life style development theory and practice. 7) Knowledge and skills in assessment of human behavior, and tests and measurement (appraisal). 8) Knowledge and skills in both traditional and social justice-oriented research and evaluations. 9) Clinical (applied) skills at the master s level. Three sources of data will be used to assess student learning in these nine areas: Source 1: The Counselor Preparation Comprehensive Examination 1. Utilize the Counselor Preparation Comprehensive Examination (CPCE), a standardized national exam required of all graduating students in our program, as a way of determining how our students compare to the national sample of students taking the exam. 2. Calculate the means for our students in each of eight areas: a. Human Growth and Development b. Social and Cultural Foundations c. Helping Relationships d. Group Work e. Career and Life Style Development f. Appraisal

2 2012 g. Research and Program Evaluation h. Professional Orientation and Ethics 3. CPCE scores will be calculated each year and compared to national scores gathered from the previous two years. 4. Based on how our students compare to the national sample, recommendations will be made regarding how to improve the curriculum. Source 2: Supervisor Evaluations of Student Clinical Performance 1. On-site and program clinical evaluations of students completing practicum or internship. 2. Have data regularly entered into spreadsheet by either GA or support staff. 3. Areas of review include: a. Counseling related Personality Characteristics, including Intellectual and Personal skills b. Counseling Related Knowledge (of interventions, ethics and individual/cultural differences [ICD]). c. Counseling Related Interpersonal Skills d. Psychological Diagnostics Skills e. Intervention Skills f. Consultation and Collaboration Skills g. Diversity and ICD Skills h. Ethics Application i. Professionalism j. Metacognitive Skills k. Basic Counseling Skills 4. Evaluations will be reviewed yearly, though rotating through practicum and internship classes. 5. Based on how our students in general achieve (or don t achieve) department-set standards, recommendations will be made regarding how to improve the curriculum.

3 2012 Source 3: Annual MA Student Program Evaluations 1. Data will be collected from the Annual MA Student Program Evaluations. This data includes satisfactory, unsatisfactory, or exceptional performance in the following areas: a. Academic Coursework b. Practitioner Skills c. Professional Activities d. Professionalism e. Writing Skills f. Independent Study/Thesis Progress (research) g. Overall Progress 2. Present all of the quantitative data from the annual reports on MA students. We will then analyze areas where it appears improvement is needed. 3. Based on this analysis, we will recommend areas for improvement and areas where we can maintain our current practice.

4 2012 ASSESSMENT OF STUDENT LEARNING SCHOOL COUNSELING MA DISTANCE PROGRAM DEPARTMENT OF COUNSELING PSYCHOLOGY AND COMMUNITY SERVICES The Department of Counseling Psychology and Community Services is committed to educating and training people who will work with students, parents, and other educators in K-12 public school settings. We developed our distance program to meet the needs of individuals who seek quality in professional school counseling education. Individuals in rural areas or those who are not able to relocate for graduate school are uniquely suited for our program. We strive to prepare knowledgeable and competent professional school counselors who will: (1) serve as providers and advocates for the academic, career, social and emotional needs of all students, (2) engage in collaborative teamwork and leadership for the purpose of ensuring opportunities for meaningful educational participation for all students, (3) commit to lifelong growth and development as professional and 4) actively engage as self-reflexive practitioners of professional knowledge and behaviors, particularly activities that promote effectiveness, community responsibility, and social justice We base our curriculum on the knowledge, skills and competencies that support the full implementation of programs developed according to the ASCA National Model and North Dakota Standards and Practices School Counselor (ESPB) Competencies. We are fully accredited by NCATE and ESPB. Our assessment of student learning plan reflects our NCATE and ESPB accreditation practices.

5 2012 SCHOOL COUNSELING SPECIALIZATION ASSESSMENT PLAN Point of (POA) Schedule for Departmental POA review Elements of Point of Point of Data Collection Procedure(s) Form of Data Timeline of Data Collection Data Review Areas impacted by data Availability of Results Comprehensive Exam 2012 we have 2 years of data to analyze Test of 8 core knowledge competencies for masters level counseling CPCE Counselor Preparation Comprehensive Exam After 2/2013 Praxis II Exam scores in each area Feb/March of 7 th semester Examination of scores in each core knowledge area Program coordinator Course content methods Students contacted individually about exam results Application to the School Counseling Program 2013 we will have 5 years of data to analyze UND Application form Statement of Purpose GPA Prerequisites required for licensure 3 letters of recommendation Phone interview Review of application materials by three faculty members Application Packet Nov 1 yearly Nov. Faculty meeting Whole faculty Recruitment Efforts Retention Procedures Procedures and resources for faculty participation in admissions process Students are notified of status by November 15 th Full Status Provisional Status Not accepted Semester Academic Program Progress 2014 We will have two years of Form use Review academic progress of student progress by semester Survey of student progress completed by faculty Form (see attachment) Rated in three areas Continuation Remediation Discontinuation Dec. yearly Electronic Consultation MA faculty Teaching assignments Course content methods Remediation or improvements plans Students notified of status upon completion of the review process Clinical Skills in 510 and Four years of data included in the First and second year student counseling skills Supervisor evaluations from Counseling 510 and Clinical Skills Evaluation Forms (see June Lab class July Lab Class Instructor of course Remediation or improvements plans Students participate in formative and

6 2012 Point of (POA) Clinical Skills Practicum and Internship Professional Skills Evaluation review competencies 527 attached) summative evaluations summer semester Schedule for Departmental POA review 2015 Four years of data included in the review 2014 We will have 2 ½ years of data collected Elements of Point of COUN 581 and 589 counseling skills competencies Ten Competency Areas Basic work requirements Knowledge and learning Professional identity Response to supervision Interactions with students being served Interactions with co-workers Counseling and group Intervention Skills Career and educational counseling skills Instructional skills Comprehensi ve counseling and guidance program Point of Data Collection Procedure(s) Supervisor observations and Evaluation External supervisor completes and reviews with the student Formative evaluation 8 weeks into the semester Summative and formal at end of semester submitted to UND Form of Data Clinical Skills Evaluation Forms (see attached) Practicum and Internship Evaluation Form (see attached) Timeline for Data Collection End of Spring and fall semesters Completed twice each semester that student is enrolled in COUN 581 or 589 Data Review End of each semester End of each semester Areas impacted by data Remediation or improvements plans Remediation or improvements plans Course content methods Availability of Results Annual Student Review December End of each Semester that the evaluation form is used

7 skills 2012

8 DEPARTMENTAL PLAN FOR ASSESSMENT OF STUDENT LEARNING ACADEMIC YEAR Department: Counseling Program: Master of Arts in Counseling Mission Statement The M.A. Program in Counseling reflects our commitment to provide training of professional counselors for work in a variety of settings. The program is appropriate for those who wish to become counselors in a range of community agencies, such as mental health centers, rehabilitation offices, addiction treatment clinics, and family service organizations. The Master of Arts is also appropriate for those planning careers in college counseling and student development services and for those who wish to become counselors in elementary, middle, junior high, or high schools. Our emphasis is on practitioner skills training within a broad theoretical foundation and exposure to the empirical bases of the profession. The Department of Counseling is also committed to seeking and valuing diversity in students and staff. Diversity, used here in a very broad sense, includes the variety of cultures, backgrounds, values, and experiences found among faculty and students; it also includes the diversity of our professional ways of practice, our ways of learning, and our personal and professional goals. In training, the M.A. Program curriculum works to integrate diversity awareness and appreciation into all course offerings, with the goal of encouraging students to explore and appreciate diversity in all situations. We are committed to training multiculturally competent counselors. Student Learning Goals & Objectives (What are our program s goals and objectives? What will students be able to think, know, do or feel because of a given educational experience?) Students are expected to : (a) acquire knowledge in eight areas of competency listed below at the level expected for an entry-level master s position in counseling; (b) demonstrate critical thinking skills through written assignments and oral presentations; (c) articulate an awareness of the needs of diverse populations and develop sensitivity and skills to meet these needs, (d) demonstrate counseling skills such as empathic listening, clarification, cognitive reframing, confrontation, and crisis intervention; (e) demonstrate

9 2 the ability to reflect upon one s values, beliefs, skills, and interventions, particularly in the context of a clinical supervisory relationship in which accepting and responding positively to feedback are expected; (f) conduct an independent research project, analyze the findings, and report the results in a manuscript; and (g) develop ethical decision-making skills demonstrated not only via knowledge (in the list below), but also in the process of counseling clients, conducting research, and various other counselor roles (such as advocate and consultant). The eight knowledge areas are: (a) Human Growth and Development (HGD), (b) Social and Cultural Foundations (SCF), (c) Helping Relationships (HR), (d) Group Work (GW), (e) Career and Lifestyle Development (CLD), (f) Appraisal (A), (g) Research and Program Evaluation (RPE), and (h) Professional Orientation & Ethics (PO). As our assessment plan develops, goals will be broken down into objectives. For example: Helping Relationships: Goal: Students will learn skills to demonstrate effectiveness in providing counseling to a variety of mental health issues with clients from diverse populations. Objective: Students will assume the counselor role in 10 or more videotaped role plays in COUN 510 (Counseling Methods). Objective: Students will complete 40 hours of direct counseling in COUN 580 (Counseling Practicum). Objective: Students will provide 100 hours of direct counseling service in the counseling internship (COUN 584, COUN 587, COUN 588, or COUN 589, depending on the student s counseling emphasis). Objective: Students will actively participate in at least 10 hours of supervision at Practicum and at least 30 hours of supervision on Internship. Objective: On the standard evaluation form (currently the ART Form), students will receive satisfactory ratings on Helping Relationship items from their supervisor at Practicum and from both their on-site supervisor and seminar supervisor at Internship. Objective: Students will obtain a raw score of 10 or more on the Helping Relationships Section of the Counselor Preparation Comprehensive Examination (CPCE). Educational Experiences How will the goals and objectives be met? What program experiences (courses, seminars, research, etc.) help students achieve the desired outcomes? Program goals and objectives are met through required classroom courses, elective courses such as research practicum, the completion of an independent study or thesis, and in the practicum and internship site experiences that accompany the classroom portion of COUN 580 and COUN 584/7/8/9. The eight knowledge areas are listed again below, with the primary courses in which learning in each area is expected to occur. Unless noted, courses are required by students in all four emphases: (a) Human Growth and Development (HGD): COUN 523 (S): Elementary School Counseling COUN 524 (elective, S): Middle School Counseling COUN 525 (S): Secondary School Counseling COUN 530: Theories of Counseling, Personality, and Development

10 COUN 531: Psychology of Women, Gender, and Development COUN 565 (elective): Child and Adolescent Development COUN 565 (elective): Mid- and Older Adult Development (b) Social and Cultural Foundations (SCF): COUN 531: Psychology of Women, Gender, and Development COUN 532: Multicultural Counseling (c) Helping Relationships (HR): COUN 510 Counseling Methods COUN 518: Group Theory and Process COUN 523 (S): Elementary School Counseling COUN 524 (elective, S): Middle School Counseling COUN 525 (S): Secondary School Counseling COUN 533: Couples and Family Counseling COUN 580: Counseling Practicum COUN 584/7/8/9: Counseling Internship (d) Group Work (GW): COUN 518: Group Theory and Process COUN 584/7/8/9: Counseling Internship (e) Career and Lifestyle Development (CLD): COUN 519: Career Counseling COUN 580: Counseling Practicum COUN 584/7/8/9: Counseling Internship (f) Appraisal (A): COUN 517 (emphases A, C, S): Psychological Testing COUN 514 (emphasis R): Rehabilitation Counseling: and Evaluation COUN 529 (emphasis A): Dynamics of Addiction COUN 580: Counseling Practicum COUN 584/7/8/9: Counseling Internship COUN 997/9: Independent Study/Thesis (g) Research and Program Evaluation (RPE): COUN 515: Research Methods COUN 516: Research Lab COUN 997/9: Independent Study/Thesis (h) Professional Orientation & Ethics (PO): COUN 501 (emphases A, C, S): Ethics in Counseling COUN 502 (emphases A, C): Professional Issues in Counseling COUN 506 (emphasis R): Rehabilitation Counseling: Foundations and Ethical Issues COUN 522: Management of School Counseling Programs COUN 580: Counseling Practicum COUN 584/7/8/9: Counseling Internship 3

11 Methods What assessment methods will we use to collect data about what our students have learned? What are our criteria for success? How will we know the objectives have been met? What level of performance meets each objective? How will we interpret and evaluate the data? 4 I. An annual evaluation is conducted by the program faculty for every student at the end of the fall semester. Students are rated on the following scale: E = exceptional, S = satisfactory, U = unsatisfactory for the following six areas: 1. Academic Coursework 2. Practitioner Skills 3. Professional Activities 4. Professionalism 5. Writing Skills 6. Independent Study/Thesis Progress The scale is also used for a final category: Overall Progress. As students who earn an A in a course may still not demonstrate exceptional skills in all areas assessed by the course, this evaluation allows us to comment on discrepancies between, for example, objective test performance and formal oral presentation skills, or between timeliness of assignment completion and quality of participation in classroom discussion. Other qualitative descriptions are offered that are consistent with expectations and feedback methods of supervisors in the field of counseling. Students who achieve ratings of Satisfactory in all areas are considered to be successfully progressing toward the degree. II. Objective : National Standardized Test As a graduation requirement, students in the Master of Arts in Counseling Program must attain passing scores on the Counselor Preparation Comprehensive Examination (CPCE), a nationally administered multiple choice test developed by the National Board for Certified Counselors (NBCC). Passing scores are determined at the local level by the Department of Counseling. We started administering the CPCE in the fall semester of 1997, shortly after the test s inception, and based our passing scores according to national averages provided by NBCC. Now that we have been administering the CPCE two to three times per year for over seven years, enough local and national data have accrued for us to have established a cutoff score of 87 for the Total Score and 10 for each of the eight subscale scores, comprised of the following content areas: (a) Human Growth and Development (HGD), (b) Social and Cultural Foundations (SCF), (c) Helping Relationships (HR), (d) Group Work (GW), (e) Career and Lifestyle Development (CLD), (f) Appraisal (A), (g) Research and Program Evaluation (RPE), and (h) Professional Orientation & Ethics (PO). Students who score below an 87 must retake the CPCE. Students who score below 10 on one or more subscales must write an essay of approximately 8-10 pages for each topic area in which they received a failing score. Each essay question is written by a faculty member and the essay is evaluated by the same faculty member. The student must receive a passing evaluation on each essay to pass the comprehensive examination. If the evaluator determines that the essay is unsatisfactory or of marginal quality, a second faculty

12 5 member evaluates the essay. If the second evaluator concurs with the first, the student must rewrite the essay. If the second evaluator determines that the essay is satisfactory, a third faculty member evaluates the essay for the final time. If the third evaluator determines that the essay passes, the student has passed; if the third evaluator determines that the essay is unsatisfactory or marginal, the student must write a new essay in response to an essay question on the same domain written by a faculty member. We have analyzed the results of the local 116 protocols since the Program started using the CPCE in 1997 and compared them to the national results of 2236 protocols, which is the total number across six administrations during the same period. We refer to individual students test results as protocols because our total sample of 116 represents 110 students, 107 of whom took the CPCE once, 4 who took it twice, and 1 who took it three times. National results may similarly reflect a small percentage of students who took the test repeatedly. The mean Total Score for our students was compared to an average national mean of (see Table 1). As the standard deviation for the national sample averaged across the six administrations sampled, the Total Score for our students is 0.80 standard deviations above the national average. The range of Total Scores for our students was 78 to 120, compared to a national range of 40 to 125, indicating that the lowest score for our students far exceed the lowest national score, whereas the top score for our students is close to the top score nationally. Table 1 MA Student Summary Scores for the Counselor Preparation Comprehensive Examination (CPCE), Fall 1997 Spring 2005 HGD SCF HR GW CLD A RPE PO Total National Mean (N=2236) Local Mean (N=116) Difference in Standard Deviations Key: HGD: Human Growth and Development SCF: Social and Cultural Foundations HR: Helping Relationships GW: Group Work CLD: Career and Lifestyle Development A: Appraisal RPE: Research and Program Evaluation PO: Professional Orientation & Ethics Since we started using the CPCE in 1997, 102 of 110 students (92.7%) received a passing Total Score on the CPCE on the first attempt. Three did not attempt the CPCE again, four passed it on the second attempt and one passed it on the third attempt. 1 The term average national mean rather than national mean is used because it was calculated as the mean of the mean of six national administrations and is not a true weighted mean (as sample sizes differed across the six administrations).

13 Of the 107 protocols, 67 (62.6%) passed without having to write any essays to show competency on weak areas of the test. There were 30 protocols (28.0%) for which one essay was required, 9 (8.4%) requiring two essays, and 1 (0.9%) requiring three essays. Most students write essays that are satisfactory to the first reviewer, demonstrating their learning in the content domain and thus expediting the completion of graduation requirements. 6 Looking now at the eight subscales of the CPCE, our students exceeded the national average on each. For Human Growth and Development (HGD), the mean score for our 116 protocols was 13.20, compared to a national average of The national standard deviation across six administration periods was 2.45, such that UND students were performing at 0.53 standard deviations above the mean in the HGD area. Standard deviations will not be reported individually here for the remaining seven areas, but range from 0.36 (A) to 0.85 (RPE) above the national average (see Table 1). For Social and Cultural Foundations (SCF), the UND student mean was 11.84, compared to the national mean of In the area of Helping Relationships (HR), the UND student mean of exceeded the national mean of In Group Work, the UND mean was 14.1, compared to nationally. There was a smaller margin for Appraisal (A), with UND students nevertheless outperforming the national average: compared to The largest score difference was in Research and Program Evaluation, in which the UND mean of surpassed the national mean of Finally, in Professional Orientation and Ethics, the UND mean of was solidly above the national average of The CPCE has served us as an excellent assessment of student learning for several reasons. First, the eight topic areas closely match our curriculum. Second, format of the test, including the eight areas subscales, match the National Counselor Examination (NCE) that is required in most U.S. states for counselor licensure; thus, the taking CPCE is excellent preparation for licensure in the field. Third, our students consistently score above the national average in the total score and in most of the eight topic areas (leading to cumulative results showing a clear UND edge over the national sample in all eight areas). Therefore, considering that most of our students enter the program with little field experience and GRE and MAT scores that are average or below average (i.e., prior knowledge of the discipline is modest and graduate level learning is not predicted to be high), we have evidence that what our students are learning from year to year is consistently more than in other counseling programs nationwide. In fact, UND scores are consistently above those of students in programs accredited by CACREP, an accreditation we plan to seek within the next five years. Finally, this consistency in scores suggests that changes in faculty and evolving changes in the curriculum have little effect on student learning of the eight areas targeted in the most commonly used counseling licensure examination. III. Evaluation of Counseling Skills and Clinical Written Work. Students complete a self-evaluation of clinical skills and are rated by their supervisor (GTA) in each clinical course, and by the site supervisor in COUN 584/7/8/9 (see Timeline for details). For the past several years, student learning in the area of counseling skills and documentation of clinical work has been assessed via the 32-item Anchored Ratings for Therapists (ART) form in the Counseling Practicum (COUN 580) and Internship in Counseling (COUN 584, 587, 588, and 589) classes. Recently the 31-item Self- of Basic Helping Skills (SBHS) has been used in Counseling Methods (COUN 510) and Counseling Practicum (COUN 580). To date, no instrument has been located or developed that we believe adequately assesses the broad range of skills that students are

14 7 expected to acquire during their three-course clinical sequence (Methods, Practicum, Internship). In September 2005 the faculty will review a proposal suggesting that the SBHS be used in Methods and Practicum while the ART be used in Practicum and Internship. The advantage to this proposal is that student learning could be assessed from Methods through Practicum via the SBHS and from Practicum through Internship via the ART. The disadvantage is the double-testing of students in Practicum. A trainee rating form still needs to settled upon for our school counseling students, who do little of the 50-minute individual session work that most counselor rating forms emphasize. We expect to make significant progress in the area of counseling skills assessment during the academic year. IV. Evaluation of Written Work: Research. The faculty are discussing preliminary ideas to evaluate completed independent studies and theses on an annual basis. In the primary idea under consideration, faculty would nominate outstanding projects, which would then be reviewed by a three-person team, including the student s advisor/thesis chair. A rubric would be developed to evaluate the different sections of the paper (typically introduction, literature review, methods, results, and discussion), plus other important qualities such as evidence of critical thinking, incorporation of cultural diversity, and adherence to the American Psychological Association Publication Manual (APA style). Ratings would be kept on file with the completed papers (students names and other identifying data will be removed) as evidence of student learning and the top paper each year would receive an award that would be publicized to all faculty, students, and in the alumni newsletter. Timeline When will we collect data? How often? I. An annual evaluation is conducted by the program faculty for every student at the end of the fall semester. II. Students take the Counselor Preparation Comprehensive Examination (CPCE) once they advance to degree candidacy. III. During COUN 510 and 580, clinical rating forms are collected from students and GTA at midterm and during the final two weeks of the semester. During COUN 584/7/8/9 data are collected from students, GTA, and site supervisor at the end of each semester of the two-semester internship. IV. We are proposing that, every September, program faculty briefly review the completed independent studies (IS) and theses for every student for whom they have been the chair, committee member, or advisor. Outstanding work will then merit a nomination for the IS/Thesis award. We expect that the team review of nominated works and award decision will be completed by October. Responsibilities Use of Results and Process for Documentation & Decision-Making Who will be responsible for collecting, interpreting and reporting the results? How will the results of assessment be used? Who needs to know the results? How can we convince them that the goals and objectives have been met?

15 How can we improve our program and assessment process? 8 The M.A. Program Coordinator will be the primary person responsible for collecting, interpreting, and reporting the results of the CPCE and annual evaluations. The Program Coordinator will provide feedback to students on CPCE results and faculty advisors meet individually with students to review the annual evaluations. Instructors of our primary clinical courses (COUN 510 and 580, as well as the Field Placement Coordinator for 584/7/8/9) will be primarily responsible for collecting the SBHS and ART forms, interpreting the data, providing feedback to students, and reporting the results to the Program Coordinator. Aggregated CPCE results will be provided to instructors of courses, broken down by CPCE topic area as delineated in the Educational Experiences section of this report. All program faculty are present at the annual student evaluations and are therefore aware of the results. Individual SBHS and ART results are provided to the student, Program Coordinator, and Faculty Advisor. Faculty will already have the IS/Theses from which nominations for the outstanding study will be drawn. The Program Coordinator will distribute nominated works to review teams. Evaluations of the studies will be shared with all faculty. The award will be announced to all students, faculty, staff, college administration, and alumni. Annual evaluation results are used to encourage students to maximize academic learning potential and, more importantly, develop the self-awareness and strategies to grow professionally beyond the classroom setting. CPCE results will be used by faculty to update and develop syllabi for current content and assignments that sufficiently attend to material required to pass the licensure exam. Clinical rating forms will be used to identify clinical strengths and weaknesses and develop a plan for continued skill development. IS/Thesis evaluation and award will be used as an incentive for students, a quality check for consistency among graduating students, and to help identify model manuscripts for current studies to use as they complete their own final research projects. As we develop and implement this plan, we will periodically review it at faculty meetings, evaluate the effectiveness and efficiency of different components, invite feedback from the Dean, Assistant Provost, and, when applicable, students, and modify program objectives as needed. We are likely to require some initial feedback before we can determine how to improve our program and assessment process beyond the proposals that have been outlined in this document. Respectfully Submitted, David Whitcomb, Ph.D. Assistant Professor of Counseling Director, M.A. Program (701) david.whitcomb@und.edu