AA.1. A review by program faculty of programs, curricular offerings, and characteristics of program applicants.

Transcription

1 Northern Illinois University Department of Counseling, Adult and Higher Education Doctorate in Counselor Education and Supervision Systematic Program Evaluation Report (in accordance with CACREP Standard I.AA. Evaluation) The following report is written to comply with the CACREP standard I.AA which charges every counseling program with the Distribution of an official report that documents outcomes of the systematic program evaluation, with descriptions of any program modifications, to students currently in the program, program faculty, institutional administrators, personnel in cooperating agencies (e.g., employers, site supervisors), and the public. CACREP Section I. EVALUATION AA. Program faculty members engage in continuous systematic program evaluation indicating how the mission, objectives, and student learning outcomes are measured and met. The plan includes the following: AA.1. A review by program faculty of programs, curricular offerings, and characteristics of program applicants. The mission statement and program objectives are evaluated using the Advisory Board, employer and site supervisor surveys, graduate surveys/focus groups, and faculty retreats. Program objectives are also evaluated with a review of course syllabi and the College of Education course outline review. Learning outcomes are evaluated using assignments in core courses, general/specialty comprehensive exams, annual advanced practice review, non-course requirements review, and doctoral dissertation evaluation process. Feedback from the Advisory Board is used to evaluate the mission statement and program objectives for the doctoral program. Feedback from graduates, employers and site supervisors has been difficult to obtain and the faculty have made additional efforts to seek feedback from these populations. In , while the doctoral site-supervisors were also the employers of doctoral graduates, the survey return rates were very low. Currently, surveys are being prepared for doctoral level supervisors and employers to occur in to keep pace with the scheduled evaluation timeline (every 3 rd year). Site supervisors will be surveyed as part of the Annual Site Supervisor Workshop held once a year in the fall. The next data collection conducted for doctoral graduates will be attempted in spring 2014 so that the master s and doctoral program surveys occur on the same timeline. Each year, the NIU Alumni Office distributes a survey of the previous year s graduates. The NIU graduate survey response rates have also been very low. Results from these surveys (graduates, site supervisors, and employers) are shared with the faculty during

2 faculty retreats. Faculty retreats are held every year in the fall and spring to evaluate feedback and develop program plans based on the data. Typically fall retreats are focused on the master s program and spring retreats are focused on the doctoral program. There are several additional evaluation elements that are also part of the evaluation timeline. Review of the course syllabi occurs annually. The last review was in spring 2008 and the next review will be in spring The College of Education course outline review is conducted once every three years. This review was conducted in and the next review is in Faculty review of assignments in the core doctoral courses is conducted once a year at spring faculty retreats. General and specialty comprehensive examinations, non-course requirement reviews and the doctoral dissertation evaluation process are conducted every semester. The advanced practice review is conducted annually at a faculty meeting. Trends from these elements are shared with the faculty during faculty retreats. Faculty retreats are held every year in the spring to evaluated feedback, make and implement program plans. Programs and Curricular Offering The faculty members are responsible for reviewing the doctoral program and courses at several points in the comprehensive program evaluation plan. Faculty members began reviewing the program objectives during the annual spring faculty retreat in 2007 and continue to review program objectives each spring. In addition, during many faculty meetings throughout the year, ongoing assessment of the program and courses takes place. The connection between learning outcomes and curricular offerings is assessed at several points, including retreats and faculty meetings. The connection between assignments in core courses, learning outcomes and program objectives is evaluated annually at the spring faculty retreat. There are many points in the program in which students are evaluated to determine whether or not they have achieved the doctoral program learning outcomes. As indicated on the comprehensive program plan, these points include assignments in core courses, the advanced practice review, comprehensive exams, non-course requirements and the doctoral dissertation. Assignments in core courses Specific core course assignments have been selected by the faculty to assess student knowledge of core areas as well as identify writing strengths and weaknesses. Students are given feedback from these individual assignments in their courses. During a faculty meeting, overall patterns from the student s performance are discussed. These discussions have pinpointed student s difficulties around writing skills, information synthesis, and APA style. The chair of the student s program committee shares feedback regarding the achievement of learning outcomes with the student. Additionally, aggregated data from these assignments are used to make program modifications. The analysis of these trends has resulted in program changes. Faculty agreed that core assignments should be moved from the end of the semester, so faculty would have more time to evaluate writing assignments and offered feedback to students.

3 It was agreed that some core assignments needed to be changed and additional feedback from these assignments needed to be given to students (other than the advanced practice review letter) to better prepare them for comprehensive exams. Advanced practice review Every spring, the program coordinator prepares an advance practice review report on each doctoral student and this report is reviewed at a faculty meeting. This report indicates whether each student is making satisfactory progress in their coursework and developing appropriate advanced clinical skills in practicum and internship. Each year, doctoral students are sent progress evaluation letters. These letters are placed in the student s file and reviewed with the student and their program chair. Students who are making satisfactory progress are sent a letter indicating their satisfactory progress. Students who are identified as having difficulty in achievement of any learning outcomes are sent a concerned letter indicating they should arrange to meet with their program chair. Students who have not followed through on their concerned letter or who have stopped working on their dissertations are sent a danger letter. This review is used to drop doctoral students from the program, especially at the dissertation stage, when they are inactive for multiple semesters. If there is no response after the danger letter, the Graduate School drops the student from the program. This process has been working effectively. To indicate this effectiveness, data from 2005 will be compared to 2009 and In March of 2005, there were 43 students enrolled in the doctoral program. Thirty five students were making satisfactory progress (81%), five students received danger letters (indicating they were at risk to be dropped from the program), three students received concerned letters (indicating that they were not making satisfactory progress), and six students were dropped from the program. In spring 2009, there were 44 students enrolled in the doctoral program. Thirty eight were making satisfactory progress (86%), five students were given concerned letters, one student was given a danger letter and no students were dropped. During the academic year, there were 47 students enrolled in doctoral program. Forty (85%) received satisfactory progress letters, five students were sent concerned letters and two students received danger letters. In 2010, nine students were dropped from the program. A letter was sent to each of them requesting they contact their advisor. The students were also asked to register for a class within the year. Eight of those nine made no contact and did not register for a class. The ninth student initiated withdrawal from the program. In spring 2011, four students were identified as being in danger. Letters requesting they contact their advisor and register for class were sent in June General/specialty comprehensive exams Each semester students are evaluated on general and specialty comprehensive exam performance. At the completion of their counseling core classes, students take general and specialty comprehensive examinations. The general comprehensive exam is designed to reflect knowledge of the CACREP core doctoral content areas.

4 An analysis of patterns and trends in general comprehensive performance for indicated that students were having difficulties with the knowledge integration and writing skills necessary to successfully complete their comprehensive exams. In 2005, three students passed their general comprehensives. In 2006, two students passed their general comprehensives and in 2007, four students passed their general comprehensives. This trend continued through While all students passed, often students do not pass all the questions on their general comps the first time they take the exam and must rewrite the questions. At least 50% of students each semester that take the general comprehensive examination had to rewrite at least one question. A formal analysis of the trends in specialty comprehensive performance has not taken place, but the students tend to experience similar problems on their specialty exams. Most students have not experienced difficulties in the oral defensive of their specialty comps. Difficulties seem to focus around APA style, writing structure and clarity, and synthesis of information. As the 2009 standards continue to be integrated, a formal analysis of comprehensive and specialty exams will be included in future program assessment. Non-course requirements Throughout their programs, doctoral students are expected to engage in a number of activities to enhance their professional development. These activities include the development of computer/technology competence, professional association involvement, research team involvement, submitting an article for publication, and presenting at a professional conference. These activities must be completed prior to the approval of their dissertation proposal and are documented on the non-course requirements for Ed.D. in Counselor Education and Supervision form. Additionally, these activities help students receive external feedback on their performance from journal editors and presentation participants. An informal review of students completion rates for these activities was conducted for the spring 2008 faculty retreat. This review suggested that all students are able systematically to achieve and document computer/technology competence, professional association involvement, and presentations during their program. As a result of this review, research team involvement was included as a required experience as part of the doctoral research course to facilitate achievement of this activity. Helping students submit papers for publication was identified as a growth area. It was noted that the faculty have consistently included doctoral students in their own presentations, publications and professional association leadership activities. Dissertation review process The Review of Dissertations are conducted by a panel of external reviewers approximately every eight years. As a requirement for the program review, the graduate school forwarded four dissertations completed by recent graduates of the counselor education program to four external reviewers. All four external reviewers found the dissertations of high quality and felt they made excellent contributions to the literature of their respective topics. The external reviewers commented on the conceptualization of the problem being well thought out and the strengths outweighed the weaknesses of the dissertations. The external reviewers did identify a few weaknesses including the need to

5 check the reference list for accuracy and the literature review being narrow. The latter may be attributed to the research topic being relatively new to the discipline of counselor education. Reports from Dean s designees were submitted for 17 of the 20 dissertations defended from 2004 through On the whole, these reports were positive with responses of strongly agree and agree to all three questions (quality of the dissertation, rigor of the defense, and fair treatment of the candidate by the committee) with three exceptions. Two responses of disagree were reported regarding the quality of the dissertation and one response of disagree was indicated in terms of the rigor of the defense. Specific comments of Dean s designee reports relative to quality of the dissertation noted there were methodological limitations, exclusion of current literature and editorial issues. A comment about the rigor of the oral examination indicated the student was unable to adequately respond to questions about the methodology but was not pushed to do so by the committee. The Dean s designee report mentioned that while current literature was not included in the written document that after discussion there were absolutely no concerns about the candidate s grasp of the literature. The external reviewers and Dean s designee reports indicate that the counseling program produces high quality dissertations that align with the standards of other doctoral programs. Furthermore, the positive internal and external reviews demonstrate that our graduates have the ability to conduct independent scholarly research. The feedback from the external and internal review process is valued and will be useful to help us further enhance the dissertation process. Characteristics of Program Applicants Applicants to the doctoral program complete an application to the Graduate School and a specific application for the program. This application process involve the submission of a research writing sample, GRE scores, letters of recommendation and an essay indicating relevant advanced practice, supervision and counselor education career goals. Other than diversity data, data on characteristics of program applications is antidotal in nature. In 2007, 34 % (15 out of 44) doctoral students were from racial/ethnic diverse backgrounds. In 2006, the Siemens Scholars program was implemented to increase the number of diverse students within the doctoral program. The faculty created a partnership with the Siemens Corporation in order to recruit and support minority students. Based on this program, there was a notable increase (approximately 20%) since the CACREP report (2003) in the number of African-American students enrolled. As of 2009, NIU Institutional Research reported that student ethnicity of doctoral students was 23% African American/Black, 3% international, and 6% listed as other. The remainder of students enrolled in the doctoral program identify as White/Caucasian. AA.2. Formal follow-up studies of program graduates to assess graduate perceptions and evaluations of major aspects of the program;

6 Graduate survey Formal follow-up studies of program graduates are conducted about the program mission statement, program objectives, as well as overall satisfaction with the program. Annually, the NIU Office of Assessment Services conducts an Alumni Survey of the previous year s graduates. This survey assesses graduates job titles, salary levels, satisfaction with current job, preparation for current job and future educational needs. It must be noted that the doctoral program in counseling is very small. There have been 24 graduates from and 10 graduates from According to the NIU alumni survey data, all of the doctoral respondents in 2006 (n=2) and 2008 (n=2) were able to find jobs after completing the counseling program as compared to 91% (n=569) and 92% (n-517) of NIU respondents. This may be attributed to the viability of jobs in the counselor education profession. Students can teach, work in mental health settings, become administrators/supervisors and/or start a private practice. In 2006 and 2008, 100% of respondents (n=2 for both years) felt the degree prepared them for their present position compared to 96% and 97% for other NIU respondents. Likewise, the counseling graduates were in agreement (100%) that the degree was a worthwhile investment which compares favorably with NIU data of 93% and 96% for 2006 and 2008, respectively. Additional responses from alumni were obtained through a survey distributed by the counseling program with assistance from the Office of Assessment Services and Office of Advancement Services (during fall 2009 and spring 2011). In 2009, 95 surveys were sent to master s and doctoral graduates, only 12 (17%) alumni from the master s program responded and 8 (40%) alumni from the doctoral program. The program faculty were interested in the perception of alumni relative to the training they received on specific learning outcomes as well as their overall experiences with the program. The data indicated that alumni of the doctoral program strongly agree (or agree) that the program has been very helpful with increasing their knowledge/skills in specific counseling areas (critical analysis of theories, advanced case conceptualizations, strong counselor educator identity, understanding of clinical supervision models, effective consultation skills, articulate research methodology). There were only two skill areas in which the responses were not 100% strongly agree (or agree). One graduate did not believe the program was helpful with developing skills in assessment and evaluation, or helping them apply, analyze and evaluate research paradigms, methods and empirical findings. In spring 2011, only two doctoral graduates responded to the counseling program survey. The data revealed that 100% of doctoral alumni strongly agree or agree that the program was helpful with enhancing their knowledge/skills in the specified counseling areas. Exit interviews/focus group While preparing this report, the faculty made one additional attempt to gather data from doctoral program graduates. In May of 2008, three requests were made to gather this data. An request (and follow-up request) was sent to 18 graduates from 2000-

7 2007. They were asked to participate in an on-line discussion group. Subsequently, a set of survey questions were individually ed to these graduates. No graduates responded to this request for feedback. The faculty spent a significant amount of time problem-solving about how to increase the amount of feedback from doctoral program graduates. Because the doctoral program is small, there are a limited number of graduates to be contacted. As a result, the faculty has agreed to implement an intensive individual attempt to gather feedback from program graduates. Discussion will also continue around ways of conducting an exit interview as part of the final dissertation completion stage of the student s program. The Department of Counseling, Adult and Higher Education in which the program is housed has another doctoral program in Adult and Higher Education which has experienced difficulties obtaining feedback from doctoral program graduates. As a result, the department plans to implement a graduate survey of its own in fall, 2012 as this may generate additional feedback. A.A.3 Formal follow-up studies of site supervisors and program graduate employers to assess their perceptions and evaluations of major aspects of the program. Formal follow-up studies of site supervisors and program graduate employers are conducted about the program mission statement, and program objectives for the doctoral program. One survey was used to obtain information from employers and site supervisors of graduates and students in the doctoral program Site supervisors Because many doctoral students enter into the program with a significant amount of clinical counseling experience their internships often involve conducting research with faculty members, team-teaching with faculty, and/or supervising master s students in counseling practicum. Consequently, many of the doctoral site supervisors are faculty members in the program. During fall 2007, pilot evaluation surveys were distributed to the on-site internship supervisors of doctoral students. The intent was to collect preliminary data on the major components of the program and to determine if the survey was appropriate for formal follow-up evaluation if conducted every three years. Fifteen current clinical supervisors were sent the survey and five completed the survey. Preliminary results from the pilot evaluation survey reveal that the site supervisors would quickly hire doctoral interns if a doctoral level position was available. Interns received outstanding ratings for their advanced counseling knowledge and skills. Competencies assessed in this area included supervision, counseling theory/ethics and practice, teaching, ability to work with diverse clientele, personality assessment, and consultation. Supervisor ratings of doctoral intern s research skills were scored between high average and outstanding (M=3.5). These skills included program evaluation, as well as design, implementation and presentation of research. Only one of the five site supervisors provided recommendations/suggestions for the doctoral program. This respondent focused on teaching skills by indicating that talented interns would be better

8 prepared if they had sole responsibility for course development, experience assessing their clinical/teaching effectiveness and more opportunities to work with difficult students. Employers Every three years (as national accreditation stipulates) employers are asked to complete a survey that requests feedback on our doctoral graduates and about general aspects of our program. There have been 18 doctoral graduates of the counseling program between 2005 and Identifying employers of NIU counseling graduates is very challenging, especially since many of them re-locate outside of the northern Illinois area. Consequently, the number of employers who completed the newly revised survey was very small. After three attempts to obtain feedback from 17 employers during fall 2007 and spring 2008 only five were returned. We considered this preliminary data from our new survey. The next time data will be collected from employers will be during the academic year. In an attempt to increase the response rate, we will send systematic reminders via and follow up with a reminder telephone call. The preliminary results reveal that former site supervisors frequently hired our interns as full-time employees after the internship ended. Our graduates received outstanding ratings for their advanced counseling knowledge and skills. Competencies assessed in this area included supervision, counseling theory/ethics and practice, teaching, ability to work with diverse clientele, personality assessment and consultation. Supervisor ratings of graduate s research skills were between high average and outstanding. These skills included program evaluation, as well as design, implementation and presentation of research. Although the return rates are low, findings indicate that doctoral students are performing well on learning outcomes. While the program does not have a great deal of feedback from employers, the program does have some basic information about where doctoral graduates are employed. From , 24 students graduated from the doctoral program. Nine graduates are either full-time or part-time counselor educators in counseling or related educational programs (37.5%); seven graduates are in private practice (29%); five are administrators in community agencies (20%); two are administrators in community college settings and one is a school counselor. Identifying employers of program graduates is challenging in a post-graduate market as diverse and widespread as the northern Illinois area. There were three attempts to contact doctoral student employers. In December of 2007, the survey was mailed directly to employers. A graduate student wrote to the doctoral graduates to obtain the addresses of employers. In spring 2008, two additional attempts were made to graduates and ask them to have employers complete the attached survey. Only one employer responded to the employer survey. The one employer survey response indicated outstanding ratings in all areas.

9 The number of doctoral students who engage in professional presentations and publications can be used as another measure of external feedback. Doctoral students are encouraged to present at professional conferences and submit a paper for publication with faculty. Presentation and publication activities are documented with the Non-Course Requirements for Ed.D. in counseling. From doctoral students and faculty copresented 20 times and published 7 times. In 2007, a doctoral student (Linda Letch Alford) received the 2007 Association for Counselor Education and Supervision Outstanding Dissertation award. The counseling faculty has indicated that that they wish to increase the number of doctoral student publications and the publication of articles from completed dissertations. Advisory Board The faculty meets once a year with the Advisory Board to seek input about into the program s mission, objectives, program changes, and changing professional needs. The faculty specifically uses feedback from the Advisory Board to consider curricular improvements. Advisory Board membership consists of site supervisors, employers, program graduates and local professional leaders in the counseling community. In 2006, the Advisory Board made the following recommendations. The Advisory Board recommended that more faculty members should attend board meetings. Additionally, the Advisory Board strongly supported the recommendation that the Ed.D. in Counseling be changed to a Ph.D. The board also recommended that the faculty reconsider the new procedures for doctoral comprehensive exams which currently allow students to rewrite questions until they can write an acceptable response. The Board questioned if the procedure was too lenient and balanced more toward mastery than excellence. The faculty agreed to continue to monitor the procedure. In 2007, the Advisory Board reviewed and approved the mission statement for the doctoral program. Little feedback regarding changes to the mission statement was offered. Other issues and recommendations by the Advisory Board regarding the doctoral program included: on-line teaching of counseling classes, increasing the emphasis on administration and management skill development and increasing the emphasis on professional identity and ethics. The faculty is also working on ways to increase return rates for doctoral program site supervisor surveys and employer surveys. The return rates for surveys for the master s program site supervisors were significantly increased when feedback was gathered during the Site Supervisor Workshop. Traditionally, this workshop has been developed for site supervisors working with the master s students. In the future, doctoral program site supervisors (who are non-faculty) will also be invited to the workshop so that feedback can be gathered from them as well. Additionally, the faculty has agreed that the same intensive process that will be implemented to gather feedback from doctoral graduates will be used to gather feedback from employers. When graduates are contacted, they will be asked about current employment and employer contact information. Then, employers will be contacted individually by and phone. Full implementation of a plan to

10 systematically gather feedback from doctoral program employers will be implemented during the academic year. A.A.4. Assessment of student learning and performance on professional identity, professional practice, and program area standards. The doctoral program in counseling is designed such that students take appropriate courses and internship experiences to satisfy the program requirements. The course work taken will enable students to learn the knowledge by reading, discussion, and exams. Each student entering the doctoral program must have met the entry-level requirements in the M.S.Ed. program in counseling. The program is divided into two sections, academic requirements and non-academic requirements. The academic area is divided into two areas of coursework: didactic and experiential. In the didactic area, there are thirteen areas that require coursework which also reflect the CACREP standards. Doctoral students must complete work in the following: CAHC 700, Professional Seminar in Counseling (3) CAHC 730, Theories of Counseling (3) CAHC 740, Group Leadership (3) CAHC 752, Supervision in Counseling (3) CAHC 764, Personality Assessment (3) CAHC 765, Multicultural Counseling (3) CAHC 790, Seminar on Research in Human Services (3) CAHE 765, Consultation in Human Services (3) EPS 710, Seminar in Lifespan Human Development (3) The following experiential course work is required: an advanced practicum CAHC 750 (3 credit hours), where students see clients in the counseling lab on campus. Students are observed in their practicum behind one-way mirrors by their team and supervisor. All sessions are recorded and are reviewed by supervisors and the students. The second experiential class is CAHC 786, Advanced Internship (12 credit hours). The required 1200 clock-hour internship is designed to enable students to acquire the skills needed to teach counseling courses, improve supervision skills, and/or to become proficient in clinical counseling. The teaching part of an internship is usually done at Northern Illinois University with a faculty member in counseling. Students who do a teaching internship may elect to teach CAHC 211, Career Planning, or may make arrangements to teach with a faculty member. These interns are mentored in the teaching process by a faculty member. Doctoral students may also gain experience in supervision by assisting a master s class in individual or group practicum and supervising master s level students. In addition to the required course work, all students must identify an area of specialized study where they complete 9-hours in course work related to this area. These courses

11 must be approved by the student s committee. Typical specialized areas are marriage and family therapy, multicultural issues, psychology, human resource development, etc. Finally, students must take at least one course in Human Growth and Development and three additional courses in research (ETR: 521, Introduction to Educational Statistics, ETR 522, Educational Statistics II and ETR 525, Qualitative Research). Near the end of their course work, doctoral students complete a two-part comprehensive examination, general comprehensive and a specialty comprehensive. In the general comprehensive, students are given 2 weeks to write on questions written by the faculty on the core counseling areas. This comprehensive is blind reviewed by three counseling faculty members. In the specialty comprehensive each student has eight weeks to prepare a 25-page paper addressing a question or area of research agreed upon by the student and the student s committee. The student presents the written paper and then defends it at an oral examination. Upon the successful completion of this exam, the student becomes a candidate for the doctorate. The student can then begin their dissertation work. Students are required to complete a minimum of 15 credit hours of dissertation (CAHC 799). A three-member faculty committee works with the student on the dissertation process. Students must defend their proposal, generally consisting of the first three chapters, and have approval from the university s IRB to conduct their research. The required dissertation is designed to have the doctoral candidate demonstrate acceptable research skills. It includes identifying a research problem and stating the research questions, reviewing the pertinent literature on which the study is based, deciding on appropriate research methodology, collecting date, interpreting the findings, and drawing appropriate conclusions and recommendations for further investigation. All doctoral students must successfully complete the non-course requirements which include: Professional association involvement Research team involvement Submission of an article for publication Presentation at a professional conference Computer technology competence (including assessment and diagnosis, career guidance, case management, and statistics) At the dissertation defense, candidates are expected to verify their completion of requirements. Program Objectives in Syllabi The results of the three program objective reviews contributed to the need to revisit the doctoral mission statement, doctoral program objectives, and student learning outcomes to ensure that they reflected the doctoral degree in counselor education and supervision.

12 The program objectives and student learning outcomes were re-evaluated during a faculty meeting in the spring 2008 semester. Some changes were made to emphasize counselor education identity, cognitive complexity for case conceptualization, and evidenceinformed practice. These changes have been targeted to emphasize the Ed.D.to Ph.D. degree change. The first step conducted to ensure that the mission statement reflected a Ph.D. degree was to review similar degree programs to benchmark peer groups (BHE/MAC/Carnegie) to which NIU belongs and national practice (e.g., the preponderance of the Ph.D. as the terminal degree in the area of study). The mission statements of the benchmark programs were compared. This comparison along with the positive feedback from the Advisory Board resulted in no changes to the mission statement of the doctoral program. However, a more in depth comparison of the nature of these programs and input from the Advisory Board and the program faculty significantly modified the program objectives to emphasize counselor identity, research and writing skills, critical analysis skills in terms of theory to practice integration, and advanced practice with an emphasis on multicultural and supervision skill development. A second program modification has resulted from the program s emphasis to prepare counselor educators. While the faculty still desires to train advanced practitioners, the characteristics of program applicants needs to be shifted so that the majority of doctoral students have counselor education as their career goal. Additionally, attracting more students from a national rather than local geographic region will necessitate a change in the characteristics of students who apply. The change to a Ph.D. program would significantly support this emphasis. This change is supported by the faculty and is consistent with CACREP-accredited doctoral programs in counseling. Review of Program and Curricular Offering and Characteristics of Program Applicants Analysis of trends in the achievement of doctoral student learning outcomes indicated the following program and curricular modifications should be made. As reflected in the student learning outcomes, doctoral students needed to develop better writing and research skills. Another result of the analysis of these trends was the awareness that assignments in core courses needed to be modified to reflect an increased emphasis on counselor education identity, cognitive complexity for case conceptualization, and evidence-informed practice, while encouraging the development of writing and research skills. Assignments in CAHC 765 (Multicultural Counseling), CAHC 790 (Research in Human Services), and CAHC 750 (Advanced Practicum) were changed to reflect this awareness. A research paper focused on socio-cultural/ethnic issues in counseling was developed as the core assignment for Multicultural Counseling was developed. This assignment included a critical analysis of a multicultural counseling issue with a review of current research literature. A rubric was developed to emphasize and evaluate critical analysis of current research and synthesis of information. Since the implementation of this core assignment (2006), rubric-based evaluations of the research papers have indicated the established criteria for knowledge pertaining to socio-cultural influences in counseling

13 was met by all students. Major improvements continue to be needed in how doctoral students identify and respond to other components of the research papers. The research paper included a few questions that were not focused on socio-cultural issues in counseling, but would measure the student s ability to engage in critical analysis and professional writing. About 50% of students did not address all the research questions in their papers. This issue is being address by making this assignment due earlier in the semester, so that constructive feedback can be given to the students so that they will be able to make paper modifications and enhance their performance on future research papers and comprehensive exams. Additionally, in the annual review of doctoral student progression it was recognized that even strong students sometime struggle with the writing elements during comprehensive examinations and in construction of the dissertation proposal. Two courses CAHC 750 (Practicum in Counseling) and CAHC 790 (Seminar in Human Services Research) were targeted for instructional delivery changes to help students build writing and research skills. In CAHC 750, many of the doctoral students enter the program already certified or licensed in the counseling field. As a result, their fundamental counseling skills set and the level of client case conceptualization provides them with a strong base. Building on this base, additional readings on the effectiveness of counseling/psychotherapy, as well as how such research is conducted were added to the practicum class. This was an attempt to integrate knowledge of evidence-informed practice to their advanced counseling skills. The final written assignment required incorporation of empirical literature into a case study and that the paper is written as if it would be submitted for publication in a journal. As a result, doctoral students advance practice skills are used to enhance critical thinking and writing skills. In the second course, CAHC 790 (Seminar in Human Services Research), several written assignments has been added to increase the doctoral students critical thinking and writing skills. These assignments have been designed to prepare them for the dissertation process. A requirement of weekly written synopsis of the readings for class has been added. These synopses must be posted to the discussion board on Blackboard 48 hours prior to class time. Then a written response to at least one classmate's postings is required. The developmental skill sought in the pre-class assessment is the capacity of the student to identify significant themes/constructs and concisely express what has been learned from the week's topic. After each class, they are asked to do an exit assessment, 3-2-1; three things that are the most important ideas, two areas for follow-up, and one lingering question about the topic. Additionally, students are asked to complete a structural analysis (not content) of a dissertation that is similar to the topic they are interested in pursuing. Finally, students submit a structural map of how they would organize their dissertation proposal with a sample section from each chapter. Follow-up Studies of Graduates, Site Supervisors, and Employers While there has been limited feedback from surveys of program graduates, site supervisors and doctoral program employers, changes have been made in the past two years. First, the new CACREP 2009 Standards outline five domains important for any

14 counselor education program: counseling, supervision, teaching, leadership and research. As the NIU Counselor Education and Supervision program faculty move toward offering the Ph.D., the curriculum was revised and aligned based on these five domains. In doing so, a research course and a professional identity course were added and there was a restructuring of the internship experience to include five different experiences that mirror those domains. Students are also engaging in more leadership and research experience, either independently or with faculty. Faculty members continue to work with students on professional writing skills by forming writing teams and co-authoring grants. For example, one faculty member recently received a grant from the Department of Mental Health and has worked with several students to develop training modules. This experience has also resulted in several online articles. Informal feedback from program graduates currently employed in counselor education programs and comparisons to current counseling education Ph.D. programs has resulted in the addition of a research methods course to the research requirements of the doctoral program. Students are required to have one 3 hour credit course in general counseling research methods and design (CAHC 790: Research in Human Services), one 3 hour credit course in quantitative methods (ETR 521: Educational Statistics), one 3 hour credit course in qualitative methods (ETR 525: Qualitative Research in Education), and one additional 3 hour credit research course chosen with the approval of their program committee. This additional research course will encourage students to have additional coursework in the specific method they will use to complete their dissertations. AA.5. Evidence of the use of findings to inform program modifications. Programmatic Change Resulting from Assessment, and Other Continuous Quality Improvement Activities During the annual spring retreat, the evaluation data that has been recently collected is discussed and implications for changes to the curriculum are proposed. The evaluation data discussed in this report spans a four-year period from During this time, the program has experienced several significant external evaluation processes including a Mid-Cycle program review and the implementation of LiveText as part of the preparation for NCATE accreditation. Both of these processes, in addition to the program evaluation plan as a part of the CACREP accreditation, have resulted in significant data collection, analysis and program modifications. These program modifications have been documented in several sources including the NIU Mid-Cycle status report, NCATE reports and Advisory Board Meeting minutes. The following discussion summarizes the program modifications resulting from the program assessment. Following is a summary of how evidence has been used to impact change. Programmatic Improvements A new orientation program has been developed for doctoral students. This is a formal threehour program that describes course requirements, assistantships/fellowship opportunities and

15 general procedures for completing the doctorate. The first annual orientation was well received by students. In order to improve student performance on the general comprehensive exam a few modifications have been implemented. Faculty who teach doctoral courses have agreed to include comprehensive type questions as part of their course assignments. A rubric for evaluating comprehensive exams has been developed. The evaluation criteria consist of the following areas (organization, integration and synthesis, knowledge of the professional literature, critical thinking, application, originality and depth of thought, APA style and mechanics/grammar). The final change for the exam was the addition of a comprehensive exam workshop conducted by the counseling faculty. The workshop is an opportunity for students to learn about the evaluation criteria and to receive tips on how to best respond to exam questions. These changes have reduced anxiety about the exam, ensured that students received early exposure to comprehensive type questions and enhanced their understanding of the process and evaluation criteria. Faculty evaluation of students independent research and feedback from dissertation reviewers has contributed to a change in the number of research courses that are required by our doctoral students. Within the past two years, the requirement has changed from 9 hours (or three courses) to 12 hours (four courses). The faculty implemented a process to annually evaluate each student for satisfactory progress in the program. The primary motivation for this policy was to encourage students to maintain an intentional plan for completing the doctoral program in a timely manner. Additionally, it was designed to send a strong message to inactive students who often remained inactive for long periods of time, especially at the dissertation stage. Since inception the list of inactive students has decreased and students have decreased the time it takes to earn their degree. The counseling faculty is working hard to involve new faculty on student committees, inviting them to serve as co-chairs of dissertation committees and encouraging these tenuretrack colleagues to teach doctoral courses. This process should help new faculty become familiar with the dissertation process and consequently, contribute to an increase in the number of doctoral graduates each year.