AA.1. A review by program faculty of programs, curricular offerings, and characteristics of program applicants.

Northern Illinois University Department of Counseling, Adult and Higher Education Master s in Counseling Systematic Program Evaluation Report (in accordance with CACREP Standard I.AA. Evaluation) 2004-2010 The following report is written to comply with the CACREP standard I.AA which charges every counseling program with Distribution of an official report that documents outcomes of the systematic program evaluation, with descriptions of any program modifications, to students currently in the program, program faculty, institutional administrators, personnel in cooperating agencies (e.g., employers, site supervisors), and the public. CACREP Section I. EVALUATION AA. Program faculty members engage in continuous systematic program evaluation indicating how the mission, objectives, and student learning outcomes are measured and met. The plan includes the following: AA.1. A review by program faculty of programs, curricular offerings, and characteristics of program applicants. Each of these elements (mission statement, program objectives, student learning outcomes) are evaluated using advisory boards, employer and site supervisor surveys, graduate surveys/focus groups, and faculty retreats. Program objectives are evaluated with a review of course syllabi and the College of Education s formal course outline review. Learning outcomes are evaluated using assignments in core courses, admissions screening, the portfolio, annual retention review, admission to practicum/internship evaluations, and the NCE examination. Learning outcomes have been evaluated using rubric data aggregated in LiveText. LiveText is a data management system adopted by the College of Education which allows students to electronically submit assignments to faculty. Assignments are graded via use of rubrics and data is aggregated to aid faculty in determining whether learning outcomes have been met. Feedback from the advisory board is used to evaluate the mission statement, program objectives, and student learning outcomes. The Advisory board meetings are held once a year, in the fall. Site supervisor feedback is also gathered annually as part of this site supervision training day. Employer surveys are typically conducted once every three years. The graduate/alumni survey is administered every year by the University. In an attempt to gather additional information from our master s students who are about to graduate, focus groups are conducted each spring

Review of course syllabi is conducted annually and the College of Education course outline review is conducted once every three years. Screenings (admissions, practicum, and internship) are conducted in the fall and spring of each semester. Portfolio evaluation happens every semester including summers. The retention review is conducted annually (in the spring) and the NCE is offered in October and April. National Counselor Examination (NCE) All master s students are strongly encouraged to take the National Counselor Examination (NCE). This national examination is designed to assess student knowledge in the major foundational counseling context areas. Student achievement on the NCE provides regular external feedback to the faculty on how well students perform on a national exam focused on the core courses. There also is a general trend toward a higher percent of NIU counseling students passing the examination. The mean score for NIU students in October, 2010 was 107.67 as compared to the mean score of 108.28 for students at all CACREP universities. Characteristics of Program Applicants The faculty evaluate the characteristics of program applicants through the pre-admission process (PAW) every fall and spring. For all applicants who have submitted their materials by the application deadline, the M.S.Ed. Admissions Coordinator and one additional faculty member, conduct a prescreening of the written application materials. These materials include the Graduate School admission application (application, letters of reference, GRE scores, and undergraduate/graduate GPA) and the program admissions application (letters of reference and essay responses). In this prescreening process, the Admissions Coordinator, reviews the application materials to determine whether minimum academic requirements have been demonstrated. This PAW allows for early identification of promising students who may need additional support in terms of writing skills and mentoring. Currently, the M.S.Ed. program consists of about 69% school applicants, 30% community, and 1% career. Applications are also considered based on area of professional preparation in order to better balance the number of students in each area. Evaluation of admission data has indicated approximately 75% of those applicants who attend the PAW are admitted into the program. Trends in admission data suggest that applicants to the master s program are still predominantly White women. As a result, increasing the diversity of students in the master s program is an ongoing concern. Admission data for 2006 indicates that of the 45 admitted students, 32 (71%) were women and 35 (77%) were white. In 2007, 50 admitted students, 39 (78%) were women and 36 (72%) were white.

AA.2. Formal follow-up studies of program graduates to assess graduate perceptions and evaluations of major aspects of the program. There are two aspects in which program graduates provide feedback on the program, NIU Alumni Survey and focus groups. Annually, the NIU Office of Assessment Services conducts an annual Alumni Survey of the previous year s graduates. This survey assesses graduate s job titles, salary levels, satisfaction with current job, preparation for current job, and future graduate education needs. The NIU Assessment Office survey of graduates of the counseling master s program in 2004-2006 indicates the results on whether the graduate has a job in the counseling field, satisfaction with their job and how well their degree prepared them for their current position. Based on the information reported from the NIU survey data, the proportion of master s students who obtain jobs in the field during and after completing their program is high. During the last five years, the percentage of students employed range from 94% to 100%. The results for the counseling program graduates who responded are slightly higher when compared to the university during this same time frame. This may be attributed to the viability of jobs in school counseling and in mental health settings. Alumni indicate that they are satisfied with their degree. While the numbers responding to this question fluctuate, there is consistent agreement (83 to 100%) that alumni are glad they received a degree from NIU. Alumni also respond similarly to the question regarding the time it takes to complete their degree. There is very strong satisfaction with the time to degree factor. During the past few years, 100% of alumni were in agreement that the time it took to complete their degree was satisfactory. The results across the four years of NIU show that graduates overall feel this is a program worthy of their time and investment. Each year 83% or more of the graduates responding felt positive about their graduate program. For all, but 2005 graduates, the level of satisfaction was slightly higher for program graduates from NIU graduates. For program specific indicators we selected the following three survey items: 1. In general, how well did your graduate degree prepare you for your present job? (response choices very well, well, adequately, inadequately, poor, very poorly) 2. I consider this degree program a worthwhile investment of my time. (response choices strongly agree, agree, disagree, strongly disagree) 3. My internship (paid or unpaid) provided experience that was professionally helpful to me. (response choices almost always to almost never on a four-point scale) In 2005, 97% of respondents felt the degree prepared them for their present position. However, in 2006 and 2007, the level of satisfaction slightly decreased to 91% and 93%, respectively. While in 2008, 100% of respondents were satisfied. The counseling rates of

satisfaction were slightly higher than those for NIU in 2005 and 2008 and slightly lower in 2006 and 2007. Beginning with a low of 91% in 2005 and increasing to 100% in 2008, program graduates agreed the degree was a worthwhile investment of time. The gradual increase from 87% to 96% of university respondents was similar to the program s increase each year relative to the perception that the degree was a worthwhile investment of time. Additional responses from alumni were obtained through a survey distributed by the counseling program with the assistance of the Office of Assessment Services and Office of Advancement Services. Approximately 95 surveys were sent to master s and doctoral students, but only 12 (17%) alumni from the master s program responded and 8 (40%) alumni from the doctoral program. The faculty is very interested in the alumni s perception of the training they received on specific learning outcomes. The preliminary data indicates alumni from the master s program, strongly agree (or agree) that the program has been very helpful with increasing their knowledge/skills in specific counseling areas. Responses show that 75 to 100% of alumni strongly agree that the training they have received while in the program was very beneficial. The perceived benefits of the training received in social justice and multicultural areas were especially high with 100% of respondents indicating strong agreement. Focus Group Results from the 2008 focus group of 12 participants had both positive encouragements and areas for growth for the master s program. Positive encouragements included that the mission statement with its emphasis on multicultural training was well reflected in course content, that the clinical skill emphasis in the program is one of the best aspects, and that a research course geared specifically to counseling students was very helpful. Growth areas were course material may be too focused on school counseling students, program changes need to be conveyed to students in a more effective and timely manner, and internship requirements that dictate that students find their own internship was time consuming and burdensome. In 2009 and 2010, the results of the focus groups were very similar which could be attributed to a stronger emphasis on objectives and learning outcomes by the group facilitators. Twenty-two students participated in focus groups during spring 2009 and twenty-four in spring 2010 (students were distributed across three focus groups). One of the major strengths mentioned from these two focus groups, is that while there are numerous reflective papers, they are instrumental in helping students enhance their empathy and acquire a heightened sense of self and one s impact on others. Other strengths included the positive support students perceived when applying skills and strategies in individual and group counseling courses and the overall view that faculty were interested in student s ability to understand and demonstrate knowledge in the stated objectives and learning outcomes.

A recommendation emphasized in the 2009 and 2010 focus groups was that students value the integration of multicultural issues across the core courses but strongly believed that a separate course on multicultural counseling would be more effective in helping them become multiculturally competent. In addition, school counselors in training would like to be assigned to a school interested in the developmental model, while they are enrolled in CAHC 570. This would provide a hands-on approach while they are learning the development model. Many of the focus group participants indicated personal counseling should be a requirement prior to admission to the master s program. Similar to 2008, students expressed a desire for assistance with Internship placements, and preparation for NCE and state certification. AA.3. Formal studies of site supervisors and program graduate employers that assess their perceptions and evaluations of major aspects of the program. Two different surveys (one for employers and a second for site supervisors) were conducted to gather data on external perceptions and evaluations of major aspects of our program. Employers During September 2005, the counseling faculty designed and implemented an employer survey. This employer list represented graduates from 1999-2004. There were approximately 100 graduates on this list. Graduates for whom addresses could be located were sent a letter requesting them to enlist the aid of their supervisor/employer in completing the survey. A month later, a follow-up letter was sent out as a reminder to the graduates. Eleven employers responded to the survey indicating about a 10% response rate. Respondents were asked 51 questions on graduate s knowledge, skills, and attributes on a 5-point Likert scale, from low to poor (1) to exceptional (5). Scores on knowledge ranged from 4.91-3.88. Scores above 4.50 on knowledge included ethical and legal issues, small group counseling, large group counseling, theories of counseling, components of the developmental school counseling model, designing a counseling curriculum, and career counseling. Scores below 4.20 included test interpretation, crisis intervention, consultation, family dynamic, program evaluation, and diagnosis. Scores on skills ranged from 4.89-3.67. Scores above 4.50 on skills were ethical and legal duties, small and large group counseling, individual counseling with adolescents and children, adult individual clients, counseling diverse clients, family and couples counseling, implementing a developmental school counseling model, performing grief and time-limited counseling and career counseling. Scores below 4.20 included test interpretation, crisis intervention, consulting, diagnosing, and planning treatment. Scores on attributes ranged from 4.80-4.40, including overall competence (4.60), responsiveness to feedback/supervision (4.70), dependable/conscientious/responsible

(4.80), collegiality (4.40), professional demeanor (4.40), dedication to counseling field (4.80), engages in professional development (4.50), and adheres to policies (4.70). Site Supervisors During fall 2007, pilot evaluation surveys were distributed to the on-site internship supervisors of current master s students completing internships at school and community mental health agencies Completed surveys for the pilot survey were received from 25 supervisors that work with M.S.Ed. interns (20 schools and 5 community agencies). This represents a 50% response rate for each type of master s level internship site during fall 2007. Overall, evaluations of our interns were very positive on a scale of one ( poor ) to four ( outstanding ). Intern s counseling skills received ratings in the high average range (M= 3.0). The two specific recommendations from community supervisors were about diagnosis: One suggested that interns were knowledgeable of DSM IV but need assistance in application. The second supervisor suggested that interns could benefit from more direct instruction on substance abuse and treatment planning for MISA clients. The school supervisors that responded were all from schools (55% high schools, 45% middle schools) that were diverse in terms of ethnicity, income level, learning styles and languages. The NIU interns received impressive ratings from their school-site supervisors on a scale of one ( poor ) to four ( outstanding ). When asked about intern s level of competency to work individually and in groups with diverse students as well their ability to promote the comprehensive developmental model, 90% of supervisors indicated competence levels of high average or outstanding and 10% rated these skills as average. Site supervisors were asked to identify additional instructional areas that would help future NIU interns better perform their job responsibilities. The areas noted addressed specific topics that needed more attention versus the need for new courses. Following is the list of focal points that supervisors believe should receive more emphasis: college and post-graduation selection strategies/procedures (n=4), test interpretations for students, teachers and parents (n=4), special education paper work (n=2), state standards for school counselors (n=1), grief counseling (n=1), academic interventions (i.e., PBIS, RTI), and various strategies for scheduling high school classes (n=1). Employer/Internship site-supervisor survey The most recent employer/internship site-supervisor survey was conducted in spring 2009, with a 50% response rate. Our graduates received impressive ratings from their supervisors on a scale of one ( poor ) to four ( outstanding ). A sample of the survey comments attest to the strong support of our graduates: NIU students are well qualified and prepared ; great training in theory as well as actual agency/school based knowledge ; your students demonstrate a very high level of

professionalism ; it s great to see motivation for additional opportunities ; very knowledgeable and competent in general population issues ; when possible we only accept NIU students. Advisory Board The faculty meets annually with the Advisory Board to seek input about into the program s mission, objectives, program changes, and changing professional needs. The faculty utilizes feedback from the Advisory Board to consider curricular improvements. In 2008, the Advisory Board received updates on the counseling program s Involvement in the aftermath of the NIU shooting. Additionally, feedback was requested on: The mission statement. Overall, members were satisfied with the mission statement. The only suggestion mentioned was that including references to ethical behavior should be considered. New CACREP Requirement: 60 hr Clinical Mental Health Program. The Advisory Board was asked to provide a list of suggested courses they believed should be included as the curriculum expanded. The major ones courses included grief & loss, comprehensive crisis, substance abuse and addiction, and vocational and college counseling. Additionally, the Advisory Board indicated faculty members should consider separate courses for ethics and multicultural issues. Creative therapies were also suggested as an elective courses (i.e., art, animal assisted therapy). (2008) Assessment Update: Recently revised assessment plans (for master s and doctoral programs) were sent electronically to board members prior to this meeting. Board members indicated the assessment plans were detailed and comprehensive. In addition, they commented that the methods and timelines were appropriate. Further, they were pleased the Advisory Board Report was part of the overall assessment plan. In regards to specific feedback on Doctoral Program mission statement: The Board believed that the term counselor education and supervision should be included in the statement. This term focuses on the specific training that students receive. Additional Comments: Members were pleased that there was movement toward the Ph.D. They encouraged faculty to keep moving on this long awaited goal. (2009) After welcome and introductions, an overview of the Program Review was presented to the board by Dr. Teresa Fisher. This report (conducted every seven years by NIU), includes a thorough perspective on how the master s and doctorate programs compare to

similar programs in Illinois and throughout the U.S. in regard to graduation rates, number of faculty and cost efficiency. Additionally, the report provides feedback about the program from alumni, Internship-site supervisors and employers. The review also includes projections that program faculty would like to obtain within the next five years. The Advisory Board was pleased that the master s and doctorate programs are doing well when compared to other programs. The Board viewed the comments from alumni and others as overwhelmingly positive. Board members cautioned faculty about the stated projections especially since they are not at full capacity. Feedback was also requested on the following: Curriculum Updates Curricular updates were presented for both the master s and doctorate programs. While the master s program had several course additions, the doctoral level program primarily consisted of name changes to existing courses Master s- New 60 hr Curriculum: Advisory Board members were glad that several of the courses they suggested the previous year were included in the new 60 hour curriculum. Proposed Courses for Doctorate: The majority of Advisory Board members considered course modifications for the Ph.D. justifiable. The new course titles seem more appropriate for what the course actually addresses. Faculty Updates -Carole Minor s Retirement- Faculty assured the Board that even though Dr. Minor has retired, one of her signature initiatives will continue, Project 211. (This project teaches undergraduates how to explore careers, while providing counselors-in-training opportunities to enhance their teaching skills). -Debra Pender s Research: Dr. Pender shared research sponsored by her grant with the Division of Mental Health. She has been working with doctoral students to develop evidence informed modules for Community Mental Health (CMH) Counselors (in the field). These computer-based modules are used for professional development by the state. -In her role as Counseling Lab Coordinator, Dr. Pender also updated the Board on her work to obtain Landro Systems for the Lab. (Landro is a digital based software for recording and tagging counseling videos.) The Advisory Board members were excited to learn that NIU was participating in the creation of electronic modules for professional development. This was viewed as an opportunity for NIU to establish a stronger connection with community mental health

agencies. Relationships that can help expand our CMH student population as well as provide internship opportunities. Additional Comments and Recommendations Overall, Advisory Board members believed that the NIU Counseling Program is headed in the right direction regarding the 60 hr curriculum for master s and developing the Ph.D. for the doctoral program. Both of these new perspectives should strengthen the programs. Board members would like to see the Counseling Program place an emphasis on the following: prevention based counseling, on-line learning, cohorts and recruitment that advertises the unique aspects of the program. (2010) Dr. Tollerud provided an overview of major changes in the doctoral program (new mission statement, modify admissions procedures to only one date in the spring each year, and new courses that created more alignment with 2009 CACREP standards). Members unanimously endorsed the mission statement stating that it was clear and reflected the ACA mission statement as well as capturing the definition of counseling. The current doctoral support structure was applauded by members, some of whom wished it existed while they were in the program. It was reiterated that the doctoral program places great emphasis on the five CACREP areas of Counseling, Research, Supervision, Teaching and leadership. Counseling faculty noted that the new 60 hour program for both School Counseling and Community Mental Health is officially in effect, starting fall 2010. Overall, Board members believed that the increase in the number of hours required for the M.S.Ed. will better prepare the students. One particular course cited as a major boost to the program is college counseling. It was revealed that the program now brings school counseling to the level of social work. AA.4. Assessment of student learning and performance on professional identity, professional practice, and program area standards. The analysis of these data lead to the program goal of creating a link between program objectives, student learning outcomes, and the CACREP standards to the core and specialized course assignments. To meet this goal, the faculty modified the program by designing a new method for assessing student outcomes. In the past, rubrics have been created for each assignment and placed in LiveText. LiveText allows for the creation of aggregated reports of data. During spring 2010, faculty assessed learning outcomes by had using a Likert-type scale. During spring 2011, a new on-line procedure for collecting learning outcomes was adopted.

This process is already beginning to happen. For example, through the review of the course core assignments (2006-2007) it was noted that master s students performance ratings in practicum and internship tend to reflect a higher level of counseling skills than the level of theory integration and cognitive complexity reflected in the writing assignments. Strategic changes were implemented in CAHC 501, CAHC 533X, and CAHC 550 to address the deficit. Changes to CAHC 501 (Mental Health) and CAHC 533X (Standardized Testing) included instructional delivery changes such as using problem-based learning, adding web-discussion of course content to assess comprehension and areas of missing information, and small group critical thinking exercises during class. Changes in CAHC 550 (Practicum) included creation of a Live Text based Group Process Note assignment for weekly session processing. Other changes included creating a master Blackboard Course Resource Board for all sections. The board contains exercises to assist counselors-in-training to conceptualize the links between case documentation such as case notes and intakes with core theories learned in core course curriculum. Assessment of the effectiveness of these instructional strategies entailed a review of CAHC 550 Final Paper Assignments in fall 2007 and spring 2008. AA.5. Evidence of the use of findings to inform program modifications. Various assessments have identified key areas that needed improvement in order to observe continuous progress on learning outcomes and overall program development. Following is a summary of how evidence has been used to inform program changes. Programmatic Improvements A new orientation program was developed for master s students and has occurred since 2009. The orientation is a formal three-hour program that describes course requirements, assistantships/fellowship opportunities, and general procedures for completing the master s program. Students have found the annual orientation session to be informative. Faculty members have infused more technology based instruction in their classes (blackboard, web board, avatar-virtual reality, Landro, and hybrid/on-line classes). The Counseling Lab has been updated with advanced software (Landro) that provides increased opportunities for faculty to enhance student s clinical and case conceptualization skills. The Landro software is an excellent tool for training and research. Landro is capable of recording video as a digital file which can be marked (for training examples) during the recording, or afterwards. The professor and the student can spend time analyzing and discussing performance instead of searching through videotapes for specific examples of behavior. New procedures for increasing student participation in the National Counselor Exam (NCE) have been implemented. This will provide an additional external method of assessing student performance on learning outcomes.

Faculty members have increased their attendance at the annual Advisory Board meetings. Program changes and assessment results are shared with Board members and their recommendations are discussed during faculty assessment retreats. Faculty members are in the process of implementing and reviewing assignments designed to improve overall skills in assessment and diagnostic skills. These specific assignments have been included in a core class as well as practicum (clinical experience prior to Internship). In addition to new assignments, faculty members have recently invested in a new electronic case note technique (Empathic) that should help students improve assessment and diagnostic skills. Faculty members have implemented a more formal evaluation process for allowing students to enter practicum and internship. The process includes a thorough review of each student and communication with adjunct faculty who are teaching core courses. The process for remediation of students who are not ready to enter practicum or internship has been formalized and implemented in a more systematic method (i.e., student handbook). New accreditation policies as well as survey results from internship-site supervisors and employers have indicated the need for specific training experiences prior to internship. New courses have been developed to address the deficiencies cited among our students (i.e., multicultural counseling, substance abuse issues in counseling, post-secondary college and career counseling, evidenced informed practices). Counseling faculty are developing plans to have more contact with site-supervisors prior to and during internship placement. Plans under consideration include more written communication, face to face visits, electronic orientations, and workshops (i.e., webinar sessions, blackboard). Faculty members are confident that these plans will clarify expectations and enhance lines of overall communication. The program assessment plan has been modified to include the comprehensive components necessary for CACREP reaccreditation, curricular changes and new assessment methods (i.e., portfolios, focus groups). In the core course, CAHC 501 (Mental Health), a developmental assignment has been added early in the semester that is designed to assist students with identifying the core concepts underlying diagnostic decision-making and links to future clinical practice. This was an attempt to enhance critical thinking skills in case conceptualization and diagnosis/treatment planning have included both individual assessment (final exam) and group assessment in which students conduct a mock staffing for a case study. These assignments were designed to directly address the need for earlier practice with case conceptualization prior to practicum. Data from students enrolled in CAHC 524 (Community Agency: Programs, Issues and Practices) in 2006-2007 indicated students were having difficulties with the final comprehensive program plan assignment. These difficulties included using the literature to write about the historical foundations of an issue, designing programs with realistic

implementation strategies, and writing with organization, clarity and APA form. Based on these findings, the comprehensive program plan assignment has been separated into several parts, so students can receive feedback and have the opportunity to rewrite before turning in their final plan. Additionally, students are being asked to interview community counselors already working in the field about their ideas, so that plans are developed in a more realistic and practical manner. Assessment of the effectiveness of these instructional strategies will entail review data collected in spring 2008. As a result of assessment results obtained through the LiveText rubic, the content of CAHC 570 (Consultation and Management in Developmental School Counseling Programs) has been modified to include a greater emphasis on the differences and distinctions between the three major types of consultation in school counseling. Students are now assessed on whether they can label triadic dependent, collaborative dependent, and collaborative independent consultation types. In CAHC 523 (School Counseling: Programs, Issues and Practices) assessment data indicated that students needed earlier feedback on assignments. In order to accommodate this feedback, the major course project was divided into four components each having a separate due date. Assessment results from the career specialty courses suggested changes to the core courses as well as the career specialty courses. CAHC 511 (Career Counseling) was modified to include two assignments to increase student s critical thinking, case conceptualization and understanding theory to practice integration. Additionally, the curriculum in career counseling assessment was altered so that students would take a core assessment course (CAHC 533x: Standardized Testing) as well as a specific career assessment course (CAHC 575: Assessments in Career Counseling). Assessment results suggested students needed two courses to master this material. As noted previously, a specific assignment has been identified in each core and specialty course and a rubric has been developed for the assignment. Several new faculty members have been hired since the program modifications were implemented. Feedback from these new faculty members has resulted in a revisiting of the core assignments and rubrics. Additionally, a portfolio assignment has been integrated into the internship course. Faculty members continue to work on the integration of finalizing the artifacts and the development of the final portfolio elements. Students are encouraged to keep artifacts from each course, especially the capstone projects. A stronger focus on portfolios has increased their use as a possible assessment for monitoring student performance on learning outcomes for other courses. The faculty has made significant revisions to the process by which students are evaluated for practicum and internship. These revisions have included using more doctoral students and doctoral program graduates to teach the basic skills and practicum class and using more systematic methods seeking feedback from them. The Counseling Lab coordinator meets monthly with practicum instructors to increase the early identification of problematic practicum students.

The syllabi and course outline process has resulted in several curricular changes. Course titles are being revised to better reflect current practices and program objectives. Course descriptions and content are being revised to more clearly integrate the mission statement and program objectives. Prerequisites and course sequencing are also under revision. Follow-Up Studies of Graduates, Site Supervisors and Employers Another significant area of program modification has been in the follow-up studies of graduates, site supervisors and employers. These studies have traditionally been plagued with very low response rates. Strategic efforts have been implemented to design better studies, increase return rates, and increase the validity of the data collected. Currently, focus groups exit interviews are conducted immediately after the conclusion of the final internship class. A site supervisor survey was conducted in fall 2010 to provide the faculty with an opportunity to revisit the survey and increase return rates. Subsequently, the return rates increased because data was collected at the conclusion of the annual workshop for site supervisors. Because site supervisors and employers of graduates are often the same individuals, future plans include inviting both site supervisors and employers to the annual site supervisor s workshop by offering CECs and opportunities for data collection. There have been program changes reflecting an increased use of the Advisory Board. Advisory Board meetings are being conducted on an annual basis with attendance from all available faculty members. Assessment information is more consistently being shared with members and recommendations are being more consistently given to the faculty. The board meetings are scheduled on the same day as the workshop for site supervisors. This allows board members to get continuing education hours, thus increasing participation on the Board from site supervisors. While the number of responses from graduates, site supervisors, and employers may be small, there have been some program changes based on feedback. These changes have included an increased focus on multicultural competencies, increased emphasis on diagnosis, and being more strategic about selecting schools for internship sites that are multicultural in their populations and developmental in their programming. The master s program has experienced several significant changes and new faculty members have been hired. Feedback from the focus group suggests that care must be taken to better inform the students of program changes and provide for continuity of programs.