M.S. & Ed.S. in School Psychology Assessment in the Major Report By Dr. Christine Peterson, Program Director Submitted: October 2014

Transcription

1 M.S. & Ed.S. in School Psychology Assessment in the Major Report By Dr. Christine Peterson, Program Director Submitted: October 2014

2 Table of Contents Description of Methods... 2 Program Disposition Review... 3 Practicum and Internship Student Evaluations... 5 PRAXIS II: Content Test Summary or National School Psychologists Exam Results PRAXIS II: Content Test Summary PRAXIS II: First Time Test Takers Other Information Plan For Communicating Assessment Results Conclusions and Action Plan to Improve Teaching and/or Advisement In the School Psychology Program Goals for School Psychology AIM Report Page 1

3 Description of Methods The following assessment tools and strategies were utilized to determine student performance, progress, and levels of satisfaction in the major: Benchmark I Faculty review/selection of program applicants Transcripts review Acceptance into the School Psychology Program Benchmark II Initial portfolio review Student disposition reviews Review of practicum I evaluations Faculty reviews of student performance, including academic progress, experiential evaluations, and professional conduct Benchmark III Second portfolio review Student disposition reviews Faculty reviews of student performance, including academic progress, experiential evaluations, and professional conduct Review of practicum II evaluations Student scores on the National Certification in School Psychology Exam (NCSP) or Praxis II Benchmark IV Survey of interns regarding their perceptions of program competencies and coursework Final portfolio review Exit Student Survey 2014 Assessment in the Major report data to synthesize and review the above listed data Enrollment Data Year Enrolled Completed School Psychology AIM Report Page 2

4 Program Disposition Review Disposition reviews are conducted at Benchmark II (end of first year in the program) and III (mid-term during second year in the program) for all school psychology students. Domain scores rated as: 1 (unsatisfactory), 2 (minimal), 3 (satisfactory) to 4 (proficient). Total composite scores range from 15 (unsatisfactory) to 32 (proficient). Figure 1 Comparative Benchmark II Means School Psychology AIM Report Page 3

5 Figure 2 Comparative Benchmark III Means School Psychology AIM Report Page 4

6 Outcome of the Disposition Review Results As a whole, the school psychology students displayed satisfactory to proficient attitudes and behaviors within the program (see Fig. 1 and 2). Scores from Benchmark II data exceeded an average of 3.6 (out of 4.0) or higher across all domains. Benchmark III data averaged 3.7 (out of 4.0) or higher across all domains. Conclusion(s): Benchmark II and III Dispositions allow faculty and students to engage in reflective dialogue regarding each student s progress and development toward the mastery of critical skills necessary for success as a grad student and as a professional. Data from suggests that students are progressing appropriately in these important skills domains. Practicum and Internship Student Evaluations Graduate students in the school psychology program complete two (2) supervised field practica totaling 600 hours during their second year in the program, and a culminating 1200 hour internship that is their third and final year in the program. Graduate students completing both Practicum I and Practicum II are evaluated through the use of The Field Practicum Evaluation tool. This newly revised tool utilizes the same items, along a developmental scale, in order to demonstrate and review growth as students advance through each experience. Scoring metrics for each tool are as follows: Unestablished- 1 Observer- 2 Novice-3 Advanced Novice-4 Practicum I Practicum II Observer- 2 Novice-3 Advanced Novice-4 Emerging Practitioner-5 Internship Advanced Novice-4 Emerging Practitioner-5 Established Practitioner-6 Advanced Practitioner-7 School Psychology AIM Report Page 5

7 Outcomes of the Practicum and Internship Student Evaluations a) Practicum Evaluations Results from the 2013 evaluations indicate Practicum I students area scores (means) ranged from 3.5 (Interventions and Instructional Support to Develop Academic Skills) to 3.8 (Legal, Ethical and Professional Practice) on the 4-point developmental scale Evaluation Guide for School Psychology. Scores 3.0 and above are rated in the Novice to Advanced Novice range, which is considered in the accepted standard for Practicum I students on the developmental scale. Practicum I students obtained an average score of 2.9 on the 3.0 Professional Work Characteristics scales. Results from the 2014 evaluations indicate Practicum II students rating scores ranged from 4.6 (Consultation and Collaboration) to 4.9 (Legal, Ethical and Professional Practice) on the 5.0 point developmental scale. Results indicate that practicum students earned above satisfactory scores (4.0 or higher) in all NASP Domains. Professional Work Characteristics ranged 2.8 out of a 3.0 point scale. Scores on the Stage of Development scale (see Figure 6) indicate that most Practicum II students were between the Advanced Novice (4.0) and the Emerging Practitioner (6.0) ratings. Though more data is required to accurately assess trends, it is the hope that with our revised developmental scale and evaluation ratings, student ratings across each applied experience will better match where they are along the developmental progression through the program. School Psychology AIM Report Page 6

8 Figure 3 Practicum I Evaluation Means School Psychology AIM Report Page 7

9 Figure 4 - Practicum II Evaluation Means School Psychology AIM Report Page 8

10 b) Internship Evaluations Results indicate that internship area mean scores ranged from 5.9 (Interventions to Support Academic Skills) to 6.7 (Legal and Ethical Practices) on the 7-point Evaluation Guide for School Psychology for the spring 2014 interns. Professional Work Characteristics ratings averaged 2.9 out of a 3.0 point scale. Results indicate that interns earned above satisfactory to highly satisfactory scores (scores of 6.0 and above) in all Evaluation Guide areas (see Figure 5). The revised Intern Evaluation scale is designed to demonstrate growth from the Emerging Practitioner at the Practicum II stage, to the Established and Advanced Practitioner at the Internship stage. Figure 5 Internship Evaluation Means School Psychology AIM Report Page 9

11 Results indicate that interns scored in the Established Practitioner or above range on all Evaluation Guide areas (Data-based Decision Making; Consultation and Collaboration; Effective Instruction and Development of Cognitive/Academic Skills; Socialization and Development of Life Skills; Student Diversity in Development and Learning; School and Systems Organization, Policy Development, and Climate; Prevention, Crisis Intervention, and Mental Health; Home/School/Community Collaboration; Research and Program Evaluation; School Psychology Practice and Development; Professional Work Characteristics; and Communication Skills). Conclusion(s): Reviews of the Evaluation Guide for School Psychology suggest that UW-Stout graduate students are adequately prepared to enter the field of practice with professional skills for success as practicing school psychologists. We are pleased with the accuracy of the data that our recently revised evaluation tools have provided. PRAXIS II: Content Test Summary or National School Psychologist Exam Results It is a program requirement to take the National Certification in School Psychology (NCSP) exam (PRAXIS II) used by the National Association of School Psychologists (NASP) as requirement for national certification. The passing score established by NASP and recognized by the Wisconsin Department of Public Instruction (DPI) is currently 165. Starting in September of 2014, NASP and the state of Wisconsin will recognize the revised passing score for Praxis II at 145. This reflects a significant revision to the test itself, including new scoring metrics. In addition to the total score received by examinees, the PRAXIS II exam also report scores addressing the following content areas: 1) Data-based Decision Making, 2) Research-based Academic Practices, 3) Research-Based Behavioral and Mental Health Practices, 4) Consultation and Collaboration, 5) Applied Psychological Foundations, and 6) Ethical, Legal, and Professional Foundations. Students and interns can take the national certification exam at any point during their graduate program, preferably prior to the start of their one-year internship. Outcomes of the NCSP Results In , 100% of the students obtained scores of 165 or higher on their first attempt at the Praxis. As such, every student who took the PRAXIS Exam met the passing criteria of NASP and DPI. School Psychology AIM Report Page 10

12 PRAXIS II: Content Test Summary School Psychology Praxis Test Code Content Test from ETS (0401) 10/11 11/12 12/13 13/14 Number of Examinees: Highest Observed Score: Lowest Observed Score: WI Score Needed to Pass: Number with WI Passing Score: Percent with WI Passing Score: 9/11 9/10 11/14 8/8 82% 90% 79% 100% Percent Correct (percentage of items answered correctly) by Area Points UW-Stout Wisconsin National School Psychology Test Category Available 10/11 11/12 12/13 13/14 13/14 13/14 Data-Based Decision Making % 67% 65% 73% 76% 70% Research-Based Academic Pract % 79% 71% 69% 82% 78% Research-Based Behavioral & Mental Health Practices % 72% 71% 81% 84% 80% Consultation & Collaboration % 69% 79% 83% 79% 78% Applied Psychological Foundation % 74% 65% 73% 79% 71% Ethical/Legal & Prof. Foundations % 59% 77% 80% 75% 68% School Psychology AIM Report Page 11

13 Figure 7 Praxis II Domain Percentages = National Average per Domain School Psychology AIM Report Page 12

14 Praxis II First Time Test Takers Beginning in the school year, all teacher preparation program in the state of Wisconsin are required to report on the passage rate on the first attempt of students and graduates of the program on examinations administered for licensure (Act 166, section 17). SPSY Content Test # first time test takers Spring # (and %) passed # first time test takers # (and %) passed 10 9 (90%) 8 8 (100%) Conclusions Overall results indicate UW-Stout s School Psychology program is preparing students to achieve the foundational knowledge necessary to pass the NCSP exam. Review of domain area trends suggest that our students are making gains and/or demonstrating strengths in knowledge related to Data-based Decision-Making, Research-based Mental Interventions, Applied Psychological Foundations and Legal and Ethical Professional Practices. Areas identified for growth include Intervention and Instructional Practices for Academic Skills. Information from PRAXIS II data review will be used to inform on-going program review with focus on continuous improvement in identified areas, as well to inform program development in general. Overall, moderate improvement is observed across all domains but one. Faculty will work to examine and remediate issues related to Research-based Academic Practices domain. School Psychology AIM Report Page 13

15 Other Information a) Portfolio Assessment (Benchmark II and Benchmark III) Students initiate and complete components of a portfolio over the course of the program. The completed portfolio includes evidences or artifacts demonstrating professional competence in all ten (10) NASP s Domains of School Psychology training and practice (same as the Department of Public Instruction content guidelines). A partially completed portfolio (Benchmark II) demonstrating competency in 8 out the 10 domains is submitted to the program faculty for review after completion of the M.S.Ed. and prior to admittance to the Ed.S. in School Psychology Program. At Benchmark III, students re-submit their portfolio to demonstrate competency in all ten (10) domains. Faculty members rate the portfolio materials. Based on a review of the portfolio ratings, academic performance, and professional conduct, a student is either recommended for, or denied admittance to, the Ed.S. degree in School Psychology (for Benchmark II) or Internship (Benchmark III). Outcomes of the Portfolio Assessment A review of the portfolio ratings in 2014 revealed all students produced satisfactory portfolios at Benchmarks II and III. Based on faculty consensus, there is however, a need to further highlight the relevancy of the portfolio for candidates, [particularly during the internship year. Faculty has discussion a number of ways to achieve this. b) Survey of Intern Result In , as in previous years, school psychology interns were surveyed to determine their perceptions regarding their previous coursework. Each intern was asked to respond to questions on a 5-point Likert-style survey (1 = No knowledge or Skill gained to 5 = A lot of knowledge or Excellent skills gained) to assess how much knowledge or skill the student gained in each course. Further, interns are asked to identify the five most important courses in terms of how helpful they were in preparing them for their internship year. They also are asked to identify which courses were not helpful or had overlapping content. Outcomes of the Intern Survey Results from the spring of 2014 survey indicate students believed that they gained above average to a lot of knowledge and skill (x = 4.33) in most program courses. However, students continue to indicate that they gained below average knowledge and skills in the Research Foundations (EDUC 740, x = 2.66). Variable opinions were expressed about the courses related to counseling skills and theory, with some students considering it a value add to the program (x=4.10), while others questioned the relevance of this coursework to their jobs in the field (x=2.8). Additional comments indicated that most students reported they believe they were well prepared for the field of school psychology after taking all program requirements. Students did have additional feedback related to the nature of research expectations and timelines in the program. School Psychology AIM Report Page 14

16 Conclusion(s): Intern survey responses continue to suggest the need for further review of the EDUC 740 Research Foundations course. Although content related to research is considered foundational knowledge for Masters and Ed.S. Thesis courses, more information is needed to determine whether the course could emphasize more theoretical and research-related information related to be more focused on student research. Some narrative feedback articulated a desire for more statistical analysis content in research foundations coursework, as well as a desire to have content tailored more toward statistics and analysis of research designs for action research (single-subject, reversal designs, etc). This issue becomes even more relevant with the implementation of our Applied Research option for Ed.S. candidates. We currently do not have a course that explicitly addresses research design and methods for applied research. The feedback related COUN 788 Counseling Process Lab in particular, is consistent with years past. Efforts to enhance students understanding and/or appreciation for its relevance to their professional skills is on-going, and efforts to c) Program Advisory Committee Results The program faculty members meet with the School Psychology Program s Advisory Committee (PAC) two times a year. Colleagues from related programs on campus, as well as practitioners and stakeholders in the community meet to discuss updates in the program and solicit information from the committee about program needs and future goals. Current program consists of 7 colleagues, including practitioners in the field, representatives from higher education, and the state department (DPI). The PAC has been consulted as we have continued to revise and update the program in order to align with the new NASP standards. Their input continues to be valuable as we further develop and evolve our program. Outcome(s) of the Program Advisory Committee Results: Most recent PAC focus continued to relate to revisions to the program. Focus of recent meetings has been on NASP accreditation efforts. New discussions will include program revisions and recommendations from the NASP report. School Psychology AIM Report Page 15

17 Plan for Communicating Assessment Results The School Psychology Program faculty meets on a bi-weekly basis to discuss program issues and plan for the future. This Assessment in the Major report will be disseminated to all members of the program faculty, and the results will be discussed by the program faculty at an upcoming meeting. This guide is a valuable resource for faculty to use in guiding program revisions. Furthermore, this report will be disseminated to the Program Advisory Committee (PAC) in the fall of the academic year. Conclusions and Action Plan to Improve Teaching and/or Advisement in the School Psychology Program Summary and review of all available data related to the School Psychology programs at UW-Stout, the following goals are targeted for the year: Targeted Goals for : A. Complete and seek full approval during the 2013 NASP review cycle. a. Update: NASP review was completed and report sent Feb, UW-Stout received full approval through fall of B. Program faculty will work in collaboration to comprehensively review research expectations, timelines and associated coursework for School Psychology, and revise accordingly. a. Update: Faculty has developed a Research course that better fits both the thesis and the action research methods, and will be incorporating that into a program revision proposal this year (see Goal B). C. Pending NASP feedback report, program faculty will work to continue revising our program in accordance with NASP guidelines and standard for graduate preparation. a. Update: Faculty worked over the summer to complete a program revision, based on a variety of sources, including NASP accreditation report, student feedback, and program advisory. This includes new course proposals and course sequence revisions. We will seek approval for this revision during the academic year. D. Program faculty will work with SOE support members to transition to an E-portfolio process for students. a. Update: Continued goal for school year E. Enhanced support for students in preparation for taking the Praxis exam. This may include: providing test prep materials, available through the Resource Library, and/or in coordination with the Praxis Lab. a. Update: Praxis conversations and prep are more clearly embedded into Practicum Seminar schedule for the fall semester. Faculty attended a Praxis revision meeting this summer, in order to get more information about upcoming changes to the Praxis II exam. School Psychology AIM Report Page 16

18 Goals for Based a variety of sources (NAPS review, program advisory, student feedback), program faculty will work to implement a program revision that aims for continuous improvement and efficiency. 2. In coordination with Goal #1, program faculty will work in collaboration to comprehensively review research expectations, timelines and associated coursework for School Psychology, and revise accordingly. 3. Faculty will work with Grad School and Stout Marketing to develop a more concrete and tangible marketing plan for the School Psychology program. 4. Program faculty will work with SOE support members to transition to an E-portfolio process for students. Respectfully submitted, Christine R. Peterson, Ph.D. Assistant Professor Program Director for School Psychology University of Wisconsin-Stout School Psychology AIM Report Page 17