Special School District Program Evaluation for Special Education Schools. Sheri Menscher Dr. Paul Bauer

Transcription

1 Special School District Program Evaluation for Special Education Schools Sheri Menscher Dr. Paul Bauer September 2005

2 TABLE OF CONTENTS List of Tables...iii List of Appendices...iv Executive Summary...v Chapter I: Introduction...1 Background and Purpose...1 Focus for the Program Evaluation...1 Design of the Report...2 Chapter II: Program Description...3 Criteria...3 Curriculum...3 Service and Placement Options...3 Enrollment Data...5 Chapter III: Literature Review...6 Missouri School Improvement...6 School Improvement Process...7 School Effectiveness...7 District Leadership...8 Principal Leadership...8 Methods for Monitoring Student Achievement...8 Analysis of Student Data...9 Chapter IV: Methodology...10 Process...10 Focus of the Evaluation...10 Methods for Data Collection and Analysis...11 Presented to Board of Education i

3 Chapter V: Results...14 Library Media Centers...14 Facilities and Safety...15 Staff Surveys...16 Student Surveys...18 Parent Surveys...21 Self-Study Results...25 Cost Analysis...27 Chapter VI: Summary and Recommendations...28 Summary...28 Limitations...29 Recommendations...30 References...32 Appendix...34 Presented to Board of Education ii

4 LIST OF TABLES Table School Enrollments...5 Table 2 Data Sources and Data Collection Methods...11 Table 3 Comparison of 2001 Results with 2005 Results for Library Media Centers...15 Table 4 Comparison of 2001 Results with 2005 Results for Facilities and Safety...16 Table 5 Strength Areas on Staff Survey...17 Table 6 Need Areas on Staff Survey...17 Table 7 Staff Survey Results...18 Table 8 Strength Areas on Elementary Student Survey...19 Table 9 Progressing Areas on Elementary Student Survey...19 Table 10 Secondary Students who Completed the Survey by School...20 Table 11 Strength Areas on Secondary Student Survey...20 Table 12 Secondary Student Survey Results...21 Table 13 Strength Areas on Parent Survey...22 Table 14 Progressing Areas on Parent Survey...22 Table 15 Need Areas on Parent Survey...23 Table 16 Parent Survey Results...24 Table 17 Scoring of Compliance with MSIP Standards...25 Table 18 Cost per Pupil...27 Presented to Board of Education iii

5 LIST OF APPENDICES Appendix ES-1 Appendix 1-1 Appendix 4-1 Appendix 4-2 Appendix 4-3 Appendix 4-4 Appendix 4-5 Appendix 4-6 Appendix 4-7 Strengths-Needs Chart Committee Members Staff/Administrator Survey Elementary Student Survey Secondary Student Survey Parent Survey Open Forum Flyer Self-Study Self-Study Scoring Guide Appendix MSIP Final Report Summary Appendix 5-2 Safety Drills Record for Presented to Board of Education iv

6 EXECUTIVE SUMMARY Introduction Special School District is committed to program evaluation as a foundation for the continuous improvement process. The program evaluation framework approved by the Board of Education (2003) guides Special School District staff in conducting program evaluation activities to measure, analyze and effectively manage special education services and operations. The Special Education Schools Program Evaluation was a formative process occurring from July, 2004 through May, The schools evaluated were Ackerman, Bridges, Career Training Program, Juvenile Detention Center, Lakeside, Litzsinger, Neuwoehner, Northview, Project Learn and Southview. Since the special education services in the Special Education Schools are evaluated through the discipline specific program evaluations, it was determined that the purpose of the program evaluation for Special Education Schools would be to evaluate the schools operational management, achievement impact for students and general leadership capacity. The state of Missouri will conduct its review of the Special School District in May of This program evaluation paralleled the Missouri School Improvement Program (MSIP) process and was planned to serve as a preassessment for selected resource, process and performance standards. The program evaluation report details the process, results and recommendations that will guide improvement activities for the Special Education Schools during the school year. Stakeholders were engaged as survey respondents, steering committee members, and work group members. The committee membership included parents, students, graduates, teacher-level staff, paraprofessionals, administrators and community members (Appendix 1-1). The questions posed by the committee and approved by the Board of Education were designed to guide the work of the committee through the evaluation process. The focus of the program evaluation for Special Education Schools was to answer three questions approved by the Board of Education: 1. What is the status of the concerns identified as a result of Special School District s 2 nd cycle MSIP review in the Special Education Schools? 2. To what degree are the Special Education Schools meeting the 3 rd cycle MSIP expectations? 3. What actions need to be taken during the school year to meet the 3 rd cycle MSIP expectations? Presented to Board of Education v

7 Related Literature Review The state of Missouri identifies best practice for schools through the Missouri School Improvement Program (MSIP) Standards and Indicators. Four documents were utilized for this portion of the related literature review. The documents were: 1) Missouri School Improvement Program, Third-Cycle Procedures Handbook (Appendix 3-1), 2) Missouri School Improvement Program, Integrated Standards and Indicators Manual (Appendix 3-2), 3) Missouri School Improvement Program, District Response to the Standards (Appendix 3-3), and 4) Report Writing Form for Third Cycle, Instructional Design and Practices Report Appendix 3-4). These documents organize the standards for review into three sections labeled as Resources, Process and Performance. The Resource area reviews information submitted by each district on the October Core Data cycle. The Resource Standards are considered basic requirements that all districts must meet and are quantitative in nature. For the purposes of this program evaluation, the focus is on the Course of Study standards for elementary, middle and high schools. The MSIP Process Standards incorporate multiple criteria to look at the instructional and administrative structures used in schools. These criteria are of a qualitative nature and are assessed through on-site review by the MSIP team. This program evaluation will report on the implementation of the standards that address curriculum, assessment and instruction as identified through the Self-Study narrative completed by each principal. Additional process standards implementation data was gathered through the staff, student and parent surveys. The Performance Standards information is collected annually and analyzed for most districts through the Annual Performance Report (APR). At the time this report was written, Special School District has not received their final APR. Additionally, a literature review in the area of leadership for school improvement and student achievement was conducted to identify effective practices for meeting standards of operation and achievement. The structure of the MSIP model, continuous improvement processes and work of the administrators in the Special Education Schools to meet the MSIP standards is validated by the professional literature. This literature review also yielded additional strategies for the administrators to consider when identifying school improvement activities for the future. In 2005, Chrisman found improved student achievement to be the product of how well a school operates and dependent upon the quality of leadership. The author further detailed what leadership was needed from teachers, principals and district Presented to Board of Education vi

8 office personnel. Instructional programs and practices implemented at the district and school level that contribute to success were detailed. Waters, Marzano and McNulty detailed leadership responsibilities of principals and the correlation with student achievement in Situational awareness and serving as a change agent to intellectually stimulate the staff were rated as highest. Lowest rankings were assigned to providing contingent rewards, visibility and direct involvement in curriculum and instruction practices. Lambert (2005) identified three phases of development in leadership skills that a school needed for improvement to occur. These phases were labeled instructive, transitional and high capacity. Lambert also cited personal attributes and behaviors that principals needed to display to sustain improvement in their schools. The attributes and behaviors could be characterized as leading by example, working as a member of a learning community and using formal authority to influence change beyond the school. In 2003, Elmore defined school improvement as a developmental process that proceeds in stages. He further indicated that performance often lags behind practice and that most of the knowledge about school improvement is found in the schools with a history of low performance. Jerald (2003) found that leading districts have adopted some form of common district-wide curriculum, instructional programs or detailed achievement targets whereas lower performing systems have not. State and district assessments are often criticized when used as the sole method to monitor student achievement. Research on student progress monitoring was analyzed by Fuchs and Fuchs in They concluded that the use of systematic methods to track student progress in order to identify students who require additional or different forms of instruction resulted in higher student achievement. Student progress monitoring with probes was first used by special educators and appeared in professional literature as early as The method is now being applied to track literacy and other curricular content progress in both general and special education. Sharkey and Murname (2003) recommended some strategies for overcoming the obstacles schools face with disaggregating achievement data. The authors summarized their work with schools in Boston to help teachers and administrators learn from student assessment results. Sharkey and Murname provided specific strategies for overcoming obstacles in the areas of technology, knowledge and opportunity. Presented to Board of Education vii

9 Methodology The Final Report for the MSIP Review of 2001 identified areas of concern in Resources, Process and Performance that were applicable to the Special Education Schools (Appendix 5-1). Specifically, Resource and Process concerns were identified for Library Media Centers (LMC). Additional Process concerns were identified in 2001 in the areas of Facilities and Safety. Performance concerns at the middle and high school level were identified as were concerns regarding reading achievement. To answer the first program evaluation question, the identified 2001 MSIP concerns were compared to two current evaluation documents: a) the 2006 MSIP Standards and the results of the Standard Program Evaluations for Library Media Centers (LMC) which the Board of Education approved in the spring of 2005 and b) the 2006 MSIP standards and the results of the Standard Program Evaluations for Facilities and Safety approved by the Board in the winter of As mentioned previously, the performance concerns will be addressed at a later time. In order to gather current information on the Resources and Process Standards sections of the evaluation and answer the second and third program evaluation questions, surveys that followed the organization of the MSIP advance questionnaire were distributed to parents, students, staff and administrators in the Special Education Schools, the Career Training Program and the Court Programs. The survey data was compiled and analyzed by the Coordinator of Program Evaluation. A second level of analysis occurred by the Director of Special Education Schools. After a review by all of the school administrators, the principals of the schools met with their staff to review and further analyze the survey results. Finally, the program evaluation committee reviewed and provided feedback on the analysis of the results. The principals of each school also completed a written self-study on the relevant state-required and optional Process Standards and submitted it to the director for review to further answer the second and third program evaluation questions. The director scored the self-study against the MSIP Standards and sent it back to the principal for additions/clarifications as needed. The revised self-study from each school was compiled into one document representing the work of all schools by the director and reviewed and scored by the steering committee using the MSIP scoring guide. Additional areas of revision were identified by the committee and subsequently completed by the principals and director. A public forum was advertised through flyers posted at all Special Education Schools and disseminated to all students, families, staff and community members. The forum was also advertised through the district web site, voice mail, the conference Presented to Board of Education viii

10 folder on SSD first class, flyers posted at central office and disseminated through the PAC. No one attended the April 14 forum. A review of available demographic information, cost analysis and comparison completed the methodology. Results The committee members met to review, organize and summarize the program evaluation data. Through discussion and analysis, the committee identified areas of strength and concern that were noted across multiple data sources. (Appendix ES-1) Status of Concerns Identified in May, 2001 The following concerns, which are applicable to the Special Education Schools, were identified through the 2 nd cycle MSIP review: Library Media Centers, Accessible entrance at Page Bridges, Cleanliness and instructional space at Northview, Earthquake drills conducted twice yearly at all schools and Student achievement. Results of the standard program evaluation for Library Media Centers indicate that five of the six concerns specific to the Special Education Schools had been rectified. The outstanding concern regarding the LMC procedural manual is due to be completed by June, An entrance that is accessible for persons with disabilities at the Page Bridges site has been added since Each Special Education School conducts two earthquake drills per school year. Space utilization for instruction at Northview continues to be at capacity and additional classroom space at a site across the street from Northview was rented during the school year. In the summer of 2005, the building where space was rented was sold to the Ferguson-Florissant school district and the space was no longer available. The cleanliness concerns at Northview appear to be rectified based upon survey results indicating that 76% of the staff agrees or strongly agrees that the school building is in good condition. Student performance concerns were identified in 2001 in grades 6-8, 9-11 and for reading achievement. At the time this report was written, the Department of Elementary and Secondary Education was reviewing Special School District s performance data. The Department had not finalized the criteria needed to provide Special with a final Annual Performance Report. Therefore, no results can be reported. Staff, students and parents were surveyed in the Fall of The return rate for the staff surveys was 80%, the student surveys return rate was 59% and the parent surveys return rate was 28%. The findings are reported below. Presented to Board of Education ix

11 Strengths Staff survey data indicates strengths in the areas of board, climate, school improvement, instruction, parents as partners and professional development expectations. The student survey identified strengths in climate, instruction and parents. Parents of students attending the Special Education Schools cite achievement, board, climate, curriculum, facilities, instruction, parental involvement and resources as strengths. Results of the self-study indicate additional strengths in student motivation, school-based professional development and reading instruction. Concerns Staff survey data indicates relative concerns in the area of analysis of achievement data for subgroups, curriculum input, resources procurement and allocation, cultural diversity awareness and teaching research skills. Survey results for students indicate concerns with climate and course offerings. The only concern parents of students at the Special Education Schools identified was that the community did not provide enough money for the schools to do their job. A concern identified through the Self-Study was needed revisions to the District s Assessment Plan to reflect current practices. Administrative follow up with staff on the survey results identified that staff did not know how to interpret and/or answer some of the survey questions. This uncertainty may have affected the staff survey results. Cost Analysis Since the final calculations regarding the cost per pupil for the school year was not available when this report was written, the cost per pupil was calculated by dividing all direct costs attributed to a school in the FY 04 budget by the number of students attending the school on the last Wednesday of September, The cost per pupil ranged from a low of $15,922 in the Court Programs to a high of $45,065 in the Bridges Program. The Court Programs have a low per pupil cost because there are no transportation or facility costs to Special School District at these sites. The Bridges site s high per pupil cost and the benefits associated with the cost was reviewed by the Board in January, Presented to Board of Education x

12 Recommendations The full committee analyzed the survey, self-study and follow-up results to determine areas of need. These recommendations address the identified needs. Curriculum and Assessment 1. Develop a process for staff, parents and students to have input into the development and revision of the curriculum. Provide opportunities for SSD teachers who have been involved in the development of the curriculum to share this involvement and knowledge with colleagues. 2. Revise the District Assessment Plan to reflect current practices and include strategies for assessing the Show-Me Standards not assessed through MAP. 3. Schedule quarterly staff reviews of disaggregated achievement and performance data. Use this data to make decisions regarding instruction and evaluate the effectiveness of the adjustments. Professional Development 1. Provide staff training regarding how the curriculum is revised and aligned with the Show-Me Standards and MAP. 2. Provide ongoing training on the teaching of research skills and using technology for instruction. 3. Provide staff development on how to use a variety of assessment data (longitudinal, demographic, disaggregated, diagnostic, surveys) to support districtwide decisions about curriculum and instruction. 4. Continue professional development in the area of diversity awareness and increase implementation of differentiated strategies. Resources 1. A process should be developed at each secondary school for meeting with students and planning the courses students will take. 2. Each high school should complete a course credits chart. Presented to Board of Education xi

13 3. Each principal should review teacher schedules to assure minimum number of minutes for instruction and course offering requirements are met. 4. Each principal should conduct a building-level review of the school s master schedule at the beginning of each semester and submit a copy to the director. A listing of exploration and elective classes indicating that the minimum number of minutes for instruction and course offering requirements are met should be included. 5. Continue to involve staff and parents in acquiring and allocating resources. School Improvement Process. 1. Continue to use the School Improvement Process and Positive Behavior Support Model to focus efforts and increase climate perception data. 2. Investigate the discrepancy between students perceptions regarding climate when compared to those of staff and parents. Identify strategies to address this discrepancy. Limitations The data collected for this evaluation provided the committee with feedback on the status of the Special Education Schools continuous improvement process. In the process of data analysis, the committee noted limitations that may have affected the reported results. These limitations should be taken into consideration when interpreting the results and designing future program evaluation activities. 1. Since application of the MSIP standards is based upon many factors, the evaluation of the committee may not match how the MSIP team interprets and evaluates the standards. 2. The Resources and Performance standards are quantitatively scored by DESE. The parameters for this scoring are unknown at this time. 3. No site observations were conducted by an outside team to validate the Self- Study findings. Presented to Board of Education xii

14 CHAPTER I INTRODUCTION Background and Purpose Special School District is committed to program evaluation as a foundation for the continuous improvement process. The program evaluation framework approved by the Board of Education (2003) guides Special School District staff in conducting program evaluation activities to measure, analyze and effectively manage special education services and operations. The Special Education Schools Program Evaluation was a formative process occurring from July, 2004 through May, The schools evaluated were Ackerman, Bridges, Career Training Program, Juvenile Detention Center, Lakeside, Litzsinger, Neuwoehner, Northview, Project Learn and Southview. Since the special education services in the Special Education Schools are evaluated through the discipline specific program evaluations, it was determined that the purpose of the program evaluation for Special Education Schools would be to evaluate the schools general leadership capacity, operational management and achievement impact for students. The state of Missouri will conduct its review of the Special School District in May of This program evaluation paralleled the Missouri School Improvement Program (MSIP) process and was planned to serve as a preassessment for selected resource, process and performance standards. The program evaluation report details the process, results and recommendations that will guide improvement activities for the Special Education Schools during the school year. Stakeholders were engaged as survey respondents, steering committee members, and work group members. The committee membership included parents, students, graduates, teacher-level staff, paraprofessionals, administrators and community members (Appendix 1-1). The questions posed by the committee and approved by the Board of Education were designed to guide the work of the committee through the evaluation process. Focus for the Program Evaluation The focus of the program evaluation for Special Education Schools was to answer three questions approved by the Board of Education: Presented to Board of Education 1

15 1. What is the status of the concerns identified as a result of Special School District s 2 nd cycle MSIP review in the Special Education Schools? 2. To what degree are the Special Education Schools meeting the 3 rd cycle MSIP expectations? 3. What actions need to be taken during the school year to meet the 3 rd cycle MSIP expectations? Design of the Report The report documents the review of current literature and the methodology used to evaluate the program. The results and discussion of data is based upon standards and indicators that have been identified through a literature review of the Missouri School Improvement Program (MSIP) documents. Limitations of the program evaluation are addressed. Recommendations of the evaluation team are based upon a comparison of the results and the findings of the literature review. The committee has developed action plans to serve as a guide for the implementation of any recommendations that the Board of Education approves. Presented to Board of Education 2

16 CHAPTER II PROGRAM DESCRIPTION Criteria This program evaluation focused on three clusters: a) the public separate day schools operated by the Special School District Ackerman, Bridges, Litzsinger, Neuwoehner, Northview and Southview; b) the schools attended by students in residential court placements Juvenile Detention Center and Lakeside and the courtassigned alternative program Project LEARN and c) the Career Training Program which is comprised of nineteen worksites throughout the area. Curriculum The curriculum for all of these students is the general education curriculum of the Special School District. The curriculum is implemented for each student in accordance with the provisions of the student s IEP. Most students IEPs note extensive adaptations and the students work from a modified curriculum. Service and Placement Options The individual needs of each child are considered by the IEP team to determine educational programming, services and placement. Educators and parents of students work together to consider the best location and services to meet the student s educational needs. The team follows three basic guidelines: a) maintaining the student in the least restrictive setting where services can meet the special education needs; b) considering all service delivery options before finalizing placement decisions; and c) considering carefully the student and family variables along with evaluation information in making educational decisions. Given these guidelines, educators and parents consider the following placement options: 1. Outside Regular Class less than 21 percent of day: Children with disabilities who receive special education and related services outside the regular classroom for less than 21 percent of the school day. Presented to Board of Education 3

17 2. Outside Regular Class at least 21 percent/no more than 60 percent: Children with disabilities who receive all of their special education and related services outside the regular classroom for at least 21 percent but no more than 60 percent of the school day. 3. Outside Regular Class more than 60 percent of day: Children with disabilities who receive all of their special education and related services outside the regular classroom for more that 60 percent of the school day. This category does not include children who received education programs in public or private separate day or residential facilities. 4. Public Separate (Day) Facility: Children with disabilities who receive all of their special education and related services for greater than 50 percent of the school day in public separate facilities. 5. Private Separate (Day) Facility: Children with disabilities who receive all of their special education and related services, at public expense, for greater than 50 percent of the school day in private separate facilities. 6. Public Residential Facility: Children with disabilities who receive all of their special education and related services for greater than 50 percent of the school day in public residential facilities. 7. Private Residential Facility: Children with disabilities who receive all of their special education and related services, at public expense, for greater than 50 percent of the school day in private residential facilities. 8. Homebound/Hospital: Children with disabilities who receive all of their special education and related services in hospital or homebound programs. Presented to Board of Education 4

18 Enrollment Data The table below contains enrollment information for each school from the school year as compared to enrollment data for September, 2003 which was the basis for the cost analysis. Enrollment is in a constant state of flux due to IEP team decisions, decisions of the Courts and the transience of students. Enrollment typically increases in the Special Education Schools as the school year progresses. Table 1. School enrollment data for September, 2003 as compared to enrollment trends. School 9/24/03 9/29/04 12/1/04 6/1/05 Ackerman Bridges Career Training Program Juvenile Detention Center Lakeside Litzsinger Neuwoehner Northview Project LEARN Southview The Missouri Department of Elementary and Secondary Education publishes a Special Education District Profile annually for each district in the fall of the school year using the December 1 census data from the previous year. The profile reports contain twelve tables compiling core data entered for the school year reporting cycle. The tables are based on five general reporting areas: 1. School Age Child Count Data 2. Early Childhood Child Count Data 3. Missouri Assessment Program Data 4. Discipline Incidents Data The Special Education District Profile is one of the data sources used to evaluate performance goals and indicators, compare local data to Missouri data, and provide information for program evaluation through special education monitoring of performance data. At the time this report was written, the Department of Elementary and Secondary Education and Special School District were working to accurately report the data. Presented to Board of Education 5

19 CHAPTER III LITERATURE REVIEW During recent years, national attention has focused on schools and their need to be accountable to the public. At first, this accountability took the form of school districts reporting on how they used the resources they were allocated based upon standards set by the district or state. Now this accountability is focused on the results that school districts achieve using their allocated resources. These results are often identified as performance and the criteria or standard that must be achieved is typically determined by an external source such as the state or federal government. The results a district attains are compared to the results of other districts and states. School districts future funding can be dependent upon the results they achieve. In addition to standards of accountability, a growing body of work has focused on the steps school leaders should take to assure that schools produce the results required. (Elmore, 2003; Mintrop 2003; Darling-Hammonds, 1997; Barth, 1990;) Qualities of effectiveness for school leaders and schools have been identified and analyzed by numerous authors. The structure of the Missouri School Improvement Program (MSIP) model, continuous improvement processes and work of the administrators in the special education schools to meet the MSIP standards is validated by the professional literature. This literature review will detail the standards and indicators of quality utilized for the Special Education Schools Program Evaluation and will report on the finding regarding school leadership that leads to higher student achievement. The review will also identify effective practices for meeting the standards and will yield additional strategies for administrators to consider when identifying school improvement activities for the future. Missouri School Improvement The state of Missouri identifies best practice for all schools through the Missouri School Improvement Program (MSIP) Standards and Indicators. Four documents were utilized for this portion of the related literature review. The documents were: 1) Missouri School Improvement Program, Third-Cycle Procedures Handbook, 2) Missouri School Improvement Program, Integrated Standards and Indicators Manual, 3) Missouri School Improvement Program, District Response to the Standards, and 4) Report Writing Form for Third Cycle, Instructional Design and Practices Report. Presented to Board of Education 6

20 These documents organize the standards for review into three sections labeled as Resources, Process and Performance. The Resource area reviews information submitted by each district on the October Core Data cycle. The Resource Standards are considered basic requirements that all districts must meet and are quantitative in nature. For the purposes of this program evaluation, the focus for the evaluation of Resources is on the Course of Study standards for elementary, middle and high schools. The MSIP Process Standards incorporate multiple criteria to look at the instructional and administrative structures used in schools. These criteria are of a qualitative nature and are assessed through on-site review by the MSIP team. This program evaluation will report on the implementation of the standards that address curriculum, assessment and instruction. Additional process standards implementation data was gathered through the staff, student and parent surveys and through the Self- Study completed by each principal. The performance standards information is collected annually and analyzed for most districts through the annual performance report (APR) prepared by the Department of Elementary and Secondary Education (DESE). At the time this report was written, Special School District had not received a final APR. School Improvement Process In 2003, Elmore defined school improvement as a developmental process that proceeds in stages. He further indicated that performance often lags behind practice and that most of the knowledge about school improvement is found in the schools with a history of low performance. Jerald (2003) found that leading districts have adopted some form of common district-wide curriculum, instructional programs or detailed achievement targets whereas lower performing systems have not. School Effectiveness In 2005, Chrisman found improved student achievement to be the product of how well a school operates and dependent upon the quality of leadership and the effectiveness of instructional programs and practices. The author further detailed the leadership needed from teachers in areas such as collaboration, action research and professional development. Principal effectiveness was typically noted in schools where the principal had served for at least the last three years. Principals at successful schools were more likely to create time for teachers to collaborate and provided teachers with structured support. Chrisman also found these same principals to be comfortable with using data to make changes when student achievement had not increased. Presented to Board of Education 7

21 District Leadership In addition to school-based practices, instructional programs and practices implemented at the district level that contribute to school success were detailed by Chrisman. These district-wide practices included professional development focused on pedagogy, disaggregating assessment data by teacher and student and use of benchmark assessments to track a student s learning and adjust instruction accordingly. Principal Leadership Waters, Marzano and McNulty detailed leadership responsibilities of principals and the correlation with student achievement in Situational awareness and serving as a change agent to intellectually stimulate the staff were rated as highest. Lowest rankings were assigned to providing contingent rewards, visibility and direct involvement in curriculum and instruction practices. Lambert (2005) identified three phases of development in leadership skills that a school needed for improvement to occur. These phases were labeled instructive, transitional and high capacity. During the instructive phase, a principal works to focus the staff on results, solves difficult problems and establishes structures and processes to engage the staff in articulating the beliefs that become the culture of the school. The transitional phase follows. During this phase, the principal releases some authority because the staff has assumed more responsibility and the principal continues to support and coach the staff to follow the vision and processes established. Lambert notes that the transitional phase is often the most difficult for a school due to the wide range of teachers leadership development. The high capacity phase is described by Lambert as the ultimate goal for schools. In this phase, the principal focuses on facilitation so that the teachers initiate actions and identify crucial questions about student learning. Lambert also cited personal attributes and behaviors that principals needed to display to sustain improvement in their schools. The attributes and behaviors could be characterized as leading by example, working as a member of a learning community and using formal authority to influence change beyond the school. Methods for Monitoring Student Achievement Bernhardt (2005) states that schools must use data to improve student learning. The challenge for educators is to identify the types of data to use and the purposes of that data. State and district assessments are often criticized when used as the sole method to Presented to Board of Education 8

22 monitor student achievement (Abrams and Madaus, 2003; Pedulla, 2003). Olson (2005) indicates that assessment should provide data that inform instruction and identify needed curriculum adjustments. Again in 2005, Hall reported on a set of assessment parameters that educators were using to guide instructional decisions and evaluate program effectiveness and student growth. The parameters are benchmarks that assist the staff in identifying the instructional focus for students who are new to the school. Research on student progress monitoring, a method to track individual student growth, was analyzed by Fuchs and Fuchs in Their analysis concluded that the use of systematic methods to track student progress in order to identify students who require additional or different forms of instruction resulted in higher student achievement. Student progress monitoring with probes which are brief, easily administered measures that are used weekly, biweekly or monthly was first used by special educators and appeared in professional literature as early as To implement student progress monitoring, the teacher determines a student s current performance level on skills that the student will be learning that school year, identifies achievement goals that the student needs to reach by the end of the year and establishes the rate of progress the student must make to meet those goals (Safer & Fleischman, 2005). The method is now being applied to track literacy and other curricular content progress in both general and special education. Analysis of Student Data Sharkey and Murname (2003) summarized their work with schools in Boston to help teachers and administrators learn from student assessment results. These authors recommended some strategies for overcoming the obstacles schools face with disaggregating achievement results. Specific strategies such as designing databases that principals and teachers can easily access and manipulate to look at data were detailed. They recommend that teams from schools with expertise in analyzing assessment results demonstrate to teachers from other schools what they have done and how it contributed to instructional improvement. Lastly, it is recommended that time be built into the schedule for data analysis and teachers should be provided with processes to use when discussing their students work such as protocols and question-formulation techniques. Olson (2005) recommends the use of assessments that deliver results that can lead to quick action and establish growth targets for each student that will provide data teachers can use to evaluate their own effectiveness with individuals and groups of students. Presented to Board of Education 9

23 CHAPTER IV METHODOLOGY The focus of this chapter includes the procedures utilized during the program evaluation process and the methods of data collection and analysis used in the program evaluation of the Special Education Schools. Process The program evaluation process involved stakeholders of the Special Education Schools. A committee was established for collaborative work on the program evaluation tasks. Committee members included students, a graduate, parents, a paraprofessional, teacher-level staff, community members, principals and a director. The committee met on a monthly basis to gather, summarize and analyze the data. Next steps for the committee included the development of recommendations and action plans. The chair was responsible for writing the report. Focus of the Evaluation The focus for this program evaluation was targeted Resources, Processes and Performance of the Special Education Schools of Special School District as identified by the 3 rd Cycle MSIP Standards. Presented to Board of Education 10

24 Methods for Data Collection and Analysis There were eight methods used to collect data. The data collection methods used in addressing the focus questions approved by the Board of Education are noted in Table 2. Table 2: Evaluation Focus Questions and Data Collection Methods Special Education Schools Program Evaluation Guiding Questions 1. What is the status of the concerns for special education schools identified as a result of Special School District s 2 nd cycle MSIP review? 2. To what degree are the special education schools meeting the 3 rd cycle MSIP expectations? 3. What actions need to be taken during the school year to meet the 3 rd cycle MSIP expectations? Literature Review Document Review Self- Study Data Collection Methods Staff / Admin. Survey Parent / Guardian Survey Student Survey X X X X Public Forum Cost Analysis X X X X X X X X X X X X X Staff / Administrator Survey The staff / administrator survey was sent to all administrators and teacher-level staff working in Ackerman, Bridges, Career Training, Juvenile Detention Center, Lakeside, Litzsinger, Neuwoehner, Northview, Project Learn and Southview schools. The survey was conducted during the month of October. The survey was sent to 250 staff and 203 were returned. The return rate was 80%. The respondents were asked to rate their perception on 92 items regarding the MSIP process standards using a five-point Likert scale with a rating of 1 denoting strongly disagree and a rating of 5 denoting strongly agree. (Appendix 4-1) Student Survey Two student surveys were conducted in October of Surveys were completed by 36 elementary students in grades 3-6. (Appendix 4-2) A total of 333 students in grades 7 and above completed the secondary student survey. (Appendix 4-3) This resulted in a 59% return rate. Presented to Board of Education 11

25 Parent / Guardian Survey A parent survey was distributed to all parents/guardians of students attending the schools at Open Houses held in the fall of (Appendix 4-4) Surveys were sent home to those parents who did not attend. A total of 1,130 parent/guardian surveys were distributed and 322 were completed. The return rate was 28%. Public Forum A public forum was held to gather input from stakeholders who had not previously had an opportunity to provide input into the program evaluation of the Special Education Schools. A public forum was advertised through flyers posted at all special education schools and disseminated to all students, families, staff and community members known to the schools. (Appendix 4-5) The forum was also advertised through the district web site, voice mail, conference folder on SSD first class, flyers posted at Central Office and disseminated through the PAC. No one attended the April 14, 2005 forum. Self-Study Written Response Each principal completed a written response for all required and optional standards that described how their school met the state requirements. (Appendix 4-6) The principal did this by referring to the MSIP Standards and Indicators Manual for standards 1.1, 1.2, 1.3, 6.1, 6.2 and 6.3 and providing a narrative response on what the school did to meet the standard. This written response was submitted to the director who then reviewed the response and provided the principal with a preliminary score based upon the MSIP Integrated Standards and Indicators Manual and DESE s Report Writing Form. Written feedback was returned to the principals indicating which areas required revision and what type of clarification was needed. (Appendix 4-7) Once each principal had completed the written response for the individual school that met the scoring criteria, the director compiled all responses into one district response. The program evaluation committee used the MSIP Report Writing Form and Standards and Indicators Manual to score the composite written response for all schools. Again, the program evaluation committee identified areas of needed clarification and revision and submitted these to the director. The written response was revised according to the scoring feedback and resubmitted to the program evaluation committee for subsequent review. The committee scored all standards as met at the second review. Presented to Board of Education 12

26 Cost Analysis The FY04 budget expenses for total cost coded to the building were utilized in analyzing costs associated with the Special Education Schools. The annual budget was divided by the number of students in each setting as of the September enrollment date of September 24, 2003 to calculate an estimated annual per pupil cost per school. Presented to Board of Education 13

27 CHAPTER V RESULTS The findings of the program evaluation for the Special Education Schools are presented in this chapter. These analyses have been included to answer the following questions posed by the Board of Education: 1. What is the status of the concerns for special education schools identified as a result of Special School District s 2 nd cycle MSIP review? 2. To what degree are the special education schools meeting the 3 rd cycle MSIP expectations? 3. What actions need to be taken during the school year to meet the 3 rd cycle MSIP expectations? The sets of findings are listed below, along with the number corresponding to the evaluation focus question related to the finding. 1. Comparison of the MSIP Summary Report July, 2001 with findings of standard program evaluations for library media centers. (BOE question 1) 2. Comparison of the MSIP Summary Report July, 2001 with findings of standard program evaluation for facilities and safety. (BOE question 1) 3. Staff survey results. (BOE questions 2, 3) 4. Student survey results. (BOE questions 2, 3) 5. Parent survey results. (BOE questions 2, 3) 6. Self-study results. (BOE questions 2, 3) 7. Cost analysis for each school. Library Media Centers The July, 2001 State Board Summary for the MSIP review of Special School District indicates seven concerns pertinent to the Special Education Schools. (Appendix 5-1) The Standard Program Evaluation for Library Media Centers was submitted to the Board of Education in April of Due to space limitations, students at Northview, Neuwoehner and Bridges utilize the library services at North and South Tech. The outstanding concern noted was that the written procedural plan needed to be updated. Action plans indicate that the procedural handbook is being revised. Presented to Board of Education 14

28 Table 3. Comparison of 2001 Results with 2005 Results for Library Media Centers Concerns Service/ procedural plans for the buildings have not been revised Not 2005 met Elementary LMCs are not open and staffed at all times LMC facilities and furnishings are not adequate Records of student and staff use of the LMC resources are not maintained No management system is in place No documentation indicates that library skills instruction is being implemented LMC Staff members not assigned to Northview, Neuwoehner and Bridges Not met Not met Not met Not met Not met Not met Met Met Met Met Met Met Facilities and Safety The July, 2001 State Board Summary for the MSIP review of Special School District indicates four concerns pertinent to the Special Education Schools. The Standard Program Evaluation for Facilities and Safety was submitted to the Board of Education in January of (Appendix 5-3) The first concern was rectified shortly after the last review. The next concern regarding cleanliness at Northview appears to have been met based upon staff survey results that indicate that 76% of the staff agree/strongly agree that the building is in good condition. In regards to the instructional space concern, the district rented space across the street from the school during the school year. In the summer of 2005, the building where space was rented was sold to the Ferguson Florissant school district. Special District has been informed the space is no longer available. An outstanding concern regarding safety drills was noted for the special education schools in January. Each of the special education schools operated in district buildings conducted the required number and type of drills during the school year. The dates of those drills is included on appendix 5-4. Presented to Board of Education 15

29 Table 4. Comparison of 2001 Results with 2005 Results for Facilities and Safety Concerns Lack of handicapped accessible entrance at Bridges Page Not Met met 2001 Cleanliness concern at Northview Instructional space concern at Northview Safety drills Not met Not met Not met Met Met Met Staff Surveys Surveys were distributed to all principals, assistant principals and teacher-level staff working at Ackerman, Bridges, Career Training Program, Juvenile Detention Center, Lakeside Center, Litzsinger, Neuwoehner, Northview, Project Learn and Southview schools. Surveys were returned from 203 staff, yielding an 80% return rate. There were 92 items on the staff survey. The ratings were based upon a Likert scale with a rating of 1 representing strongly disagree and a rating of 5 denoting strongly agree. An analysis of the staff and administrator survey results was conducted by calculating the mean score for all responses and identifying strengths and needs based upon the mean or average of all staff responses on each item. The mean score for all items was The lowest mean score was This score was for the statement, Our classes are often interrupted. The highest mean score was 4.58 for the statement, Teachers in this school really care. The statements with a mean score above 4.25 are considered strengths and are represented by the table below. Presented to Board of Education 16