North Carolina investing in Rural Innovative Schools. (NCiRIS)

Transcription

1 for Traditional Comprehensive High Schools North Carolina investing in Rural Innovative Schools (NCiRIS) Table of Contents Response to Priorities Introduction Need for the Project Quality of the Project Design Quality of the Project Evaluation Quality of the Management Plan RESPONSE TO PRIORITIES Absolute Priority 5 Improving Achievement and High School Graduation Rates (Rural Local Educational Agencies): This proposed project will implement the successful design and instructional principles of the North Carolina New Schools Project (NCNSP) in 18 high schools in ten Rural Low Income Schools (RLIS) program eligible LEAs. This project addresses the unique challenges of high need students in rural areas by radically changing expectations for college readiness, teaching and learning, personalization, professionalism, leadership, and school design with a program that is affordable and scalable for rural LEAs. The program facilitates the use of online courses for students and teacher and administrator use of online tools for school improvement and professional development. Teacher and principal effectiveness is improved through increasing locally sustainable professional development capacity. This project also meets Absolute Priority 3 Innovations that Complement the 1

2 Implementation of High Standards and High-Quality Assessments by increasing the success of under-represented student populations in academically rigorous programs including Honors, Advanced Placement, and college courses by implementing crucial elements of successful Early College High Schools (ECHS) and the NCNSP Design Principles in rural traditional comprehensive high schools. Based on the NCNSP Design Principles implemented using NCNSP s Integrated System of School Support Services (IS 4 ), ECHSs have been shown in rigorous evaluation studies to increase the success of underrepresented and high need students in demanding programs of study and to close gaps in achievement and college preparatory course success. The NCNSP Design Principles and ECHS strategies encompass a whole school reform effort in which all students are expected to participate in college preparatory courses and encouraged to complete 21 college credits or more while in high school. This requires schools to align their academic program with state standards as well as with the standards adopted by a postsecondary partner for entrance into and success in college courses. To help schools make this significant transformation the proposed project involves training leaders and teachers to use formative assessments aligned with college ready standards, and to translate standards and data from assessments into classroom practices specifically designed to support high-need students. Competitive Preference Priority 7 - Innovations that support College Access and Success: ECHS strategies and the NCNSP Design Principles are expressly designed to prepare high-need high school students for success in postsecondary education. The project will address students preparedness and expectations through: 1) a college-ready academic program that includes an instructional framework aligned to college-ready standards; 2) an aligned sequence of college courses and supports as part of the high school program of study; 3) significant exposure to the culture and norms of college; 4) assisting students and their parents in 2

3 understanding issues of college affordability and financial aid; and 5) providing comprehensive college application and financial aid advising and assistance. The NCNSP approach provides support to students from peers and adults through wraparound services (including academic support and advising) and a strong college-going school culture that establishes an expectation that all students will achieve postsecondary success. Competitive Preference Priority 8. The NCNSP Design Principles and ECHS strategies include instructional practices and academic support strategies designed to increase the academic outcomes and college readiness of Students with Disabilities (SWD) and Limited English Proficient (LEP) students. The instructional coaching helps teachers implement a common instructional framework that supports SWD and LEP achievement, emphasizing literacy rich classroom practices such as collaborative group work, writing-to-learn, and oral inquiry and questioning strategies. ECHS strategies include finding ways to provide tutoring and supplemental instruction in precollege and college courses, academic advising, and immersion in a college-going culture to support the success of struggling students. Independent evaluation research (American Institutes for Research/SRI International, 2009) points to the ECHS model as particularly beneficial for students from homes where a language other than English dominates. At three of the 18 participating high schools, LEP students are greater than 11% of the enrollment, and the overall number for the 18 schools is more than 500 LEP students. INTRODUCTION The North Carolina New Schools Project (NCNSP), a 501c(3) non-profit organization chartered in 2003, in partnership with ten rural Local Education Agencies (LEAs) (Alleghany County Schools, Beaufort County Schools, Hertford County Schools,Jones County Schools Madison County Schools, Rutherford County Schools, Surry County Schools, Warren County 3

4 Schools, Wilkes County Schools, Yancey County Schools) and supported by the organization s continuing partnerships with the North Carolina Community College System and the North Carolina State Board of Education is pleased to apply for an i3 validation grant to expand ECHS strategies and the successful NCNSP Design Principles to 18 rural traditional high schools in order to advance the development of schools that graduate every student ready for college, careers, and life in the 21st century. The NCNSP is a statewide public-private partnership that supports innovation in secondary schools. Since its inception and initial support from the Bill & Melinda Gates Foundation in 2003, it has helped develop 117 schools in 70 school districts in every region of the state. NCNSP promotes a vision that every high school will graduate every student ready for college, career and life by accelerating systemic sustainable innovation in secondary schools across the state. NCNSP master teachers and accomplished school administrators provide a full range of services and supports to enhance the knowledge and skills of educators. NCNSP advances change by:1) Demonstrating how new approaches to education can succeed with all students; 2) Providing effective professional support to educators and administrators leading to the creation of innovative schools; and 3) Working with individuals and organizations to build an irresistible demand for innovation in a growing number of North Carolina schools. Since the first 13 ECHSs emerged in 2005 as partners with NCNSP, the North Carolina ECHS Initiative has grown to include 74 schools in , enrolling nearly 13,000 students the most ECHSs of any state. NCNSP also works with redesigned traditional high schools serving a population of 12,000 students. Early results from NCNSP innovative high schools indicate that more 9 th graders stay in school and advance to eventual graduation, more students succeed at college-level work and graduate with significant college credit, achievement gaps are decreasing and more teachers 4

5 believe in their schools and say their schools are a good place to work and learn. A. NEED FOR THE PROJECT A.1. Magnitude of the need for the services or activities to be provided by the project The ten partner LEAs partnering include some of the poorest and most rural districts in North Carolina as demonstrated by data in Table 1. In the targeted high schools, 42.6% to 76.2% of the students qualified for Free and Reduced Lunch in Scores on state standardized tests at these schools tend to be low, particularly in mathematics. Among the 18 schools in this project, the rate of students scoring at/above grade level on the state s Algebra I End-of-Course exam ranged between 35.8% to 83.8% with nine of the schools below 70.0% ( NC Report Cards). These schools often lack the staff to offer a range of honors and advanced classes and participate at a lower rate than suburban schools in on-line and dual enrollment programs. At the 18 schools, the rate of participation in college prep courses (i.e., AP, IB, community college, university) is abysmally low, ranging between 0-5% ( NC Report Cards) compared to the state s 5% average. Working with these districts, NCNSP has helped establish one ECHS in each partner district. While these new schools are on track to show impressive gains in graduation rates, the traditional high schools that serve the majority of the students still have 4- year cohort graduation rates ranging from 62.6 to 84.7 ( NC Report Cards). Adding to the need for services, these already stressed communities have been devastated by the recession. The 24-month average unemployment is 10.82% and 2009 per capita money income (3-year American Community Survey) was $18,311 compared to $27,100 for the U.S., calculated for counties using Stats America ( While successful ECHSs exist in these districts, these schools cannot serve all of the high 5

6 school students in the district. Most of the ECHSs are located on the campuses of community colleges. Due to the high demand for community college services, exacerbated by the current economic situation, the facilities on these college campuses are now operating beyond their capacity. Therefore, to extend the proven advantages of the NCNSP Design Principles and ECHS strategies, there is a need to carry out professional development and redesign at the traditional high schools. A.2. An Exceptional Approach to i3 Priorities The NCNSP and partner LEAs propose to expand an exceptional approach to increasing college readiness and success of high need students in rural and low-income schools by implementing the NCNSP Design Principles and ECHS strategies in their high schools. This approach holds exceptional promise for transforming existing schools where more students can be impacted, revolutionizing high school education in North Carolina. NCNSP assists schools in implementation of these principles and strategies with a comprehensive professional development program titled IS 4, designed specifically to help high schools transform their vision and practices to those embodied by successful innovative schools and enable young people underrepresented in higher education (students from low-income families, first-generation college students, English language learners, minority) to earn a high school diploma and work towards a college degree simultaneously. In 2003, when NCNSP began working with the Bill & Melinda Gates Foundation to redesign high schools, the impetus was that challenge, not remediation, would make a difference for those young people who are least likely to attend college and for whom society has historically had low aspirations for achievement, and that accordingly, an accelerated academic program could dramatically raise postsecondary attainment rates of low-income students. In the eight years since, the NCNSP Design Principles and the IS 4 6

7 professional development program, in conjunction with partnerships with the NC Community College System, LEAs and local IHEs has indeed proven to be an exceptional approach for increasing the likelihood that high-need students graduate on time and prepared for college. NCNSP is currently working with 74 ECHSs in North Carolina, the most of any state in the nation. These ECHSs, targeting students traditionally underrepresented in colleges (minority, economically disadvantaged, and or first generation in family to attend college), have significantly lower dropout rates, higher graduation rates, and higher enrollment and achievement in rigorous college preparatory courses than other schools with similar demographics. Providing a fifth year of enrollment and college dual enrollment opportunities, it is possible for students to earn an Associate Degree through the community college system. The NCNSP Design Principles under gird the success of these 74 schools. The Six NCNSP Design Principles 1. Ready for College: Innovative high schools are characterized by the pervasive, transparent, and consistent understanding that the school exists for the purpose of preparing all students for college and work, and recognizes that, in the 21st century, the skills to succeed in post-secondary education and in viable employment are the same. They maintain a common set of high standards for every student to overcome the harmful consequences of tracking and sorting. 2. Require Powerful Teaching and Learning: Innovative high schools are characterized by the presence of commonly held standards for high quality instructional practice. Teachers in these schools use a Common Instructional Framework to design instruction that ensures the development of critical thinking, application, and problem-solving skills. 3. Personalization: Staff in innovative high schools understand that knowing students well is an essential condition of helping them achieve academically. These high schools ensure that staff 7

8 leverage knowledge of students in order to improve learning. 4. Redefined Professionalism: The responsibility to the shared vision of the innovative high school is evident in the collaborative, creative, and leadership roles of all adult staff in the school. The staffs of these schools take responsibility for the success of every student, hold themselves accountable to their colleagues, and are reflective about their roles. 5. Leadership: Staff in NCNSP schools work to develop a shared mission for their school and work actively as agents of change, sharing leadership for improved student outcomes in a culture of high expectations. 6. Purposeful Design: Innovative high schools are designed to create the conditions that ensure the other five Design Principles are evident. The organization of time, space, and the allocation of resources ensure that these best practices become common practice. The NCNSP Integrated System of School Support Services (IS 4 ) To help schools meet the challenging ideals set forth in the Design Principles, NCNSP has developed a set of comprehensive professional development supports (IS 4 ) and a set of rubrics that articulate growth standards for each Design Principle. These standards are used by all NCNSP ECHSs to do rigorous self-assessments and guide development. This NCNSP professional development program includes four interactive programs: 1. Professional Development for Leaders: Principals at NCNSP ECHS s are supported by a Leadership Coach who conducts regular on-site visits. Leadership Coaches are experienced school leaders who have worked in schools with a focus on innovation. Among the Leadership Coaches major responsibilities are the following: a) Establishing trusting relationships with the principal and school staff; b) Building understanding of the NCNSP Design Principles and best practices; c) Identifying specific needs for support and assistance related to successfully 8

9 implementing the model; d) Identifying potential obstacles to success while helping develop strategies to eliminate them and assure support for initiatives within the scope of NCNSP expectations; and e) Guiding and focusing school leaders on innovation, reflective practice and the strategic planning process to ensure that all students will graduate prepared for college and work. In addition, the Leadership Coaches design and deliver regular professional development services to principals in regional groups called Leadership Innovation Networks (LINs). These professional development sessions are designed to broaden and deepen principals skills and knowledge as they work toward the ideals set forth in the NCNSP Leadership Design Principle. 2. Instructional Coaching: Each school receives services from a proven educator who has knowledge, experience, and skills in working with teachers and who understands and is committed to NCNSP s mission, vision, and support system. In the first year of a school s work with NCNSP, a leadership coach facilitates a shared understanding of and commitment to the Design Principles. Based on this understanding an Instructional Coach works directly with the teaching staff individually and collectively to improve their skills in using the Common Instructional Framework (CIF). The CIF includes six instructional strategies: writing to learn, cooperative learning, scaffolding learning, classroom talk, literacy groups, and questioning. Instructional Coaches help teachers to integrate the strategies so that all students are challenged to read, write, think, and talk every day in every class. 3. Teaching for Results: Each year, teachers in NCNSP ECHSs take part in a series of intensive professional development activities that sustain their focus on instruction, academic rigor and professional learning communities. The sessions stress differentiating instruction, teaching literacy across the curriculum, facilitating meaningful learning, and providing effective student support. During the school year, the sessions involve visits to peer schools in which 9

10 teachers use a medical rounds model to improve their practice collaboratively. In this model teachers observe each other and discuss student work and instructional practices focusing on implementation of the Common Instructional Framework. These visits provide a basis for using the rounds model in each school, so that teachers work as true professional peers, providing critical feedback and learning from each other through facilitated classroom observations. 4. NCNSP Portfolio Manager: NCNSP s School Development Team, made up of highly accomplished teachers and administrators, provides ongoing support. Each school is assigned to a Portfolio Manager from this team. The Portfolio Managers coordinate the leadership and instructional coaching, ensure training of guidance counselors, promote relationships with IHEs, facilitate visits to peer schools, design and deliver professional development institutes, ensure the delivery of integrated supports, and act as a primary point of contact with NCNSP. The NCNSP ECHS strategies include a focus on completing college courses, providing significant experience with college culture, and assisting families with understanding and completing the college application process. Partnerships with the North Carolina Community College System and local community colleges allow NCNSP to continue to expand access to college courses and allow NCNSP to develop college campus experiences for students whose primary campus will remain the traditional high school. Additionally, IHE partners help with providing training to volunteers who assist students with college and financial aid applications. A.3. Importance and magnitude of the effect expected as supported by the research The NCNSP model is the subject of a longitudinal experimental study (Edmunds, J. A., Bernstein, L., Unlu, F., Glennie, E., Arshavsky, N., Smith, A., 2011), funded by an IES grant that used a randomized lottery system to assign students to either an early college (treatment) or a traditional high school (control). Because this study started with 9 th graders and is following 10

11 students as they progress through the early college, final outcomes such as graduation rates and college enrollment rates are not yet available but intermediate outcomes measured show that of the students in the initial applicant pool, early college students are on-track for accomplishing these outcomes at significantly higher rates than those in traditional schools. Findings include: More students in the early college setting were taking the courses they needed to be ready for college. By the end of 9 th grade, 82% of treatment students had completed at least one college preparatory math course (Algebra I, Geometry or Algebra II), compared to 70% of control students. The adjusted impact estimate 1 was 6 percentage points (effect size of 0.18), a statistically significant difference. By the end of 10 th grade, 73% of treatment students had completed at least two college preparatory math courses compared to 57% of control students. The adjusted impact estimate was 6 percentage points (effect size of 0.15). Significantly more treatment students remained enrolled in school. In 10 th grade, 96% of treatment students were still enrolled in a North Carolina public school compared to 89% of control students (Edmunds, Bernstein, Unlu, Glennie, Arshavsky, & Smith, April, 2011), a statistically significant adjusted impact estimate of 5% percentage points (effect size of.40). If this pattern continues, the impact on graduation rates should be at least 10 percentage points. This grant-funded project will extend early college strategies to 18 traditional, comprehensive high schools currently serving more than 12,200 students in ten rural low-income districts and will create a well-codified professional development and district capacity building 1 The percentages reported are unadjusted means. The study team conducted regression analyses that incorporated covariates and that included a site-specific indicator to develop an adjusted impact estimate, which serves as the primary impact estimate. 11

12 platform to support further national scaling. Based on the results from the experimental study cited above and results from other existing NCNSP IS 4 -supported high schools, the implementation of this project should produce an effect size of.40 on continuing enrollment in school in the 10 th grade going up to.53 in the 11 th grade and eventually producing an effect size of at least.40 on graduation rate. The expected effect size on college preparatory course taking is.18. In this project, the following measurable outcomes are expected: Approximately 21,442 students impacted over the five-year grant period; The 4-year cohort graduation rate will increase an average of 10 percentage points across the 18 project schools by the end of the fifth year of the grant program; Schools will increase the percentage of students successfully completing Algebra 1 by the end of ninth grade an average of 10 percentage points by the end of the second year of implementation; At least 50% of students who experience four years in project schools will successfully complete at least 21 units of college credit. 90% of project schools and LEAs will continue implementation of the NCNSP Design Principles and ECHS strategies with active participation in NCNSP Network after the completion of three years of IS 4 services. North Carolina will enact legislation and policy changes to expand access to college courses for high school students. B. QUALITY OF THE PROEJCT DESIGN B.1. Clear set of goals and explicit strategy Goal 1: Improved Student Outcomes 12

13 Strategy 1a - Implementation of ECHS strategies and NCNSP Design Principles in rural traditional high schools: The proposed project, including 18 schools in ten LEAs, which together serve more than 12,200 students each year, provides the opportunity to apply and validate the widespread application of NCNSP Design Principles and ECHS strategies to a range of traditional high schools of varying sizes in rural districts serving a largely low-income, high need population of students. The ten LEAs were selected based on their successful track records of developing early colleges for high need students, their commitment to taking this design to scale, and the support of key stakeholders. Each of the selected LEAs has one innovative early college school that will serve as an exemplar to help guide expansion of the design to other high schools in the district. Participating LEAs agreed to make the NCNSP Design Principles and ECHS strategies the default philosophy and pedagogy for participating traditional high schools and all students enrolled therein. We anticipate replicating the ECHS success in our traditional comprehensive high schools! Strategy 1b - Provide IS 4 services: NCNSP will provide each participating school IS 4 services focused on implementing NCNSP Design Principles and ECHS strategies. As described in section A.2, IS 4 services are coordinated by a Portfolio Manager and include job embedded instructional coaching for teachers and leadership coaching for principals, professional development institutes, visits to peer schools, and management support. Focus is placed on implementation of the Common Instructional Framework and Leadership Innovation Network to develop a coherent environment across classes. These practices engage students of all skill levels, as well as staff and administrators, in learning and teaching critical skills needed in the 21 st Century. The amount of coaching services to be provided is based on school size. For three years, each teacher will annually receive the equivalent of three full days of instructional 13

14 coaching and for three years each principal will annually receive 15 days of leadership coaching with additional days for other administrators. Strategy 1c - Enhancing partnerships with local IHEs: Each district will work with their local IHE partner(s) to increase student and family understanding of college access and opportunities through tours and activities on the IHE campus. IHE admissions staff will work with school staff to educate students and families and to train volunteers to assist students with applications to college and for financial aid. Schools and partner IHEs will develop an aligned sequence of college courses to be taken on the high school or college campus or through online delivery with support to ensure successful completion. Funding to support the partial cost of college course tuition and textbook purchase is included in the budget for this proposal as new state budget restrictions will otherwise limit access to these courses. In addition to the funds provided in the budget for this, project schools will access other programs to allow their students to earn college credit, including North Carolina s College and Career Ready Initiative, dual enrollment, Learn and Earn Virtual course taking, and AP courses through the North Carolina Virtual Public High School. Strategy 1d - Expand access and support for online college course-taking to enhance scalability: NCNSP will continue to validate and refine innovations on the most effective ways to utilize online courses to accelerate student learning. This activity will build on NCNSP s creation in 2008 of six ECHSs currently serving 600 students through online college courses. These schools have seen strong results to date, including a 98% pass rate (with a C or better) on college courses taken through 2010 and a dropout rate of less than 1%. Goal 2: Build Capacity for Sustainable Implementation Strategy 2a: Build partner LEA capacity to sustain and grow effective practices through 14

15 a strategic process to engage school districts in the work of school innovation: This includes the introduction of the NCNSP Design Principles Rubric to district and school staff in order to build shared understanding and use as a district self-assessment tool, development of a communication plan to build stakeholder support, study visits for district staff and community stakeholders (e.g. school board members) to other innovative districts, redesign of relevant central office functions and budgeting, and retraining of curriculum and instructional staff at the central office. Strategy 2b: Develop the Leadership Innovation Network to support district innovation beyond the period of grant funding: NCNSP will continue to develop the capacity of its Leadership Innovation Network (LIN) in order to support new and continuing administrators beyond the period of grant funding. Principals and district leaders will be encouraged to continue participation beyond the grant period. The goal of the LIN group is to build leadership capacity and sustainable innovation to positively impact student outcomes. Focusing on the fundamentals of effective leadership, principals actively participate in small group professional development sessions designed to support the leadership initiatives needed in an innovative high school. In addition, LIN groups participate in collaborative problem-solving opportunities and peer sharing. Leveraging the influence of their peers in a professional community of learners is a key feature of the LIN group model. With an emphasis on empowering principals to assume ownership for their professional growth, a sustainable model of leadership development is cultivated. Strategy 2c - Use peer-learning to leverage results and accelerate scale: NCNSP will promote innovation and rapid dissemination of best practices emerging from early adopters though strategic documentation and targeted face-to-face and online meetings across the network of participants and other interested educators. Social networking tools will be used to facilitate 15

16 virtual interaction. This network will sustain current schools and support expansion. Goal 3: Create Platform to Support Large-Scale Expansion Strategy 3a: Document and disseminate lessons learned and policies needed to support innovative teaching and learning. NCNSP will document lessons learned and policies needed to support the implementation of the Design Principles and ECHS strategies. This will be disseminated locally with groups such as the NC School Boards Association and the NC Association of School Administrators and nationally through collaboration with organizations such as the Southern Regional Education Board, Jobs for the Future, the National Governor s Association and the Council of Chief State School Officers. Dissemination will also include publications and participation in policy forums and academic conferences. NCNSP will make policy recommendations to the legislature and State Board of Education around issues such as professional development standards, funding mechanisms and leadership development related to a comprehensive approach to supporting access to college credit for high school students. Strategy 3b- Leverage learning to further statewide capacity development: In a variety of public engagement strategies, NCNSP will further disseminate lessons learned from this use of ECHS strategies with strategic policy briefings and co-development of forums to publicize results across the state. Working with groups such as the NC ECHS Initiative of the NC Department of Public Instruction, the NC Education Cabinet, and the JOBS Commission, NCNSP will promote innovation and rapid dissemination of best practices so that learning will not be isolated within a limited number of school districts. B.2. Costs are reasonable in relation to objectives, design and potential significance With a total cost of $15 million, this project will serve more than 21,000 students over the five years of the grant at a per student cost of approximately $715/student. A large part of the 16

17 cost to each school is to support tuition and books for college courses. These courses help reset student and teacher understanding of the demands of college and exert upward pressure on expectations of student performance of high standards. The potential increase in graduation rates will result in significantly more high school graduates over the length of the grant with continued lifetime benefits to these students, their families and communities. Increasing the graduation rate by 10 percent at each school would result in 225 more graduates per year across the 18 project schools. If the financial lifetime benefit of high school graduation is estimated at $200,000 per student then the benefit of this project for one cohort of students would be $45 million. The grant is budgeted to provide for three years of instructional coaching for each of 491 teachers leading to lasting adoption of effective classroom practices. In addition the impact of the grant-funded project will lead to long term cultural changes in these rural schools benefiting future students. NCNSP will then use these schools as a springboard to replicate the model. B.3. Extent to which the services reflect up-to-date knowledge The NCNSP Design Principles embody effective practices in instruction and school design consistent with a large body of research and numerous other recommendations. For example, the What Works Clearinghouse (WWC) Practice Guide addressing college access (Tierney, W. G., Bailey, T., Constantine, J., Finkelstein, N., & Hurd, N. F., 2009) makes five recommendations in the area of academic preparation, expectations, and support with steps for college entry similar to the NCNSP Design Principles. The WWC Practice Guide addressing dropout prevention (Dynarski, M., Clarke, L., Cobb, B., Finn, J., Rumberger, R., and Smink, J., 2008) makes two recommendations for school wide interventions: 1) personalize the learning environment, and 2) provide rigorous and relevant instruction. All of these recommendations are built into the NCNSP Design Principles and ECHS strategies that school staff will implement in this project. 17

18 B.4. Estimate of the cost of reaching 100,000, 250,000, and 500,000 students The total cost of this project is $15 million and will reach more than 21,000 students giving a per student cost of approximately $715. Based on these costs, the estimated cost of extending this to a further 100,000 students after the grant period is $71,500,000, to 250,000 students is $178,750,000 and 500,000 is $357,500,000. It is important to note that approximately 18.3% of the total budget for this project (almost one-fifth of the total) is to pay for tuition and textbooks for college level courses. Investment in these courses for high school students may reduce state and family costs for college education later. Also textbooks are likely to come down in price as digital versions are implemented reducing printing, distribution, and updating costs. B.5. The potential and planning for the incorporation of the project into ongoing work The NCNSP vision is Every student in North Carolina graduates ready for college, careers and life and mission is To accelerate systemic, sustainable innovation in secondary schools across the state. Since 2003, NCNSP has worked with 117 schools in 74 districts including 74 ECHSs. This project extends NCNSP s current work into additional schools and will serve as a demonstration to communities and legislators that significantly better results can be achieved in the traditional, comprehensive high school given the implementation of NCNSP Design Principles and IS 4 professional development strategies. Each partner district has demonstrated their commitment to the NCNSP Design Principles in implementing and sustaining their respective ECHSs. The success of these schools has built support and understanding in the school and wider community for the significant changes in design and culture required by this project in order to achieve significantly better student outcomes in all of the districts schools. C. QUALITY OF PROJECT THE EVALUATION 18

19 C.1. Extent methods include a well-designed experimental or quasi experimental study Through a high quality combination of an experimental and quasi-experimental study, the independent evaluation conducted by the SERVE Center at UNC Greensboro will assess the extent to which the implementation of the NCNSP Design Principles and ECHS strategies have a positive impact on student outcomes associated with staying in school and increased readiness for college. SERVE is currently leading an IES-funded experimental study of the impact of the ECHS model which will complement their evaluation of this project. The evaluation will examine the fidelity of implementation and provide useful feedback to the program developers. Research Questions: Impact Question: 1. To what extent does the implementation of NCNSP Design Principles and ECHS strategies result in improved student outcomes, including increased course taking and success, higher student attendance, reduced dropout rates, and increased graduation rates? Implementation Questions: 1. To what extent have participating schools implemented NCNSP Design Principles and ECHS strategies with fidelity to the project design? What is the pattern of implementation in the schools over the five years of the grant? 2. What services have been provided to the schools and districts? What has been the perceived quality and utility of those services? 3. What challenges occurred in the implementation of the program. How were those challenges addressed? 4. What specific actions have been taken to create a platform for expansion? To what extent have these actions resulted in changes in policy? 19

20 Table 2 provides a logic model for the program and its anticipated outcomes. 20

21 Table 2. Logic Modeling for a Target Population of Students at Rural, Low-Income, Comprehensive High Schools Goals Strategies Intermediate Outcomes Long Term Outcomes Goal 1: Improved Student Outcomes Goal 2: Increased District Capacity Design Principles plus ECHS strategies IS 4 professional development Enhanced IHE partnerships Expand access and support for online college courses Strategic process of engagement Leadership Innovation Network Implementation of Design Principles in schools Increased student attendance Decreased dropout rate Increased enrollment and success in college preparatory and college courses Increased graduation rate Continued participation in Leadership Innovation Network Sustained innovation Goal 3: Platform for Expansion Policy influence Peer learning Policies which expand access to college courses for high school students. Strong virtual community to support expansion 21

22 Methodology and Analysis: To determine the impact on student and school-level outcomes, this evaluation will use a staggered entry experimental design, supplemented by a quasi-experimental comparative interrupted time series. To examine implementation, the evaluation will collect descriptive information from the program staff and from participants. Impact Evaluation Methodology. Both student and school-level outcomes for Goal 1 will be assessed using experimental and quasi-experimental approaches. A total of 18 high schools were recruited for participation in the program. In Year One, six schools selected randomly from the list of 18 schools that agreed to participate will serve as an initial treatment group with the remaining 12 schools serving as a control group. In Year Two, an additional six randomly selected schools will be served. These schools will serve as another treatment group with the last remaining six schools serving as a control group. In Year Three, the remaining six schools will be served. Once schools have begun the program they will continue to receive services for three years. NCNSP has found that it takes time to institutionalize reform particularly at large traditional schools. This experimental design will compare the results for schools with some exposure to the intervention to those with no exposure to the program. Because all participating schools will have received at least some of the treatment by the time the three years of services are completed, the evaluation will supplement the experimental component with comparative interrupted time series analyses that will examine performance on core student outcomes over time for both the 18 schools receiving the treatment and a matched set of comparison schools (more information on the comparison schools is found in the Sample section below) for the entire duration of the project. A power analysis suggests that the sample size of 18 schools for the experimental study will be sufficient to detect effect sizes of.22 or greater. Including comparison schools, the sample of 22

23 36 schools in the quasi-experimental study will be sufficient to detect effect sizes of more than 0.16 standard deviations. Assumptions for the power analysis (completed using Optimal Design software) include a statistical significance level of.05, 80% power, 160 students per grade per school, an intra-class correlation of.10 (Hedges & Hedberg, 2007), and an estimate of R 2 =.80 explained by 8 th grade achievement data used as a covariate (R 2 of.90 and higher have been identified for high school outcomes (Bloom, Richburg-Hayes, & Black, 2007). Impact Evaluation Outcomes: The following specific outcomes will be examined: 1. College preparatory course taking and success. One of the main goals of the project is to increase the college readiness of students. A core part of this is enrolling students in the courses required for college entrance and helping them succeed in those courses. This study looks at the proportion of students taking and succeeding in a core set of college preparatory courses. The courses to be examined include those that are required for entrance into the University of North Carolina system (e.g., 4 years of English, four years of college prep math, etc.). The evaluation will look at two outcomes for each course. The first will be the percentage of a given grade of students who are taking the course and serves as a measure of access and the extent to which schools are providing opportunities for students to get ready for college. The second will be successful course completion or the percentage of a given grade of students who took and passed the course. In courses for which there is a North Carolina mandated End-of- Course (EOC) exam, a passing score on the exam will be used. In courses for which there are no EOC exams, students final grades will be used. This second measure of successful course completion captures both access and success in school and does not penalize schools that are expanding access to more students. The anticipated impact is at least 10 percentage points on both course taking and course success by the second year of the intervention. 23

24 2. Attendance. Student attendance has been positively associated with progress in school (Lee & Burkham, 2003); changes in student attendance are seen as a reliable indicator of students likelihood of remaining in school. The evaluation will examine the number of days that a student is absent from school. The intervention is expected to result in a reduction of two days absence. 3. Dropout and continued enrollment in school. The intervention is designed to keep more students in school and on track for graduation. As a result, the evaluation will look at the number of students who drop out. Because experience indicates that the dropout data are not always complete, the evaluaiton will also look at the proportion of students who remain enrolled in school in each year. By Year Two of implementation in a school, the intervention is expected to result in an increase of 5 percentage points in the proportion of students who remain enrolled in school, consistent with findings in the previously cited experimental study. 4. College credits accrued while in high school. One of the key goals of the program is to increase the number of students receiving college credit while in high school. The evaluation will examine the proportion of students receiving college credit. 5. Graduation from high school. Given that the grant program will last only five years, the evaluation will be able to examine graduation rates only for the first cohort of students (those students who were in 9 th grade in Year One of the project). Based on results being obtained by the early colleges, the intervention is expected to result in an increase of 10 percentage points on graduation rates for those students who experience four years in an implementation school. Impact Evaluation Analysis: For the experimental design, the analysis for the primary student impact estimates will be a two-level model with students nested in schools. To increase precision, the evaluation will include the following 8 th grade student level covariates in the model: race/ethnicity; English Language Learner status; free and reduced price lunch status; 24

25 attendance; and 8 th grade reading and math performance. Given that these schools will be enrolling in the initiative in a staggered basis, evaluators will be able to conduct a set of analyses based on years of exposure to the program. For example, in Year One, evaluators can compare results from one year of treatment for six schools to no exposure to the program for the 12 remaining schools. In Year Two, evaluators will be able to compare results from one year of treatment for 12 schools to the six remaining schools. The evaluators will also be able to conduct more exploratory analyses such as the difference between two years of treatment (six schools) to no treatment (six schools). It is recognized that because these exploratory analyses are using a subset of the sample, they will thus have less power. To supplement the experimental design, the study will utilize a comparative interrupted time series (ITS) approach in order to estimate the effects of the early college expansion initiative over and above the outcomes that would be observed in the absence of the initiative. This analytic approach combines two particularly strong quasi-experimental evaluation methods: an interrupted time series analysis and a comparison schools technique that will allow us to develop relatively reliable and unbiased estimates of the impact of the initiative on student outcomes. For the 18 treatment schools, the evaluators will identify a comparison school using propensity score matching (more detail on the identification of comparison schools is found in the description of the Sample). The evaluators will then compare measures of student performance in schools that implemented the intervention to performance of similar students in the same schools prior to implementation of the intervention. The difference between performance levels in the two groups is referred to as a deviation from the baseline. The evaluators will then conduct a second interrupted time series analysis for the matched comparison schools in the same district that have student characteristics and baseline outcomes 25

26 similar to those of the intervention schools. The difference between the deviations from the baseline in the intervention schools and the deviations from the baseline in the comparison schools will represent the estimated impact of the intervention. The evaluators will also conduct exploratory analyses by subgroup to determine the impact for specific populations including minority students, low income students, those who are the first in their family to go to college, LEP students, and students with disabilities. Implementation Evaluation Methodology. The evaluation is also designed to assess the extent to which the program has been implemented as intended, including implementation of the Design Principles. Under the ongoing experimental study of the impact of ECHSs, SERVE has developed a staff survey to assess implementation of the Design Principles in the schools. The survey includes scales related to college readiness, rigorous and relevant instruction, personalization (support and relationships), and the professional working environment. Reliability has been calculated on all these scales. This survey will be modified to also measure the recently added leadership Design Principle and to more closely reflect implementation of the Common Instructional Framework. The survey will be administered at baseline and then annually to see if the level of implementation of the Design Principles increases. All schools in both the experimental and quasi-experimental portions of the study will complete this survey. To assess the implementation of the Design Principles, an average score will be calculated for each Design Principle from the surveys. Scores on each Design Principle will be compared to both the school s scores at baseline as well as the scores of the schools that have not yet received the treatment. This will be done using a multiple regression analysis that includes a dummy variable for receiving the treatment and school-level covariates including school size, percent free and reduced price lunch, percent minority, and composite score for the end-of-course exams. 26

27 The evaluation will also document the services received by the participating schools through annual reviews of program records and annual interviews with program staff. In addition, as part of the Design Principles staff survey, the evaluators will ask all staff in the experimental design sample to indicate the services they have received through the IS 4 grant and to indicate any other services that have been received. This will allow us to determine the extent to which the services received by the treatment group differ from the services received by the control group. The evaluation will conduct annual interviews with each of the leadership and instructional coaches to identify their perspective on challenges faced by the schools and ways in which the schools have overcome those challenges. In the first and second years, these interviews will be used to identify a set of three or four schools that appear to be making significant changes due to the IS 4 treatment. Starting in Year Two, the evaluation will carry out case studies of these schools to identify factors that appear to help and hinder implementation of the Design Principles and Common Instructional Framework. These case studies will include annual interviews with the principal and assistant principal, lead teachers, and guidance counselor. At least one student focus group will be conducted per school to obtain the students perception on the intervention and the extent to which they observe any changes in the school. To examine the implementation of Goal 2 strategies, these site visits will also include interviews with district staff to understand the services they have received and their perception of the impact of these services. Finally, to examine the implementation of the Goal 3 strategies, the evaluators will also conduct interviews with project staff and review State Board of Education minutes to describe the actions taken to influence policy and the extent to which those actions have been successful. And, the evaluation team will participate in the virtual community to examine the extent and nature of participants involvement in this community. 27

28 Sample: The sample for the staggered entry experimental study will include a total of 18 schools located throughout rural North Carolina. All of these schools have committed to participate in the program. Each of the schools receiving the treatment will be matched to one comparison school. Schools will be matched first on a three year pre-intervention pattern of the outcomes of interest, including graduation rates, percent of students enrolled in Algebra II (which will serve as a proxy for students on a college preparatory path), and percent of students participating in AP /IB/college credit courses. Schools will then be matched on school-level characteristics, including school size, percentage minority, percentage low-income and percentage Limited English Proficient. Data Sources and Collection: The student outcome analyses will rely on data collected annually by the North Carolina Department of Public Instruction. These data are housed at the North Carolina Education Research Data Center where they are cleaned, de-identified, and linked to longitudinal student records. School-level administrative data are also available through this dataset. The evaluation team has used this dataset for the current experimental study of the impact of ECHSs. The data to be obtained through this dataset include: student demographic data, course transcript data, test scores (End-of-Course exams and 8 th grade End-of-Grade exams), attendance data, enrollment data, AP exam data, and dropout and graduation data. These data will be supplemented by data from the state s community college system, which will provide information on the number of college credits accrued. School-level data on the Design Principles will be collected using surveys (described in more detail below). C.2. Extent the methods will provide high quality implementation data and feedback As noted above, measuring implementation of the Design Principles in the school will rely partly on a survey to be administered to staff in the treatment schools. These surveys will be 28

29 based on the Early College Implementation Survey developed by SERVE as part of the current early college study. The surveys will include scales that measure the following aspects of the Design Principles: college-going expectations; implementation of the strategies in the Common Instructional Framework; quality of student-staff relationships; amount of academic and personal support provided to students; amount of staff collaboration; and perceptions of shared leadership. This survey will be administered annually in an online format at a time that is convenient for the school. All staff will be asked to respond to the survey. The evaluation team will develop protocols for the coach and site visit interviews that focus on understanding the services provided to the schools, any changes being made in the school, and any challenges that have been faced. In later years, the content of these protocols may be revised to explore themes emerging from the data collection. The evaluation team will meet with the NCNSP management team on a quarterly basis to review findings and assess progress. In addition, the evaluation team will submit annual reports to the project team that will include findings on implementation and interim results for outcomes. C.3. Extent the evaluation will provide sufficient information about the key elements NCNSP has created a well-defined set of key elements for their intervention, including a set of rubrics they use as part of their program to encourage schools to reflect on their implementation. The evaluation will use the NCNSP tools as a starting point for clearly describing the desired elements of the intervention. As described in more depth above, the data collected through the surveys, interviews, and case studies will provide detailed information on what the key elements look like in practice. These data will be supplemented by interviews with program staff and reviews of project records to ensure that all of the services received by the participants to support implementation of these key elements are adequately described. In 29

30 addition, the evaluation team will work with NCNSP project team staff to ensure an accurate and complete description of the intervention. C.4. Sufficient resources to carry out the evaluation effectively. Proposed Resources: A total of $1,320,172 is allocated for the evaluation. Based on the evaluation team s experience, this will be sufficient to conduct an in-depth process and outcome evaluation. The evaluation will also be able to capitalize on administrative data already collected and submitted through the state. In addition, the evaluation will build on knowledge gained through the current study of the early college model. Qualifications of Evaluation Staff: Dr. Julie Edmunds at SERVE Center at UNC- Greensboro will be leading the evaluation effort. Dr. Edmunds is Principal Investigator for two IES grants for a longitudinal experimental study of the impact of the ECHS model in North Carolina. She has an in-depth knowledge of the early college effort and of the issues associated with measuring the impact and implementation of this model. Dr. Laura Feagans Gould at SERVE Center will serve as Senior Quantitative Lead. Dr. Gould has spent the past 14 years conducting and applying research on at-risk youth to community and school-based programs that target social, emotional, and behavioral competencies of youth. She is a quantitative researcher by training with expertise in analytic frameworks relevant to longitudinal and nested data including, hierarchical linear modeling, growth curve modeling, structural equation and growth mixture modeling. D. QUALITY OF THE MANAGEMENT PLAN D.1. Adequacy of the management plan Dr. Dana Diesel Wallace, NCNSP Vice President for School Development and a member of the NCNSP Executive Team, will serve as PI devoting 10% of her time to this project. She will 30

31 have final decision making authority and make regular reports on program progress to the NCNSP Executive Team. Dr. John Denning, Senior Director for Policy and Engagement, will serve as a Co-PI on the project, providing 10% of his time and effort to supporting the project s administration and implementation particularly with regard to Goal 3, building a platform for project expansion and communicating with policy leaders in North Carolina and beyond. Ms. Sofi Frankowski, Senior Director for School Development, manages the instructional coaching team and process at the NCNSP. She will manage the Instructional Coaches for the project schools devoting 10% of her time to this role. Ms. Wendi Knapke, NCNSP s Director of Finance and Administration will manage the financial operation of the grant, devoting 25% of her time and effort to this work. The NCNSP Executive Team comprised of: Dr. Tony Habit, President; Dr. Wallace; Lynne Garrison, Vice President of Strategic Partnerships and Engagement; Sofi Frankowski, and Dr. Denning will support all operations including raising the necessary private matching funds as described in the budget narrative. A full time Project Director will be hired to manage the day-to-day operations of the project, coordinate all program activities, supervise the Leadership and Instructional Coaches, liaison with the partner school districts, IHEs and independent evaluators, be in charge of submitting required reports, work closely with Dr. Wallace as an advisor on key decisions, and be the lead contact for the U. S. Department of Education. Supporting the Project Director, the grant also budgets for a 25% full-time equivalent Contract Manager/Budget Analyst to work closely with the eight school districts, and a 25% full-time equivalent Administrative Assistant to support all operations. This project will also use the existing support structure of the NCNSP network of 106 schools throughout the state as a management platform. The NCNSP employs and contracts with a strong group of professional educators who serve as Leadership and Instructional Coaches to provide the professional development workshops that make up the IS 4 services. Each school will 31

32 also be assigned a Portfolio Manager who will oversee the integration of services. Table 3 shows the project timeline for representative major tasks in initiating the project and beginning with the first cohort of schools. It is based on a start date of April The second and third cohorts of schools will follow a similar pattern with initial sessions earlier in the spring semester to facilitate appropriate student scheduling. Additional tasks related to dissemination will be scheduled in the third through fifth years of the project. Table 3: Major tasks and Milestones. Task/Milestone Responsible * When Sign award agreement, contracts, charts of accounts in place WK Y1Q1 Hire Project Director DW, JD Y1Q1 Meet with evaluator to assure data collection mechanisms in place DW, JE, PD Y1Q1 Strategic planning meeting with SBE and Community College DW, JD Y1Q1 Meet with all Y1 districts and schools and IHE partners SF, PD Y1Q1 Explain budgeting procedures to LEA financial staff WK Y1Q1 Assign Portfolio Managers, Leadership and Instructional Coaches DW, SF, PD Y1Q1 Begin Leadership Coaching Services PD Y1Q1 Carry out initial school assessments using Design Principles Rubric Develop School Action Plan based on assessment Principals/PMs Y1Q1 Principals/PMs Y1Q1 Conduct awareness sessions for guidance counselors PMs Y1Q1 Conduct District Leadership Professional Development (ongoing) PD Y1Q1 Schedule Summer Professional Development Sessions PD Y1Q1 Districts develop communication plans for community engagement LEAs Y1Q2 Summer Professional Development Institutes (annual) PD Y1Q2 32

33 Task/Milestone Responsible * When Begin Instructional Coaching (ongoing) PD Y1Q2 Quarterly Leadership Innovation Network meetings LCs/Principals Every Q Project Staff meet with evaluators DW, JE Every Q Budget meetings WK, DW monthly Attend Department of Education i3 meetings DW, JD, PD As req. Annual updates to State Board of Education, IHE and LEA partners DW, JD annually Submit progress reports to Department of Education DW As req. Carry out school assessments for each year s progress Update school action plan based on assessment Principals/PMs annually Principals/PMs annually Continue IS 4 Services PMs ongoing * Note: DW=Dana Diesel Wallace, JD= John Denning, SF=Sofi Frankowski, WK= Wendi Knapke, PD= Project Director (to be hired), JE= Julie Edmunds (SERVE- lead evaluator), LC= Leadership Coaches, PM=Portfolio Managers D.2. Qualifications (training and experience) of project director and key personnel Dr. Wallace has a history of leading progressive initiatives in a variety of teaching and administrative positions including Director of Middle School Education in Wake County Schools, North Carolina, and Superintendent of West Fargo, North Dakota School District. She has a BS in education from Old Dominion University, a M.Ed. from Harvard University Graduate School of Education, and an Ed.D. from Teachers College, Columbia University. Dr. John Denning is responsible for management and leadership for strategic partnerships, research, communications and government relations for NCNSP. Previously, he was senior manager for federal and state policy initiatives with the All Kinds of Minds Institute in Chapel 33

34 Hill, a non-profit organization that helps teachers, parents and students better understand and use brain research to support powerful teaching and learning. Denning helped establish the Southeast Center for Teaching Quality, now the Center for Teaching Quality, a regional office of the National Commission on Teaching and America's Future. As associate director there, Denning led the organization s teaching quality policy tracking teams and provided outreach and public engagement for its policy agenda. Denning holds a bachelor s degree in history and political science from Elon College, a master s degree in public administration from Exeter University, and a doctoral degree in educational leadership and policy from UNC-Chapel Hill. With 12 years of experience as a classroom teacher, including being named 2002 Teacher of the Year in Wake County schools, Sofi Frankowski has worked with strategies to de-track high school classrooms to ensure an academically rigorous experience for all students. A National Board Certified teacher, she has served in a variety of leadership roles, including mentor teacher, member of the Wake County Superintendent's Teacher Advisory Council and member of the Wake Education Partnership s Board of Directors. Frankowski holds a bachelor s degree in public policy studies from Duke University and a master s degree in education from Stanford University. She is also a graduate of the NC Education Policy Fellowship Program. Wendi Knapke brings more than 14 years of experience in the field of accounting. The majority of her career was spent at Ernst & Young, LLP, in the audit and business advisory group, serving both the not-for-profit and healthcare industries. Knapke holds a bachelor's degree in accounting from Wright State University in Dayton, Ohio, and is a certified public accountant. The project director to be hired will be experienced in working with high need school districts, have administrative experience in North Carolina schools and be a leader in school reform. The successful candidate will have experience in managing large, complex, grant funded projects and 34

35 will have knowledge and understanding of the North Carolina policy environment. The SERVE Center at UNC-Greensboro will carry out the independent evaluation. Dr. Julie Edmunds will be the Lead Program Evaluator. See Section C.4. for evaluators qualifications. D.3. Eligible applicant s capacity to bring the project to scale The NCNSP works to accelerate systemic, sustainable innovation in secondary schools across the state. NCNSP was established in 2003 by the state s Education Cabinet and the Office of the Governor with the support of the Bill & Melinda Gates Foundation. Operating as an independent not-for-profit corporation, NCNSP is governed by a Board of Directors chaired by Burley Mitchell, former chief justice of the North Carolina Supreme Court. NCNSP carries out its mission through an aggressive, three-pronged strategy of: 1) Creating innovative, highly effective high schools across North Carolina; 2) Building a statewide consensus for significant change; and 3) Advancing policies that promote innovation, higher standards and improved performance. With local, state and national partners, NCNSP has launched an unprecedented effort and created more than 100 innovative high schools across North Carolina. These innovative high schools offer all students an academically rigorous curriculum grounded in the skills needed to succeed in college and the 21st century workplace. While NCNSP has developed the funding and relationships to support an expanding network of innovative schools, the support of i3 grant funding will allow NCNSP to extend and validate its Design Principles, ECHS strategies and IS 4 professional development strategies and to advocate for scaling these in all North Carolina high schools. Furthermore, through close ties with similar programs in other states and the Bill & Melinda Gates Foundation, the NCNSP is well positioned to support scaling up the project to other states by sharing tools and serving as a model program. 35