An Evaluation of the National Institute for School Leadership: Executive Development Program in Milwaukee Public Schools INTERIM REPORT YEAR TWO

Transcription

1 An Evaluation of the National Institute for School Leadership: Executive Development Program in Milwaukee Public Schools INTERIM REPORT YEAR TWO Roisin P. Corcoran, Ph.D. Joseph M. Reilly Steven M. Ross, Ph.D. The Center for Research and Reform in Education Johns Hopkins University September, 2014

2 NISL Cohort #3 Report 1 Introduction The National Institute for School Leadership s (NISL) Executive Development Program (EDP) is a comprehensive professional development series designed to improve the administrative skills of school principals across the United States. Through traditional classroom instruction, interactive online curriculum, coaching, and other resources; the EDP provides for a yearlong professional development program that includes four courses and 13 units. This report describes the early evaluation results of the program as implemented in the Milwaukee Public School System (MPS). To date, NISL has produced three cohorts of approximately 30 principals each who have successfully completed the yearlong professional development program while serving as fulltime principals in the MPS district. Currently, NISL has a fourth cohort of MPS participants scheduled to complete the program in summer, The present evaluation study employs a mixed-method design to determine the impact of the NISL-EDP in Milwaukee Public Schools, as perceived by participating principals and as measured by student achievement and school climate outcomes. During the study s first year, the program s impact on the MPS principals who participated in EDP Cohorts 1 and 2 (a total of approximately 60 principals) was evaluated and reported. The present interim report summarizes the interview and survey data collected from EDP Cohort 3 during the study s second year, and addresses only the outcomes related to the quality of the program and its impact on principals activities in the field (Evaluation Questions 1 and 2). Study Design A mixed-method approach was used to collect both quantitative and qualitative data from EDP Cohort 3. CRRE solicited the participation of all individuals who participated in the NISL-EDP as a part of Cohort 3 in a survey covering participant reactions to the program. A sample of five individuals from Cohort 3 was selected to be interviewed as well in order to provide more in-depth feedback pertaining to

3 NISL Cohort #3 Report 2 the program. This interim report reflects analysis of the data collected through these activities. The participant survey and interview instruments are both presented in the Appendix section. Evaluation Questions Listed below are the specific evaluation questions for the present study: 1. How do program participants evaluate the NISL-EDP with regard to perceived (a) quality of curriculum and instruction/training, and (b) reasonableness of requirements and demands? What components of the program appeared strongest? Which were weakest? 2. How do program participants evaluate their NISL-EDP experiences with regard to improving their leadership activities in the field? Which components were most helpful? Which were least helpful? What types of skills and knowledge were used most? What types of skills and knowledge were used least? 3. How do trends in math performance differ between schools served by NISL-EDP principals and comparison schools served by non-participating principals at the elementary, middle, and high school levels? 4. How do trends in reading performance differ between schools served by NISL-EDP principals and comparison schools served by non-participating principals at the elementary, middle, and high school levels? 5. How do trends in school climate ratings differ between schools served by NISL-EDP principals and comparison schools served by non-participating principals at the elementary, middle, and high school levels? Participants All 30 NISL Cohort 3 participants were invited to take the survey in the winter of The survey administration window lasted from March-June of To help increase the participant

4 NISL Cohort #3 Report 3 response rate, MPS assisted in contacting participants to solicit participation in the survey. Ultimately, 15 total principals participated in the survey for a response rate of 50.0%. For the interview activity, a sample of five Cohort 3 participants was selected and interviewed during the spring of Measures Principal survey. The survey was comprised of open-ended short answer questions and questions using multiple variations of a 5-point Likert scale, where a higher Likert scale rating would signify a greater degree of support for the program. Response means (M) and standard deviations (SD) across the entire participating population were computed and descriptively analyzed (i.e., the numeric averages of Likert responses for all participant surveys combined). For the survey s open-ended questions, qualitative analysis was performed to identify notable trends among participant responses using a method of categorizing and totaling different types of participant answers. Principal interview. Using similar methods to the open-ended survey question analysis, interview responses were also analyzed qualitatively to identify trends among participants responses. Specifically, participant responses were categorized and totaled for each of the interview questions. Participant responses that reflected divergent opinions from the overall group consensus were analyzed in relation to the participant s other responses in order to gain clarity and context. Results Survey Participants All 15 principals from Cohort 3 who completed the survey successfully completed the NISL- EDP program. Furthermore, 100% are currently head principals and were head principals at the time they completed the program (N = 15). Participants ranged from having 3 to 30 years of experience as administrators and all participants indicated having experience teaching either on the elementary or secondary level prior to becoming an administrator. Lastly, the participants reported having a variety of

5 NISL Cohort #3 Report 4 professional development and training experiences prior to participating in the program. In this area, most principals listed university programs (often on a master s degree level) and principal certification coursework. A small portion of participants also listed other training experiences including district professional development initiatives and doctoral -level programs. Survey Findings Participation in the NISL-EDP program. Participants were asked what prompted them to participate in the program and what skills they hoped to develop from participating. All 15 participants responded to this open ended question. The following types of responses were reported most frequently: The desire to improve skills in school administration/improve school outcomes (N = 5) A colleague spoke highly of the program/recommended the program (N = 5) The district recommended the program/asked them to participate (N = 4) NISL-EDP program components. Participants reported, on average, that they found all five of the key program components useful (M = , SD = ). As shown in Table 1, participants rated the classroom instruction component (M = 4.33, SD = 0.98) and the Instructional Leadership Instrument component (M = 4.27, SD = 0.88) the highest. In turn, although participants generally expressed positive opinions toward the utility of the online curriculum (M = 3.20, SD = 1.32) and coaching component (M = 3.53, SD = 1.73), less consensus existed pertaining to the degree of these components usefulness. Table 1 Principal Survey Item #12 a-e, Response Statistics and Frequencies Survey Item a. The Instructional Leadership Instrument Strongly disagree Somewhat disagree Neither agree nor disagree Somewhat agree Strongly agree 0.0% 6.7% 6.7% 40.0% 46.7% M SD

6 NISL Cohort #3 Report 5 was useful in helping me assess and reflect upon my strengths and weaknesses as an educational leader. b. I believe the classroom instruction component of the Executive Development Program 0.0% 6.7% 13.3% 20.0% 60.0% was useful in helping me improve as an educational leader. c. I believe the online curriculum component of the Executive Development Program 13.3% 13.3% 33.3% 20.0% 20.0% was useful in helping me improve as an educational leader. d. *I believe that the Action Learning Project was useful in helping me 14.3% 0.0% 7.1% 28.6% 50.0% improve as an educational leader. e. I believe that the coaching component was useful in helping me 26.7% 0.0% 13.3% 13.3% 46.7% improve as an educational leader. *Likert Scale: 1-Strongly disagree, 5-Strongly agree Note. N = 15 response rate for all items except 12d (N = 14 response rate for this item). For item 12d, valid percentages are reported (the percentage of participants who completed the survey item). Participants were also asked follow-up questions on the survey pertaining to what (if anything) they would change about the program components. The majority of participants did not suggest any changes to the program components when asked these questions. Of those who made recommendations, the following suggestions were made most frequently: Pertaining to the coaching component, three participants suggested having more visits/more time spent with their coaches.

7 NISL Cohort #3 Report 6 Pertaining to the online curriculum, two participants indicated that they were not always able to successfully access the component. Pertaining to the online curriculum, two participants indicated that they preferred more interaction than the component offered. Pertaining to the Action Learning Project, two participants indicated that they would have liked more time to process and implement this project. Program units. Participants reported, on average, that they found all 13 of the program units useful in helping them improve as educational leaders (M = , SD = ). Participants rated The Principal as a Strategic Thinker (M = 4.00, SD = 1.71), Leadership for Excellence in Science (M = 4.00, SD = 1.36), and Leading for Results (M = 4.00, SD = 1.51) the highest. Although also rated on average by participants as being useful, the simulation activities were generally not rated as favorably as the other units. Specifically, the Culminating Simulation (M = 3.53, SD = 1.51) and the Phase One Simulation (M = 3.67, SD = 1.40) received the lowest and third lowest participant ratings of the 13 units respectively. Table 2 Principal Survey Item #13 a-m, Response Statistics and Frequencies I believe that the unit (items A-M, below) helped me improve as an educational leader Survey Item (Unit Name) Strongly disagree Somewhat disagree Neither agree nor Somewhat agree Strongly agree M SD disagree a. The Educational 20.0% 0.0% 6.7% 13.3% 60.0% Challenge b. *The Principal 21.4% 0.0% 7.1% 0.0% 71.4% as a Strategic Thinker c. Elements of Standards- Based Instructional Sys/School Designs 13.3% 6.7% 6.7% 20.0% 53.3%

8 NISL Cohort #3 Report 7 d. Foundations of 20.0% 0.0% 6.7% 33.3% 40.0% Effective Learning e. Leadership for 13.3% 13.3% 6.7% 33.3% 33.3% Excellence in Literacy f. Leadership for 0.0% 13.3% 20.0% 26.7% 40.0% Excellence in Math g. Leadership for 6.7% 13.3% 6.7% 20.0% 53.3% Excellence in Science h. Promoting 6.7% 20.0% 13.3% 20.0% 40.0% Professional Learning and Phase One Simulation i. The Principal as 20.0% 6.7% 6.7% 13.3% 53.3% Instructional Leader and Team Builder j. *The Principal 21.4% 0.0% 7.1% 21.4% 50.0% as Ethical Leader k. The Principal as 20.0% 0.0% 6.7% 13.3% 60.0% Driver of Change l. Leading for 13.3% 6.7% 6.7% 13.3% 60.0% Results m. Culminating 20.0% 0.0% 20.0% 26.7% 33.3% Simulation *Likert Scale: 1-Strongly disagree, 5-Strongly agree Note. N = 15 response rate for all items except 13b and 13j (N = 14 response rate for these items). For items with N = 14 response rate, valid percentages are reported (the percentage of participants who completed the survey item). Participants were also asked follow up questions concerning their reactions to the EDP Units. Specifically, participants were asked questions concerning the characteristics of the units they found most and least useful, and what (if anything) they d change about the course units.

9 NISL Cohort #3 Report 8 Participants were first asked; of the course units that they thought helped them improve the most as educational leaders, what made these units most effective. Of those that responded (N = 12, 80.0%), the following areas were reported most frequently: Unit specific instruction or content (N = 6) Classroom discussions/collaboration with classmates and instructors (N = 5) Program materials including readings and videos (N = 2) Participants were then asked, of the course units that they thought helped them improve the least as educational leaders, what they felt made these units least effective. Of those that responded (N = 9, 60.0%), the instruction or content specific to certain units was reported most frequently (N = 7). Units mentioned in this area included the Simulation Activities (N = 3), Leadership for Excellence in Literacy (N = 2), and Leadership for Excellence in Math (N = 2). Participants were also asked what (if anything) they would change about any of the course units. Of those that responded (N = 4, 26.7%), two participants suggested that the program make improvements to how well the instruction is connected with district led initiatives. One participant indicated they would like Feedback on the simulations and one participant indicated that the course units could be more in depth. Program and district demands. Participants reported, on average, that they felt the curriculum materials used by the program were relevant to their professional development needs (M = 3.87, SD = 1.60). Participants, on average, also reported that they felt the requirements of the program were reasonable to effectively complete while working full time as principals (M = 3.80, SD = 1.47). Table 3 Principal Survey Item #17 a-b, Response Statistics and Frequencies Survey Item Strongly disagree Somewhat disagree Neither agree nor Somewhat agree Strongly agree M SD

10 NISL Cohort #3 Report 9 disagree a. I feel the curriculum and materials used by the Executive Development Program 20.0% 0.0% 6.7% 20.0% 53.3% were relevant to my professional development needs. b. I feel that the requirements and demands of the Executive Development Program are reasonable for a 13.3% 6.7% 13.3% 20.0% 46.7% participant to effectively complete while working full time as a principal. *Likert Scale: 1-Strongly disagree, 5-Strongly agree Also pertaining to this area, participants were asked if there were any hurdles in being able to meet the demands of the program while serving as a full-time principal and if so how did they address them. Of those who responded (N = 12, 80.0%), the following hurdles were reported most frequently: Amount of time spent out of their school buildings (N = 7) Lack of time (N = 3) Participants were also asked how supportive their supervisor and the district central office is of the changes they were making as a result of the program and also if there are any barriers in the district that make it difficult for them to implement the program s teachings. Of those who responded (N = 13, 86.7%), the following types of responses were reported most frequently: The district was supportive (N = 6) Lack of alignment between district professional development and NISL/District initiatives interfere with fully implementing NISL teachings (N = 2 )

11 NISL Cohort #3 Report 10 Lastly in this area, participants were asked what (if any) changes needed to be made in the MPS district to help make principals more effective. Of those who responded (N = 13, 86.7%), the following suggestions were made most frequently: NISL certify more district staff/ have the district more fully embrace NISL teachings (N = 3) Have the district exhibit more trust in principals/ provide principals with greater autonomy (N = 2) Move away from a one size fits all approach (N = 2) Employ a more direct focus on student achievement (N = 2) Increase opportunities for principals to interact (N = 2) Program influence on school and student outcomes. Participants, on average, reported that their schools performance had improved since their completion of the program in multiple areas. Specifically, participants reported the greatest improvements in school climate (M = 4.57, SD = 0.51) and school attendance (M = 4.36, SD = 0.63). Although not as large, participants also reported improvements in school-wide achievement across all four core content areas: language arts (M = 3.86, SD = 0.53), mathematics (M = 3.86, SD = 0.66), science (M = 3.86, SD = 0.53), and social studies (M = 3.86, SD = 0.53). Table 4 Principal Survey Item #21 a-f, Response Statistics and Frequencies For each (items a-f, below) please rate your school s performance now compared to your school s performance prior to your completion of the Executive Development Program Survey Item Much Somewhat No Somewhat Much M SD weaker weaker difference stronger stronger a. School-wide 0.0% 0.0% 21.4% 71.4% 7.1% student achievement in language arts b. School-wide student achievement in mathematics 0.0% 0.0% 28.6% 57.1% 14.3%

12 NISL Cohort #3 Report 11 c. School-wide 0.0% 0.0% 21.4% 71.4% 7.1% student achievement in science d. School-wide 0.0% 0.0% 21.4% 71.4% 7.1% students achievement in social studies e. Overall student 0.0% 0.0% 7.1% 50.0% 42.9% attendance f. Overall school 0.0% 0.0% 0.0% 42.9% 57.1% climate *Likert Scale: 1-Much weaker, 5-Much stronger Also pertaining to this area, participants were asked an open-ended question concerning the performance of their school. Specifically, participants were asked if the performance of their school had improved since their participation in the program (if so how) and whether or not they feel the improvements are a result of the program. Of those participants who responded (N = 14, 93.3%), 11 (78.6%) participants indicated that their school s performance had improved in some way; two (14.3%) participants indicated that performance had not yet improved, and one (7.1%) described how their administrative skills had improved. Of those who indicated that improvements had occurred in their school (N = 11) the following types of improvements were listed most frequently: Improvements in school climate (N = 5) Improvements in academic performance (N = 5) Improvements of the school s vision (N = 5) Decreased suspensions (N = 3) Improvements in attendance (N = 3) General school-based improvements (N = 2)

13 NISL Cohort #3 Report 12 Program influence on professional skills and competencies. As shown in Table 5, participants reported that the program had a positive impact on improving their performance as principals across a variety of domains. In particular, participants rated the strongest improvements occurring in their abilities to formulate clear visions for their school communities (M = 4.57, SD = 0.65) and use data to produce improvements in instruction and student achievement (M = 4.43, SD = 0.76). Table 5 Principal Survey Item #24 a-f, Response Statistics and Frequencies For each (items a-f, below) please rate your performance now compared to your performance prior to your completion of the Executive Development Program Survey Item Much Somewhat No Somewhat Much M SD weaker weaker difference stronger stronger a. Formulating a clear 0.0% 0.0% 7.1% 28.6% 64.3% vision to inspire others in the school community b. Implementing fullyaligned, 0.0% 0.0% 7.1% 50.0% 42.9% standards- based instructional systems c. Building effective 0.0% 0.0% 14.3% 57.1% 28.6% instructional programs in the core academic subjects, particularly math, language arts, and science d. Using data to produce 0.0% 0.0% 14.3% 28.6% 57.1% continuous improvements in instruction and student achievement e. Providing effective 0.0% 0.0% 14.3% 42.9% 42.9% training programs to build a professional learning community for school faculty and staff f. Creating integrated school improvement plans that reflect strategic and systemic thinking 0.0% 0.0% 14.3% 42.9% 42.9%

14 NISL Cohort #3 Report 13 *Likert Scale: 1-Much weaker, 5-Much stronger Also pertaining to this area, participants were asked what elements of the program they have been able to implement in their schools and whether or not they felt that these implementations helped them become more successful administrators. Of those who responded (N = 14, 93.3%), the following responses were reported most frequently: The Principal as a Strategic Thinker (N = 8) Refining/improving their schools vision (N = 3) Elements of Standards-Based Instruction Systems (N =2) The use of safety nets (N = 2) Use of NISL activities with staff (N = 2) The Action Learning Project (N = 2) Refining/improving their philosophical approach to school administration (N = 2) Instruction related to teaching and learning (N = 2) Participants were also asked which components of the program they felt to be most beneficial in helping them improve as educational leaders. Of those participants who responded (N = 11, 73.3%), the following responses were reported most frequently: Instruction from specific program units (N = 15) Instruction on creating a school vision (N = 3) Collaboration with other principals (N = 2) Cultivating professional awareness or systemic thinking (N = 2) Furthermore, participants were also asked which components of the program they felt to be least beneficial in helping them improve as educational leaders. Of those participants who responded (N = 10, 66.7%), the following responses were reported most frequently:

15 NISL Cohort #3 Report 14 None (N = 3) Online component (N = 3) Changes and closing thoughts concerning the program. Participants were asked open-ended questions pertaining to what (if anything) they would change about the program and were also asked if there was anything else they would like to share concerning their experience with the program. When asked what (if anything) they would change about the program, 10 participants responded (66.7%). Of those who responded, the following answers were reported most frequently: The program shouldn t change anything (N = 3) Scheduling (amount of time out of their school buildings) (N = 3) Increase the duration of the program (N = 2) When asked if there was anything else they would like to share about the program, nine participants provided additional comments (60.0%). Below is a sample of the additional comments shared: Great program I love the NISL staff! They were inspiring and made connections to our work. It was a pleasure to share the time with them. Too much time spent listening to the presenter share personal experience The program was instrumental in structuring my team, professional learning communities, and overall growth plan for improving culture and student achievement. The leaders from the program were effective, caring and knowledgeable. I enjoyed my opportunity in the program. I would have appreciated more time to process the material. I miss meeting with my colleagues in a format of professional learning and networking. The instructors are very well informed and brings to the program unique ideas and information Principal Interview Findings Telephone interviews were conducted during the spring of 2014 with a sample of five principals from NISL Cohort 3. The interview covered topics pertaining to the participants perceptions of the

16 NISL Cohort #3 Report 15 program s the strengths and weaknesses as well as the program s influence on their professional practice. The principal interview script can be found in the Appendix section. Interview participants. Participants ranged from having 5 to 22 years of experience and reported a variety of different experiences in the field of education prior to becoming administrators including teaching, district curriculum development, and university administration. All five participants indicated that they were assistant principals before becoming head principals and reported a variety of principal training activities prior to participating in NISL. Two participants indicated that they received their initial administrative training through the New Leaders for New Schools Program while three others indicated that their administrative training came through university coursework. Additionally, principals reported participating in district led professional development initiatives (often in the form of single-day workshops), and multiple participants indicated that they had received Master s degrees in education based fields. Recruitment and participation in NISL. Participants were asked how they learned about the NISL-EDP program and what ultimately led them to participate. All five participants indicated that the district communicated that the program was being offered. Two participants indicated that they heard about the program during district led principal meetings and one participant indicated that the district recruited them to participate. Two participants indicated that the program was appealing to their personal interests in the field. One participant indicated that they participated because they felt it was the politically correct thing to do at the time. The program s most beneficial components. Participants were asked which components of the program they found to be most beneficial in helping them improve as educational leaders. Four of the five participants listed multiple facets of the program that they felt to be most beneficial (one participant indicated that they felt the program largely repeated what they learned through participating in the New

17 NISL Cohort #3 Report 16 Leaders for New Schools program and did not list any components as strengths). Three participants indicated that they found the Principal as a Strategic Thinker instruction to be most beneficial. Two participants indicated that they felt the sense of urgency that the program helped them develop was highly beneficial. Other responses included instruction on strategically refining a schools vision, The Elements of Standards Based Instruciton unit, The Principal as an Ethical Leader unit, the content units for math and science, the applied learning project, and the professional learning communities. Selected participant remarks included: I would have to say really being able to refine my school s vision and making sure that once we did finally refine and revise it so that it had a sense of urgency and a focus on students to make sure that everything else um was aligned to the vision of the school. (P02) So I found that the principal as a strategic thinker I use most often in terms of um thinking about not just what kind of change I want to make but how do I communicate that change? What might be my resistors to change, how might I identify some barriers and obstacles in advance and create some allies along the way? (P03) The program s least beneficial components. Participants were asked which components of the program they found to be least beneficial in helping them improve as educational leaders. Four of the five participants listed facets of the program they thought were least beneficial and one participant indicated that they found everything to be beneficial. Three of the participants indicated they felt the content units (i.e., Leadership for Excellence in Literacy, Math, and Science) were least beneficial. One participant indicated that they felt the simulation activities were not as beneficial as they could have been because the participants did not receive feedback on these assignments. Selected participant remarks included: In the middle of the course and then at the end of the course we did what they called simulation activities which were um like web-based questionnaires and in this situation what would you do as the leader? and I, I didn t really see the purpose of that because we got no feedback. (P03)

18 NISL Cohort #3 Report 17 All that dealt with literacy, math, and science we re not content experts within all content areas so I thought it would be beneficial to have a content area leader/department chair present for that specific professional development. (P05) District hurdles. Participants were asked if there were any hurdles in their school or district that they had to overcome in implementing what they learned in the program. Four out of the five participants indicated that they did experience hurdles of this sort although the type of hurdle varied between participants. One participant indicated that getting their staff to buy into the NISL paradigm shift has been a challenge. One participant reported that they felt the NISL teachings didn t always coincide with what the district required principals to do. Another participant indicated that many district initiatives that were in place before the NISL program was offered didn t directly align with NISL instruction. Lastly, one participant indicated that they experienced some criticism from the district as they tried to implement components of the NISL program in their framework for tier two. Selected participant remarks included: The paradigm shift probably was the biggest hurdle that I had to um work on and am still working on with my staff, um for really being able to turn the building around..if there was like a NISL for teachers that would be awesome. (P02) Fitting in NISL philosophy and with sort of already in place expectations (P04) Program demands. Participants were asked if there were any hurdles in being able to meet the demands of the program as well as meet the demands of their job as a principal while completing the program. Three of the five participants indicated that demands of this sort did exist. All three of these participants indicated that demands with scheduling; particularly leaving their schools for two consecutive days each month, was challenging. One participant indicated that attending the NISL sessions during the week of Thanksgiving break and around winter break was particularly difficult. Selected participant remarks included: It just seemed to feel while going through the process that I was out of the building a lot and so um, being able to get the staff to understand that this is why I m out of the building, this is how

19 NISL Cohort #3 Report 18 its gonna benefit us, and then really being strategic about uh, rotating who was covering the building because I don t have an assistant principal (P02) I do see the value of having the two back to back days but from the principal point of view, ya know, always in the back of your mind is what s going on at the building? it s a distraction to have that going on while you are trying to focus on this (P03) Application of NISL-EDP teachings. Participants were asked how they have applied what they learned in the program. Four of the five participants indicated that they have applied what they learned, and one participant indicated that they were already doing everything taught in the program before they participated. Those participants who indicated they have applied teachings from NISL in their practice provided a variety of examples. Two participants indicated that they now think more strategically as principals as a result of the NISL instruction. Two participants indicated that they have used some of the materials (including articles and YouTube videos) they received through the program with their staff. Other examples shared by participants included modelling school professional development sessions after NISL instruction, more keenly focusing on the school vision, communicating a greater sense of urgency with school staff, and utilizing the NISL Wheel. Selected participant remarks included: I probably would say that professional development that I m able to offer is directly influenced by this professional development that I received (P03) Well I m a strategic thinker now---everything has a contingency plan---think vision first, resources second, um, generating consensus to make change, creating the conditions to um build consensus, analyzing resources, using data more effectively. Ya know..it s routine. (P05) Participant performance. Participants were asked if they felt the program made an impact on their performance as administrators or in the performance of their school. Four of five participants indicated that they felt the program had an impact on their performance as administrators and three of the five participants gave examples of how the program positively influenced their school s performance. Two participants indicated that they looked at data differently and more often as a result of the program. One participant indicated that they were able to use their Action Learning Project to help

20 NISL Cohort #3 Report 19 improve student writing achievement. One participant indicated that the program helped them change their expectations about what a successful school should look like. One participant indicated that they are in classrooms more often and participate more in school meetings as a result of the program. Lastly, one participant cited a variety of personal attributes they felt the program helped them cultivate including strategic thinking and the ability to effectively communicate school wide-initiatives. Selected participant remarks included: Our school climate data is, has been great---i mean our attendance has been up, our suspensions are down they re way down ya know would we have done that anyway if I wasn t in NISL last year? Maybe but it certainly, it certainly spurred action. (P04) I m more efficient. I m more effective. I think I think more clearly. I think more strategically. And I can communicate school-wide initiatives more cogently. And that leads to greater achievement and more cohesiveness. (P05) Student and school achievement. Participants were asked if they thought the program helped them raise student achievement in their schools. Three of the five participants reported that they felt the program has helped them raise student achievement. One participant indicated that improving their ability to think strategically and communicate a sense of urgency within their building was helping improve student performance. One participant who reported being in their first year as a principal at their school indicated that her school increased their MAP scores 12% from spring to fall. She attributed this gain to having meetings with teachers and students on understanding student test scores. Lastly, one participant indicated that although they had not seen an increase in student performance on the City Report Card they had seen improvements in the data they were collecting internally. Selected participant remarks included: We saw a 12% increase in our MAP data in the fall! Yes, 12% higher proficiency rate. And that was with no instruction between spring and fall just telling them take this seriously, this is what your, this is the number that means you re on grade level, this is the number that means you re college ready just that awareness was powerful (P03)

21 NISL Cohort #3 Report 20 According to City Report Card no. But what we know, and what we can see via our data is that we are meeting the needs of more minimal learners and moving them towards basic (P05) Recommending the NISL program. Participants were asked if they would recommend the program to another principal. Four of the five participants indicated they would recommend the program to other principals. The remaining participant indicated that they would recommend the program to principals who came through a traditional university based training program but not those who went through the New Leaders for New Schools program. Participants were then asked a follow up question pertaining to what type of principal (if any) the program would be best suited for. Three participants indicated that they felt the program was best for principals with a few years of experience in the field as opposed to brand new principals. The participants reasons varied for recommending this, but generally centered on making sure that principals had sufficient experience and expertise in school administration before participating. One participant also suggested that the program would be beneficial for principals with substantial experience but who may have become uneasy after witnessing many education reform initiatives. Finally, one participant recommended that assistant principals also be invited to participate. Selected participant remarks included: I would recommend the program to two different types of principals: One, um someone who is within their first five years of being a principal who um, has this passion but they ve had like a year to actually be a principal, and um can have something to root everything in I would also say someone who still has this passion to be a principal but they re feeling a little uneasy about all of the change and how do I stay focused in the midst of change? I think that uh, would be an ideal. (P02) I actually think it is better to have maybe a couple of years of experience before you go into this so that you have something to compare to and some kind of maybe past practices to challenge. I don t know if you re a brand new principal if ya know, if you would have that same perspective. (P03)

22 NISL Cohort #3 Report 21 Changes to the NISL program. Participants were asked what (if anything) they would change about the program. Four of the five participants made suggestions for changes that could be made to the program. The remaining participant indicated that they wouldn t change anything but would like the district to add a NISL booster session every few months for principals that had already completed the program. Of the participants who recommended changes, three suggested changes that would reduce the amount of time they had to leave their school buildings to attend the program sessions. Participant suggestions in this area included condensing the program sessions to one day per month, holding the program sessions on Saturdays, and holding more of the program sessions during the summer and over winter and spring break. Lastly, one participant suggested having school principals bring their building content leaders to attend the program units in literacy, math, and science. Conclusions

23 NISL Cohort #3 Report 22 Appendix NISL Executive Development Program: Milwaukee Public Schools Principal Survey The purpose of this survey is to determine your reactions regarding the NISL Executive Development Program. The results will be used by NISL Executive Development Program developers as well as Milwaukee Public Schools to identify the program s strengths, weaknesses, and directions needed for improvement. Thus, it is very important that you react honestly and candidly. Your responses and identity will be treated confidentially. Thank you for your participation! Please tell us about your administrative experience. 1. I have been an administrator for years. 2. I have been an administrator in my current school for years. 3. What is your current administrative role at your school? o Head Principal o Assistant Principal o Academic Department Chair o Instructional Support Teacher o Other 4. If you have already completed the NISL Executive Development Program, what was your administrative role at your school when you were participating with the program? o Head Principal o Assistant Principal o Academic Department Chair o Instructional Support Teacher o Other 5. Prior to becoming an administrator, how many years did you teach on the elementary level (grades K- 5 th )? 6. Prior to becoming an administrator, how many years did you teach on the secondary level (grades 6 th - 12th)? 7. Prior to becoming an administrator, please describe other administrative or teaching experiences you have had (subjects taught, other administrative roles, etc.).

24 NISL Cohort #3 Report Please describe the principal training you received prior to participating in the Executive Development Program (university coursework, professional development workshops, etc.). 9. What prompted you to participate in the NISL Executive Development Program? What administrative skills did you hope to improve upon as a result of participating with the program? 10. Did you fully complete the NISL Executive Development Program? o Yes o No o In Progress The Executive Development Program has several main components including the Instructional Leadership Instrument, classroom instruction, online curriculum, the Action Learning Project and coaching. Please tell us about your experience with each of these components. 11. The Instructional Leadership Instrument was useful in helping me assess and reflect upon my strengths and weaknesses as an educational leader. o Strongly disagree o Somewhat disagree o Neither agree nor disagree o Somewhat agree o Strongly agree 12. What (if anything) would you change about The Instructional Leadership Instrument? 13. I believe the classroom instruction component of the Executive Development Program was useful in helping me improve as an educational leader.

25 NISL Cohort #3 Report 24 o o o o o Strongly disagree Somewhat disagree Neither agree nor disagree Somewhat agree Strongly agree 14. What (if anything) would you change about the classroom instruction component? 15. I believe the online curriculum component of the Executive Development Program was useful in helping me improve as an educational leader. o Strongly disagree o Somewhat disagree o Neither agree nor disagree o Somewhat agree o Strongly agree 16. What (if anything) would you change about the online curriculum component? 17. I believe that the Action Learning Project was useful in helping me improve as an educational leader. o Strongly disagree o Somewhat disagree o Neither agree nor disagree o Somewhat agree o Strongly agree 18. What (if anything) would you change about the Action Learning Project? 19. I believe that the Coaching component was useful in helping me improve as an educational leader. o Strongly disagree

26 NISL Cohort #3 Report 25 o o o o Somewhat disagree Neither agree nor disagree Somewhat agree Strongly agree 20. What (if anything) would you change about the Coaching component? Please tell us about your experience with each of the Executive Development Program course units. For each, please rate your level of agreement that each of the course units helped you improve as an educational leader. 21. I believe that the unit (items A-M, below) helped me improve as an educational leader. Unit Strongly disagree Somewhat disagree Neither agree nor disagree Somewhat agree Strongly agree A 1: The Educational Challenge B 2: The Principal as a Strategic Thinker C D E F G H I J K 3: Elements of Standards-Based Instructional Systems and School Design 4: Foundations of Effective Learning 5: Leadership for Excellence in Literacy 6: Leadership for Excellence in Math 7: Leadership for Excellence in Science 8: Promoting Professional Learning and Phase One Simulation 9: The Principal as Instructional Leader and Team Builder 10: The Principal as Ethical Leader 11: The Principal as Driver of Change

27 NISL Cohort #3 Report 26 L 12: Leading for Results M 13: Culminating Simulation 22. Of the course units above that you thought helped you improve the most as an educational leader, what did you feel made these units most effective? 23. Of the course units above that you thought helped you improve the least as an educational leader, what did you feel made these units least effective? 24. What (if anything) would you change about any of the course units above? Please tell us about your experience with the Executive Development Program in terms of program relevancy and program demands. 25. I feel the curriculum and materials used by the Executive Development Program were relevant to my professional development needs. o Strongly disagree o Somewhat disagree o Neither agree nor disagree o Somewhat agree o Strongly agree 26. I feel that the requirements and demands of the Executive Development Program are reasonable for participants to effectively complete while working full time as administrators. o Strongly disagree o Somewhat disagree o Neither agree nor disagree o Somewhat agree

28 NISL Cohort #3 Report 27 o Strongly agree 27. Were there any hurdles in being able to meet the demands of the program, as well as meet the demands of your job as an administrator while you were participating with the program? If so, what were they and how did you address them? 28. What from the Executive Development Program have you been able to implement in your school? Do you feel these implementations have helped you become more successful as an administrator? 29. Since participating in the Executive Development Program has the performance of your school improved? If so, how specifically has it improved (student test scores, school climate, school vision and plan, etc.)? Do you feel your participation in the Executive Development Program is related to these improvements? 30. For each (items A-F, below) please rate your schools performance now compared to your schools performance prior to your completion of the Executive Development Program. A B C Much weaker Somewhat weaker No difference Somewhat stronger Much stronger School-wide student achievement in language arts School-wide student achievement in mathematics School-wide student achievement in science D School-wide student achievement in social studies

29 NISL Cohort #3 Report 28 E Overall student attendance F Overall school climate 31. How supportive are your supervisor and other district central office staff to the changes you are making based on your Executive Development Program training? What (if any) barriers exist in your district that make it difficult to implement what you learned from the Executive Development Program? 32. What (if any) changes need to be made at Milwaukee Public Schools to help make principals more effective? Please think about the following aspects of your skills and knowledge as an educational leader. 33. For each (items A-F, below) please rate your performance now compared to your performance prior to completing the Executive Development Program. A B C D Formulating a clear vision to inspire others in the school community Implementing fully-aligned, standards-based instructional systems Building effective instructional programs in the core academic subjects, particularly math, language arts and science Using data to produce continuous improvements in instruction and student achievement Much weaker Somewhat weaker No difference Somewhat stronger Much stronger

30 NISL Cohort #3 Report 29 E F Providing effective training programs to build a professional learning community for school faculty and staff Creating integrated school improvement plans that reflect strategic and systemic thinking Please tell us your concluding thoughts concerning the Executive Development Program. 34. Overall, which components of the Executive Development Program did you find most beneficial in helping you improve as an educational leader? 35. Overall, which components of the Executive Development Program did you find least beneficial in helping you improve as an educational leader? 36. Overall, what would you change (if anything) about the Executive Development Program? 37. Is there anything else you would like to share concerning the Executive Development Program?

31 NISL Cohort #3 Report 30 NISL Executive Development Program: Milwaukee Public Schools Principal Interview Please tell us about your administrative experience. 1. Please describe your background as an administrator. Specifically how long have you been an administrator, and what kind of administrative training did you receive prior to participating in the Executive Development Program (university coursework, professional development workshops)? 2. How did you learn about the Executive Development Program? Were you recruited to participate? Why did you ultimately decide to participate with the program? Please tell us your lasting thoughts concerning the Executive Development Program 3. Which components of the Executive Development Program did you find most beneficial in helping you improve as an educational leader? 4. Which components of the Executive Development Program did you find least beneficial in helping you improve as an educational leader?

32 NISL Cohort #3 Report Were there any hurdles in your school or in your district you had to overcome in implementing what you have learned from the Executive Development Program? If so what are they and how have you addressed them? 6. Were there any hurdles in being able to meet the demands of the program as well as meet the demands of your job as a principal while you were participating with the program? If so what were they and how did you address them? 7. How have you applied what you learned from the Executive Development Program? 8. Do you think your participation with Executive Development Program has made an impact in your performance as an administrator? Has it made an impact in the performance of your school? Have student assessment scores and school climate data changed since your participation in the program? 9. If you believe the Executive Development Program helped you raise student achievement, how do you think it helped accomplish this?

33 NISL Cohort #3 Report Would you recommend the Executive Development Program to another principal? If so who do you think the program is ideally suited for? 11. What would you change (if anything) about the Executive Development Program?