RESEARCH. Assessing School Leader and Leadership Programme Effects on Pupil Learning. Conceptual and Methodological Challenges

Transcription

1 RESEARCH Assessing School Leader and Leadership Programme Effects on Pupil Learning Conceptual and Methodological Challenges Kenneth Leithwood Professor OISE/University of Toronto Ben Levin Deputy Minister of Education Government of Ontario Research Report RR662

2 Research Report No 662 Assessing School Leader and Leadership Programme Effects on Pupil Learning Conceptual and Methodological Challenges Kenneth Leithwood Professor OISE/University of Toronto Ben Levin Deputy Minister of Education Government of Ontario The views expressed in this report are the authors and do not necessarily reflect those of the Department for Education and Skills. Kenneth Leithwood and Ben Levin 2005 ISBN

3 Table of Contents Summary...3 Introduction...6 The Meaning of Leadership...6 Leadership Effects...7 Leadership Programme Effects...10 A General Framework to Guide Research on Programme and Leader Effects on Pupil Learning...12 Leadership Practices: The Independent Variables...14 Leadership Effects: The Dependent Variables...18 Mediating Variables for Leadership Effects Research...23 Variables Moderating Leadership Effects...29 The Antecedents of Leaders Practices...33 Evaluating Leadership Programme Effects...35 Methodological Challenges...40 Conclusions and Recommendations

4 Summary Leadership is widely considered a variable critical to school improvement; considerable evidence now justifies the claim that leadership has important effects on pupil learning. While largely indirect, evidence summarized in this paper indicates that such effects explain as much as a quarter of the variation in pupil learning across schools accounted for by school factors. These leadership effects, furthermore, are usually greatest where they are needed most; leadership acts as a catalyst unleashing the potential of other factors contributing to the improvement of pupil learning. Given the key role of leadership in school improvement, the Department for Education and Skills (DfES) commissioned this paper to identify the pathways by which school leadership programmes and practices influence, or positively impact on, pupil learning and to inform future research and development in this field. More precisely, our goals were to develop a model or framework of variables and relationships clarifying how leadership programmes and practices impact on pupil outcomes. Our paper describes challenges associated with conceptualizing the relationship between leadership programmes, changes in leader practices and the effects of such changed practices on schools and pupils. We also grapple with some of the methodological challenges facing evaluators and researchers with an interest in programme and leader effects, offering suggestions about how these challenges might be addressed. Conceptual Challenges Four conceptual challenges are addressed in our paper: How to most usefully frame the relationship between leadership programmes, leadership practices and pupil outcomes? We offer a relatively generic response to this question, suggesting that any comprehensive framework would include independent, dependent, antecedent, moderating and mediating categories of variables. A review of research illustrates what we know about each category of variables and the relationships among them. Frameworks guiding a handful of current and recently completed studies illustrate variations on the generic framework; How the nature of leadership practices might be conceptualized and what intellectual resources are available to assist in such conceptualization? We review a wide range of alternative leadership models to assist in thinking about this challenge; these are models developed in both school and non-school organizational contexts; What can leadership theory and research developed outside of schools contribute to our understanding of school leadership practices and their effects? Building on our earlier description of leadership models in non-school contexts, we identified perspectives on leadership yet to be explored in school-based leadership research but with potential. How to define and what do we know about - dependent, moderating and mediating variables in school leader and leadership programme effects studies? We illustrate the state-of-the-art of knowledge about these variables through a review of leadership research carried out exclusively in U.K. contexts. It is clear from this review that 3

5 recently published U.K. leadership research is almost exclusively small scale and qualitative in nature. Methodological Challenges Although not as extensive as our treatment of conceptual challenges, the paper also responds to a series of methodological challenges typical of field based leadership research; these challenges are addressed more fully in a subsequent paper 1. One of the problems we take up in this paper is the measurement of leadership practices, examining a small set of instruments commonly used for this purpose. We also discuss such common difficulties in conducting both programme and leadership effects research as the narrow and unreliable nature of commonly used student assessment instruments and how to deal with missing data for individual schools. The paper presents evidence that, while the school is the unit of analysis in much leadership effects research, there is typically greater within - than across - school variation in measures of leadership an important, largely unaddressed, challenge for future research. Promising approaches to the evaluation of leadership programme effects are also outlined. Conclusions and Recommendations Our conclusions and recommendations bear on selected aspects of future DfES - sponsored research that might be carried out on both leadership, and leadership programme, effects. Future leadership effects research should: measure a more comprehensive set of leadership practices than has been included in most research to date: these measures should be explicitly based on coherent images of desirable leadership practices. Such research is likely to produce larger estimates of leadership effects on pupil outcomes than has been provided to date; measure an expanded set of dependent (outcome) variables: these are variables beyond just short-term pupil learning including, as well, longer term effects. Examples of such long - term effects include pupil success in tertiary education, employment and commitment to learning over the life span; systematically describe how leaders successfully influence the condition of variables mediating their effects on pupils: we now have considerable evidence about what are the most potentially powerful variables mediating school leader effects but we know much less about how leaders influence these mediators; attend more systematically to variables moderating (enhancing, reducing) leadership effects: lack of attention to this category of variables seems likely to be a major source of conflicting findings in the leadership research literature. Furthermore, when studies do attend to moderators, their choice has often been difficult to justify and largely atheoretical. 1 See Leithwood, K., Levin, B. (2005). Assessing Leadership Effects on Pupil Learning: Methodological Issues. Toronto: Report submitted to the DfES, January. 4

6 reflect greater methodological variety than is evident in recently published U.K. leadership research: this will be essential if a robust body of context - relevant knowledge is to be developed. Future leadership programme effects research should: be guided by conceptual frameworks similar to those we have recommended for leadership effects research: our review suggests that the vast majority of previous efforts to evaluate leadership programme effects, in most parts of the world, have not generated the type and quality of evidence required to confidently answer questions about their organizational or pupil effects. This problem could be addressed by introducing, into the conceptual frameworks we have suggested for leadership effects studies, leadership programmes conceptualized as one category of antecedent variables stimulating changes in leadership practices. provide comparative information about programme effects: formal programmes are just one of many influences on leaders practices; to fully appreciate the value of such programmes, their effects need to be compared to the effects of such other antecedents as on-the-job learning, leaders traits and early family experiences, for example. Such information would inform not only programme improvement efforts but leadership selection processes, as well. Information of this sort would also assist with cost-effectiveness judgements in the context of planning for leadership development. be funded at levels consistent with the expectations for what is to be accomplished: few documented programme evaluations provide the type of comprehensive data we call for here and funding is part of the reason. If future leadership programme evaluations are to assess the direct and indirect effects of such programmes on pupil learning as well as leaders practices, then a different level of funding will be required than has been typical to date. Implementing these recommendations will require considerable attention to a significant number of conceptual and technical issues at the core of designing and conducting high quality, high impact research and evaluation. Just doing more research of the type that is typically being done in the country now - at least as it is reflected in the published literature - seems unlikely to significantly advance our understandings about how leadership programmes and leaders most productively improve pupil learning. 5

7 Introduction Purposes and Methods The Department for Education and Skills (DfES) commissioned this paper to identify the pathways by which school leadership programmes and practices influence, or positively impact on, pupil learning and to inform future research and development in this field. More precisely, our goals were to develop a model or framework of variables and relationships clarifying how leadership programmes and practices impact on pupil outcomes. In this paper we describe alternative approaches to conceptualizing the relationship between leadership programmes, changes in leader practices and the effects of such changed practices on schools and students. We also identify some of the methodological challenges facing evaluators with an interest in programme and leader effects and offer some suggestions about how these challenges might be addressed. Our methods for developing this paper entailed systematic analyses of current research and theory. In the case of each challenge or issue taken up in the paper, we aimed to reflect the best of current thinking and to be quite comprehensive about the relevant literatures to which we paid attention; at least a sample of our sources are cited throughout the paper. We have grappled with many of these issues in our own recent research, as well, and our commentary also reflects that experience. The Meaning of Leadership We begin by reflecting on the thorny question of Just what is leadership, anyway? not because we are able to provide a precise answer but because the concept at least deserves some preliminary exploration in a paper with the focus of this one. It is perfectly reasonable to ask How could we understand leadership effects if we don t have a clear notion of what produced those effects? That said, we raise this question in the face of many claims that there is no agreed definition (e.g., Antonakis, Cianciolo & Sternberg, 2004). Indeed, the leadership literature contains literally hundreds of at least slightly different conceptions of the concept. One way often used to clarify the meaning of leadership is to compare it to the concept of management. Some of these comparisons seem largely unhelpful, as in Bennis and Nanus (1985) claim that management is doing things right and leadership is doing the right things. More helpful, we think, is a distinction offered by Kotter (1990). According to this source, management is about producing order and consistency, whereas leadership is about generating constructive change. Adopting this perspective, the primary effect of organizational leadership would be significant change in a direction valued by the organization. In practice, of course, distinguishing between leadership and management behaviours can be extremely difficult. This is because the distinction rests not on the nature of the behaviour but its effects. If behaviour produces order and consistency then it must be management; if it produces change in a valued direction it must be leadership. 6

8 Most conceptions of leadership do associate it with productive change. And at the core of most of these conceptions are two functions generally considered indispensable to its meaning: Direction-setting: helping members of the organization establish a widely agreed on direction or set of purposes considered valuable for the organization; and Influence: encouraging organizational members to act in ways that seem helpful in moving toward the agreed on directions or purposes. Each of these functions can be carried out in different ways, such differences distinguishing many models of leadership from one another. As Yukl notes, leadership influences: the interpretation of events for followers, the choice of objectives for the group or organization, the organization of work activities to accomplish objectives, the motivation of followers to achieve the objectives, the maintenance of cooperative relationships and teamwork, and the enlistment of support and cooperation from people outside the group or organization (1994, p. 3). One must assume, in the case of this conception, that the objectives being referred to entail some sort of change. This is it not a very precise way of defining leadership and may be vulnerable to the occasionally-heard charge that such lack of precision severely hampers efforts to better understand the nature and effects of leadership. But leadership is a highly complex concept. Like health, law, beauty, excellence and countless other equally complex concepts, efforts to define it too narrowly are more likely to trivialize than help bring greater clarity to its meaning. Leadership Effects Policy makers aiming to improve schools on a large scale in today s context invariably assume that the success with which their policies are implemented has much to do with the nature and quality of local leadership, especially leadership at the school level (e.g., Caldwell, 2000; Murphy & Datnow, 2003; Young, Peterson & Short, 2001). This has not always been the case. As David Day (2001) reminds us, it was popular several decades ago, at least in academic circles, to claim that the effects of those in organizational leadership roles were substantially outweighed by the effects of other organizational actors and conditions. Conducted in non-school organizations, for example, Meindl s (1995) research about the romance of leadership argued that leadership was simply the easiest and simplest explanation for organizational effects that were actually the result of a host of more complex and harder to understand relationships and conditions. Evers and Lakomski (2000) have mounted similar arguments about school leadership, in particular. However, more recently arguments that leadership does not matter have been overtaken by empirical evidence indicating that it matters a great deal. We summarize this evidence 7

9 here very briefly with an exclusive focus on pupil effects. It is important to note that most of this evidence has come from research on school-level leaders, especially heads, deputy heads or their equivalent in schools outside of the United Kingdom (U.K.). Local Education Authority (LEA) or district leadership effects on pupils have, until recently, been considered too indirect and complex to sort out, and research on teacher leadership has rarely inquired about pupil effects (Barr & Duke, 2004). 2 Claims about the important effects of leadership on pupils are justified by five quite different types of research evidence: Large-scale quantitative studies of overall leadership effects on pupil test scores: evidence of this type reported between 1980 and 1998 (approximately four dozen studies across all types of schools) has been reviewed in a series of papers by Hallinger and Heck (1996a, 1996b, 1998). These reviews conclude that the combined direct and indirect effects of school leadership on pupil test scores (primarily math and language scores) are small but educationally significant. While leadership explains only three to five percent of the variation in pupil test scores across schools (not to be confused with the very large within-school effects that are likely), this is actually about one quarter of the total across-school variation (10 to 20 percent) explained by all school-level variables, after controlling for pupil intake or background factors (Townsend, 1994; Creemers & Reetzig, 1996). The quantitative school effectiveness studies providing much of these data indicate that classroom factors explain more than a third of the variation in pupil test scores. Overall leadership effects on pupil engagement in schools: using designs comparable to the leadership effects studies summarized above, a second source of evidence about leadership effects on pupils replaces test scores with pupil engagement in school as the criterion variable. Building on Finn s (1989) early work, participation is the behavioural component of engagement and includes students actions in the classroom and school. Identification with school is the psychological component of engagement, its meaning captured in such terms as affiliation, involvement, attachment, bonding and commitment. Two overlapping research programmes on leader effects, one in Canada (e.g., Leithwood & Jantzi, 1999), the other in Australia (e.g. Silins & Mulford, 2002) have used engagement as their dependent measure and both have found significant indirect effects of transformational approaches to leadership on student engagement. Evidence from these programmes and others (Fredricks, Blumenfeld & Paris, 2004) also suggests significant relationships between engagement and both retention and a wide range of achievement outcomes in elementary and secondary schools. Effects of specific leadership practices on pupil test scores: a third source of evidence about leadership effects on pupils is also large-scale and quantitative in nature. However, instead of examining overall leadership effects, it inquires about the effects of specific leadership practices on pupil test scores. Evidence of this sort can be found 2 This very comprehensive review of empirical evidence was able to locate only five studies assessing the effects of teacher leadership on pupils. Only four of these studies actually included direct measures of pupil learning and three of the four found no effects of teacher leadership on such learning. In light of how popular teacher leadership has become as a strategy for school reform, this is an astonishing demonstration of just how susceptible are schools to evidence-free claims about what they ought to do. 8

10 sporadically in the research alluded to above, but a recent meta-analysis by Waters, Marzano and McNulty (2003) has significantly extended this type of research. This study identifies 21 leadership responsibilities and calculates an average correlation between each and whatever measures of pupil achievement were used in the original studies. From this data, estimates are calculated of the effects on pupil test scores (e.g. a 10 percentile point increase in pupil test scores resulting from the work of an average principal who improved her demonstrated abilities in all 21 responsibilities by one standard deviation (p. 3). 3 Primarily qualitative case study evidence: studies providing this type of evidence typically are conducted in exceptional school settings (e.g. Gezi, 1990; Reitzug and Patterson, 1998 ). These are settings believed to be contributing to pupil learning significantly above or below normal expectations as, for example, effective schools research based on outlier designs. Such studies usually report very large leadership effects not only on pupil learning but on an array of school conditions, as well (e.g. Mortimore, 1993; Scheurich, 1998). What is lacking from this evidence, however, is external validity or generalizability. Research on leader succession is the final and, arguably, most compelling source of evidence about leadership effects. This evidence demonstrates, for example, that few school improvement initiatives survive a change in principal leadership and that important attitudes toward leadership influence are significantly and negatively influenced by frequent changes in such leadership. Teachers come to ignore or otherwise inoculate themselves against the influence of their schools administrative leaders when these leaders are perceived to rotate through the school every two or three years (Hargreaves, Moore, Fink, Brayman, & White, 2003; Macmillan, 1996). Thus inoculated, it takes considerable time and a prodigious effort on the part of a committed leader prepared to stay the course to recover the cooperation of her school s staff in the interests of school improvement. Similarly, leader succession research in non-school organizations provides powerful evidence of very large leader effects, especially the effects of leaders at the apex of their organization s hierarchy. Evidence from Day and Lord (1988) and Thomas (1988) indicates, for example, that: When properly interpreted and methodologically sound, the research on leadership succession---has demonstrated a consistent effect for 3 While this quantitative synthesis of research produces clearly interesting data, estimates from such data to principal effects on pupil learning in real world conditions must be treated with considerable caution. First of all, the data are correlational in nature, but cause and effect assumptions are required for the extrapolated effects of leadership improvement on pupil learning. Second, the illustrative effects on pupil achievement described in the study depend on a leaders improving their capacities across all 21 practices at the same time, an extremely unlikely occurrence! Some of these practices are dispositional in nature, or rooted in deeply held beliefs unlikely to change much, if at all, within adult populations (e.g., ideals/beliefs, flexibility). And just one of the 21 practices, increasing the extent to which the principal is knowledgeable about current curriculum, instruction and assessment practices would be a major professional development challenge, by itself. Nonetheless, this line of research is a useful addition to other lines of evidence which justify a strong belief in the contributions of successful leadership to pupil learning. 9

11 leadership that explained 20 to 45 percent of the variation in relevant organizational outcomes (Day, 2001). These five sources of evidence add up to powerful support for the importance widely attributed to leadership by policy makers and the public at large. However, without minimizing the considerable progress that has been made over the past 15 years, we still have a great deal to learn about: the nature of school leadership practices that are successful in improving pupil learning; under what conditions or circumstances are some forms of school leadership more successful than others? how successful leadership practices are connected to changes in the school organization and eventually to improvements in pupil learning; and how successful leadership is best developed. Leadership Programme Effects Because effective leadership is so critical to school improvement, massive amounts of energy and considerable resources have been devoted to its initial development and ongoing improvement over the past 15 years, in particular. The National College of School Leadership is a highly visible manifestation of this attention in England. Formal training programmes appear to be the most direct strategy for building leadership capacity but they are by no means uniformly effective. Indeed, calls for more rigorous programme evaluation have become ubiquitous. These calls have escalated significantly in the past decade in response to an outcomes-oriented accountability policy environment for education and a widely shared belief in the importance of effective leadership. Programme evaluators have struggled to keep up with these calls for more rigorous evaluation for several reasons. Most obviously, first of all, the funds available for leadership programme evaluation rarely match the ambition of the expectations. Second, the most ambitious expectations to assess leadership programme effects on pupil learning require highly sophisticated theoretical frameworks, frameworks that lie outside the theoretical repertoire of those typically charged with programme evaluation responsibilities. Finally, and closely related to this second reason, such highly sophisticated frameworks (called indirect effects or mediated models) potentially include all of the variables at the school and classroom level that are themselves the focus of independent lines of active research with the usual debates and uncertainties about their effects on pupil learning. Examples of such variables at the school level include school culture, school improvement planning processes, and shared decision-making. At the classroom level, such variables include, for example, classroom instructional processes, opportunity to learn and class size. In a recent analysis of leadership preparation programmes across the United States, McCarthy (1999) concluded that we do not actually know whether, or the extent to which, such programmes actually achieve the goal of producing effective leaders who create school environments that enhance pupil learning? (p. 133). This gap in our 10

12 knowledge is not because leadership preparation programmes are never evaluated; rather, the vast majority of such evaluations do not provide the type and quality of evidence required to confidently answer questions about their organizational or pupil effects. Most evaluations are limited to assessing participants satisfaction with their programmes and sometimes their perception of how such programmes have contributed to participants work in schools (McCarthy, 2002). 11

13 A General Framework to Guide Research on Programme and Leader Effects on Pupil Learning Most efforts to conceptualize the relationships among leadership programmes, leaders practices and pupil learning has assumed that the effects of leaders on pupil learning are largely indirect. Based on this well-justified assumption, then, one of the primary challenges for research on programme and leaders effects is to locate the most defensible set of variables mediating and moderating programme and leaders effects. A second significant challenge is to uncover the nature of the relationships among these variables and between leaders and such variables. Figure 1, reflecting these assumptions, is the general framework guiding our account of how to better understand programme and leader effects. This figure indicates that leadership practices (overt behaviours - or properties of the organization - aimed at direction setting and influence) have direct effects on potentially a wide range of variables; they stand between or mediate the effects of leadership, particularly when those effects are conceptualized as pupil learning. Below we discuss some defensible alternatives to pupil outcomes as potential dependent variables in leadership effects research but also assume that in the current U.K. educational environment pupil learning is likely to be at least among dependent variables necessarily included in DfES or the National College for School Leadership (NCSL) endorsed studies of educational leadership effects. Figure 1 also includes a set of moderating variables. As we explain more fully below, these are features of the organizational or wider context in which leaders work that interact with the dependent and/or mediating variables. These interactions potentially change the strength or nature of relationships between, for example, the independent and mediating variables or the mediating and dependent variables. For example, if previous evidence suggested that male and female teachers respond differently to the same set of headteachers leadership behaviours, then teachers gender would be a promising moderating variable to include in a study of headteacher effects. Some of the antecedents variables in Figure 1 are internal to the leader, including for example, leaders traits, values, cognitions, and emotions. There are external antecedents, as well. These would include leadership programmes, of course, but also such variables as LEA relationships, government educational policies, and leader family and socialization experiences (Popper & Mayseless, 2002). The next several sections of the text address issues in the conceptualization and measurement of each of the variables in Figure 1. 12

14 Antecedents Moderating Variables e.g. family background family culture gender formal education reward structure Independent Variables leadership practices Mediating Variables school conditions class conditions individual teachers professional community Dependent Variables short term long term Figure 1: A General Framework for Guiding Leadership Effects Research 13

15 Leadership Practices: The Independent Variables Direction-setting and influence, as we argued above, is the generic meaning of leadership. Both of these functions can be exercised in many different ways and more or less successfully. In this section we illustrate the range of ways in which the successful or effective exercise of leadership has been conceptualized both within and outside of the education sector; these are alternative leadership models or theories. We also identify some important challenges in the measurement of successful or effective leadership. Leadership Models Developed in the Education Sector In their extensive review of research, Leithwood and Duke (1999) developed a five-fold classification of leadership models reflected in the educational literature. These are summarized along with related theory and research published since Instructional leadership: focuses on the behaviours of teachers as they engage in activities directly effecting the learning of pupils. The more fully developed models in this category (e.g., Hallinger, 2003) also include attention to broader sets of organizational variables, such as school culture or climate, thought to influence teachers classroom practices. Transformational leadership: focuses on the commitments and capacities of organizational members, as well as their willingness to engage in extra effort on behalf of their organizations. While the bulk of the evidence about this approach to leadership has been collected in non-school contexts (e.g., Avolio & Yammarino, 2002), educational researchers have recently begun to redress this imbalance (e.g., Nguni, 2004; Lunenburg, 2004). Moral leadership: is concerned with the ethics and values of those exercising leadership. Specifically, it aims to clarify the nature of the values used by leaders in their decision making and how conflicts among values are best adjudicated (Begley & Leonard, 1999; Begley & Johansson, 2003). A strand within this approach to leadership specifically aims to promote democratic values and the empowerment of a large proportion of organizational members (e.g. Starratt, 2003: Johansson, 2003). Participative leadership: shines a spotlight on group decision-making processes. Educational research inquiring about this approach builds on a strong foundation of research in other sectors dating back to seminal studies in the early 1930 s (e.g., Mayo, 1933) about increases in organizational effectiveness associated with greater participation of employees in meaningful decisions about their work. The extensive body of research on teacher participation in decision making reasonably can be viewed as part of the body of evidence about this model of leadership (e.g., Conley, 1991). Rapidly growing literatures on both teacher leadership (York-Barr & Duke, 2004; Harris & Chapman, 2002) and distributed leadership (Gronn, 2002; Spillane, Halverson & Diamond,2002) are the most recent evolutions of this approach. Managerial and strategic leadership: encompass a range of tasks or functions found in the classical management literature (reviewed in Rost, 1991), including tasks such as coordination, planning, monitoring and the distribution of resources. Educational literature from the United Kingdom reflects a far greater interest in this form of leadership than does the North American literature. Also addressed much more 14

16 extensively in the UK than the North American literature 4 is the entrepreneurial, creative and change oriented strategic leadership sometimes thought to be the exclusive purview of those occupying senior levels of the organizational hierarchy (Yukl & Lebsinger, 2004). Contingent leadership: emphasizes the need for leaders to be responsive to the unique demands of their organizations and the contexts in which those organizations function. While this approach is quite mature in both education and non-education sectors (e.g., Blake and Mouton, 1964), its original conception was limited to a very small number of dimensions along which leadership styles could vary in response to context (primarily the initiation of structure and demonstrations of consideration for employees). Current leadership research continues to call for more sensitivity to the context in which leaders work and greater flexibility on the part of leaders across a much larger number of dimensions (Yukl & Lepsinger, 2004). Leadership Models Developed in Other Sectors With the exception of instructional leadership, all of the approaches to leadership explored by the educational research community are also active areas of research in other sectors. But academic leadership research 5 in these other sectors reflects an additional range of approaches or models. The classification of this research offered by Dansereau, Yammarino & Markham (1995) illustrates this range. We provide only a cursory description of their categories here, our aim being to simply alert readers to this larger field of work. A total of 13 approaches to leadership appear in the classification system of Dansereau and his colleagues and these are nested within four superordinate categories: Classical approaches: including several contingency-oriented, as well as participative, approaches to leadership. This category assumes that leaders can and ought to change their styles over time in response to the circumstances in which they find themselves. Contemporary approaches: with a more explicit focus on both the leaders and the development of their followers (p. 254), includes charismatic and transformational approaches to leadership. This superordinate category also includes Leader-Member Exchange theory which argues that leaders have unique relationships with individual organizational members depending on such factors as trust, perceived competence and the like. 6. Alternative approaches: expand the focus of attention beyond either the individual leader or followers to relationships and interactions. Included in this superordinate 4 See, for example, the special issue of School Leadership and Management (2004, vol. 24, no. 1) edited by Brent Davies. 5 We use this term academic leadership research in reference to systematic, theoretically informed, empirical inquiry about leadership - as distinct from the highly popular genre of leadership literature which is autobiographical, anecdotal and/or exclusively case based. 6 For one of the few studies of this model of leadership in a school context, see Devereaux s (2004) recent dissertation. 15

17 category are information processing, substitutes for leadership, and romance of leadership models. New wave approaches: this category includes a quite eclectic set of leadership models held together as much by their recent emergence as anything more conceptually coherent. Self leadership, a multiple linkages model, multi-level theory and individualized approaches to leadership are included. Measuring Leadership Practices Some of the education sector leadership models have been specified in detail and tested with instruments which are quite well developed. This is the case, for example, with Hallinger s instructional leadership model (see Hallinger & Murphy, 1985), Leithwood s transformational school leadership model (Leithwood & Jantzi, 2000) and Marks and Printy s (2003) synthesis of both of these forms of leadership. Several versions of Bass (1985) Multifactor Leadership Questionnaire have been used extensively to study transformational leadership effects primarily in non-school organizations, but in schools and districts, as well (e.g., Nguni, 2004). Many other instruments are available for measuring leadership, especially for doing so in non-education organizations (Clark & Clark, 1990). Without diminishing their many other contributions, one limitation of some of the most widely cited reviews of leadership effects on pupil learning is that they confound estimates of such effects by failing to distinguish among alternative approaches to, or models of, leadership (e.g., Hallinger & Heck, 1996b). Other reviews seem to infer comprehensive assessments of leadership effects while actually limiting themselves to the behaviours associated with a particular model of leadership (e.g., Witzier, Bosker & Kruger, 2003). Furthermore, some original studies of leadership effects use secondary data sources originally created for other purposes, typically estimating the effects of potentially incoherent or incomplete models of leadership. Virtually all large-scale quantitative leadership effect studies in education restrict their attention to only part of what it is that leaders do. Such studies are usually guided by a leadership model which is intentionally reductionist in nature. The aim of this work is to assess the explanatory power of a particular set of leadership behaviours, which are associated with the chosen model. But, of course, real leaders do much more in their schools than provide, for example, instructional or transformational leadership. On occasion, almost all leaders demonstrate the use of practices associated with strategic leadership, moral leadership and the like. Such outside-the-model practices are often captured in the much wider net of qualitative case study research and this is one of the reasons for the discrepancy between the size of leadership effects reported in qualitative and quantitative studies. A review of research (Bell, Bolam & Cubillo, 2003) prepared for the British Evidencebased Policy and Practice Centre (EPPI) started with more than 4000 potential references but found that only eight met the criteria of a well-specified leadership model and empirical evidence on pupil outcomes. In addition, many leadership programme 16

18 evaluations neither specify nor measure the leadership practices which they aim to improve, electing instead for more global measures of participant satisfaction with the contribution of the programme to participants personal and implicit leadership efforts or espoused leadership theories. These shortcomings in the actual measurement of leadership point to the importance of clearly specifying those leadership practices which are hypothesized to effect pupil outcomes. Failure to do this arises from both practical and conceptual sources. Practically, available resources will often press researchers and evaluators to rely on existing evidence, evidence that is an imperfect match for their purposes. Conceptually, a major source of the problem is lack of agreement about the definition of leadership, as we discussed earlier. These challenges notwithstanding, subsequent improvements in our understanding of leadership effects on pupils depends, to a significant degree, on the use of reliable leadership measures explicitly based on clearly defined and conceptually coherent images of desirable leadership practices. 17

19 The Big Picture Leadership Effects: The Dependent Variables David Day s (2001) views on the assessment of leadership outcomes in the business sector, are useful stimulants for thinking more comprehensively about the outcomes of educational leadership. He identifies three levels of leadership outcomes which he terms financial performance, operational performance, and organizational effectiveness. Financial performance is the first and narrowest level, the equivalent in the education sector to pupil test results. The next broadest level is operational performance. Subsuming financial performance, operational performance in the business sector is a function of, for example, market share, product quality, and measures of technological efficiency. Research on educational leadership has typically conceptualized outcomes of this sort as mediating variables (discussed more full below) or mechanisms through which the influence of leaders eventually impact on pupils. Finally, at what Day terms the broadest and most subsuming level (2001, p. 388) is organizational effectiveness, the primary indicator being the health or survival of the organization. In Day s words, For those leaders at the highest organizational levels in which a system perspective is imperative for successful performance [survival] may be primarily a function of the organization s identity, image, and reputation (2001, p. 389). In the remainder of this section, we describe the educational equivalent of Day s first and third levels of leadership outcomes, leaving our treatment of the second level (mediating variables) to the next section. Level One Effects in Education: Pupil Learning In this section we consider some of the major challenges associated with determining and measuring pupil outcomes in inquiries about leadership effects. Academic achievement, as it is typically measured, is just one of several indicators of pupil outcomes. Others of a more long-term nature include graduation rates, drop out rates, and engagement in school, for example. These two sets of outcomes are quite different. Achievement measures reflect pupils skills and knowledge in a specific curriculum domain. Secondary school graduation rates, however, reflect not only specific curricular goals but also course selection decisions, course load, exam difficulty and the like. Achievement test scores are necessarily informed by pupils entire previous school careers, as well as their personal lives. Indeed, there is a strong case to be made that the important outcomes of education are, in fact, the broader and longer-term measures such as participation in further or higher education, employment, and other measures of social participation. Many people care far more about these kinds of outcomes than they do about, for example, science test scores at age 15. Moreover, broader measures tend to present fewer data problems. 18

20 As these arguments make clear, our current preoccupation with pupil test scores, as the dependent measure of choice in inquiries about leadership effects, is open to serious challenge. That said, the preference for assessing leadership effects on pupil test scores is not likely to go away anytime soon. In fact, Hallinger and Heck (1996a) decried the extent of use of such measures in studies of leadership a decade ago. But the press to use them has grown rather than diminished in the interim. So what are the challenges associated with this measure of pupil outcomes? While purpose-built achievement measures could be used by researchers and evaluators (although they would have their own limitations), in practice, both levels of funding and national, state or LEA policies mean that most research studies and programme evaluations end up using existing measures. These measure are typically part of national, state or LEA pupil testing programmes which have three well-known limitations as estimates of leadership effects: a narrow focus, questionable or unknown reliability, and the questionable accuracy with which they are able to estimate change over time. We have encountered a handful of less pervasive, practical limitations in some of our own recent work which we also identify in this section. Narrow focus. First, most large-scale testing programmes confine their focus to maths and language achievement with occasional forays into science. Only in relatively rare cases (e.g. Kentucky) have efforts been made to test pupils in most areas of the curriculum not to mention cross-curricular areas such as problem-solving or teamwork. Technical measurement challenges, lack of resources and concerns about the amount of time for testing explain this typically narrow focus of large-scale testing programmes. But this means that evidence of leaders effects on pupil achievement using these sources is evidence of effects on pupils literacy and numeracy. Because improving literacy and numeracy are such pervasive priorities in so many schools at the moment, this is a limitation that will not concern many researchers and evaluators. There is evidence, however, that leadership effects are of a different magnitude for even these two areas of achievement. The lesson for researchers and programme evaluators is that the size and significance of leadership effects on other areas of achievement cannot be assumed or extrapolated. Reliability. Lack of reliability at the school level is a second limitation of many large-scale testing programmes. Most of these programmes are designed to provide reliable results only for large groups of pupils. So results aggregated to national, state or LEA levels are likely to be reliable. But as the number of pupils diminishes, as in the case of a single school or even a small district or region, few testing systems claim to even know how reliable their results are (e.g., Wolfe, Childs & Elgie, 2004). The likelihood, however, is that they are not very reliable, thereby challenging the accuracy of judgments about leadership effects. Researchers and programme evaluators would do well to limit analysis of achievement to data aggregated above the level of the individual school or leader. Estimating change. Conceptually speaking, monitoring the extent to which a school improves the achievement of its pupils over time is a much better reflection of a school s 19

21 (and leader s) effectiveness than is its annual mean achievement scores. Technically speaking, however, arriving at a defensible estimate of such change is difficult. Simply attributing the difference between the mean achievement scores of this year s and last year s Key Stage Two pupils on the country s literacy test to changes in a school s (and/or leader s) effectiveness overlooks a host of other possible explanations: Cohort differences: This year s pupils may be significantly more or less advanced in their literacy capacities when they entered the cohort. Such cohort differences are quite common, as any teacher will attest; Test differences: While most large-scale assessment programmes take pains to ensure equivalency of test difficulty from year to year, this is an imperfect process and there are often subtle and not-so subtle adjustments in the tests that can amount to unanticipated but significant differences in scores; Test conditions differences: Teachers are almost always in charge of administering the tests and their class s results on last year s tests may well influence the nature of how they administer this year s test (more or less leniently) even within the guidelines offered by the testing agency; External environment differences: Perhaps the weather this winter was more severe than last winter and pupils ended up with six more snow days - six fewer days of instruction, or a teacher left half way through the year, or was sick for a significant time; Regression to the mean: this is a term used by statisticians to capture the highly predictable tendency for extreme scores on one test administration to change in the direction of the mean performance on a second administration. So schools scoring either very low or very high one year can be expected to score less extremely the second year, quite aside from anything else that might be different. Linn(2003) has demonstrated that these challenges to change scores become less severe as change is traced over three or four years. It is the conclusions drawn from simply comparing this year s and last year s scores that are especially open to misinterpretation. Unfortunately, it is the year over year comparisons that are most commonly made by those who report achievement results. The lesson here for researchers and programme evaluators is to use, as measures of their dependent variable, changes in pupil achievement over relatively long periods of time (three or more years). Two further limitations. While the three limitations, reviewed above, of national or state achievement data challenging researchers and evaluators have attracted considerable recent attention, we encountered two others in the course of conducting a recent leadership programme evaluation (Leithwood et al., 2003) using U.S. state (Louisiana) achievement data as the measure of programme and leadership effects. One of these limitations was changing measures over time. Ours was a five year longitudinal evaluation which was complicated by changes in the state s achievement measures. Given the frequency of policy shifts in pupil assessment practices in many jurisdictions, it may become impossible to maintain a consistent set of data over several years. Not only can the tests change, but scoring rubrics and cut-offs may also be modified. 20